Broadcom NIC leads to VNET jails being unable to make DNS requests

P

pswilde

Hi everyone, I hope you're all well.

I just wanted to raise a post here about a problem I'm experiencing after upgrading a server to 15.0-RELEASE where jails seem to be unable to make DNS requests.
I first raised this issue as I thought it was a problem with how BastilleBSD handles networking here: https://github.com/BastilleBSD/bastille/issues/1468
However, now after some testing it seems the problem may be more related to Broadcom network cards as in one of my problem servers (I have two problem ones), switching over to the installed Intel card resolve the issue.

More information on the actual issue:
Some of my jails are VNET jails (they have a network accessible IP address) and they have been working absolutely fine on FreeBSD 14.3 and previously. However, if I upgrade the host to 15.0, these jails become unable to make any DNS requests. They can ping any IP address on the internet absolutely fine, they can ping the DNS server's IP address, but any DNS request just seems to get lost.
i.e. a ping to an IP address as below is fine
Code: 
[postgres]:
root@postgres:~ # ping 9.9.9.9
PING 9.9.9.9 (9.9.9.9): 56 data bytes
64 bytes from 9.9.9.9: icmp_seq=0 ttl=60 time=14.501 ms
64 bytes from 9.9.9.9: icmp_seq=1 ttl=60 time=14.475 ms
64 bytes from 9.9.9.9: icmp_seq=2 ttl=60 time=14.617 ms

but a ping to quad9.net just times out

Code: 
root@postgres:~ # ping quad9.net
ping: cannot resolve quad9.net: Name could not be resolved at this time

My /etc/resolv.conf is fine, as below:

Code: 
root@postgres:~ # cat /etc/resolv.conf
nameserver 9.9.9.9
options edns0

but if we ignore that anyway and try a direct DNS request to the server, it times out:

Code: 
root@postgres:~ # drill quad9.net @9.9.9.9
Error: error sending query: Could not send or receive, because of network error

If I roll the host back to 14.3-RELEASE, normally functionality returns - DNS requests are fine.

I have two servers that have this issue, different server models, but similar network cards - Broadcom BCM5719 and BCM5720.
One of these servers had an additional Intel NIC in it as well, so I switched that server over to that card today, and the jails work absolutely fine on 15.0-RELEASE again.

To double-down on it being Broadcom as the problem here, on the server above with the Intel NIC I just created a new test VNET jail, setting it's host NIC to the Broadcom rather than Intel, and it suffers the same problem mentioned above (ping to IP fine but DNS requests time out) whilst the Intel connected jail continues to operate fine.

So, my question really is - where should I look next if I want to ensure connectivity to the VNET jails remains robust when using 15.0 on these Broadcom cards? I know Broadcom are a bit of a pest, but I'd like to think this can be made to work as it did using 14.3 with a little tweak here and there.

Happy to perform any diagnostics and dump the results here if anybody can offer their knowledge!

Edit: adding some tcpdump output:

tcpdump on the host machine when using broadcom NIC: we don't see any responses
Code: 
> $ sudo tcpdump -i bge0 -v port 53
tcpdump: listening on bge0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:50:08.606533 IP (tos 0x0, ttl 64, id 33187, offset 0, flags [none], proto UDP (17), length 55)
    <local ip of jail>.39194 > dns9.quad9.net.domain: 35593+ A? quad9.net. (27)
16:50:13.628043 IP (tos 0x0, ttl 64, id 33188, offset 0, flags [none], proto UDP (17), length 55)
    <local ip of jail>.11771 > dns9.quad9.net.domain: 35593+ A? quad9.net. (27)
16:50:18.752908 IP (tos 0x0, ttl 64, id 33189, offset 0, flags [none], proto UDP (17), length 55)
    <local ip of jail>.34978 > dns9.quad9.net.domain: 35593+ A? quad9.net. (27)

But on the intel card, we get a response:

Code: 
> $ sudo tcpdump -i igb0 -v port 53
tcpdump: listening on igb0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:53:13.968660 IP (tos 0x0, ttl 64, id 8468, offset 0, flags [none], proto UDP (17), length 55)
    <hostname of jail>.21816 > dns9.quad9.net.domain: 3771+ A? quad9.net. (27)
16:53:13.983571 IP (tos 0x0, ttl 60, id 58815, offset 0, flags [none], proto UDP (17), length 71)
    dns9.quad9.net.domain > <hostname of jail>.21816: 3771 1/0/0 quad9.net. A 195.176.255.138 (43)
 
I find it odd that one type of NIC works and the other one doesn't with regards to DNS. It's just packets going out, hardware doesn't (shouldn't) care about the contents.

The DNS lookup is happening in a jail bound to either the intel or the bcm nic, is that a correct statement?

What happens if you put the nameserver as say the OpenDNS?
208.67.222.222 and 208.67.220.220
Google has 8.8.8.8 and 8.8.4.4

If none of those work, then I would suspect something blocking packets to port 53 or not routing correctly.
The nc command to try and connect to dest ip port 53 may help.
 
mer said:
I find it odd that one type of NIC works and the other one doesn't with regards to DNS. It's just packets going out, hardware doesn't (shouldn't) care about the contents.
Click to expand...
Same. Very much doubt its specific to a driver and only fails for specific type of packet (DNS query).

Isn't it something super obvious? Typo in a firewall rule for example?

And I'd run tcpdump(1) on the host's egress interface, just to verify you can see the DNS queries going out (or not).
 
Thanks mer, thanks SirDice, yes it is very odd!
I have gone through my configs with a fine tooth comb and cannot see any typos. There certainly could be something, but even so, why would rolling back to 14.3 resolve the problem, or just simply changing the interface the host and jails use?
I can indeed see DNS queries going out on tcpdump, but I don't see anything coming back in

Yes to the best of my understanding the jail is bound to either the Intel or Broadcom NIC, i.e. you'd specify the interface when creating the jail with bastille like "bastille create -V <jailname> <FreeBSD release> <ip address> <interface>". Incidentally non-VNET jails (that use a shared IP on the host card or loopback networking) operate fine - this issue is specific to VNET jails.
It works fine on the Intel NIC, but not on the Broadcom on 15.0-RELEASE (specifically p2 at the time of writing).
When I switched over to the Intel NIC I changed nothing other than the interface name in the jail configuration, rc.conf and pf.conf - but it just worked.
I've tried numerous DNS resolvers already, Quad9, GoogleDNS, OpenDNS, local network resolvers, etc. all the same - timeouts.

It's is a weird one, that is to be said!
 
SirDice said:
Then it's pretty much out of our hands. And you'll need to look for a potential cause further upstream.
Click to expand...
Thanks - yep. I will, but first I'm going to migrate all things off one of the problem servers and reinstall the base OS to double-triple-quadruple check that I just haven't done something stupid.
 
Maybe a traceroute to quad9 or other server may give information as to the path the packets go, which may give a clue as to where they drop off. traceroute uses icmp so may not be hitting the same problem as UDP to port 53 (DNS)
 
did you try any other dns servers?
quad9 has had some routing issues in the past where I also had weird behaviour (e.g. ping was OK, dns queries were not).

Other than that, what covacat suggested was my initial thought too: checksums or broken offloading (new driver/firmware in 15.0?)
 
Thanks mer and sko

traceroute returns results - a bit slow, but checking tcpdump during the process shows it's trying to resolve names as it goes so the speed is likely related to the overall problem.

And yes, I have tried every public DNS resolver under the sun and also local network DNS resolvers - nothing. Just to say again though... it all works absolutely fine if I bind to a different NIC or rollback to 14.3.

I'm just working through some options I have found here: https://github.com/pfsense/docs/blo...hooting-network-cards.rst#broadcom-bce4-cards
Specifically the "Packet loss with many small UDP packets" bit. Will report back..... actually, no, this had no effect.
 
Just had something with similar symptoms, turned out I messed up the MTU size. Once the MTU size was returned to normal everything worked again. Just a thought...
 
same as drill output for quad9.net in original post, but yes

Code: 
root@postgres:~ # drill freebsd.org
Error: error sending query: Could not send or receive, because of network error

And as before, if I switch to bind on the Intel NIC it resolves fine

Code: 
root@postgres:~ # drill freebsd.org
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 53099
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; freebsd.org.    IN    A

;; ANSWER SECTION:
freebsd.org.    3600    IN    A    96.47.72.84

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 21 msec
;; SERVER: 9.9.9.9
;; WHEN: Mon Feb  2 17:54:15 2026
;; MSG SIZE  rcvd: 45

For completeness, when I switch from the Broadcom to Intel NIC, the only value I am changing is "bge0" to "igb0" in the jail.conf - everything else is identical.
Code: 
exec.prestart += "jib addm postgres <interface>";


Such a bizarre problem from what I can see.
 
  • Thanks
Reactions: sko
sko interesting. on 14.3-RELEASE man resolv.conf under "options" does not have edns0, but the example shows it.
My 15.0-RELEASE system shows the same info in man resolv.conf.
with openDns and google, that option seems to work.
 
mer said:
sko interesting. on 14.3-RELEASE man resolv.conf under "options" does not have edns0, but the example shows it.
My 15.0-RELEASE system shows the same info in man resolv.conf.
with openDns and google, that option seems to work.
Click to expand...
Perhaps a red herring in this particular problem, I'm only giving quad9 details for ease of explanation - the issues I'm having are not specific to quad 9
 
pswilde said:
Perhaps a red herring in this particular problem, I'm only giving quad9 details for ease of explanation - the issues I'm having are not specific to quad 9
Click to expand...
Understood, but sko implied he had working configuration with quad9 and then added the edns0 option and had failures. So perhaps take your config to quad9 and comment out the edns0 options?
 
mer said:
Understood, but sko implied he had working configuration with quad9 and then added the edns0 option and had failures. So perhaps take your config to quad9 and comment out the edns0 options?
Click to expand...
Sure, no problem doing that at all

Removed edns0, same problem.
Left it removed and set nameservers to 8.8.8.8 and 8.8.4.4. Problem remains.


OpenDNS - same.
Local network DNS server looking up the name of a local network FQDN - same problem.


Bind the jail to the Intel NIC and all of the above works fine.
Stay on the Broadcom NIC and roll the host back to 14.3 and all of the above works fine.

I really don't think this is an issue with what DNS server I'm using.
 
pswilde said:
Sure, no problem doing that at all

Removed edns0, same problem.
Left it removed and set nameservers to 8.8.8.8 and 8.8.4.4. Problem remains.


OpenDNS - same.
Local network DNS server looking up the name of a local network FQDN - same problem.


Bind the jail to the Intel NIC and all of the above works fine.
Stay on the Broadcom NIC and roll the host back to 14.3 and all of the above works fine.

I really don't think this is an issue with what DNS server I'm using.
Click to expand...
I agree with your last sentence. So maybe something going on with routing for the 2 different cards? What does ifconfig on the host show (config for the BCM and Intel) and then ifconfig in the jail when on BCM and on Intel?
 
You must log in or register to reply here.
Back
Top