Solved Jailed ping(8) can not resolve names using hosts unbound(8), but host(1) and drill(1) can

D

dautor

EFFECT
drill(1) resolves r0.z201 correctly to 192.168.201.1.
host(1) resolves r0.z201 correctly to 192.168.201.1.
ping(8) works when given argument 192.168.201.1.
ping(8) doesn't work when given argument r0.z201. (fails to resolve)

OBSERVATIONS
When using truss(1) I can see that
ping(8) does:
  1. socket(PF_INET,SOCK_DGRAM|SOCK_CLOEXEC,0) = 3
  2. connect(3,{ AF_INET 127.0.0.1:53 },16) = 0
  3. sendto(3,"\M-+\M-!\^A\0\0\^A\0\0\0\0\0\0"...,25,0,NULL,0) = 25
host(1) and drill(1) do:
  1. socket(PF_INET,SOCK_DGRAM,IPPROTO_UDP) = 3
  2. sendto(3,"\M^^\^E\^A\0\0\^A\0\0\0\0\0\0\^B"...,25,0,{ AF_INET 127.0.0.1:53 },16) = 25
Apart from that, under truss(1) I can see that ping(8) is doing
open("/lib/libcasper.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,066000) = 3 (0x3)
open("/lib/casper/libcap_dns.so.2",O_RDONLY|O_CLOEXEC|O_VERIFY,066000) = 3 (0x3)
while other commands that successfully resolve do not load in casper.

When I use a different resolver than the locally set up unbound ping(8) does not fail to resolve.

SETUP
I have some jails set up with nameserver 127.0.0.1 in /etc/resolv.conf.
Jails use a separate loopback interface that is NAT-ed to the outside via pf(4).
Jails are on subnet 10.201.0.0/24.
Host is running unbound(8) with interface set to 0.0.0.0 and ::0 and is accepting both TCP and UDP.
I have static local records in the configuration e.g.:
local-zone: "z201." static
local-data: "r0.z201. IN A 192.168.201.1"
local-data-ptr: "192.168.201.1 r0.z201."
and I forward unknown requests to 1.0.0.1, 1.1.1.1, 8.8.8.8, 8.8.4.4.

QUESTION
Am I missing something?
Where should I look?
How can I make it work?
 
UPDATE: I now ran /usr/share/dtrace/udptrack | grep :53 -A6 on host and can confirm that ping(8) does indeed send query the same way drill(1) does and is getting correct response.
I have no idea why it isn't reading it, though.
 
dautor said:
I have some jails set up with nameserver 127.0.0.1 in /etc/resolv.conf.
Click to expand...
'Standard' jails don't have a lo0 interface, thus no 127.0.0.1 either. From the jail's point of view that IP address doesn't exist.
 
kalleboy said:
But inside a jail, when you ping 127.0.0.1 (also ping localhost), you get replies back correctly?
Click to expand...
Yes, when I ping 127.0.0.1 it does reply.

Code: 
# ping r0.z201
ping: Unknown host
# host r0.z201
r0.z201 has address 192.168.201.1
# drill r0.z201
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 4563
;; flags: qr aa rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 
;; QUESTION SECTION:
;; r0.z201.    IN    A

;; ANSWER SECTION:
r0.z201.    3600    IN    A    192.168.201.1

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 0 msec
;; SERVER: 127.0.0.1
;; WHEN: Thu Jul 28 23:13:43 2022
;; MSG SIZE  rcvd: 41
#
# ping 192.168.201.1
PING 192.168.201.1 (192.168.201.1): 56 data bytes
64 bytes from 192.168.201.1: icmp_seq=0 ttl=64 time=0.100 ms
64 bytes from 192.168.201.1: icmp_seq=1 ttl=64 time=0.110 ms
64 bytes from 192.168.201.1: icmp_seq=2 ttl=64 time=0.370 ms
^C
--- 192.168.201.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.100/0.193/0.370/0.125 ms
# 
# ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.073 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.112 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.202 ms
^C
--- 127.0.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.073/0.129/0.202/0.054 ms
#
 
SirDice said:
I suggest using nameserver <ip address of host> in your jail's resolv.conf
Click to expand...
Which one?
I have multiple interfaces with 9 addresses on the host.
Code: 
# ifconfig | grep '\..*\..*\.'
    inet 127.0.0.1 netmask 0xff000000
    inet 10.201.0.2 netmask 0xffff0000
    inet 10.201.0.5 netmask 0xffff0000
    inet 10.201.0.6 netmask 0xffff0000
    inet 10.201.0.254 netmask 0xffff0000
    inet 192.168.199.1 netmask 0xffffff00 broadcast 192.168.199.255
    inet 192.168.201.1 netmask 0xffffff00 broadcast 192.168.201.255
    inet CENSORED --> CENCORED netmask 0xffffffff
    inet 192.168.230.1 netmask 0xffffff00
I now tried to set every one of them and none worked with ping(8), but each worked with host(1) and drill(1).
 
You've configured PF on the host (at least for the NAT), any specific firewall rules that might be blocking the traffic?
 
Even with pass in quick all as the first rule it does not work.
Why would it even interfere when ping(8) is asking but not when other programs are?
Port 53 is served as intended and ICMP is working.
Everything looks to be correctly set up but something is misbehaving.
 
I spent ages struggling with this issue as well, one of the key hints was from this thread: https://forums.freebsd.org/threads/local-dns-not-working-in-jail.84979/.

It turns out that the request to unbound() can go through on `127.0.0.1` but it will respond on the address you came from. Unfortunately then the resolver library code will discard the response because it didn't come from the address it was requested from. One solution is to make each jail request from its own local address, but a much simpler and more portable option is actually to set `resolv.conf` to contain the line nameserver 0.0.0.0

This has the effect that it will make the request against localhost but then accept any response address. You can also leave out the `nameserver` entry to `resolv.conf` and the get same behaviour (or indeed delete the file all-together) but then drill() wont work, as it needs to parse this file itself instead of relying on system defaults.
 
Thank you! This makes sense, albeit it is really unintuitive and hard to debug.

It would also be nice if nameserver could be a unix socket and we could share DNS via VFS.
 
You should *never* use the localhost address for any communication between hosts/jails. That address comes with many implications by various pieces of software which will cause a myriad of problems if you misuse that address for actual communication between hosts.
Always use proper (rfc1918) addresses to communicate between jails and the host. This will make troubleshooting much easier (i.e. jails are treated just like hosts connected to a switch like in a physical network) as well as migrations.

You can also share sockets between host and jails via nullfs mounts, but make sure to use distinct mounts for each service/socket to minimize the security impact.
 
You must log in or register to reply here.
Back
Top