No IPv6 when using qjail but it works with vanilla jails?

[Thorny]

Aloha,

i'm having a quite dubious problem. Currently there are 6 jails on my server and i want to migrate them to be managed by sysutils/qjail. All jails are currently build via make buildworld etc.

Now i created with qjail create my first jail. I copied /etc/resolv.conf and /etc/rc.conf and also all the rest of the data. qjail start and everything is fine. Great.

I started the second qjail created, migrated all the data, but i could not connect to the internet. After multiple tries i can say:
- i can ping to IPv6 addresses at the same server
- i can't do anything else
- this limitation are jail-based - but only when created with qjail; i tried various IPv6 addresses and other jails.

Here my configuration. My rc.conf:

test / >cat /etc/rc.conf
# No network interfaces in jails
network_interfaces=""

# Prevent rpc
rpcbind_enable="NO"

# Prevent jails from doing their cron jobs at the same time
cron_flags=" -J 60"

# Prevent syslog from opening sockets
syslogd_enable="YES"
syslogd_flags="-ss"

# Prevent sendmail from trying to connect to localhost
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

ifconfig_em0_ipv6="inet6 2a01:4f8:10b:9c3::10/64"
ipv6_defaultrouter="fe80::1%em0"



My resolv.conf

test / >cat /etc/resolv.conf
# Generated by resolvconf
search your-server.de
nameserver 2620:0:ccc::2
nameserver 2620:0:ccd::2

This is ifconfig:

test / >ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 90:1b:0e:cb:ce:37
hwaddr 90:1b:0e:cb:ce:37
inet6 2a01:4f8:96b:9c3::10 prefixlen 128
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo


Ping IPv6 to the same server:
test / >ping6 -c 2 2a01:4f8:10b:9c3::3
PING6(56=40+8+8 bytes) 2a01:4f8:96b:9c3::10 --> 2a01:4f8:10b:9c3::3
16 bytes from 2a01:4f8:10b:9c3::3, icmp_seq=0 hlim=64 time=0.093 ms
16 bytes from 2a01:4f8:10b:9c3::3, icmp_seq=1 hlim=64 time=0.096 ms
--- 2a01:4f8:10b:9c3::3 ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.093/0.095/0.096/0.001 ms


Ping IPv6 to another of mine server:

test / >ping6 -c 2 2a01:4f8:150:50a5::6
PING6(56=40+8+8 bytes) 2a01:4f8:96b:9c3::10 --> 2a01:4f8:150:50a5::6


Can't even ping my gateway:

test / >ping6 -c 2 fe80::1
ping6: UDP connect: Network is unreachable
test / >ping6 -c 2 fe80::1%em0
PING6(56=40+8+8 bytes) 2a01:4f8:96b:9c3::10 --> fe80::1%em0
^C
--- fe80::1%em0 ping6 statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss


How can i debug / fix this?

 

[SirDice]

The IP address is assigned by the qjail configuration, I'm guessing you only assigned an IPv4 address to the jail.

 

[Thorny]

The IP address is assigned by the qjail configuration, I'm guessing you only assigned an IPv4 address to the jail.

No, i already checked this. Also the IPv4 address should have shown up in ifconfig, shouldn't it?
This is the qjail config:

# cat  /usr/local/etc/qjail.global/test 
name="test"
ip4=""
ip6="2a01:4f8:96b:9c3::10"
path="/usr/jails/test"
interface="em0"

# cat /usr/local/etc/qjail.local/test 
name="test"
ip4=""
ip6="2a01:4f8:96b:9c3::10"
path="/usr/jails/test"
interface="em0"

# cat /usr/local/etc/qjail.config/test 
test { 
host.hostname       =  "test";
path                =  "/usr/jails/test";
mount.fstab         =  "/usr/local/etc/qjail.fstab/test";
exec.consolelog     =  "/var/log/qjail.test.console.log";
mount.devfs;
ip6.addr            =  2a01:4f8:96b:9c3::10;
interface           =  "em0";
allow.raw_sockets;
devfs_ruleset       =  "4";
allow.sysvipc;
exec.start          =  "/bin/sh /etc/rc";
exec.stop           =  "/bin/sh /etc/rc.shutdown";
}

Also the first jail created with qjail is IPv6 only too. Just for another test i removed the ipv4 lines and tried again - nothing changed.

 

[fbsd1]

1. What version of FreeBSD are you running?
2. Is your FreeBSD host & jails running on real hardware or VM?
3. Have you read the "qjail" ip6 man page and tested using the example shown there?
man qjail-ipv6-testing
4. The jails /etc/rc.conf doesn't require the ipv6 ifconfig statements you added. Remove them.
5. The jails /etc/resolv.conf is auto populated by the "jail create" command. Do not manually add any thing to it.
6. On the host did you allocate an alias for the hosts ipv6 ip address range assigned to the jails? The example demonstrates qjail works with ipv6 address. You must have an ipv6 assignment problem.
7. To issue console commands from a jail you have to use the "qjail console" command or if ssh has been activated in the jail, ssh log into the jail. How are you issuing jail console commands?

 

[Thorny]

1. What version of FreeBSD are you running?
2. Is your FreeBSD host & jails running on real hardware or VM?
3. Have you read the "qjail" ip6 man page and tested using the example shown there?
man qjail-ipv6-testing
4. The jails /etc/rc.conf doesn't require the ipv6 ifconfig statements you added. Remove them.
5. The jails /etc/resolv.conf is auto populated by the "jail create" command. Do not manually add any thing to it.
6. On the host did you allocate an alias for the hosts ipv6 ip address range assigned to the jails? The example demonstrates qjail works with ipv6 address. You must have an ipv6 assignment problem.
7. To issue console commands from a jail you have to use the "qjail console" command or if ssh has been activated in the jail, ssh log into the jail. How are you issuing jail console commands?

1. 11.1-RELEASE
2. Real Hardware
3. Yes, the example do not work either
4. It also do not work without them. I tested both variants
5. In this case nothing would work. The /etc/resolv.conf populated by qjail create only included IPv4 Nameserver, but qjail create was done with -6 (and no -4 given).
6. No, i didn't. The new address seems to be assigned automatically since it is displayed in ifconfig. I also expected this behavior since its the same for vanilla jails. I tested it right now: no changes (except from all other jails now having no more IPs )
7. I'm using "jexec test" to log into the jail and issue the commands. There should be no different to qjail console or is there?

Also: thanks for your reply!

 

[fbsd1]

Your first post stated you already have native jails IPV4 working. Post the host /etc/jail.conf file that defines your native jails.
You have to use different jail names and IPV4 address than your existing native jail environment use to be able
to have the native jails and the qjail jails running at the same time.
I would suggest you first try getting qjail IPV4 setup working.

You stated that the qjail IPV6 example did not work for you.
This means you have something incorrectly setup with your host network.
Is this your first time working with IPV6 addresses?

Is your ISP assigning you IPV6 address range?

If no IPV6 addresses from your ISP then you need to use one of the local IPV6 address ranges
reserved for local use just like 10.0.0.0/8 is one of the ranges reserved for IPV4 local use.
Local uses means non-routeable on the public internet.

Describe your host & jail network.
Are you creating non-vnet jails or vnet jails?
Do you have a business account with your ISP or a single home account?
Does your ISP offer IPV6 service? Most do not. Have you contacted your ISP to verify they offer IPV6 service?
You can use IPV6 on jails and a local LAN only if you use one of the IPV6 address ranges reserved for that purpose.

Post the 'qjail create' statement of each jail you want to use.
After a clean boot of the host system post the HOST'S /etc/rc.conf and /etc/resolv.conf and ifconfig -a

 

[Thorny]

Your first post stated you already have native jails IPV4 working. Post the host /etc/jail.conf file that defines your native jails.
You have to use different jail names and IPV4 address than your existing native jail environment use to be able
to have the native jails and the qjail jails running at the same time.
I would suggest you first try getting qjail IPV4 setup working.

I did use different jail names. And i have no IPv4 addresses to spare. That is the reason i use IPv6 only jails.

You stated that the qjail IPV6 example did not work for you.
This means you have something incorrectly setup with your host network.
Is this your first time working with IPV6 addresses?

I must disappoint you and cut the road your taking. My host network is fine and its not the first time i'm working with IPv6 adresses. Its a problem only occurring when using qjail - which strongly indicates a bug.

At least i stopped putting hours into debugging this qjail-issue and just wrote some short shell-scripts. Now everything works fine the way i need, while managing vanilla jails got much easier. Thanks for your time!