Solved symlink issue with VNET Thin Jail Using NullFS

S

SamKook

I've created a thin jail using nullfs following the handbook jail page that is also using zfs and VNET so I can run some apps in a separate vlan and almost everything is working as expected, except that the jail is missing access to usr/local so I have none of the apps I installed, which I assume the jail is supposed to point to the ones from the host system once it is running, but I'm not too clear on that since the handbook only says how and I'm pretty new to jails.

The only deviation from that doc was the use of 14.3-RELEASE instead of 14.4-RELEASE(is that out?) and this line I found in this post since moving the var folder to the skeleton one was creating an error because of the empty folder inside.
Code: 
chflags -Rvx noschg /usr/local/jails/templates/14.3-RELEASE-base/var
But I also tried it exactly like the doc first, ignoring that error. I deleted the snapshot and re-did it properly once I saw it had issues, but it did not help.

First, I'd like to know if my understanding that my installed apps should be usable by the jail is correct since I'm just wasting my time if not.

And second, what should I look at to fix this? If I look into the final thinjail-nullfs-base folder, both the home and usr/local folder in the skeleton folder are completely empty and the rest of them have stuff but it differs from the host system.

The host is 14.3-RELEASE-p5 if that is relevant and the vnet stuff does seem to work since I have the correct ip if I do ifconfig using the toor user(since both my root and user use zsh, I can't use the jail with either currently).
 
usually that is by design, as thin jails are expected to reference a common base system but have separate packages installed. if you want to null mount the host's /usr/local you'd have to do that on your own, but then the package system in the jail would be out of sync with the actual filesystem. that might be ok for your use case, but you'd have to make sure not to actually use the pkg command in the jail.
 
I can't even get pkg in the jail though since I'm just getting a bunch of "Certificate verification failed" errors if I do pkg update, unless that's a toor specific issue, I haven't tried temporarily fixing my root shell.

edit: Changed the root shell to try and same problem.

A portion of the errors:
Code: 
Certificate verification failed for /C=US/O=Let's Encrypt/CN=E8
0810C586723A0000:error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:/usr/src/crypto/openssl
/crypto/store/store_register.c:237:scheme=file
0810C586723A0000:error:80000002:system library:file_open:No such file or directory:/usr/src/crypto/openssl/providers/
implementations/storemgmt/file_store.c:267:calling stat(/etc/ssl/certs/4042bcee.0)
0810C586723A0000:error:16000069:STORE routines:ossl_store_get0_loader_int:unregistered scheme:/usr/src/crypto/openssl
/crypto/store/store_register.c:237:scheme=file
0810C586723A0000:error:80000002:system library:file_open:No such file or directory:/usr/src/crypto/openssl/providers/
implementations/storemgmt/file_store.c:267:calling stat(/etc/ssl/certs/4042bcee.0)
0810C586723A0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:/usr/src/c
rypto/openssl/ssl/statem/statem_clnt.c:1890:
 
Oh, it's copied from the base system, which is an issue since the jail is in a different vlan, didn't think of that.
 
From the handbook, resolv and localtime are the only 2 things that are copied:
Code: 
cp /etc/resolv.conf /usr/local/jails/templates/14.3-RELEASE/etc/resolv.conf
cp /etc/localtime /usr/local/jails/templates/14.3-RELEASE/etc/localtime

Changing the resolv file to the correct gateway for the vlan does make the errors go away, but apparently I have a dns issue in the vlan since no hosts resolve, which is odd since it was working in a VM, but that is likely a separate issue.
 
Hmm, the weird error is back now that hosts resolve so I guess using either gateway works.

Is /usr/src supposed to be empty?

Edit: Running "certctl rehash" fixed the problem, I can now use pkg, finally.
 
You must log in or register to reply here.
Back
Top