HOWTO: 9.1-RELEASE setup from scratch (Basic tools, ZFS, NFS, Qjails, and more)

J

junovitch@

Developer
Good evening FreeBSD forums!

I've been lurking the forums and reading for some time while I've been putting together my own FreeBSD based home server for backups, testing, and personal hosting. I've finally compiled all my notes together into one place and posted them to my Github page. For me, this was mostly copy/paste once I did a clean install and SSH'd in for the first time. Although it is specific to my setup, there is a lot of good info here in one place and hopefully you can find this useful if you are just trying out FreeBSD for the first time and wondering where to start.

The following are the key areas I covered:

- Pkg setup and Ports installations (using Portmaster) of several utilities
- Setup of OpenNTPD for time and NUT for UPS monitoring
- OpenSMTPD for external emailing of periodic emails (including IPMI checks)
- ZFS setup for AF 4K drives and snapshotting script
- Mostly jail friendly NFSv4 support (with v2/v3 support commented)
- Qjail setup split to DMZ/LAN segments with sysloging to the host system
- Basic FTP/iRedMail/Wordpress/Owncloud/OpenVPN Certificate Authority jails

I would be glad to come up with a more specific write-up or answer any questions on anything here. Feel free to let me know!

Link to the repo:
https://github.com/junovitch/my-freebsd-build

Direct link to the guide:
https://github.com/junovitch/my-freebsd-build/blob/master/freebsd-install-guide-public
 
Just to add, there are a few areas I still have to finalize and merge into the Github script.

1. My current NUT config doesn't shutdown on upon the UPS battery going critical. Once it works I will add it in.

2. Jails don't send packets on the physical interface they are bound to. Although my LAN jails are bound to em1, they will send out packets on em0 and get blocked by my firewall. The static route is just a work-around so packets internal to my LAN go to the LAN default gateway. I may switch to PF route-to statements as that seems like the appropriate fix action but I still have to read up on it.

3. Logcheck produces way too much output with everything pointed at it so I still have to clean that up.

4. My NFS setup isn't 100% jail friendly and still causes these warnings when creating new jails.

Code: 
Warning: Some services already seem to be listening on all IP, (including 10.100.102.12)
  This may cause some confusion, here they are:
root     rpcbind    50338 8  udp6   *:872                 *:*
root     rpcbind    50338 13 udp4   *:984                 *:*
root     nfsuserd   50326 3  udp4   *:788                 *:*
root     nfsuserd   50325 3  udp4   *:788                 *:*
root     nfsuserd   50324 3  udp4   *:788                 *:*
root     nfsuserd   50323 3  udp4   *:788                 *:*
root     nfsuserd   50322 3  udp4   *:788                 *:*

5. Write some Unison/Rsync scripts that will work for me.

6. Learn Puppet and see if I can get the Puppet jail to handle configuration management of my Linux desktops and laptop.
 
You state "Static IP address setup with extra aliases for jails" But you code 10.100.0.0 this is not a public routable ip address. Maybe static is wrong word to use here. Static generally means static ip assigned by your ISP. What is true meaning? How about adding a drawing of your network to clarify what it looks like.
 
That's quite true, my wording could be clearer. I meant static in the sense that it was statically assigned in general rather than a statically assigned address by my ISP. As I said this is a home project so I only have 1 publicly routeable IPv4 address. My IPv6 addresses are provided by a Hurricane Electric's tunnel broker service and are globally routeable. I will come up with some kind of diagram and fix the wording a bit in the next few days. Thanks for your input.

In the meantime here is a very quick summary.

em0 (DMZ), 192.168.102.2/24, .10-.19/32
Netgear router eth0.102, (DMZ default gateway), 192.168.102.1

em1 (LAN), 10.100.102.2/24, .10-.19/32
Unmanaged Netgear switch
Netgear router eth0.103, (LAN default gateway), 10.100.102.1

My Netgear router runs port forwarding from the WAN-DMZ only and allows forwarding LAN-WAN, LAN-DMZ, and DMZ-WAN.
 
Are you saying your ISP issues you this IP address 192.168.102.2? If so then your ISP really has you on a wlan and is NATing your traffic at the ISP before it's released to the public Internet. Or the ISP modem/router at your location is assigned a real dynamic public routable IP address and the modem/router NATs all traffic for you and assigns private IP address for all wired and wifi connections. This is how ATT Internet access works. All this is transparent to you. But this fact should be mentioned in your script to clarify your network configuration for public readers. I comment my own scripts as reminders to what I an doing and why, But to other readers there is so much info missing no one can follow what is going and why. Just a suggestion.

I see no reason why you populate /etc/resolv.conf manually and why you don't use ifconfig_em0="DHCP" statement which you have commented out. Using DHCP will automatically update your host's /etc/resolv.conf with new ISP info when the ISP changes your home service dynamic IP address.

Using your method your whole network loses public Internet access whenever your ISP changes your dynamic IP address. Another missing piece of info is, do you have registered domains that point to your equipment and what process do you use to update your domain name with your new dynamic IP address when it changes?

In your follow up post you say, "few areas I still have to finalize" maybe I can help you with some of those items.

Item number 2. This problem is caused by ezjail. Ezjail doesn’t have option to bind IP address to NIC name even though this option is available when coding jails natively in rc.conf. This option IS available in qjail.

Item number 4. Those warning messages pop out of ezjail for every jail you create. Haven’t you noticed that before? There is a PR reporting this to the author but no action in last 3 years.

On the subject of NFS in a jail. There are PR's on this posted on the FreeBSD PR system. If you read through the PR thread the recommendation is not to use NFS in a jail but run it on the host. It’s just not reliable in a jail if you can get it to work at all. Something about NFS being supported in the kernel and jail being support in userland if I remember correctly. I saw a post on current mailing list early last year where someone was working on jailed NFS support for 10.0. I have not seen any comment of any work for NFS jail support so maybe work has come to an end for now. Just don’t know. I run NFS on the host with no problems and no fooling around to get it functional.
 
Good day,

Please see the diagram below for a better visual on my topology. My ISP does issue a routeable address. My main reason for all static addressing was primarily because ezjail doesn't make an alias for you. Configuring an ezjail using an IP not already assigned gives an "IP x.x.x.x not configured on a local interface" error. I decided that since I was already setting my aliases statically to accommodate ezjails that the interface itself would be set the same way for consistency (along with resolv.conf). I don't see Comcast providing IPv6 addresses for some time so the addresses that I use will be stay exactly as set for a few years whether I used DHCP or static. It seemed less cumbersome to set all the aliases and use ezjail compared to setting up jails directly in rc.conf. However, after reviewing the man page for sysutils/qjail, having the "-n" flag to pick an interface seems like it would be the way to go and would certainly simplify things while keeping the convenience of ezjail.

For your mention of item 2, my understanding is that since jails aren't fully virtualized and still share the routing table of the host, that this would still be an issue. After all, you can't do a `service routing restart` from within a jail. I'm still learning the ins and outs of jails but I would have thought any jail implementation would have this until the network stack has change or you use setfib in some fashion to deal with it.

For your mention of item 4, I am aware of the PR on rpcbind not honoring the "-h" flag to bind to certain IPs. The man page for nfsuserd doesn't offer any flag to bind to any address. It's not completely ideal, but my firewall filters requests to those ports before anything can come in from the WAN so no real impact. Regarding NFS and jails, my NFS runs on the host system and I'm aware of issues running it in jails. I don't intent to run NFS that way although I did come across some userspace NFS implementations that could fit the bill. No need in my situation though.

Thanks for your comments on the matter, I didn't know about the qjail utility and it seems more actively maintained based on recent activity on freshports. I think the "-n" flag alone would make it worth it and would simplify my setup. I'll be doing some reading on it. Your website as well has a lot of good information as well. Cheers!

r1lqhu.jpg
 
Fbsd1, I took your advise and changed my primary interface to DHCP. I did leave my LAN side interface manually set as I have my NFS, SSH, syslogd, and OpenSMTPD bound to that address. My guide has been revised to reflect using sysutils/qjail and taking advantage of the fact that it creates and destroys aliases as it goes.

I did run into one issue and had to come up with some kind of fix action. As you are the maintainer of qjails we'll discuss it over email.

EDIT:
After our discussion the best fix would be to remove the following directories to prevent accidently overwriting files when creating new jails. This has been worked into the steps.
# rm -rf /usr/jails/flavors/default/usr/share
# rm -rf /usr/jails/flavors/ssh-default/usr/share
 
Junovitch,

Have you had any problems with pkg on FreeBSD9.1 release? I installed 9.1 release tonight and was following your guide, I did the /usr/sbin/pkg to bootstrap the install of pkg and then tried to install portmaster and I'm getting all kinds of build errors for pkg.

Code: 
<snip>
cc1: warnings being treated as errors
backup.c: In function 'copy_database':
backup.c:64: warning: 'done' may be used uninitialized in this function
backup.c:63: warning: 'total' may be used uninitialized in this function
cc1: warnings being treated as errors
backup.c: In function 'copy_database':
backup.c:64: warning: 'done' may be used uninitialized in this function
backup.c:63: warning: 'total' may be used uninitialized in this function
*** [backup.o] Error code 1
*** [backup.So] Error code 1
2 errors
*** [all] Error code 2
1 error
*** [do-build] Error code 1

Stop in /usr/ports/ports-mgmt/pkg.
*** [run-depends] Error code 1

Stop in /usr/ports/ports-mgmt/portmaster.
*** [install] Error code 1
 
That's very peculiar. When you run /usr/sbin/pkg that bootstraps the system with pkg-1.0.2. Once you run the portsnap fetch extract you end up with pkg-1.0.4 in the ports collection. Installing ports-mgmt/portmaster means that both portmaster and the latest pkg will get installed. I setup a VM and tried running the portmaster install before the portsnap. That installs it just fine. However, the same problem comes up in that once you update the ports collection it always will automatically try to update pkg.

I've been able to replicate the error by doing cd /usr/ports/ports-mgmt/pkg && make install to show it and I've saved all the output to a text file. I'm going to look into it. I'm sorry but the best thing I can suggest for today is to skip out on using pkg for the time being. Although pkg is cool and has some potential, it's still a work in progress.
 
I applied the above fix manually but now I got another error message... I think it would be better to wait until port maintainer fixes the issue. Hopefully it won't take long.
 
I applied same fix in files (under work directory, i used find to locate them) with missing uninitialized vars:

pkg_elf.c: In function 'analyse_elf':
pkg_elf.c:199: warning: 'sh_link' may be used uninitialized in this function
pkg_elf.c:198: warning: 'numdyn' may be used uninitialized in this function

pkg_repo.c: In function 'pkg_create_repo':
pkg_repo.c:523: warning: 'ret' may be used uninitialized in this function

create.c: In function 'exec_create':
create.c:76: warning: 'format' may be used uninitialized in this function

update.c: In function 'pkgcli_update':
update.c:52: warning: 'retcode' may be used uninitialized in this function

It works.
I cross my finger waiting an "official" patch.
 
Try removing these files in /usr/ports/ports-mgmt/pkg and rebuilding again:

files/patch-libpkg__Makefile
files/patch-pkg__Makefile

Do make clean before you rebuild the port.
 
I followed the part about setting up OpenSMTPD from your guide, but I can't get it to work for the life of me.
After following your example, my smtpd.conf looks like this:
Code: 
#       $OpenBSD: smtpd.conf,v 1.5 2012/10/11 21:16:28 gilles Exp $

# This is the smtpd server system-wide configuration file.
# See smtpd.conf(5) for more information.

# To accept external mail, replace with: listen on all
#
# XXX try to find a portable way to get the IP interface                                                   
#listen on lo0                                                                                             
listen on 10.23.1.11                                                                                       
                                                                                                           
                                                                                                           
table aliases db:/usr/local/etc/mail/aliases.db                                                            
                                                                                                           
# Uncomment the following to accept external mail for domain "example.org"                                 
#                                                                                                          
# accept from any for domain "example.org" alias <aliases> deliver to mbox                                 
accept for local alias <aliases> deliver to mbox                                                           
accept for any relay                                                                                       
accept from local for domain "gmail.com" relay via tls+auth://smtp.gmail.com:587 auth secrets              
accept from 10.23.0.0/16 for local alias aliases deliver to mbox

However, when I run
# newaliases
I get the following errors:
Code: 
# newaliases 
/usr/local/etc/mail/smtpd.conf:13: invalid backend configuration for table aliases
/usr/local/etc/mail/smtpd.conf:20: invalid use of table "<dynamic:8>" as AUTH parameter
/usr/local/etc/mail/smtpd.conf:21: invalid use of table "<dynamic:12>" as ALIAS parameter

For added fun, the error about the invalid backend also happens when I use smtpd.conf.sample without modifications.
My /etc/aliases is default, besides aliasing root to my gmail address. It's also copied to /usr/local/etc/mail.
What did I do wrong? Google did not turn up any helpful results.

Cheers,

-joe
 
Good day. It's all fixed. The issue was because of the new syntax in the latest mail/opensmtpd snapshots. I just updated myself to version 201301281310 which was released a few hours ago and fixed my guide to match. I had the same issue earlier this month when it was first release with the syntax changes and couldn't figure it out. I found out there was a problem with the port so I downgraded back to version 201210090136. Sorry for the hassle. See the discussion on it below for more on the syntax changes and initial issues.

http://forums.freebsd.org/showthread.php?t=36745
 
You must log in or register to reply here.
Back
Top