Solved FreeBSD 12.1-RELEASE + Apache 2.4.x + (problems installing ap24-mod_qos)

[StreetDancer]

Hey everyone!

I want to apply some stricter policies to my apache 2.4.x configuration to allow heavier hits. I am hardening with ap24-mod_qos; however when I go to install it and update pkg repositories; I am getting some strange errors that is leading me to believe it could be a security issue.

Could somebody please ease my mind and perhaps explain to me what is happening and how to remedy this?

Thank you very much!

OS is: FreeBSD 12.1-RELEASE
Updating inside a Jail [SERVER1]

What I did:

# pkg update
Updating FreeBSD repository catalogue...
[SERVER1] Fetching meta.txz: 100%    916 B   0.9kB/s    00:01    
[SERVER1] Fetching packagesite.txz: 100%    6 MiB   3.2MB/s    00:02    
Processing entries:   0%
Newer FreeBSD version for package py37-mlt:
To ignore this error set IGNORE_OSVERSION=yes
- package: 1202000
- running kernel: 1201000
Ignore the mismatch and continue? [Y/n]: n
pkg: repository FreeBSD contains packages for wrong OS version: FreeBSD:12:amd64
Processing entries: 100%
Unable to update repository FreeBSD
Error updating repositories!
root@

# pkg install ap24-mod_qos
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    ap24-mod_qos: 11.66

Number of packages to be installed: 1

90 KiB to be downloaded.

Proceed with this action? [y/N]: y
[SERVER1] [1/1] Fetching ap24-mod_qos-11.66.txz: 100%   89 KiB  91.6kB/s    00:01    
pkg: cached package ap24-mod_qos-11.66: size mismatch, fetching from remote
[SERVER1] Fetching ap24-mod_qos-11.66.txz: 100%   89 KiB  91.6kB/s    00:01    
pkg: cached package ap24-mod_qos-11.66: size mismatch, cannot continue
Consider running 'pkg update -f'
root@SERVER1:/usr/local/etc/apache24/extra # pkg install ap24-mod_qos
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    ap24-mod_qos: 11.66

Number of packages to be installed: 1

90 KiB to be downloaded.

Proceed with this action? [y/N]: n
root@

Thanks in advanced!

~ TruthSword

 

[VladiBG]

Update your version to 12.2-RELEASE
Don't forget to make a backup first.

 

[SirDice]

I am getting some strange errors that is leading me to believe it could be a security issue.

Yes, FreeBSD 12.1 is end-of-life and not supported any more. And looking at the version string you never applied the security patches for it either. So your biggest concern regarding security is to update the OS and keeping up with security patches. Keeping the OS updated is just one part, your next issue is going to be to update your packages. Because it looks like you never updated those either.

 

[StreetDancer]

Update your version to 12.2-RELEASE
Don't forget to make a backup first.

How do I update without breaking the server or the jails that are running? When you say "make a backup"; how does one backup their remote server in whole? With ezjail-admin I can archive the jails and pull those down. And what steps need to be taken to safely achieve this update.

I thought 12.1 was covered until 2024 when I looked at the FreeBSD website. I must have been wrong.

 

[SirDice]

How do I update without breaking the server or the jails that are running?

freebsd-update(8).

With ezjail-admin

     ezjail-admin command arguments... update -U -s 8.0-RELEASE
             Upgrade the base jail to the host system's release using
             freebsd-update(8). This may be used only to upgrade an existing
             installation. Tell freebsd-update which OS version to expect in
             the basejail via the -s option.

And 15.6.4. Updating Jails

When you say "make a backup"; how does one backup their remote server in whole?

I don't backup the OS and the applications because it's easier and quicker to install from scratch (I have everything documented how things are set up). You backup your data because that's what's important and can't be replaced.

I thought 12.1 was covered until 2024 when I looked at the FreeBSD website. I must have been wrong.

Major versions are supported for at least 5 years (so 12 is supported till 2024) but only the last minor version of that major release. Every minor version expires three months after the release of a new minor version. FreeBSD 12.2 was released in October 2020, that means that 12.1 expired February 2021. When 12.3 will be released (expected some time at the end of the year), 12.2 will expire three months later.

 

[StreetDancer]

freebsd-update(8).

     ezjail-admin command arguments... update -U -s 8.0-RELEASE
             Upgrade the base jail to the host system's release using
             freebsd-update(8). This may be used only to upgrade an existing
             installation. Tell freebsd-update which OS version to expect in
             the basejail via the -s option.

And 15.6.4. Updating Jails



I don't backup the OS and the applications because it's easier and quicker to install from scratch (I have everything documented how things are set up). You backup your data because that's what's important and can't be replaced.


Major versions are supported for at least 5 years (so 12 is supported till 2024) but only the last minor version of that major release. Every minor version expires three months after the release of a new minor version. FreeBSD 12.2 was released in October 2020, that means that 12.1 expired February 2021. When 12.3 will be released (expected some time at the end of the year), 12.2 will expire three months later.

SirDice,

Makes perfect sense; Thank you for your detailed response and clarification. I will make sure to implement updates in my FreeBSD administration duties and ensure that I am ready for E.O.L. minor revision updates to latest.

Appreciate it sir!

 

[VladiBG]

How do I update without breaking the server or the jails that are running?

Chapter 24. Updating and Upgrading FreeBSD

Information about how to keep a FreeBSD system up-to-date with freebsd-update or Git, how to rebuild and reinstall the entire base system, etc

docs.freebsd.org

During major upgrade you are breaking all installed userland applications that's why there's a step of rebuilding/reinstalling all of them so they match the new version of shared libs. For the minor upgrades most of the time the installed application doesn't require to be rebuild but i always rebuild them (i'm using only ports) just to be sure that there's none dependencies left pointing to the old version of the libs.

When you say "make a backup"; how does one backup their remote server in whole?

It depend of the file system. For UFS i'm using dump(8) and send the backup over ssh to remote server. For ZFS you can use zfs(8) send/recieve to backup your pool. For the virtualized host you can make a backup on the hypervisor level.
After you are done with the backup, test the backup for actual restore on virtual machine and on the same virtual machine you can test the upgrade process while you are taking notes of the steps which you perform. After you see that the upgrade is successful on the virtual test machine you can perform the actual upgrade on the server.

 

[StreetDancer]

Chapter 24. Updating and Upgrading FreeBSD

Information about how to keep a FreeBSD system up-to-date with freebsd-update or Git, how to rebuild and reinstall the entire base system, etc

docs.freebsd.org

During major upgrade you are breaking all installed userland applications that's why there's a step of rebuilding/reinstalling all of them so they match the new version of shared libs. For the minor upgrades most of the time the installed application doesn't require to be rebuild but i always rebuild them (i'm using only ports) just to be sure that there's none dependencies left pointing to the old version of the libs.


It depend of the file system. For UFS i'm using dump(8) and send the backup over ssh to remote server. For ZFS you can use zfs(8) send/recieve to backup your pool. For the virtualized host you can make a backup on the hypervisor level.
After you are done with the backup, test the backup for actual restore on virtual machine and on the same virtual machine you can test the upgrade process while you are taking notes of the steps which you perform. After you see that the upgrade is successful on the virtual test machine you can perform the actual upgrade on the server.

VladiBG:

I am currently running ZFS file system. I will read regarding zfs(8); Thank you for the detailed information as usual! Do you happen to know if there is a temperary way to ignore the version flag so that I can still install packages from ports?

Best Regards!

 

[VladiBG]

You will need to rebuild most of installed packages and most likely you will be unable to do it so it's better to upgrade to 12.2-RELEASE and continue to use packages. Before that make sure that you have full backup in case your mess up with something and you can restore that backup.

 

[richardtoohey2]

I'd also recommend - if you can - having a test machine with a set-up as close to your production/main machine(s). Then you can test installs/upgrades on the test machine and see what (if any) issues you encounter and how to resolve them.

It's near-impossible to get a second set-up exactly the same as another, but better than nothing.

 

[StreetDancer]

You will need to rebuild most of installed packages and most likely you will be unable to do it so it's better to upgrade to 12.2-RELEASE and continue to use packages. Before that make sure that you have full backup in case your mess up with something and you can restore that backup.

VladiBG,

Duly noted! I have had such good uptime and mitigated mid-range layer 7 ddos attacks; that I am frustrated that E.O.L. struck a knife in my juggler during intensive live development. Thank you for your help!

 

[StreetDancer]

I'd also recommend - if you can - having a test machine with a set-up as close to your production/main machine(s). Then you can test installs/upgrades on the test machine and see what (if any) issues you encounter and how to resolve them.

It's near-impossible to get a second set-up exactly the same as another, but better than nothing.

richardtoohey2,

Good to see you also! I am glad you mentioned that; I was thinking that I may need to purchase a laptop as a mirror of my production/live development server. I will replicate the best that I can and perhaps throw up a site under maintenance page(s) on domains/subdomains on a temp host until I can sort things out once I have made a backup as you recommended.

I agree; better than nothing. On a positive note; It would be good to get the server setup with 12.2-RELEASE so that I won't have another forced upgrade until 3 years.

Thanks again for everything and inviting me into the community of FreeBSD! The most beautiful operating system I have ever had the honor to geek to!

Best Regards,

~ TruthSword

 

[SirDice]

It would be good to get the server setup with 12.2-RELEASE so that I won't have another forced upgrade until 3 years.

Ehm, 12.3-RELEASE is likely to be released some time in September/October, 12.2 will be EoL three months after the release. Current schedule will have a release every 6 months. In March 13.0 will be released, 6 months later 12.3, 6 months later 13.1, then 12.4 (probably the last of the 12 series), 13.2, etc. So expect an upgrade for each major version once a year.

 

[richardtoohey2]

If you've got a server on the internet, you need to be patching frequently (regardless of OS and server programs used) - never going to be something you can park on the internet and leave for 3 years. Apache updates, MySQL updates (there have been lots of remote attacks recently that don't need authentication even), Intel CPU micro-code updates, OpenSSL, etc. So you might as well go whole hog and keep the OS updated as well.

In fact you will probably be forced to update the OS as well - the ports/packages will refuse to install/work on EOL FreeBSD versions.

 

[StreetDancer]

Ehm, 12.3-RELEASE is likely to be released some time in September/October, 12.2 will be EoL three months after the release. Current schedule will have a release every 6 months. In March 13.0 will be released, 6 months later 12.3, 6 months later 13.1, then 12.4 (probably the last of the 12 series), 13.2, etc. So expect an upgrade for each major version once a year.

SirDice,

Thank's for the detailed breakdown and what to expect!

Best Regards,

 

[StreetDancer]

If you've got a server on the internet, you need to be patching frequently (regardless of OS and server programs used) - never going to be something you can park on the internet and leave for 3 years. Apache updates, MySQL updates (there have been lots of remote attacks recently that don't need authentication even), Intel CPU micro-code updates, OpenSSL, etc. So you might as well go whole hog and keep the OS updated as well.

In fact you will probably be forced to update the OS as well - the ports/packages will refuse to install/work on EOL FreeBSD versions.

richardtoohey2:

Yes; you are absolutely dead on. I better understand the need to keep things updated and ready for full updates.

I took your advice and wiped a drive for a clean install on my Thinkpad T430 with FreeBSD 12.2-RELEASE. I have it running with a GUI also; now for the ezjail's and see if migration works (However; I am having a gut feeling that since the userland is updated to 12.2-RELEASE, that an ezjail-admin restore won't work properly since it was archived on 12.1-RELEASE).

I now see the need for batch scripts like bash for managing large and complicated configurations and site(s) pools.

Does everyone here use ipfw(8) as their Live Server firewall? I don't want to deviate from the topic; however you mentioned a good point regarding security.

Appreciate all your help as usual!

Best Regards,

 

[StreetDancer]

Just an update; I have successfully restored an ezjail archive from FreeBSD 12.1-RELEASE (Live Server) onto my brand new Thinkpad T430 FreeBSD 12.2-RELEASE installation (Offline New Server / Development).

I had to edit various factors; however once I did it correctly, I was able to start the jail up and re-configure the settings within; and eventually starting my MySQL Server. *got a little excited and wanted to say thanks and update my progress*.

What would be a good place on the forums here for me to write up a thread on ezjail migration from FreeBSD 12.1-RELEASE to FreeBSD 12.2-RELEASE?

Best Regards,

~ TruthSword

 

[Mjölnir]

I now see the need for batch scripts like bash for managing large and complicated configurations and site(s) pools.

I'd recommend to use the standard Bourne sh(1)ell for (reasonably simple) scripts instead, simply because it's in base. shells/bash may be more convenient, but if you think about it, it saves you a few keystrokes, but the price is that you invite some nasty bugs, too. You can find a lot of mature solutions for sh(1)ell scripting in rc.subr(8) and the like. For non-trivial scripts, please use an advanced scripting language that has modules (->namespaces) & all bells & whistles. Maybe LUA is the way to go, and IIUC it's in base, too. YMMV.

Does everyone here use ipfw(8) as their Live Server firewall? I don't want to deviate from the topic; however you mentioned a good point regarding security.

Better than nothing: use one of the standard rc.firewall(8) (manpage outdated, RTSL instead) packet filters. I.e. FMLU, the sysrc firewall_type=workstation "firewall" (it's not, it's a packet filter) can be configured with the existing configuration variables, which you can set into rc.conf(5) with sysrc firewall_myservices="80/tcp 22/tcp"

Appreciate all your help as usual!

You'll help 2. Read either way.

Spoiler: rc.firewall.patch

--- /etc/rc.firewall.orig
+++ /etc/rc.firewall
@@ -148,6 +148,7 @@

setup_loopback
setup_ipv6_mandatory
+${fwcmd} add 5000 reset ip from table\(22\) to me

############
# Network Address Translation. All packets are passed to natd(8)
@@ -436,6 +437,10 @@
# firewall_logdeny: Boolean (YES/NO) specifying if the
# default denied packets should be
# logged (in /var/log/security).
+ # firewall_logamount: Max # of packets to log. If empty, use
+ # net.inet.ip.fw.verbose_limit.
+ # Zero: unlimited. Default: 500.
+ # See also: RTFM ipfw(8).
# firewall_nologports: List of TCP/UDP ports for which
# denied incoming packets are not
# logged.
@@ -538,7 +543,7 @@
# Deny and (if wanted) log the rest unconditionally.
log=""
if [ ${firewall_logdeny:-x} = "YES" -o ${firewall_logdeny:-x} = "yes" ] ; then
- log="log logamount 500" # The default of 100 is too low.
+ log="log logamount ${firewall_logamount:-500}" # The default of 100 is too low.
sysctl net.inet.ip.fw.verbose=1 >/dev/null
fi
${fwcmd} add deny $log ip from any to any

As you already have guessed right, that 1st change is for sshguard-setup(7) to block brute-force attackers from your ssh(1) port. It should be in security/sshguard's service(8) script, but it isn't (yet).
PS: I have a RCS subdirectory in {/usr/local,}/etc & those sub-folders I make changes in, and use oldy but goldy devel/rcs ( alias ci ci -l; alias co co -l) because it can be used by root & it's perfectly enough to do the task of a VCS for config files.

 

[Mjölnir]

P.S.: psearch -s 'configuration management' shows (a subset of) what's available in the ports(7) tree. Maybe SirDice can comment on some, IIRC he's using sysutils/puppet? Beeing somehow oldschool, I'd go for sysutils/cfengine317sysutils/cfengine315, but "Cfengine is an autonomic maintenance system not merely a change management roll-out tool.", so there might be good reasons that puppet had it's hipe.

 

[StreetDancer]

uccessfully restored an ezjail archive from FreeBSD 12.1-RELEASE (Live Server) onto my brand ne

I'd recommend to use the standard Bourne sh(1)ell for (reasonably simple) scripts instead, simply because it's in base. shells/bash may be more convenient, but if you think about it, it saves you a few keystrokes, but the price is that you invite some nasty bugs, too. You can find a lot of mature solutions for sh(1)ell scripting in rc.subr(8) and the like. For non-trivial scripts, please use an advanced scripting language that has modules (->namespaces) & all bells & whistles. Maybe LUA is the way to go, and IIUC it's in base, too. YMMV.

Better than nothing: use one of the standard rc.firewall(8) (manpage outdated, RTSL instead) packet filters. I.e. FMLU, the sysrc firewall_type=workstation "firewall" (it's not, it's a packet filter) can be configured with the existing configuration variables, which you can set into rc.conf(5) with sysrc firewall_myservices="80/tcp 22/tcp"

You'll help 2. Read either way.

Spoiler: rc.firewall.patch

--- /etc/rc.firewall.orig
+++ /etc/rc.firewall
@@ -148,6 +148,7 @@

setup_loopback
setup_ipv6_mandatory
+${fwcmd} add 5000 reset ip from table\(22\) to me

############
# Network Address Translation. All packets are passed to natd(8)
@@ -436,6 +437,10 @@
# firewall_logdeny: Boolean (YES/NO) specifying if the
# default denied packets should be
# logged (in /var/log/security).
+ # firewall_logamount: Max # of packets to log. If empty, use
+ # net.inet.ip.fw.verbose_limit.
+ # Zero: unlimited. Default: 500.
+ # See also: RTFM ipfw(8).
# firewall_nologports: List of TCP/UDP ports for which
# denied incoming packets are not
# logged.
@@ -538,7 +543,7 @@
# Deny and (if wanted) log the rest unconditionally.
log=""
if [ ${firewall_logdeny:-x} = "YES" -o ${firewall_logdeny:-x} = "yes" ] ; then
- log="log logamount 500" # The default of 100 is too low.
+ log="log logamount ${firewall_logamount:-500}" # The default of 100 is too low.
sysctl net.inet.ip.fw.verbose=1 >/dev/null
fi
${fwcmd} add deny $log ip from any to any

As you already have guessed right, that 1st change is for sshguard-setup(7) to block brute-force attackers from your ssh(1) port. It should be in security/sshguard's service(8) script, but it isn't (yet).
PS: I have a RCS subdirectory in {/usr/local,}/etc & those sub-folders I make changes in, and use oldy but goldy devel/rcs ( alias ci ci -l; alias co co -l) because it can be used by root & it's perfectly enough to do the task of a VCS for config files.

Mjölnir,​

Thank you for the wealth of information. I will conduct my studies on these next level ups and learning curves.

Best Regards,