certbot suddenly fails

tsk · Jun 13, 2023

Hi,

We are using certbot to update certificates from letsencrypt. It used to work for several years but since two days it fails.
First problem was that it doesn't find mod_ssl.so any more because it searched in a different directory.
Then it wasn't able to restart apache any more with:
Unable to restart apache using ['apachectl', 'graceful']
I didn't change anything in the configuration, but some weeks ago I updated from apache 2.4.54 to 2.5.56.

Can anybody help?

SirDice · Jun 13, 2023

What version of FreeBSD? And how up to date are your installed ports/packages? I have certbot running just fine on a 13.2-RELEASE machine.

tsk · Jun 13, 2023

FreeBSD 13.2-Release
py38-certbot-1.22.0,1
py38-certbot-apache-1.22.0
apache24-2.4.56
As I said, 3 days ago it worked fine. I didn't change anything.

SirDice · Jun 13, 2023

What is the exact error message you're getting?

hruodr · Jun 13, 2023

SirDice said:
I have certbot running just fine on

I use this under debian:

GitHub - dehydrated-io/dehydrated: letsencrypt/acme client implemented as a shell-script – just add water

letsencrypt/acme client implemented as a shell-script – just add water - dehydrated-io/dehydrated

github.com

Few and non strange dependencies. I found certbot too complicated.

It would be nice if FreeBSD had a standard acme client in base like OpenBSD, or better, the same one:

acme-client(1) - OpenBSD manual pages

tsk · Jun 13, 2023

Code:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /usr/local/etc/letsencrypt/renewal/mecadtron.de.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for mecadtron.de and [URL="http://www.mecadtron.de"]www.mecadtron.de[/URL]
Error while running apachectl graceful.
apache24 not running? (check /var/run/httpd.pid).


Unable to restart apache using ['apachectl', 'graceful']
Error while running apachectl graceful.
apache24 not running? (check /var/run/httpd.pid).


Unable to restart apache using ['apachectl', 'graceful']
Encountered exception during recovery: certbot.errors.MisconfigurationError: Error while running apachectl graceful.
apache24 not running? (check /var/run/httpd.pid).
Failed to renew certificate mecadtron.de with error: Error while running apachectl graceful.
apache24 not running? (check /var/run/httpd.pid).



- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /usr/local/etc/letsencrypt/live/mecadtron.de/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at [URL]https://community.letsencrypt.org[/URL]. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
r

hruodr · Jun 13, 2023

tsk said:
Unable to restart apache using ['apachectl', 'graceful']
Encountered exception during recovery: certbot.errors.MisconfigurationError: Error while running apachectl graceful.
apache24 not running? (check /var/run/httpd.pid).

A http server must run, you must care to keep it running, but why should certboot do it?
As said, cerbot too complicated.

SirDice · Jun 13, 2023

hruodr said:
It would be nice if FreeBSD had a standard acme client like OpenBSD, or better, the same one:

security/acme.sh

SirDice · Jun 13, 2023

I've configured certbot to simply use a directory. And I have that directory mapped in nginx (don't use Apache here but you could do the same thing) to /.well-known/acme-challenge.

Code:

        location /.well-known/acme-challenge {
                alias /usr/local/www/acme/.well-known/acme-challenge;
                autoindex off;
        }

Then used a couple of simple scripts in renewal-hooks/deploy/ and renewal-hooks/post/ to copy the certificates to where I need them and restart/reload various services. Never used the apache or nginx plugins from certbot itself.

Does Apache properly reload if you use service apache24 reload? I never use apachectl(8) directly.

SirDice · Jun 13, 2023

hruodr said:
A http server must run, you must care to keep it running, but why should certboot do it?

At the very least it needs to be reloaded in order to pick up the changed certificates.

hruodr · Jun 13, 2023

SirDice said:
security/acme.sh

Indeed there is a portable version of OpenBSD acme client, but it is not a sh script, namely not that.
But it would be perhaps good to have such a client in base.

SirDice said:
At the very least it needs to be reloaded in order to pick up the changed certificates.

What needs to be reloaded? The configuration file? No, it is not!

facedebouc · Jun 13, 2023

tsk said:
FreeBSD 13.2-Release
py38-certbot-1.22.0,1
py38-certbot-apache-1.22.0
apache24-2.4.56
As I said, 3 days ago it worked fine. I didn't change anything.

My security/py-certbot version is currently py39-certbot-2.4.0,1.

SirDice · Jun 13, 2023

Now that you mention it. It should be Python 3.9, not 3.8. The default Python changed some time ago. And freshports is showing no versions available for FreeBSD:13:amd64, which indicates some build issues but I can't find issues with security/py-certbot itself. And at the moment I can't check the actual build logs (need IPv6 for that) of the build clusters.

tsk · Jun 13, 2023

I upgraded to python39-3.9.16_2, same result:
Well apache runs when I start certbot, but it apparently stops apache but cannot restart it. Am I the only one to encounter that problem?

Code:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /usr/local/etc/letsencrypt/renewal/mecadtron.de.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for mecadtron.de and [URL="http://www.mecadtron.de"]www.mecadtron.de[/URL]

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: mecadtron.de
  Type:   connection
  Detail: 95.217.187.226: Fetching [URL]http://mecadtron.de/.well-known/acme-challenge/hyZPIttXOYhiwzHTOyRJGfG8J0EhKtLsgSup_Z9_cYY[/URL]: Connection refused

  Domain: [URL="http://www.mecadtron.de"]www.mecadtron.de[/URL]
  Type:   connection
  Detail: 95.217.187.226: Fetching [URL]http://www.mecadtron.de/.well-known/acme-challenge/sAHHc0OPtCFyyYSzhb3mo2vweeyYdq-TPmNv79zc5zQ[/URL]: Connection refused

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Error while running apachectl graceful.
apache24 not running? (check /var/run/httpd.pid).


Unable to restart apache using ['apachectl', 'graceful']
Encountered exception during recovery: certbot.errors.MisconfigurationError: Error while running apachectl graceful.
apache24 not running? (check /var/run/httpd.pid).
Failed to renew certificate mecadtron.de with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /usr/local/etc/letsencrypt/live/mecadtron.de/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at [URL]https://community.letsencrypt.org[/URL]. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
r

hruodr · Jun 13, 2023

tsk said:
Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

But why certbot makes "temporary Apache configuration changes"?

facedebouc · Jun 13, 2023

tsk said:
Renewing an existing certificate for mecadtron.de and www.mecadtron.de

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: mecadtron.de
Type: connection
Detail: 95.217.187.226: Fetching http://mecadtron.de/.well-known/acme-challenge/hyZPIttXOYhiwzHTOyRJGfG8J0EhKtLsgSup_Z9_cYY: Connection refused

Domain: www.mecadtron.de
Type: connection
Detail: 95.217.187.226: Fetching http://www.mecadtron.de/.well-known/acme-challenge/sAHHc0OPtCFyyYSzhb3mo2vweeyYdq-TPmNv79zc5zQ: Connection refused

The problem is here, Certbot can't fetch his acme-challenge.

diizzy · Jun 13, 2023

hruodr said:
I use this under debian:

GitHub - dehydrated-io/dehydrated: letsencrypt/acme client implemented as a shell-script – just add water

letsencrypt/acme client implemented as a shell-script – just add water - dehydrated-io/dehydrated

github.com

Few and non strange dependencies. I found certbot too complicated.

It would be nice if FreeBSD had a standard acme client in base like OpenBSD, or better, the same one:

acme-client(1) - OpenBSD manual pages

Lego is a nice alternative, https://www.freshports.org/security/lego

SirDice · Jun 13, 2023

tsk said:
I upgraded to python39-3.9.16_2,

Did you also change py38-certbot to py39-certbot? Or else it would still use Python 3.8.

Code:

20220626:
  AFFECTS: users of python
  AUTHOR: thierry@FreeBSD.org

  The default version of python3 and python was switched to 3.9.

  For ports users wanting to keep version 3.8 as default,
  add DEFAULT_VERSIONS+= python=3.8 python3=3.8 to make.conf

  Following procedures may ease the upgrade:

  For users of pre-build packages:
  # sh
  # for i in $(pkg query -g %n 'py38-*'); do pkg set -yn ${i}:py39-${i#py38-}; done
  # pkg upgrade

  For portmaster users:
  # sh
  # portmaster -o lang/python39 python38
  # REINSTALL="$(pkg info -o "*py38*" | awk '{printf "%s ", $2}')"
  # pkg delete -f "*py38*"
  # portmaster $REINSTALL
  # REBUILD=$(pkg query -g "%n:%dn" '*' | grep py3 | grep -v py39 | cut -d : -f 1 | sort -u)
  # portmaster $REBUILD
  # REBUILD2=$(pkg list | grep python-38 | xargs pkg which | awk '{print $6}' | sort -u)
  # portmaster $REBUILD2

  Final steps (for pre-built packages & portmaster):

  If no longer required, Python 3.8 can be removed via
  "pkg remove python38" and the directory /usr/local/lib/python3.8 can
  then be deleted afterwards, if not empty.

tsk · Jun 13, 2023

I deleted python38.

richardtoohey2 · Jun 13, 2023

There’s an outstanding issue with Apache if you are using PHP as a module:

268318 – www/apache24 with www/mod_php8{0,1,2,3?}: opcache + ASLR turned on crashes Apache

bugs.freebsd.org

ericbsd · Jun 24, 2023

tsk said:

I upgraded to python39-3.9.16_2, same result:
Well apache runs when I start certbot, but it apparently stops apache but cannot restart it. Am I the only one to encounter that problem?

Code:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /usr/local/etc/letsencrypt/renewal/mecadtron.de.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for mecadtron.de and [URL="http://www.mecadtron.de"]www.mecadtron.de[/URL]

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: mecadtron.de
  Type:   connection
  Detail: 95.217.187.226: Fetching [URL]http://mecadtron.de/.well-known/acme-challenge/hyZPIttXOYhiwzHTOyRJGfG8J0EhKtLsgSup_Z9_cYY[/URL]: Connection refused

  Domain: [URL="http://www.mecadtron.de"]www.mecadtron.de[/URL]
  Type:   connection
  Detail: 95.217.187.226: Fetching [URL]http://www.mecadtron.de/.well-known/acme-challenge/sAHHc0OPtCFyyYSzhb3mo2vweeyYdq-TPmNv79zc5zQ[/URL]: Connection refused

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Error while running apachectl graceful.
apache24 not running? (check /var/run/httpd.pid).


Unable to restart apache using ['apachectl', 'graceful']
Encountered exception during recovery: certbot.errors.MisconfigurationError: Error while running apachectl graceful.
apache24 not running? (check /var/run/httpd.pid).
Failed to renew certificate mecadtron.de with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /usr/local/etc/letsencrypt/live/mecadtron.de/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at [URL]https://community.letsencrypt.org[/URL]. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
r

I have the same problem with forums.ghostbsd.org.

I never got that in the past and Apache, PHP, and Certbot were not updated on that jail.

ericbsd · Jun 24, 2023

Looks like the jail can't access the subdomain


# curl http://forums.ghostbsd.org
curl: (7) Failed to connect to forums.ghostbsd.org port 80: Connection refused

ericbsd · Jun 24, 2023

A disregard the apache was down.

richardtoohey2 · Jun 24, 2023

Apache will be down because it won’t restart properly after the graceful command.

ericbsd · Jun 24, 2023

Yeah, it is weird that it started to fail without any package updates.

Anyway, I have deleted the certificate and created one that does not use Apache.

certbot suddenly fails

tsk

SirDice

Administrator

tsk

SirDice

Administrator

hruodr

GitHub - dehydrated-io/dehydrated: letsencrypt/acme client implemented as a shell-script – just add water

tsk

hruodr

SirDice

Administrator

SirDice

Administrator

SirDice

Administrator

hruodr

facedebouc

SirDice

Administrator

tsk

hruodr

facedebouc

diizzy

GitHub - dehydrated-io/dehydrated: letsencrypt/acme client implemented as a shell-script – just add water

SirDice

Administrator

tsk

richardtoohey2

268318 – www/apache24 with www/mod_php8{0,1,2,3?}: opcache + ASLR turned on crashes Apache

ericbsd

ericbsd

ericbsd

richardtoohey2

ericbsd