sendmail STARTTLS doesnt work with Let'sEncrypt

PMc · Apr 29, 2022

Maybe somebody here knows how this would be supposed to work:
I am trying to get TLS work with sendmail, and I managed to get to it with my own CA, but not with Let'sEncrypt.
I am getting this error:

Code:

2022-04-28T23:18:46.704257+02:00 <mail.info> intra.daemon.contact sm-mta[69293] STARTTLS: TLS cert verify: depth=3 /O=Digital Signature Trust Co./CN=DST Root CA X3, state=0, reason=certificate has expired
2022-04-28T23:18:46.705103+02:00 <mail.info> intra.daemon.contact sm-mta[69293] STARTTLS=server, get_verify: 10 get_peer: 0x801c3af00
2022-04-28T23:18:46.705122+02:00 <mail.info> intra.daemon.contact sm-mta[69293] STARTTLS=server, relay=gate.intra.daemon.contact [xx.xx.xx.xx], version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
2022-04-28T23:18:46.705131+02:00 <mail.info> intra.daemon.contact sm-mta[69293] STARTTLS=server, cert-subject=/CN=moon.daemon.contact, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=certificate has expired

The certificate itself is certainly not expired, but it appears we have a crappy root cert in the tree:
/usr/src/secure/caroot/trusted/DST_Root_CA_X3.pem

Code:

    Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
    Validity
        Not Before: Sep 30 21:12:19 2000 GMT
        Not After : Sep 30 14:01:15 2021 GMT
    Subject: O = Digital Signature Trust Co., CN = DST Root CA X3

This is Rel. 13.1, so I am really wondering what does this do here?

OTOH, some software does not bother about expired certs, so maybe there is good reason to keep it.

I am now looking for insight on how this should work, what is going wrong, and where it should be fixed: in FreeBSD, in sendmail or in Let'sEncrypt?

trev · Apr 29, 2022

It's simply a side-effect of the Let's Encrypt root certificate expiring, something which happens to all certificate providers eventually.

As per the error message, Let's Encrypt's Root CA X3 certificate expired on September 30, 2021. That means that if you do not have the new ISRG Root X1 certificate in your certificate store, you will get certificate errors.

To read more, see https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ for more gory details.

PMc · Apr 29, 2022

trev said:
It's simply a side-effect of the Let's Encrypt root certificate expiring, something which happens to all certificate providers eventually.

Yes. but I wouldn't call it a "side effect" that the certificates do not work. And this expiry was in September, and the certificates will probably not work for as long as the expired one is present on the system, and I do not know for how long that will be. So what is the use of certificates when they do not work?

trev said:
As per the error message, Let's Encrypt's Root CA X3 certificate expired on September 30, 2021. That means that if you do not have the new ISRG Root X1 certificate in your certificate store, you will get certificate errors.

I know that, and I do not see how that matters. This is FreeBSD 13.1, and obviousely the new ISRG Root X1 is in the store. It is here: https://cgit.freebsd.org/src/tree/secure/caroot/trusted/ISRG_Root_X1.pem?h=releng/13.1
The problem is not with the new cacert, but with the old one. And the problem is not that there are errors, but that the certificate does just not work (apparently for as long as the old outdated cacert is there).

trev · Apr 29, 2022

I'm using FreeBSD 12-STABLE and have no certificate issues. In my sendmail .mc file I have:

Code:

define(`confCACERT',`/etc/mail/certs/ca-bundle.crt')

In the ca-bundle.crt there is no Root CA X3 certificate, but there is a ISRG Root X1 certificate.

Code:

-rw-r--r--  1 root  wheel   707K 29 Apr 04:15 ca-bundle.crt

My ACME script also does this:

cp /usr/local/share/certs/ca-root-nss.crt /etc/mail/certs/ca-bundle.crt

before restarting sendmail overnight.

drhowarddrfine · Apr 29, 2022

I'm using LetsEncrypt on my server with sendmail. I'll try to look to see what I did but I'm really, reeeeeally busy and last set it up years ago.

PMc · Apr 29, 2022

trev said:
I'm using FreeBSD 12-STABLE and have no certificate issues. In my sendmail .mc file I have:

Code:

define(`confCACERT',`/etc/mail/certs/ca-bundle.crt')

In the ca-bundle.crt there is no Root CA X3 certificate, but there is a ISRG Root X1 certificate.

Yes, then it will likely work. But where is Your intermediate then?

I mean, when you point confCACERT to the bundle - which should contain the self-signed ISRG Root X1, and point confSERVER_CERT to your cert, signed by Let'sEncrypt R3, then there need to be an intermediate where Let'sEncrypt R3 is signed by ISRG Root X1. This is usually in the chain.pem file supplied by Let'sEncrypt.

And I would have thought that this chain.pem file is what should go into confCACERT. (There is mention in the sendmail doc that You should not put more than a few ca certs into confCACERT, otherwise TLS handshake may fail.)

DutchDaemon · Apr 29, 2022

Just link to the live environment of LetsEncrypt from your .mc file:

Code:

(`CERT_DIR', `/usr/local/etc/letsencrypt/live/your.site.name')dnl
define(`confCACERT_PATH', `CERT_DIR')dnl
define(`confCACERT', `CERT_DIR/chain.pem')dnl
define(`confSERVER_CERT', `CERT_DIR/cert.pem')dnl
define(`confSERVER_KEY', `CERT_DIR/privkey.pem')dnl
define(`confCLIENT_CERT', `CERT_DIR/cert.pem')dnl
define(`confCLIENT_KEY', `CERT_DIR/privkey.pem')dnl

This is all maintained by and updated from LE, so always accurate.

hruodr · Apr 29, 2022

@DutchDaemon

Either client:

Code:

define(`CERT_DIR',         `/etc/mail/certs')dnl  * Certs must be created
define(`confCACERT_PATH',  `CERT_DIR')dnl
define(`confCACERT',       `CERT_DIR/cacert.pem')dnl
define(`confCLIENT_CERT',  `CERT_DIR/host.cert')dnl
define(`confCLIENT_KEY',   `CERT_DIR/host.key')dnl
define(`confDH_PARAMETERS',`CERT_DIR/dh.param')dnl

Or server:

Code:

define(`confCACERT_PATH', `/usr/opt/etc/cert/')dnl
define(`confCACERT', `/usr/opt/etc/cert/fullchain.pem')dnl
define(`confSERVER_CERT', `/usr/opt/etc/cert/fullchain.pem')dnl
define(`confSERVER_KEY', `/usr/opt/etc/cert/privkey.pem')dnl

DutchDaemon · Apr 29, 2022

You can also keep the defaults and symlink from there:

Code:

define(`CERT_DIR', `/etc/mail/certs')dnl
define(`confCACERT_PATH', `CERT_DIR')dnl
define(`confCACERT', `CERT_DIR/cacert.pem')dnl
define(`confSERVER_CERT', `CERT_DIR/host.crt')dnl
define(`confSERVER_KEY', `CERT_DIR/host.key')dnl
define(`confCLIENT_CERT', `CERT_DIR/host.crt')dnl
define(`confCLIENT_KEY', `CERT_DIR/host.key')dnl

Code:

find /etc/mail/certs/ -type l | xargs ls -l
lrwxr-xr-x  1 root  wheel  62 Oct 13  2021 /etc/mail/certs/cacert.pem -> /usr/local/etc/letsencrypt/live/your.site.name/fullchain.pem
lrwxr-xr-x  1 root  wheel  57 Oct 13  2021 /etc/mail/certs/host.crt -> /usr/local/etc/letsencrypt/live/your.site.name/cert.pem
lrwxr-xr-x  1 root  wheel  60 Oct 13  2021 /etc/mail/certs/host.key -> /usr/local/etc/letsencrypt/live/your.site.name/privkey.pem

The only thing you have to do is schedule a sendmail restart whenever the certificate renews. It can be something as simple as this cronjob:

Code:

0 0,12 * * *    /usr/local/bin/certbot renew && /usr/bin/find /usr/local/etc/letsencrypt/live/your.site.name/ -type l -name fullchain.pem -mtime -1h | /usr/bin/grep -q fullchain.pem && /usr/sbin/service sendmail restart

PMc · Apr 29, 2022

DutchDaemon said:
Just link to the live environment of LetsEncrypt from your .mc file:

Code:

(`CERT_DIR', `/usr/local/etc/letsencrypt/live/your.site.name')dnl define(`confCACERT_PATH', `CERT_DIR')dnl define(`confCACERT', `CERT_DIR/chain.pem')dnl define(`confSERVER_CERT', `CERT_DIR/cert.pem')dnl define(`confSERVER_KEY', `CERT_DIR/privkey.pem')dnl define(`confCLIENT_CERT', `CERT_DIR/cert.pem')dnl define(`confCLIENT_KEY', `CERT_DIR/privkey.pem')dnl

This is all maintained by and updated from LE, so always accurate.

This is exactly the way that it does not work.

Furthermore, You definitely also need this:
define(`confCACERT_PATH', `/etc/ssl/certs')dnl
Otherwise verification of your clients cannot succeed ("self signed cert encountered in tree"), because the actual CA root certs are nowhere in the live environment. (They are either in the system path /etc/ssl/certs or in the ca_root_nss.)

Lets clarify this a bit more: there are four different verifications done:
1. host A (client) connects to host B (server). Host A authenticates, host B verifies.
2. host A (client) connects to host B (server). Host B authenticates, host A verifies.
3. host B (client) connects to host A (server). Host A authenticates, host B verifies.
4. host B (client) connects to host A (server). Host B authenticates, host A verifies.

And I seem to be the only person who expects all four of them to be OK (with Let'sEncrypt certs on both sides).
There appears to be a bug in openssl 1.1.1n; I can reproduce the verification failure with openssl s_client.

PMc · Apr 29, 2022

hruodr said:
Either client:

Or server:

Both at the same time.

drhowarddrfine · Apr 29, 2022

DutchDaemon said:
Just link to the live environment of LetsEncrypt from your .mc file:

Yep. This is what I do.

PMc · Apr 30, 2022

You can test Your installation:

have Your mailserver running. Then run this command and see what happens:


openssl s_client -connect <your-mailserver>:25 -starttls smtp -CAfile <your confCACERT> -chainCAfile <your confCACERT> -key <your confCLIENT_KEY> -cert <your confCLIENT_CERT> -CApath /etc/ssl/certs -chainCApath /etc/ssl/certs

hruodr · Apr 30, 2022

PMc said:
Both at the same time.

Did you try try that? How it works?

Does it receive (as server) mails to relay them (as client) all to another same server?
Then all mails relied appear to be from the same address on the second server?

VladiBG · Apr 30, 2022

Code:

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = **************
verify return:1
---
Certificate chain
 0 s:CN = **************
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
**************
-----END CERTIFICATE-----
subject=CN = **************

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4840 bytes and written 430 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
250 CHUNKING
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: **************
    Session-ID-ctx:
    Resumption PSK: **************
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:**************

    Start Time: 1651304952
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

quit
221 2.0.0 Bye

PMc · Apr 30, 2022

hruodr said:
Did you try try that? How it works?

Does it receive (as server) mails to relay them (as client) all to another same server?
Then all mails relied appear to be from the same address on the second server?

Oh. Hm, where to begin with? In short, it can do all of this. This is what Eric Allman originally wrote it for: to dispatch the mails between Internet, DECnet, Bitnet, Usenet and almost everything else that was there at that time.

And when I started to use it, I had it dispatch the mails between the systems I was connected to: a few Usenet sites and some mailbox networks. (I wrote a suitable cf file from scratch at that time.)
But to run it over TCP, one would need a permanent uplink with permanent DNS working. And I got that rather late, when they became affordable for private use. (I always did this for fun, I was never doing business.)

Sendmail is a mail relay by nature; it will natively receive mails from somewhere as a server, and then move them onwards to somewhere as a client. Only for TCP had this to be blocked, due to common abuse.

hruodr said:
Does it receive (as server) mails to relay them (as client) all to another same server?

Same or other server, yes. Or rewrite the addresses into the notation of a different network technology first (rarely used today, where everybody speaks rfc821).

hruodr said:
Then all mails relied appear to be from the same address on the second server?

Normally not, the addresses are kept when relaying. This works very much like physical mail, there are three level involved:

The From: and To: from the mail header (what you usually see in the mail client) - these are not normally used in transport; they are basically the same as when you write a letter with "hello annie, .... cheers, george".
The "envelope" addresses. You don't see them normally, but these are used for transport. These say where the mail has to go, or where it should be returned if undeliverable.
The TCP connection peers. These are normally irrelevant: any sendmailer will try on best effort to connect to somewhere where the mail can be moved onwards - or, if it gets stuck, to connect to somewhere where the error report can be sent back to the (envelope) sender address.
Only due to common abuse was this limited.

You can make the mails appear to come from the relaying server's address, by rewriting them. This is done with the features genericstable, allmasquerade, masquerade_envelope, etc. (But you must see that undeliverables get back to some place where you want to receive them.)

You can decide on where to send specific mails with the feature mailertable. You can also decide on this from the sending address with a little ruleset:

Code:

Klsender hash -o -f /etc/mail/lsendertable
LOCAL_RULE_0
R $+                    $: $1 <!>                       set marker
R $+ < @ $=w . > <!>    $1 < @ $2 . >                   local
R $+ < @ $=w > <!>      $1 < @ $2 >                     local
R $* $=O $* <!>         $: $1 $2 $3 <!> $>canonify $&f  not local -> from
R $+ <!> $+ < @ $+ . >  < $(lsender $3 $: @ $) > $1     search lsender
R <@> $+                $1                              none, default
R < $- :: $+ > $*       $# $1 $@ $2 $: $3               got mailer+host
R $* <!> $*             $: $1                           revert marker

(must use TABs in the rules, to distinguish from spaces)

So normally you do not need to run different mailer instances to do different jobs; sendmail does fork on it's own for different tasks.

The only case where you need this is the HELO name: there can be only one per configuration. So if your machine runs multi-homed on different IP addresses, and your TCP peers expect the HELO name to match the IP address (what they commonly do today, with PICKY_HELO_CHECK, then you need multiple sendmail processes with different confHELO_NAME.

hruodr · Apr 30, 2022

PMc said:
Sendmail is a mail relay by nature; it will natively receive mails from somewhere as a server, and then move them onwards to somewhere as a client.

I have sendmail configured without confCLIENT_CERT and confCLIENT_KEY, only with Server certs.
Look how looks the headers of the mail it sends like:

Received: from xxx.cf ([NN.NNN.NN.NNN]) by mx-ha.web.de (mxweb112
[212.227.17.8]) with ESMTPS (Nemesis) id 1N002m-1o6KQ50hXi-00x4tg for
<nnnnn@web.de>; Sat, 30 Apr 2022 18:02:47 +0200

Received with ESMTPS, I suppose, this uses TLS.

Perhaps are these client certs only for sendmail as smarthost?

PMc · Apr 30, 2022

hruodr said:
I have sendmail configured without confCLIENT_CERT and confCLIENT_KEY, only with Server certs.
Look how looks the headers of the mail it sends like:

Received with ESMTPS, I suppose, this uses TLS.

If they at web.de have a cert, that should do for encryption. With https it's the same: only the webserver has a cert, normally.

But the purpose of certs is not (only) encryption, it is authentication: the client want to make sure that the server is who they claim to be, and the server may want to make sure the same thing of the client.
And therefore both sides need a cert - depending on who may want to check this.

Currently it seems not be checked at all by default. There can be any CommonName, any SubjectAltName in the cert, it will always get verified as long as the CA can be found.
But these names from the cert are retrieved, and are accessible, One can create configurations that change the behaviour, e.g. allow relaying for certain clients who have proven their identity

So it all depends on what you want to do.

I for my part just wanted to figure out how to get it fully working.

hruodr · Apr 30, 2022

PMc said:
But the purpose of certs is not (only) encryption, it is authentication: the client want to make sure that the server is who they claim to be, and the server may want to make sure the same thing of the client.
And therefore both sides need a cert - depending on who may want to check this.

With or without client certs I get the following lines in sendmail log file when sending a mail:

Apr 30 16:53:15 nnnn sendmail[2166555]: 23UGrFsv2166555: from=nn, size=169, class=0, nrcpts=1, msgid=<202204301653.23UGrFsv2166555@nnnn.cf>, relay=nn@localhost
Apr 30 16:53:15 nnnn sendmail[2166555]: STARTTLS=client, relay=[127.0.0.1], version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
Apr 30 16:53:15 nnnn sm-mta[2166557]: STARTTLS=server, relay=nnnn.cf [127.0.0.1], version=TLSv1.3, verify=NO, cipher=TLS_AES_256_GCM_SHA384, bits=256/256

verify=FAIL, verify=no? What it is being verified?

PMc · Apr 30, 2022

hruodr said:
verify=FAIL, verify=no? What it is being verified?

Now we are at what I'm talking about.

What is verified: the certificate authority that did sign your cert.

Details are in /usr/share/sendmail/cf/README:

Code:

${verify} holds the result of the verification of the presented cert.
        Possible values are:
        OK       verification succeeded.
        NO       no cert presented.
        NOT      no cert requested.
        FAIL     cert presented but could not be verified,
                 e.g., the cert of the signing CA is missing.
etc.etc.

And if You want to see why it failed: check your /etc/mail/sendmail.cf for the line
O LogLevel=9. Change it to 20 or something, and restart the sendmail. It will then write the reason codes into the log (and a lot more).

PMc · Apr 30, 2022

And here is how things will go on: the StartTLS can be coupled with DNSSEC:

Better mail security with DANE for SMTP | APNIC Blog

Guest Post: DNS-based Authentication of Named Entities (DANE) for SMTP provides a more secure method for mail transport.

blog.apnic.net

Quite nice article that is. And it seems fairly recent.

hruodr · May 1, 2022

PMc said:
And here is how things will go on:

After you try, tell me please what happens

facedebouc · May 1, 2022

I am running a mail server with mail/sendmail from ports because I need SASL AUTH.
All is working flawlessly with TLS and certificates from Letsencrypt.
I am also using SPF, DKIM, DMARC.
Here is how is configured my sendmail mail server:

Code:

define(`CERT_DIR', `/etc/mail/certs')dnl
define(`confSERVER_CERT', `CERT_DIR/host.cert')dnl
define(`confSERVER_KEY', `CERT_DIR/host.key')dnl
define(`confCLIENT_CERT', `CERT_DIR/host.cert')dnl
define(`confCLIENT_KEY', `CERT_DIR/host.key')dnl
define(`confCACERT', `CERT_DIR/cacert.pem')dnl
define(`confCACERT_PATH', `/etc/ssl/certs')dnl
define(`confDH_PARAMETERS', `CERT_DIR/dh.param')dnl

This mail server is running in a jail and certificates are renew from host and copied into the jail:

Code:

root@host:~ # cp /usr/local/etc/letsencrypt/live/mail.example.com/chain.pem /usr/local/bastille/jails/mail/root/etc/mail/certs/cacert.pem
root@host:~ # cp /usr/local/etc/letsencrypt/live/mail.example.com/cert.pem  /usr/local/bastille/jails/mail/root/etc/mail/certs/host.cert
root@host:~ # cp /usr/local/etc/letsencrypt/live/mail.example.com/privkey.pem  /usr/local/bastille/jails/mail/root/etc/mail/certs/host.key
root@host:~ # cp /usr/local/etc/letsencrypt/live/mail.example.com/fullchain.pem /usr/local/bastille/jails/mail/root/etc/mail/certs/

root@mail:~ # cd /etc/mail/certs
root@mail:/etc/mail/certs # ln -sf cacert.pem `openssl x509 -hash -noout < cacert.pem`.0

hruodr · May 1, 2022

facedebouc said:
All is working flawlessly with TLS and certificates from Letsencrypt.

Did you check the logfile to see if there is a "verify=FAIL"?

facedebouc · May 1, 2022

hruodr said:
Did you check the logfile to see if there is a "verify=FAIL"?

For example an email sent to a user@outlook.com give "verify=OK" when connecting to the server:

Code:

May  1 09:59:34 mail sm-mta[9271]: STARTTLS=client, relay=eur.olc.protection.outlook.com., version=TLSv1.2, verify=OK, cipher=ECDHE-RSA-AES256-GCM-SHA384, bits=256/256

sendmail STARTTLS doesnt work with Let'sEncrypt

PMc

trev

PMc

trev

drhowarddrfine

PMc

DutchDaemon

Administrator

hruodr

@DutchDaemon

DutchDaemon

Administrator

PMc

PMc

drhowarddrfine

PMc

hruodr

VladiBG

PMc

hruodr

PMc

hruodr

PMc

PMc

Better mail security with DANE for SMTP | APNIC Blog

hruodr

facedebouc

hruodr

facedebouc

sendmail STARTTLS doesnt work with Let'sEncrypt

Administrator

@DutchDaemon​

Administrator

@DutchDaemon