Nginx KTLS support on FreeBSD 13/13.1?

Today did a pkg upgrade and my nginx updated to: nginx-devel-1.21.5_3 on FreeBSD 13, bringing support to KTLS.

Tried to activate the KTLS within the kernel with command:
kldload ktls_ocf
then: sysctl kern.ipc.tls.enable=1

afterwards, I added; ssl_conf_command Options KTLS;" under SSL server block of my nginx.conf file, tried to refresh nginx, got an error like;
nginx: [emerg] SSL_CONF_cmd("Options", "KTLS") failed (SSL: error:1414E180:SSL routines:SSL_CONF_cmd:bad value:cmd=Options, value=KTLS)
root@:~ # uname -a
FreeBSD myhost 13.0-RELEASE-p4 FreeBSD 13.0-RELEASE-p4 #0: Tue Aug 24 07:33:27 UTC 2021 [EMAIL][/EMAIL]:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

Could anyone point me please where do I make mistake?

covacat Nginx docs say:
"FreeBSD 13.0+. As of November 2021, FreeBSD 13.0+ is the only OS that supports kTLS in NGINX without a manual build of NGINX to incorporate OpenSSL 3.0.0+"
options KTLS seems to be passed to the ssl lib and it does not know about it
i look the Base openssl and it's not there
openssl 1.1m from ports has it
i don't think it is available as a package
can you verify ktls is working if you remove nginx options ktls ?
it may be activated by default ?
covacat Nginx docs say:
"FreeBSD 13.0+. As of November 2021, FreeBSD 13.0+ is the only OS that supports kTLS in NGINX without a manual build of NGINX to incorporate OpenSSL 3.0.0+"

from the history of www/nginx:
www/nginx: Update to 1.20.2

*) Feature: OpenSSL 3.0 compatibility.

I suspect the note in the manpage only accounts to the nginx-side of OpenSSL 3.0 / KTLS support - i.e. "nginx is built with OpenSSL 3.0 support enabled on FreeBSD 13, but not on other OS yet". It does not explicitly say that OpenSSL 3.0 and/or KTLS is already supported on FreeBSD in base.
Maybe you could try installing security/openssl-devel, which is currently version 3.0.1. But I wouldn't feel good about running a development version of something as crucial as OpenSSL...

Are you hitting any performance limits or problems with the way TLS is currently handled (i.e. not in kernel)? If not I'd wait until this is battle-tested and becomes supported in base or at least with an OpenSSL release-version from pkg/ports, not a development version (at least for anything remotely related to a production system and/or connected to the outside world).

according to this commit it should already be available in base with OpenSSL 1.1.1
So it seems creating a src.conf file with WITH_OPENSSL_KTLS=1 line, then moving from 13-RELEASE to 13-STABLE (compiling from sources) the best option to enable KTLS in Nginx on FreeBSD 13?

and to answer sko , indeed, I'd like to have it due to performance purposes.

From what I gather, WITH_OPENSSL_KTLS=1 works on RELEASE as well; it requires the applications to be tailored for it, however. So - without having it tried myself - I suppose, chances are good that nginx's KTLS works the same principle as the port I tried - and that did require STABLE to work properly.

Rick Macklem's documentation explicitly pointed out that running with WITH_OPENSSL_KTLS will likely break applications like apache; with nginx "understanding" KTLS, I expect chances are good with RELEASE and still better with STABLE since their handling of connections appear to be different.
Just for those looking into this a bit later (as I am), it appears WITH_OPENSSL_KTLS is now the default in FreeBSD 13.1 RELEASE for amd64/amd64 and arm64/aarch64.

See notes here too, about how to enable KTLS in Nginx, and enable Kernel TLS on FreeBSD 13:

New kernel TLS feature is available starting with FreeBSD 13.0, and it requires OpenSSL 3.0, compiled with "enable-ktls" option.
Further, KTLS needs to be enabled in kernel, and in OpenSSL, either via OpenSSL configuration file or with

ssl_conf_command Options KTLS;

in nginx configuration.

To enable kernel TLS on FreeBSD 13 and above:

# kldload ktls_ocf
# sysctl kern.ipc.tls.enable=1

to load a software backend, see man ktls(4) for details.
Bink thanks for the link.

Under FreeBSD 13.1, I don't get any more the error at the top of my post, from nginx. The ssl_conf_command Options KTLS; works now. But;

It seems KTLS needs sendfile on; for nginx. BUT sendfile is not recommended/supported when you're on ZFS.

Please see:

And I think KTLS isn't get enabled when sendfile is off, even though you did anything else for it;
kldload ktls_ocf

Using both commands return nothing from Nginx (in debug mode);
grep BIO /var/log/nginx/error.log
grep SSL_sendfile /var/log/nginx/error.log

So, how can I proceed with KTLS while sendfile is off in nginx, under ZFS?
Or I better should ask, does the following nginx package (on FreeBSD 13.1) supports KTLS with the mentioned OpenSSL version?

nginx version: nginx/1.22.0
built with OpenSSL 1.1.1k-freebsd 24 Aug 2021 (running with OpenSSL 1.1.1o-freebsd 3 May 2022)
I didn't really get it suntzu00

Obviously, FreeBSD 13.1's default base OpenSSL is not 3.0. BUT the link says the base one is compiled with KTLS support?