I am trying to serve cached assets from NGINX on a Vultr server running in Dallas, Texas with 8G RAM, 180GB nvme and 4 intel vcpus (with FreeBSD, of course)
FreeBSD does not have BBR enabled by default, so I downloaded the source for 13.2 and compiled kernel with kernel GENERIC + "makeoptions WITH_EXTRA_TCP_STACKS=1" and "options TCPHPTS", added "net.inet.tcp.functions_default=bbr" to the sysctl.conf, installed the kernel then rebooted (init 6 + reboot button on Vultr panel. I verified with uptime that it did reboot). (Also, net.inet.tcp.cc.algorithm=htcp on my server, and BBR isn't an option. Is that correct, or is it only an option for functions_default?)
I have 1Gbps internet and the server is 12ms away from me, and when I request the cached 1.5mb image now with BBR enabled, it sends it in 12ms + 20-30ms minimum.
Before I enabled BBR, it would send it slowly at first, but as long as I was spamming refresh, it would eventually send it at full speed (24ms), and then not at full speed if I wasn't refreshing the page quickly.
Was I supposed to compile world and not just the kernel? I thought BBR was supposed to fix this issue.
I have aio on; with nginx, sendfile on; and an SSL buffer size of 4kb. I've tried tweaking all of those settings to different values with no different outcome.
Here is my NGINX config https://bytebin.lucko.me/ChpcvlYR0M
I've also tested different sysctl settings like the ones below
#kern.ipc.maxsockbuf=67108864 #default vultr one
#kern.ipc.maxsockbuf=16777216
#net.inet.tcp.sendbuf_max=67108864
#net.inet.tcp.recvbuf_max=67108864
#net.inet.tcp.sendbuf_auto=1
#net.inet.tcp.recvbuf_auto=1
#net.inet.tcp.sendbuf_inc=16384
Do you have any clue what I might be doing wrong? This doesn't happen with Linux btw, only FreeBSD. I've tested many Linux servers at Vultr with Varnish, Varnish Enterprise, NGINX, HAProxy, Squid, Apache Traffic Server and this problem only happens with FreeBSD with and without BBR, but whenever I enable BBR on Linux, it fixes the issue.
Any help would be great. I'm trying to use FreeBSD with NGINX for caching because Netflix does it and apparently it performs better than any other solution, but I'm having trouble finding out more information online about what specific kernel tweaks and nginx settings Netflix is using. And obviously a VM and a likely cheap NIC will make it harder to get good performance, but once again, it works on Linux just fine)
Thanks!
FreeBSD does not have BBR enabled by default, so I downloaded the source for 13.2 and compiled kernel with kernel GENERIC + "makeoptions WITH_EXTRA_TCP_STACKS=1" and "options TCPHPTS", added "net.inet.tcp.functions_default=bbr" to the sysctl.conf, installed the kernel then rebooted (init 6 + reboot button on Vultr panel. I verified with uptime that it did reboot). (Also, net.inet.tcp.cc.algorithm=htcp on my server, and BBR isn't an option. Is that correct, or is it only an option for functions_default?)
I have 1Gbps internet and the server is 12ms away from me, and when I request the cached 1.5mb image now with BBR enabled, it sends it in 12ms + 20-30ms minimum.
Before I enabled BBR, it would send it slowly at first, but as long as I was spamming refresh, it would eventually send it at full speed (24ms), and then not at full speed if I wasn't refreshing the page quickly.
Was I supposed to compile world and not just the kernel? I thought BBR was supposed to fix this issue.
I have aio on; with nginx, sendfile on; and an SSL buffer size of 4kb. I've tried tweaking all of those settings to different values with no different outcome.
Here is my NGINX config https://bytebin.lucko.me/ChpcvlYR0M
I've also tested different sysctl settings like the ones below
#kern.ipc.maxsockbuf=67108864 #default vultr one
#kern.ipc.maxsockbuf=16777216
#net.inet.tcp.sendbuf_max=67108864
#net.inet.tcp.recvbuf_max=67108864
#net.inet.tcp.sendbuf_auto=1
#net.inet.tcp.recvbuf_auto=1
#net.inet.tcp.sendbuf_inc=16384
Do you have any clue what I might be doing wrong? This doesn't happen with Linux btw, only FreeBSD. I've tested many Linux servers at Vultr with Varnish, Varnish Enterprise, NGINX, HAProxy, Squid, Apache Traffic Server and this problem only happens with FreeBSD with and without BBR, but whenever I enable BBR on Linux, it fixes the issue.
Any help would be great. I'm trying to use FreeBSD with NGINX for caching because Netflix does it and apparently it performs better than any other solution, but I'm having trouble finding out more information online about what specific kernel tweaks and nginx settings Netflix is using. And obviously a VM and a likely cheap NIC will make it harder to get good performance, but once again, it works on Linux just fine)
Thanks!