Hi all,
We host all our client wordpress website on a Freebsd 12.0-RELEASE host and we put every client on a separate jail to keep them apart from each other.
Each webjail run /www/nginx and all webjail share 1x common /databases/mariadb101-server and 1x common nginx web proxy.
In the last couple of month, all our WordPress site had the following error
When this happen, we are unable to do anything on the server and the team at the datacenter have to do a cold reboot for us and everything is ok again.
This is happening every 3 to 4 weeks.
Today, we look at the error log but we couldn't really make head and tail of the issue so we are hoping someone here could help us get to the bottom of it.
the /var/log/messages has the following up to the point of reboot:
The log for /var/db/mysql/mariadb.trinitech.co.uk.err is in pastbin bellow due to its size
https://pastebin.com/MjeJEH6b
Here is my entire config /var/db/mysql/my.cnf
Here is my /boot/loader.conf from the FreeBSD host:
From what I found on the internet it could be a network saturation but I am unsure.
Thank you all in advance
We host all our client wordpress website on a Freebsd 12.0-RELEASE host and we put every client on a separate jail to keep them apart from each other.
Each webjail run /www/nginx and all webjail share 1x common /databases/mariadb101-server and 1x common nginx web proxy.
In the last couple of month, all our WordPress site had the following error
Code:
Error establishing a database connection
This is happening every 3 to 4 weeks.
Today, we look at the error log but we couldn't really make head and tail of the issue so we are hoping someone here could help us get to the bottom of it.
the /var/log/messages has the following up to the point of reboot:
Code:
Sep 20 13:06:48 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8004ceac000: Listen queue overflow: 193 already in queue awaiting acceptance (36 occurrences)
Sep 20 13:07:50 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8004ceac000: Listen queue overflow: 193 already in queue awaiting acceptance (57 occurrences)
Sep 20 13:08:52 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8004ceac000: Listen queue overflow: 193 already in queue awaiting acceptance (41 occurrences)
Sep 20 13:09:52 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8004ceac000: Listen queue overflow: 193 already in queue awaiting acceptance (66 occurrences)
Sep 20 13:10:54 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8004ceac000: Listen queue overflow: 193 already in queue awaiting acceptance (36 occurrences)
Sep 20 13:11:54 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8004ceac000: Listen queue overflow: 193 already in queue awaiting acceptance (69 occurrences)
Sep 20 13:12:54 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8004ceac000: Listen queue overflow: 193 already in queue awaiting acceptance (86 occurrences)
Sep 20 13:13:41 FreeBSD-node1 cbsd[24331]: cbsdd: update remote inventory for 209.95.xx.x via loop_per_retrinv [10]
Sep 20 13:13:54 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8004ceac000: Listen queue overflow: 193 already in queue awaiting acceptance (70 occurrences)
Sep 20 13:14:54 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8004ceac000: Listen queue overflow: 193 already in queue awaiting acceptance (106 occurrences)
Sep 20 13:15:54 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff80b4ddda000: Listen queue overflow: 193 already in queue awaiting acceptance (83 occurrences)
Sep 20 13:16:55 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff80b4ddda000: Listen queue overflow: 193 already in queue awaiting acceptance (88 occurrences)
Sep 20 13:17:56 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8004ceac000: Listen queue overflow: 193 already in queue awaiting acceptance (80 occurrences)
Sep 20 13:18:56 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8004ceac000: Listen queue overflow: 193 already in queue awaiting acceptance (104 occurrences)
Sep 20 13:19:56 FreeBSD-node1 kernel: sonewconn: pcb 0xfffff8004ceac000: Listen queue overflow: 193 already in queue awaiting acceptance (138 occurrences)
https://pastebin.com/MjeJEH6b
Here is my entire config /var/db/mysql/my.cnf
Code:
[mysqld]
log-bin=mysql-bin
expire-logs-days=3
sync-binlog=1
# DATA STORAGE #
innodb_data_home_dir = /var/db/mysql-innodb
innodb_log_group_home_dir = /var/db/mysql-innodb-logs
# INNODB #
innodb-flush-method = O_DIRECT
# UTF-8 encoding settings #
collation-server = utf8_unicode_ci
init-connect ='SET NAMES utf8'
character-set-server = utf8
Here is my /boot/loader.conf from the FreeBSD host:
Code:
# ZFS root boot config
zfs_load="YES"
opensolaris_load="YES"
fdescfs_load="YES"
nullfs_load="YES"
# load the lagg module
if_lagg_load="YES"
# Set zfs pool output to show the GPT id for each drive
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gpt.enable="1"
kern.geom.label.gptid.enable="0"
# Pf firewall kernel modules, preload
pf_load="YES"
pflog_load="YES"
# Pf firewall kernel modules, preload
pf_load="YES"
pflog_load="YES"
# load the PF CARP module
if_carp_load="YES"
# load the uhid module
uhid_load="YES
# Jail resource limits
kern.racct.enable="1"
# Broadcom NetXtreme bce(4) kernel driver, preload
if_bce_load="YES"
# Advanced Host Controller Interface (AHCI)
ahci_load="YES"
# H-TCP Congestion Control for increase in speed
cc_htcp_load="YES"
# How many seconds to sit at the boot menu before booting the server.
autoboot_delay="-3" # (default 10) seconds
# Disabling hostcache cachelimit
net.inet.tcp.hostcache.cachelimit="0"
# Interface Maximum Queue Length
net.link.ifqmaxlen="2048" # (default 50)
# Enable the optimized version of soreceive() for stream (TCP) sockets.
net.inet.tcp.soreceive_stream="1" # (default 0)
# Disable destructive dtrace
security.bsd.allow_destructive_dtrace=0
# Load Nginx Filter
accf_http_load="YES"
accf_data_load="YES"
aio_load="YES"
# Load module for CBSD
#vmm_load="YES"
#nmdm_load="YES"
#if_bridge_load="YES"
#if_tap_load="YES"
kern.racct.enable=1
From what I found on the internet it could be a network saturation but I am unsure.
Thank you all in advance