ping: sendto: No buffer space available

I

i.am.the.problem

Hi,

I have an issue with a server. One symptom is that its intermittently sluggish to server web pages.
It also has 17 openvpn instances running with about 500 clients spread across them.
When its acting up I see the following error when trying to ping clients:
Code: 
ping: sendto: No buffer space available
I don't see anything obvious here:
Code: 
sysctl dev.bge.0
dev.bge.0.stats.tx.BroadcastPkts: 1
dev.bge.0.stats.tx.MulticastPkts: 0
dev.bge.0.stats.tx.UnicastPkts: 18436510
dev.bge.0.stats.tx.LateCollisions: 0
dev.bge.0.stats.tx.ExcessiveCollisions: 0
dev.bge.0.stats.tx.DeferredTransmissions: 0
dev.bge.0.stats.tx.MultipleCollisionFrames: 0
dev.bge.0.stats.tx.SingleCollisionFrames: 0
dev.bge.0.stats.tx.InternalMacTransmitErrors: 0
dev.bge.0.stats.tx.XoffSent: 0
dev.bge.0.stats.tx.XonSent: 0
dev.bge.0.stats.tx.Collisions: 0
dev.bge.0.stats.tx.ifHCOutOctets: 5974763911
dev.bge.0.stats.rx.UndersizePkts: 0
dev.bge.0.stats.rx.Jabbers: 0
dev.bge.0.stats.rx.FramesTooLong: 0
dev.bge.0.stats.rx.xoffStateEntered: 0
dev.bge.0.stats.rx.ControlFramesReceived: 0
dev.bge.0.stats.rx.xoffPauseFramesReceived: 0
dev.bge.0.stats.rx.xonPauseFramesReceived: 0
dev.bge.0.stats.rx.AlignmentErrors: 0
dev.bge.0.stats.rx.FCSErrors: 0
dev.bge.0.stats.rx.BroadcastPkts: 211496
dev.bge.0.stats.rx.MulticastPkts: 115065
dev.bge.0.stats.rx.UnicastPkts: 15596793
dev.bge.0.stats.rx.Fragments: 0
dev.bge.0.stats.rx.ifHCInOctets: 8154504167
dev.bge.0.stats.RecvThresholdHit: 0
dev.bge.0.stats.InputErrors: 0
dev.bge.0.stats.InputDiscards: 0
dev.bge.0.stats.NoMoreRxBDs: 3
dev.bge.0.stats.DmaWriteHighPriQueueFull: 0
dev.bge.0.stats.DmaWriteQueueFull: 0
dev.bge.0.stats.FramesDroppedDueToFilters: 0
dev.bge.0.forced_udpcsum: 0
dev.bge.0.msi: 1
dev.bge.0.forced_collapse: 0
dev.bge.0.%domain: 0
dev.bge.0.%parent: pci3
dev.bge.0.%pnpinfo: vendor=0x14e4 device=0x165f subvendor=0x1028 subdevice=0x001f class=0x020000
dev.bge.0.%location: slot=0 function=0 dbsf=pci0:4:0:0 handle=\_SB_.PC00.RP06.NIC1
dev.bge.0.%driver: bge
dev.bge.0.%desc: Broadcom NetXtreme Gigabit Ethernet, ASIC rev. 0x5720000
Code: 
sysctl dev.bge.1
dev.bge.1.stats.tx.BroadcastPkts: 11
dev.bge.1.stats.tx.MulticastPkts: 0
dev.bge.1.stats.tx.UnicastPkts: 1500
dev.bge.1.stats.tx.LateCollisions: 0
dev.bge.1.stats.tx.ExcessiveCollisions: 0
dev.bge.1.stats.tx.DeferredTransmissions: 0
dev.bge.1.stats.tx.MultipleCollisionFrames: 0
dev.bge.1.stats.tx.SingleCollisionFrames: 0
dev.bge.1.stats.tx.InternalMacTransmitErrors: 0
dev.bge.1.stats.tx.XoffSent: 0
dev.bge.1.stats.tx.XonSent: 0
dev.bge.1.stats.tx.Collisions: 0
dev.bge.1.stats.tx.ifHCOutOctets: 202431
dev.bge.1.stats.rx.UndersizePkts: 0
dev.bge.1.stats.rx.Jabbers: 0
dev.bge.1.stats.rx.FramesTooLong: 0
dev.bge.1.stats.rx.xoffStateEntered: 0
dev.bge.1.stats.rx.ControlFramesReceived: 0
dev.bge.1.stats.rx.xoffPauseFramesReceived: 0
dev.bge.1.stats.rx.xonPauseFramesReceived: 0
dev.bge.1.stats.rx.AlignmentErrors: 0
dev.bge.1.stats.rx.FCSErrors: 0
dev.bge.1.stats.rx.BroadcastPkts: 1230
dev.bge.1.stats.rx.MulticastPkts: 5653
dev.bge.1.stats.rx.UnicastPkts: 1970
dev.bge.1.stats.rx.Fragments: 0
dev.bge.1.stats.rx.ifHCInOctets: 2595772
dev.bge.1.stats.RecvThresholdHit: 0
dev.bge.1.stats.InputErrors: 0
dev.bge.1.stats.InputDiscards: 0
dev.bge.1.stats.NoMoreRxBDs: 0
dev.bge.1.stats.DmaWriteHighPriQueueFull: 0
dev.bge.1.stats.DmaWriteQueueFull: 0
dev.bge.1.stats.FramesDroppedDueToFilters: 0
dev.bge.1.forced_udpcsum: 0
dev.bge.1.msi: 1
dev.bge.1.forced_collapse: 0
dev.bge.1.%domain: 0
dev.bge.1.%parent: pci3
dev.bge.1.%pnpinfo: vendor=0x14e4 device=0x165f subvendor=0x1028 subdevice=0x001f class=0x020000
dev.bge.1.%location: slot=0 function=1 dbsf=pci0:4:0:1 handle=\_SB_.PC00.RP06.NIC2
dev.bge.1.%driver: bge
dev.bge.1.%desc: Broadcom NetXtreme Gigabit Ethernet, ASIC rev. 0x5720000

Is there anything else I can look at?
 
SirDice said:
Firewall? Maybe you're running out of states.
Click to expand...
Code: 
service pf status
Status: Enabled for 1 days 23:46:29           Debug: Urgent

State Table                          Total             Rate
  current entries                        0               
  searches                      3803578530        22115.2/s
  inserts                                0            0.0/s
  removals                               0            0.0/s
Counters
  match                         3803578530        22115.2/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  map-failed                             0            0.0/s
 
Could you provide output from next commands when you get subject error?
Code: 
netstat -m
vmstat -z | grep mbuf
Also, check if you tune next sysctl variables. They should be at least, not less:
Code: 
hw.bge.rxd=4096
hw.bge.txd=4096
kern.ipc.nmbclusters=524288
kern.ipc.maxsockbuf=1048576
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288
Also, you can increase some kernel variables via rebuilding kernel with next options:
Code: 
maxusers        512
options         NBUF=4096
I get such error a few years ago and all of this changes helped me.

Also, I had another FreeBSD box which had a cheap NIC (some Realtek) and it didn't help with tuning, only replacing to Enterprise NIC helped solve this issue.
 
maxusers has been a tunable since 4.5, only on versions before 4.4 does it need to be compiled in the kernel.

And depending on the amount of memory and a few other variables, it's typically much higher than 512.

Code: 
dice@maelcum:~ % sysctl kern.maxusers
kern.maxusers: 1343
 
Hi, These are my settings at the moment:

Code: 
kern.ipc.nmbclusters: 8150088
kern.ipc.maxsockbuf: 2097152
net.inet.tcp.sendbuf_max: 4194304
net.inet.tcp.recvbuf_max: 4194304
net.inet.tcp.sendbuf_inc: 8192
net.inet.tcp.recvbuf_inc: 16384

Nothing set at all for hw.bge.rxd and hw.bge.txd

I'll tweek these and see how things go.
 
You must log in or register to reply here.
Back
Top