tap0 interface - "No buffer space available"

Hi,

I'm trying to use ShrewSoft VPN Client on PC-BSD 10. It seems that there are some problems with my tap0 interface used by the VPN client to route the traffic to the remote network. The VPN client connects to my firewall and I can communicate with the internal network, but after some time (or after passing some traffic - I don't understand) the tunnel does not work any more. It is still shown as connected and there is no errors detected by the VPN client, but the traffic does not pass to the remote network and I have a strange error trying to ping a host from the remote network:
Code:
ping: sendto: No buffer space available

I use NAT-T (UDP encapsulation of ESP packets) - I recompiled the kernel to activate it.
The version of ShrewSoft VPN Client is 2.2.1 (installed from package).

Is there any sysctl(8) parameters to adjust buffers of tap interface? How can I debug the problem?

I had no problem with the same configuration under PC-BSD 9.0 on this laptop.
 
Re: tap0 interface - "No buffer space available"

Peter2121 said:
I had no problem with the same configuration under PCBSD 9.0 on this laptop.
It is generally recommended to rebuild/re-install all ports after upgrading from FreeBSD-9.X to FreeBSD-10.0. Did you?
 
Re: tap0 interface - "No buffer space available"

It is generally recommended to rebuild/re-install all ports after upgrading from FreeBSD-9.X to FreeBSD-10.0. Did you?
It is not an upgrade - the OS was reinstalled.

PC-BSD DesktopBSD FreeNAS NAS4Free m0N0WALL pfSense ArchBSD
I think that the problem is in the base system which is the same in FreeBSD and PCBSD. I hope to see more people with advanced network experience here than in PCBSD forums.
 
Re: tap0 interface - "No buffer space available"

The error you are getting can be caused by the likes of:
  • Network memory buffer exhaustion
  • Faulty NIC and/or driver issue
  • Faulty cable
  • Overloaded NIC
  • Other various switch/buffer/connectivity issues
You can use netstat -m to monitor network memory buffers.
 
Re: tap0 interface - "No buffer space available"

You're not alone. I've been messing around with FreeBSD 10 on a Soekris and unplugging it when I'm not using it since it's just been a temporary cable going across the floor and have seen the same error for dhclient. Doing service netif restart fixes it until I unplug again. I'm going to be putting it into use soon and will have to see if the errors go away when it's plugged in all the time or if they don't and what we are seeing is somehow related.
 
Re: tap0 interface - "No buffer space available"

The netstat -m output does not give anything to me.

Just after tap0 creation:
Code:
[peter@pcbsd-peter ~]$ netstat -m
1172/1408/2580 mbufs in use (current/cache/total)
1024/692/1716/504890 mbuf clusters in use (current/cache/total/max)
1024/684 mbuf+clusters out of packet secondary zone in use (current/cache)
64/82/146/252444 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/74798 9k jumbo clusters in use (current/cache/total/max)
0/0/0/42074 16k jumbo clusters in use (current/cache/total/max)
2597K/2064K/4661K bytes allocated to network (current/cache/total)
237/21/1583 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
75/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

Just after the problem with tunneling:
Code:
[peter@pcbsd-peter ~]$ netstat -m
1218/1362/2580 mbufs in use (current/cache/total)
1037/679/1716/504890 mbuf clusters in use (current/cache/total/max)
1037/671 mbuf+clusters out of packet secondary zone in use (current/cache)
65/81/146/252444 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/74798 9k jumbo clusters in use (current/cache/total/max)
0/0/0/42074 16k jumbo clusters in use (current/cache/total/max)
2638K/2022K/4661K bytes allocated to network (current/cache/total)
245/21/1583 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
75/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

I tried another command - vmstat -z at the same manner
Just after tap0 creation:
Code:
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               384,      0,     127,       3,     127,   0,   0
UMA Zones:             1152,      0,     127,       2,     127,   0,   0
UMA Slabs:               80,      0,   30558,      92,  192575,   0,   0
UMA RCntSlabs:           88,      0,    1004,      31,    1004,   0,   0
UMA Hash:               256,      0,       8,       7,      11,   0,   0
4 Bucket:                32,      0,      98,     777, 1265294,   0,   0
8 Bucket:                64,      0,      40,     766,   33237,   0,   0
16 Bucket:              128,      0,      72,     579,     851,  28,   0
32 Bucket:              256,      0,      98,     217,    5753, 100,   0
64 Bucket:              512,      0,     104,     160,   13061,  95,   0
128 Bucket:            1024,      0,     317,     163,   30546,  26,   0
vmem btag:               56,      0,   36991,    1846,   51335, 548,   0
VM OBJECT:              256,      0,   14071,    1919,  855731,   0,   0
RADIX NODE:             144,      0,   40006,    1466, 2414121,  47,   0
MAP:                    240,      0,       3,      61,       3,   0,   0
KMAP ENTRY:             128,      0,      70,     457,      96,   8,   0
MAP ENTRY:              128,      0,    7079,    4422, 2698910,   0,   0
VMSPACE:                448,      0,      85,     149,   33717,   0,   0
fakepg:                 104,      0,    6962,    1056,    7829,   0,   0
mt_zone:               4112,      0,     400,       0,     400,   0,   0
16:                      16,      0,   24723,     377,16389584,   0,   0
32:                      32,      0,   14284,    1091, 3976457,   0,   0
64:                      64,      0,  131104,     708, 5779655,   0,   0
128:                    128,      0,   24957,     649,10737736,   0,   0
256:                    256,      0,   14942,    4438, 1367248,   0,   0
512:                    512,      0,   63211,      77,  538534,   0,   0
1024:                  1024,      0,     910,      74,  357266,   0,   0
2048:                  2048,      0,     728,      48,  284020,   0,   0
4096:                  4096,      0,    3984,      35,  173420,   0,   0
SLEEPQUEUE:              80,      0,     508,     267,     508,   0,   0
uint64 pcpu:              8,      0,    2402,     158,    2402,   0,   0
Files:                   80,      0,     786,     714,  312179,   0,   0
TURNSTILE:              136,      0,     508,     112,     508,   0,   0
rl_entry:                40,      0,     118,     382,     118,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1208,      0,     109,      38,   34057,   0,   0
THREAD:                1168,      0,     449,      58,    4034,   0,   0
cpuset:                  72,      0,     247,     413,     391,   0,   0
audit_record:          1248,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 3231300,    1024,     684,  405215,1583,   0
mbuf:                   256, 3231300,     148,     724,11671005, 237,   0
mbuf_cluster:          2048, 504890,    1708,       8,    1717,  21,   0
mbuf_jumbo_page:       4096, 252444,      64,      82, 1459598,  75,   0
mbuf_jumbo_9k:         9216,  74798,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,  42074,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,       0,     432,  655526,   0,   0
ttyinq:                 160,      0,     195,      55,     795,   0,   0
ttyoutq:                256,      0,      96,     159,     408,   0,   0
ata_request:            336,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
nv_stack_t:           12288,      0,       7,       7,   16226,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,     581,     807,   0,   0
VNODE:                  472,      0,   87557,      83,   89750,   0,   0
VNODEPOLL:              112,      0,      54,     471,      54,   0,   0
BUF TRIE:               144,      0,       9,   52614,       9,   0,   0
NAMEI:                 1024,      0,       0,      80, 1727096,   0,   0
S VFS Cache:            108,      0,   52619,     406,   63403,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,    2354,      82,    2471,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0
NCLNODE:                528,      0,       0,       0,       0,   0,   0
Mountpoints:            816,      0,      52,      28,      52,   0,   0
space_seg_cache:         64,      0,   11007,     277,  155965,155965,   0
zio_cache:              944,      0,       1,    1751, 2901941,2901941,   0
zio_link_cache:          48,      0,       0,    1826, 1012381,1012381,   0
sa_cache:                80,      0,   55179,      21,   56371,56371,   0
dnode_t:                864,      0,   56890,       6,   58074,58074,   0
dmu_buf_impl_t:         224,      0,   75476,       4,   84478,84478,   0
arc_buf_hdr_t:          216,      0,   27141,       3,   39064,39064,   0
arc_buf_t:               72,      0,   25121,      14,   39123,39123,   0
zil_lwb_cache:          192,      0,       5,     155,    2349,2349,   0
zfs_znode_cache:        368,      0,   55179,      11,   56371,56371,   0
pipe:                   744,      0,     145,      65,   29932,   0,   0
procdesc:               128,      0,       0,       0,       0,   0,   0
ksiginfo:               112,      0,     134,     916,   44540,   0,   0
itimer:                 352,      0,       1,      32,       1,   0,   0
KNOTE:                  128,      0,      72,     703,  973938,   0,   0
socket:                 696, 259730,     232,     173,   23989,   0,   0
unpcb:                  240, 259744,     185,     183,    4747,   0,   0
ipq:                     56,  15833,       0,       0,       0,   0,   0
udp_inpcb:              392, 259730,      22,     158,   15091,   0,   0
udpcb:                   16, 259785,      22,     480,   15091,   0,   0
tcp_inpcb:              392, 259730,      17,     393,    2473,   0,   0
tcpcb:                 1024, 259732,      17,     167,    2473,   0,   0
tcptw:                   88,  27810,       0,     900,    1292,   0,   0
syncache:               160,  15375,       0,       0,       0,   0,   0
hostcache:              136,  15370,     114,     205,     134,   0,   0
tcpreass:                40,  31600,       0,     500,   29608,   0,   0
sackhole:                32,      0,       0,     125,       1,   0,   0
sctp_ep:               1408, 259730,       0,       0,       0,   0,   0
sctp_asoc:             2352,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80012,       0,     415,       7,   0,   0
sctp_raddr:             728,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400026,       0,       0,       0,   0,   0
sctp_readq:             104, 400026,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400026,       0,       0,       0,   0,   0
sctp_asconf:             40, 400000,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0
ripcb:                  392, 259730,       1,      89,      15,   0,   0
rtentry:                200,      0,      30,     230,      31,   0,   0
selfd:                   56,      0,     434,     844,98991044,   0,   0
SWAPMETA:               288, 1009788,       0,       0,       0,   0,   0
NetGraph items:          72,   4123,       0,     527,    1660,   0,   0
NetGraph data items:     72,    527,       0,     527,     854,   0,   0
UDF translation buffer, zone:    510,      0,       0,       0,       0,   0,   0
UDF Node zone:           40,      0,       0,       0,       0,   0,   0
UDF Dirstream zone:      72,      0,       0,       0,       0,   0,   0
pefs_node:               88,      0,       0,       0,       0,   0,   0
pefs_dircache:           64,      0,       0,       0,       0,   0,   0
pefs_dircache_entry:    544,      0,       0,       0,       0,   0,   0
pefs_ctx:              1024,      0,       0,       0,       0,   0,   0
pefs_key:                88,      0,       0,       0,       0,   0,   0
fuse_ticket:            224,      0,       0,       0,       0,   0,   0
pf states:              296,  10010,       7,     526,    4026,   0,   0
pf state keys:           88,      0,       7,     758,    4026,   0,   0
pf source nodes:        136,  10005,       0,       0,       0,   0,   0
pf mtags:                40,      0,       0,     700,     974,   0,   0
pf table entries:       160, 200000,       1,      74,       1,   0,   0
pf table counters:       64,      0,       0,       0,       0,   0,   0
pf frags:                80,      0,       0,     250,       2,   0,   0
pf frag entries:         32,   5000,       0,     375,       4,   0,   0
pf state scrubs:         40,      0,       0,       0,       0,   0,   0
IPFW dynamic rule:      120,   4125,       0,       0,       0,   0,   0

Just after the problem with tunneling:
Code:
[peter@pcbsd-peter ~]$ vmstat -z
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               384,      0,     127,       3,     127,   0,   0
UMA Zones:             1152,      0,     127,       2,     127,   0,   0
UMA Slabs:               80,      0,   30824,     126,  206696,   0,   0
UMA RCntSlabs:           88,      0,    1004,      31,    1004,   0,   0
UMA Hash:               256,      0,       8,       7,      11,   0,   0
4 Bucket:                32,      0,      96,     779, 1301386,   0,   0
8 Bucket:                64,      0,      52,     754,   36223,   0,   0
16 Bucket:              128,      0,      71,     580,     886,  28,   0
32 Bucket:              256,      0,     102,     213,    5777, 100,   0
64 Bucket:              512,      0,      91,     173,   13390,  95,   0
128 Bucket:            1024,      0,     314,     166,   31504,  26,   0
vmem btag:               56,      0,   37186,    1651,   52145, 548,   0
VM OBJECT:              256,      0,   14447,    1543,  907641,   0,   0
RADIX NODE:             144,      0,   40604,     868, 2501110,  47,   0
MAP:                    240,      0,       3,      61,       3,   0,   0
KMAP ENTRY:             128,      0,      70,     457,      96,   8,   0
MAP ENTRY:              128,      0,    7709,    3792, 2854439,   0,   0
VMSPACE:                448,      0,      88,     146,   35083,   0,   0
fakepg:                 104,      0,    6962,    1056,    7829,   0,   0
mt_zone:               4112,      0,     400,       0,     400,   0,   0
16:                      16,      0,   25092,     510,17632591,   0,   0
32:                      32,      0,   14343,    1032, 4061063,   0,   0
64:                      64,      0,  131285,     651, 6176381,   0,   0
128:                    128,      0,   24995,     611,11111600,   0,   0
256:                    256,      0,   15048,    4332, 1439413,   0,   0
512:                    512,      0,   63272,     104,  552243,   0,   0
1024:                  1024,      0,     913,      71,  382819,   0,   0
2048:                  2048,      0,     731,      53,  311705,   0,   0
4096:                  4096,      0,    3994,     101,  183789,   0,   0
SLEEPQUEUE:              80,      0,     508,     267,     508,   0,   0
uint64 pcpu:              8,      0,    2402,     158,    2402,   0,   0
Files:                   80,      0,     825,     675,  328548,   0,   0
TURNSTILE:              136,      0,     508,     112,     508,   0,   0
rl_entry:                40,      0,     122,     378,     122,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1208,      0,     112,      35,   35445,   0,   0
THREAD:                1168,      0,     451,      56,    4264,   0,   0
cpuset:                  72,      0,     247,     413,     391,   0,   0
audit_record:          1248,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 3231300,    1037,     671,  422805,1583,   0
mbuf:                   256, 3231300,     181,     691,12290283, 245,   0
mbuf_cluster:          2048, 504890,    1708,       8,    1717,  21,   0
mbuf_jumbo_page:       4096, 252444,      65,      81, 1506144,  75,   0
mbuf_jumbo_9k:         9216,  74798,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,  42074,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,       0,     432,  700100,   0,   0
ttyinq:                 160,      0,     210,      40,     810,   0,   0
ttyoutq:                256,      0,     104,     151,     416,   0,   0
ata_request:            336,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,     540,    1145,   0,   0
cryptodesc:              72,      0,       0,     550,    2290,   0,   0
nv_stack_t:           12288,      0,       7,       7,   17558,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,     664,     882,   0,   0
VNODE:                  472,      0,   87611,     117,   89906,   0,   0
VNODEPOLL:              112,      0,      54,     471,      54,   0,   0
BUF TRIE:               144,      0,       9,   52614,       9,   0,   0
NAMEI:                 1024,      0,       0,      80, 1834257,   0,   0
S VFS Cache:            108,      0,   52683,     342,   63630,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,    2353,      83,    2487,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0
NCLNODE:                528,      0,       0,       0,       0,   0,   0
Mountpoints:            816,      0,      52,      28,      52,   0,   0
space_seg_cache:         64,      0,   11074,     210,  168188,168188,   0
zio_cache:              944,      0,       1,    1751, 2993187,2993187,   0
zio_link_cache:          48,      0,       0,    1826, 1100412,1100412,   0
sa_cache:                80,      0,   55190,      10,   56454,56454,   0
dnode_t:                864,      0,   56977,       3,   58164,58164,   0
dmu_buf_impl_t:         224,      0,   75582,       0,   85330,85330,   0
arc_buf_hdr_t:          216,      0,   27488,      16,   40119,40119,   0
arc_buf_t:               72,      0,   25309,      46,   40178,40178,   0
zil_lwb_cache:          192,      0,       5,     155,    2627,2627,   0
zfs_znode_cache:        368,      0,   55190,      10,   56454,56454,   0
pipe:                   744,      0,     149,      76,   31506,   0,   0
procdesc:               128,      0,       0,       0,       0,   0,   0
ksiginfo:               112,      0,     134,     916,   53888,   0,   0
itimer:                 352,      0,       1,      32,       1,   0,   0
KNOTE:                  128,      0,      76,     699,  976673,   0,   0
socket:                 696, 259730,     255,     150,   25477,   0,   0
unpcb:                  240, 259744,     197,     171,    4902,   0,   0
ipq:                     56,  15833,       0,       0,       0,   0,   0
udp_inpcb:              392, 259730,      27,     153,   16141,   0,   0
udpcb:                   16, 259785,      27,     475,   16141,   0,   0
tcp_inpcb:              392, 259730,      23,     387,    2622,   0,   0
tcpcb:                 1024, 259732,      23,     161,    2622,   0,   0
tcptw:                   88,  27810,       0,     900,    1356,   0,   0
syncache:               160,  15375,       0,       0,       0,   0,   0
hostcache:              136,  15370,      99,     220,     147,   0,   0
tcpreass:                40,  31600,       0,     500,   29608,   0,   0
sackhole:                32,      0,       0,     125,       1,   0,   0
sctp_ep:               1408, 259730,       0,       0,       0,   0,   0
sctp_asoc:             2352,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80012,       0,     415,       7,   0,   0
sctp_raddr:             728,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400026,       0,       0,       0,   0,   0
sctp_readq:             104, 400026,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400026,       0,       0,       0,   0,   0
sctp_asconf:             40, 400000,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0
ripcb:                  392, 259730,       1,      89,      17,   0,   0
rtentry:                200,      0,      30,     230,      31,   0,   0
selfd:                   56,      0,     449,     829,108458227,   0,   0
SWAPMETA:               288, 1009788,       0,       0,       0,   0,   0
NetGraph items:          72,   4123,       0,     527,    1792,   0,   0
NetGraph data items:     72,    527,       0,     527,     920,   0,   0
UDF translation buffer, zone:    510,      0,       0,       0,       0,   0,   0
UDF Node zone:           40,      0,       0,       0,       0,   0,   0
UDF Dirstream zone:      72,      0,       0,       0,       0,   0,   0
pefs_node:               88,      0,       0,       0,       0,   0,   0
pefs_dircache:           64,      0,       0,       0,       0,   0,   0
pefs_dircache_entry:    544,      0,       0,       0,       0,   0,   0
pefs_ctx:              1024,      0,       0,       0,       0,   0,   0
pefs_key:                88,      0,       0,       0,       0,   0,   0
fuse_ticket:            224,      0,       0,       0,       0,   0,   0
pf states:              296,  10010,       6,     527,    4211,   0,   0
pf state keys:           88,      0,       6,     759,    4211,   0,   0
pf source nodes:        136,  10005,       0,       0,       0,   0,   0
pf mtags:                40,      0,       0,     700,    1111,   0,   0
pf table entries:       160, 200000,       1,      74,       1,   0,   0
pf table counters:       64,      0,       0,       0,       0,   0,   0
pf frags:                80,      0,       0,     250,       2,   0,   0
pf frag entries:         32,   5000,       0,     375,       4,   0,   0
pf state scrubs:         40,      0,       0,       0,       0,   0,   0
IPFW dynamic rule:      120,   4125,       0,       0,       0,   0,   0

I don't see anything interesting here neither.

The error you are getting can be caused by the likes of:
Network memory buffer exhaustion
Faulty NIC and/or driver issue
Faulty cable
Overloaded NIC
Other various switch/buffer/connectivity issues

Probably, the first reason is the nearest to my problem, but I cannot find any diagnostic :(
The NIC is not overloaded, the problems with switch and cables cannot be here as the interface is virtual.
Maybe there are some problems with tap driver but I don't think to be the only user of tap interface in FreeBSD 10 - probably someone would already detected this issue.
 
Code:
Re: tap0 interface - "No buffer space available"

It seems that the problem is solved after the update of PCBSD to the last version (10.0.1).
 
Last edited by a moderator:
The problem was back on 10.0.3, it is still here on 10.1.
BTW, there is an open PR 123892 that seems to address the issue, but the status is not updated since 2008 (!!)
What is really strange - the problem was NOT present in 10.0.1 version!
 
Back
Top