Solved Increase buffer with maxsockbuf for udp streaming

NapoleonWils0n

Well-Known Member

Reaction score: 115
Messages: 259

HiI need to increase my buffer size for UDP video streaming

My current buffer size is

Bash:
kern.ipc.maxsockbuf: 2097152
When I use ffmpeg to capture my desktop and send the stream over udp to another computer to record, the maximum bit rate and buffer size I can use is 500k with this command:

Bash:
ffmpeg -f x11grab -r 15 -s 1366x768 -i :0.0+0,0 \
-c:v libx264 -preset ultrafast -b 500k \
-tune zerolatency \
-maxrate 500k -bufsize 500k \
-pix_fmt yuv420p \
-f mpegts 'udp://192.168.1.102:6881?pkt_size=1316'
If I use a higher bit rate the command fails and gives me an error saying the buffer size is too small

I tried to increase the buffer size with this command but I still get the same error with a higher bitrate

Bash:
# sysctl -w kern.ipc.maxsockbuf=16777216
Do I need to add set the new buffer size in my /etc/sysctl.conf and then reboot, and are there any other settings that control the buffer size I should set

The reason I'm looking into this is because I have figured out how to send the output from obs studio over an UDP stream with ffmpeg to a second machine which I can then record the stream on either using obs or ffmpeg.

This is useful because it means you can spread the load between 2 computers, the first computer using obs to capture you web cam, record the desktop and audio etc, then send the stream over udp using ffmpeg command in obs to the second machine to record

I have also figured out how you take the output from obs and multicast stream to all device on the network at onc ewith less than 1 second of latency which is very cool as it means you can create your own lan tv station with obs.

My laptop has 4 gig of ram if that has any bearing on the buffer size I have used the same laptop running OS-X and Linux to send higher bitrate streams over UDP before
so the bottleneck is buffer size on FreeBSD, if anyone could give me a prod in the right direction would be really helpful.

I'll make a video about obs and udp streaming soon I just need to resolve the buffer size issue on FreeBSD first.
 
OP
NapoleonWils0n

NapoleonWils0n

Well-Known Member

Reaction score: 115
Messages: 259

Cheers SirDice i did come across a blog post mentioning the setting could be changed on the fly,
but generally im not one to trust random blog posts

I did try increasing the maxsockbuf value and then re running the ffmpeg command with an increased bitrate,
but i still got the buffer space error

Maybe i didnt increase the maxsockbuf value enough, ill try again
Is there anything else that determines the buffer size, i did see someone mention kern.ipc.nmbclusters

Also on a side note there is a obs-ndi plugin in ports as well which will make obs on freebsd a lot more usefull
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 8,931
Messages: 33,327

Is there anything else that determines the buffer size, i did see someone mention kern.ipc.nmbclusters
I looked in tuning(7) first. Most of the network tuning information seemed to be centered around TCP though.
 
OP
NapoleonWils0n

NapoleonWils0n

Well-Known Member

Reaction score: 115
Messages: 259

Cheers SirDice for having a look at tuning

I increase maxsockbuf to a million

Bash:
doas sysctl -w kern.ipc.maxsockbuf=1000000000
and then streamed my desktop over udp to a multicast address at 1000k

Bash:
ffmpeg -f x11grab -r 15 -s 1366x768 -i :0.0+0,0 \
-c:v libx264 -preset ultrafast -b 1000k \
-tune zerolatency \
-maxrate 1000k -bufsize 1000k \
-pix_fmt yuv420p \
-f mpegts 'udp://239.253.253.4:1234?pkt_size=1316'
and this is the error i get

Bash:
[x11grab @ 0x8054b2000] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from ':0.0+0,0':
  Duration: N/A, start: 1593457289.204475, bitrate: N/A
    Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1366x768, 15 fps, 1000k tbr, 1000k tbn, 1000k tbc
Please use -b:a or -b:v, -b is ambiguous
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[swscaler @ 0x80553aec0] Warning: data is not aligned! This can lead to a speed loss
[libx264 @ 0x805457e00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 @ 0x805457e00] profile Constrained Baseline, level 3.2, 4:2:0, 8-bit
Output #0, mpegts, to 'udp://239.253.253.4:1234?pkt_size=1316':
  Metadata:
    encoder         : Lavf58.29.100
    Stream #0:0: Video: h264 (libx264), yuv420p, 1366x768, q=-1--1, 1000 kb/s, 15 fps, 90k tbn, 15 tbc
    Metadata:
      encoder         : Lavc58.54.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 1000000/0/1000000 buffer size: 1000000 vbv_delay: -1
av_interleaved_write_frame(): No buffer space available.13 bitrate= 546.0kbits/s speed=1.01x     
Error writing trailer of udp://239.253.253.4:1234?pkt_size=1316: No buffer space available01x   
frame=  251 fps= 15 q=26.0 Lsize=    1201kB time=00:00:16.66 bitrate= 590.4kbits/s speed=   1x   
video:1092kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 10.048042%
[libx264 @ 0x805457e00] frame I:2     Avg QP:31.39  size: 83787
[libx264 @ 0x805457e00] frame P:249   Avg QP: 8.55  size:  3816
[libx264 @ 0x805457e00] mb I  I16..4: 100.0%  0.0%  0.0%
[libx264 @ 0x805457e00] mb P  I16..4:  0.4%  0.0%  0.0%  P16..4:  8.1%  0.0%  0.0%  0.0%  0.0%    skip:91.5%
[libx264 @ 0x805457e00] coded y,uvDC,uvAC intra: 40.5% 18.3% 16.3% inter: 2.9% 3.3% 2.7%
[libx264 @ 0x805457e00] i16 v,h,dc,p: 67% 22%  9%  2%
[libx264 @ 0x805457e00] i8c dc,h,v,p: 75% 17%  7%  1%
[libx264 @ 0x805457e00] kb/s:534.39
Conversion failed!
if i stream at 500k it works but any higher bitrate i get the No buffer space available error

strange thing is i have used udp streaming at much higher bit rates on linux and mac osx on the same hardware without any issue
I couldnt see any other kern.ipc sysctl settings that might be causing the issue, perhaps nmbclusters

i would have thought that increasing maxsockbuf to 1 million should allow udp streaming at 1000k

i sometimes have issues when im downloading torrents over a vpn and i get the same No buffer space available error message,
which makes me think maybe its something i have in my pf.conf that causing the issue

Bash:
#=========================================================================#
# variables, macro and tables                                             #
#=========================================================================#

int_if="ue0" # usb to ethernet adaptor
#int_if="bge0" # thunderbolt to ethernet adaptor
#int_if="wlan0" # ralink usb wifi
vpn_if="tun0" # vpn interface
all_networks="0.0.0.0/0"
vpn_network="$vpn_if:network"
tcp_services = "{ ntp, 6881, 22000, 24800 }" # tcp services - rtorrent, syncthing , synergy
udp_services = "{ ntp, 6882, 21025 }" # udp services - rtorrent, syncthing
icmp_types = "{ echoreq, unreach }"
tcp_state="flags S/SA keep state"
udp_state="keep state"

#table <internet> { $all_networks, !self, !$int_if:network } # internet
#table <lan> { $int_if:network, !self }                      # lan network
table <myself> { self }                                     # self
table <martians> { 0.0.0.0/8 10.0.0.0/8 127.0.0.0/8 169.254.0.0/16     \
            172.16.0.0/12 192.0.0.0/24 192.0.2.0/24 224.0.0.0/3 \
            192.168.0.0/16 198.18.0.0/15 198.51.100.0/24        \
            203.0.113.0/24 }                         # broken networks

#=========================================================================#
# global policy                                                           #
#=========================================================================#

set block-policy drop
set loginterface $int_if
set fingerprints "/etc/pf.os"
set skip on lo0
scrub in all fragment reassemble no-df max-mss 1440
antispoof log quick for { lo $int_if } label "block_spoofing"

#=========================================================================#
# block                                                                   #
#=========================================================================#

block log all # block log all
block return out quick inet6 all tag IPV6 # block ipv6
block in quick inet6 all tag IPV6 # block ipv6

# block broken networks - turned off for synergy
# block in quick from { <martians> no-route urpf-failed } to any tag BAD_PACKET

#=========================================================================#
# anchors                                                                 #
#=========================================================================#

# emerging threats - anchor
anchor "emerging-threats"
load anchor "emerging-threats" from "/etc/pf.anchors/emerging-threats"

# openvpn - anchor
anchor "openvpn"

#=========================================================================#
# traffic tag                                                             #
#=========================================================================#

# icmp
pass inet proto icmp all icmp-type $icmp_types keep state tag ICMP

# Allow the tcp and udp services defined in the macros at the top of the file
pass in on $int_if inet proto tcp from any to ($int_if) port $tcp_services $tcp_state tag TCP_IN
pass in on $int_if inet proto udp from any to ($int_if) port $udp_services $udp_state tag UDP_IN

# outbound traffic
block out on $int_if all
pass out quick on $int_if all modulate state
#pass out quick on $int_if from <myself> to <lan> modulate state tag LAN_OUT
#pass out quick on $int_if from <myself> to <internet> modulate state tag INTERNET_OUT
 
OP
NapoleonWils0n

NapoleonWils0n

Well-Known Member

Reaction score: 115
Messages: 259

i disabled pf to see if that was the issue by running

Bash:
doas pfctl -d
Then i tried steaming at 1000k but i still get the No buffer space available error

So it doesnt seem to be maxsockbuf or pf causing the error,
have to look in the handbook and see if theres anything i missed

Ill also have a google for freebsd vpn buffer space and see if that turns up anything

Usually when i get the no buffer space error when using a vpn taking down my network interface and then bring it up again resolves the issue

These are sysctl kern.ipc settings

Bash:
sysctl kern.ipc
Bash:
kern.ipc.maxsockbuf: 2097152
kern.ipc.sockbuf_waste_factor: 8
kern.ipc.max_linkhdr: 16
kern.ipc.max_protohdr: 60
kern.ipc.max_hdr: 76
kern.ipc.max_datalen: 92
kern.ipc.maxsockets: 126924
kern.ipc.numopensockets: 437
kern.ipc.soacceptqueue: 128
kern.ipc.shm_allow_removed: 1
kern.ipc.shm_use_phys: 0
kern.ipc.shmall: 131072
kern.ipc.shmseg: 128
kern.ipc.shmmni: 192
kern.ipc.shmmin: 1
kern.ipc.shmmax: 536870912
kern.ipc.semaem: 16384
kern.ipc.semvmx: 32767
kern.ipc.semusz: 632
kern.ipc.semume: 50
kern.ipc.semopm: 100
kern.ipc.semmsl: 340
kern.ipc.semmnu: 150
kern.ipc.semmns: 340
kern.ipc.semmni: 50
kern.ipc.msgseg: 2048
kern.ipc.msgssz: 8
kern.ipc.msgtql: 40
kern.ipc.msgmnb: 2048
kern.ipc.msgmni: 40
kern.ipc.msgmax: 16384
kern.ipc.aio.lifetime: 30000
kern.ipc.aio.target_procs: 4
kern.ipc.aio.num_procs: 4
kern.ipc.aio.max_procs: 32
kern.ipc.aio.empty_retries: 0
kern.ipc.aio.empty_results: 0
kern.ipc.piperesizeallowed: 1
kern.ipc.piperesizefail: 0
kern.ipc.pipeallocfail: 0
kern.ipc.pipefragretry: 0
kern.ipc.pipekva: 1822720
kern.ipc.maxpipekva: 64983040
kern.ipc.umtx_max_robust: 1000
kern.ipc.umtx_vnode_persistent: 0
kern.ipc.nmbufs: 1564185
kern.ipc.nmbjumbo16: 20366
kern.ipc.nmbjumbo9: 36207
kern.ipc.nmbjumbop: 122201
kern.ipc.nmbclusters: 244404
kern.ipc.maxmbufmem: 2002151424
The manual page on udp mentions a few udp settings in net.inet

The udp protocol implements a number of variables in the net.inet branch
of the sysctl(3) MIB.

UDPCTL_CHECKSUM (udp.checksum) Enable udp checksums (enabled by de-
fault).

UDPCTL_MAXDGRAM (udp.maxdgram) Maximum outgoing UDP datagram size

UDPCTL_RECVSPACE (udp.recvspace) Maximum space for incoming UDP data-
grams

udp.log_in_vain For all udp datagrams, to ports on which there is no
socket listening, log the connection attempt (disabled
by default).

udp.blackhole When a datagram is received on a port where there is
no socket listening, do not return an ICMP port un-
reachable message. (Disabled by default. See
blackhole(4).)

looking at my sysctl settings

Bash:
sysctl inet.net.udp
i see this

Bash:
net.inet.udp.checksum: 1
net.inet.udp.maxdgram: 9216
net.inet.udp.recvspace: 42080
net.inet.udp.blackhole: 0
net.inet.udp.log_in_vain: 0
not sure if im digging in the right place or not

i tried doubling nmbclusters, maxsockbuf, maxdgram and recvspace

Bash:
doas sysctl -w kern.ipc.nmbclusters=488808
doas sysctl -w kern.ipc.maxsockbuf=4194304
doas sysctl -w net.inet.udp.maxdgram=18432
doas sysctl -w net.inet.udp.recvspace=84160
but it actually made the stream cut out in 15 seconds,
whereas if i dont change those settings the stream cuts out after 30 seconds

googling freebsd no buffer space error turns up posts going back 15+ years with the same issue
 
OP
NapoleonWils0n

NapoleonWils0n

Well-Known Member

Reaction score: 115
Messages: 259

The issue is the ffpmpeg -bufsize anything above 500k cause the udp error

I can set the ffmpeg crf value to 18 or to 2000k and as long as the -bufsize is set to 500k then it works

This code streams my desktop over udp with a crf of 18 to a multicast address
which can then played on all devices on the network

The latency is less than half a second

Bash:
ffmpeg -f x11grab -r 15 -s 1366x768 -i :0.0+0,0 \
-c:v libx264 -preset ultrafast -crf 18 \
-tune zerolatency \
-maxrate 4000k -bufsize 500k \
-pix_fmt yuv420p \
-f mpegts 'udp://239.253.253.4:1234?pkt_size=1316'
to play the video use mpv and not vlc because vlc 3 doesnt work with udp streams

Bash:
mpv udp://239.253.253.4:1234
ill make a video about this and how you can either stream and record with ffmpeg
or use obs studio to stream to another machine or multicast to all devices on the network

Another reason this is cool is because you can use obs to multicast to several other machines on the network
which could then relay the streams to both youtube and twitch at the same time
 
Top