NFS with OpenBSD client

freezmi

freezmi

I have a FreeBSD 14.3-p8 NFS v3 server that works ok with linux and windows clients.

However I tried to copy any >1kB file from an OpenBSD client and the obsd kernel refuses to ever interact with the nfs mountpoint from that point on - everything that tries to access it stalls, including the shutdown process.

the target file gets created but it's empty. no error logs are generated anywhere.

read operations work fine no matter how large the file is.
write operations with a simple `echo "foobar" > ` work fine.

/etc/exports simply reads
/mnt/space -mapall=ftp -network 10.212.0.0/16
Click to expand...

both devices are on the same subnet, server has a nice Intel I340 nic with 1500 MTU, laptop has both wifi and copper - both break the same way. all filesystems on the server are ufs.

wireshark shows RPC retransmissions of the same packets until all quiets down.

should I configure the NFS server in a special way to enhance OpenBSD compatibility?
 
MG said:
Why >1K files? And you're copying to a client?
I think you need to be more specific about the situation.
Click to expand...

the NFS client is an OpenBSD 7.8 laptop, the NFS server is FreeBSD 14.3.
all operations are initiated on the NFS client, that's how this protocol works.

read anything from the NFS server works.
write one-liner files to the NFS server works.
write even small (1kB) files stalls.

later edit:
if I completely disable the pf firewall on the server the problem goes away.
which is weird given the fact that Linux and windows nfs clients do not get filtered out in any way.
 
freezmi said:
the NFS client is an OpenBSD 7.8 laptop, the NFS server is FreeBSD 14.3.
all operations are initiated on the NFS client, that's how this protocol works.

read anything from the NFS server works.
write one-liner files to the NFS server works.
write even small (1kB) files stalls.

later edit:
if I completely disable the pf firewall on the server the problem goes away.
which is weird given the fact that Linux and windows nfs clients do not get filtered in any way.

later-later edit:
in wireshark I see a lot of 'Fragmented IP protocol' UDP packets with no source or destination ports during a copy operation. how do I let these thru in the server's pf.conf ?
Click to expand...
If you create a file in the mounted nfs directory in OpenBSD, it can only be very small, otherwise the nfs-client (and its mount point) gets unrespopnsive?
I assume it's not the server. If any remote process can make a server hang we have a security exploit...
 
MG said:
If you create a file in the mounted nfs directory in OpenBSD, it can only be very small, otherwise the nfs-client (and its mount point) gets unrespopnsive?
I assume it's not the server. If any remote process can make a server hang we have a security exploit...
Click to expand...
correct, the client hangs until I do a 'pfctl -d' on the server.

and it's definitely a pf related problem on the server side, here is what gets dropped:

# tcpdump -nn -i pflog0

16:59:50.074528 IP 10.212.0.191.884 > 10.212.0.3.2049: NFS request xid 906834233 1464 write fh 1647,605008/44951813 8192 (8192) bytes @ 16384
16:59:50.074538 IP 10.212.0.191 > 10.212.0.3: ip-proto-17
16:59:50.074562 IP 10.212.0.191 > 10.212.0.3: ip-proto-17
16:59:50.074567 IP 10.212.0.191 > 10.212.0.3: ip-proto-17
16:59:50.074809 IP 10.212.0.191 > 10.212.0.3: ip-proto-17
16:59:50.074813 IP 10.212.0.191 > 10.212.0.3: ip-proto-17
16:59:59.288517 IP 10.212.0.191.884 > 10.212.0.3.2049: NFS request xid 2101143500 1464 write fh 1647,605008/44951813 8192 (8192) bytes @ 0
16:59:59.288529 IP 10.212.0.191 > 10.212.0.3: ip-proto-17
16:59:59.288538 IP 10.212.0.191 > 10.212.0.3: ip-proto-17
16:59:59.288543 IP 10.212.0.191 > 10.212.0.3: ip-proto-17
16:59:59.288582 IP 10.212.0.191 > 10.212.0.3: ip-proto-17
16:59:59.288586 IP 10.212.0.191 > 10.212.0.3: ip-proto-17
Click to expand...

10.212.0.3 is the server and I do allow packets toward port 2049:
# nfs
pass in quick on $ext_if proto tcp to port { 111 755 790 2049 16001 40755 } keep state
pass in quick on $ext_if proto udp to port { 111 755 790 2049 16001 40755 } keep state
[..]
block in log on $ext_if all
Click to expand...

it looks like _sometimes_ packets toward 10.212.0.3:2049 do not get the 'pass in' treatment from the rule above.
how can that rule work for most packets toward the nfsd port and sometimes not get matched?

looking closer to the pflog file in wireshark for all these packets
Action: block (1)
Reason: match (0)
Rule number: 50
Click to expand...
where rule 50 is the 'block in log' shown above.

the plot thickens since this looks less and less as a PEBKAC.

later edit:
in case someone is wondering, those ports are where rpc, statd, lockd, mountd are forcefully bound.
 
freezmi said:
correct.

and it's definitely a pf related problem on the server side, here is what gets dropped:



10.212.0.3 is the server and I do allow packets toward port 2049:


it looks like _sometimes_ packets toward 10.212.0.3:2049 do not get the 'pass in' treatment from the rule above.

looking closer to the pflog file in wireshark for all these packets

where rule 50 is the 'block in log' shown above.

the plot thickens since this looks less and less as a PEBKAC.

later edit:
in case someone is wondering, those ports are where rpc, statd, lockd, mountd are forcefully bound.
Click to expand...
If packet filter is the supposed problem, shouldn't you disable it to verify that? In case it's sure, there must be something wrong with data sent to the nfs-server. I must say it sounds kind of interesting that something blocked by pf still can generate an empty file... Because it works with other systems it can only be a small detail. That's probably a deep ananysis but it must be possible to find a the protocol difference if you compare with a linux or FreeBSD nfs client using netcat or something to read bytes and see what the communication looks like with the same action but different clients.
 
MG said:
In case it's sure, there must be something wrong with data sent to the nfs-server. I must say it sounds kind of interesting that something blocked by pf still can generate an empty file... Because it works with other systems it can only be a small detail. That's probably a deep ananysis but it must be possible to find a the protocol difference if you compare with a linux or FreeBSD nfs client using netcat or something to read bytes.
Click to expand...
the problem is 100% due to the server-side pf.

the 'allow' rule sometimes does not get matched and the catch-all 'block' ends up dropping the packet at rule 50. I will do a deeper analysis tomorrow - I have captured the failure from both the client and server side.

problem is that these packets are re-assembled locally from 5 1506byte fragments (I do not have jumbo frames enabled on my LAN).
 
freezmi said:
the problem is 100% due to the server-side pf.

the allow rule sometimes does not get matched and the catch-all block ends up dropping the packet at rule 50. I will do a deeper analysis tomorrow - I have captured the failure from both the client and server side.

problem is that these packets are re-assembled locally from 5 1506byte fragments (I do not have jumbo frames enabled on my LAN).
Click to expand...
Then how can it work with linux? What is OpenBSD expecting and how is it different?
 
AlfredoLlaquet said:
You do well ignoring me, but according to my investigations, forcing TCP would avoid UDP fragmentation, and this would help. Please, keep ignoring me if I'm talking nonsense.
Click to expand...
thanks, but I'm not interested in what the chatbot-du-jour AI POS thinks. I'm not looking for a bandaid to the symptom but for finding the root problem. because this problem can be a bug in one of these subsystems and having it fixed is much more important in the grand scheme of things.
 
You must log in or register to reply here.
Back
Top