Solved Help Finding Cause of FBSD Sending RST Packets

610Drew

610Drew

I have a FreeBSD 10.3 box running as a VM on VMWare ESXi. It's just a personal server that hasn't been touched in several years. It runs Apache 2.4, Postfix 3.3.1 and sshd. For whatever reason, attempts to connect to Apache started failing. The cause is that FreeBSD is sending a RST packet back after a client sends the inital SYN packet. Apache is running and there are no errors in the Apache logs. It seems like the RST is being produced by FreeBSD OS. I do not have any firewall software running on this FreeBSD box and the client is on the same network. No firewall is between them. Here's the network diagram:
temp.jpg

I can see the server is listening:
Code: 
 # sockstat -4l             
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS   
www      httpd      82132 3  tcp4   *:80                  *:*
www      httpd      82131 3  tcp4   *:80                  *:*
root     httpd      82130 3  tcp4   *:80                  *:*

And here is the tcpdump output showing the RST being sent immediately:
Code: 
 # tcpdump -v host 192.168.1.14
tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:18:44.022326 IP (tos 0x0, ttl 128, id 11601, offset 0, flags [DF], proto TCP (6), length 52)
    tagalong-ii.mykitchentable.net.25684 > blacklamb.mykitchentable.net.http: Flags , cksum 0x8026 (correct), seq 3837723177, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
16:18:44.022455 IP (tos 0x0, ttl 64, id 50584, offset 0, flags [DF], proto TCP (6), length 40)
    blacklamb.mykitchentable.net.http > tagalong-ii.mykitchentable.net.25684: Flags [R.], cksum 0x837d (incorrect -> 0xe0df), seq 0, ack 3837723178, win 0, length 0
I am at a loss as to what is causing this behavior. Any suggestions of things to check would be greatly appreciated.

Cheers,

Drew
 
Last edited:
610Drew said:
I have a FreeBSD 10.3 box running as a VM on VMWare ESXi.
Click to expand...
FreeBSD 10.3 has been end-of-life since April 2018 and is not supported any more. Upgrade to a supported version.

www.freebsd.org

Unsupported FreeBSD Releases

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.
www.freebsd.org www.freebsd.org
forums.freebsd.org

Topics about unsupported FreeBSD versions

The FreeBSD Forums cater primarily to end-users and systems administrators. As such, the Forums focus almost exclusively on FreeBSD versions that are officially supported according to the official FreeBSD website. Since resources are scarce, the FreeBSD Forums strongly suggest that anyone asking...
forums.freebsd.org forums.freebsd.org
 
As Elazar wrote, I strongly recommend to update to a supported version of FreeBSD.
10.3 was released in April 2016 and went end-of-life in April 2018. In other words, it hasn’t received updates (including security fixes) for at least 2.5 years.
 
olli@ said:
As Elazar wrote, I strongly recommend to update to a supported version of FreeBSD.
10.3 was released in April 2016 and went end-of-life in April 2018. In other words, it hasn’t received updates (including security fixes) for at least 2.5 years.
Click to expand...
I agree and plan to do so. But what made a working version stop?
 
Could be all kind of things. Like a TCP/IP bug that was triggered after some time for some reason. Or someone hacked into the machine and caused damage. I mean, it didn’t get security updates for several years, it doesn’t have a firwall or packet filter, and apparently it’s running a public web server, so …
 
olli@ said:
... it doesn’t have a firewall or packet filter, and apparently it’s running a public web server, so …
Click to expand...
There is a firewall between it and the Internet. I was just trying to explain that the issue wasn't due to a firewall blocking the traffic because I was testing from my internal network.
 
eldaemon said:
What happens if you do something like curl localhost?
Click to expand...
Good idea. That works! And so does "curl 192.168.1.4" and even "curl drew.mykitchentable.net" on the web server. So what might be causing it to send RST packets to other machines? Puzzling...

Thanks,

Drew
 
It's likely VMWare that's causing it. I've seen that happen before, with various different guest OS, not just with FreeBSD. Shutdown the guest and restart it. At some point VMWare just disconnects the guest and you'll get weird connection errors.
 
SirDice said:
It's likely VMWare that's causing it. I've seen that happen before, with various different guest OS, not just with FreeBSD. Shutdown the guest and restart it. At some point VMWare just disconnects the guest and you'll get weird connection errors.
Click to expand...
Thank you for your suggestion. However, after a complete shutdown, power off, and power on, the problem persists.

Cheers,

Drew
 
anonymous9 said:
It isn't going through the switch and it is also not going in/out two network adapters. Since it is a VMWare guest it is also processed on the host. Would need to see the routing table, but 192.168.1.0/24 is probably using 0.0.0.0 as the gateway, so everything will be routed locally and not leave the external adapter.
Click to expand...
I'm not quite sure I understand. But here is the route table on the FBSD VM. Is that what you wanted to see? Or the route table on the ESXi server?


Code: 
 # netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            192.168.1.2        UGS         em0
127.0.0.1          link#2             UH          lo0
192.168.1.0/24     link#1             U           em0
192.168.1.4        link#1             UHS         lo0

Also, I experience the same RST behavior if I test using another guest on the ESXi host. I don't know if that's an important clue or not.

Thanks,

Drew
 
What is the ESXi version? If it is below 6.5 I do bet on VMware causing those issues ... I have had quite a lot of different network problems on ESXi before 6.5
 
rootbert said:
What is the ESXi version? If it is below 6.5 I do bet on VMware causing those issues ... I have had quite a lot of different network problems on ESXi before 6.5
Click to expand...
It is 6.5.0 Update 2 (Build 9298722)

Because it is an HP server, it's using this custom build by HP:

HPE-ESXi-6.5.0-Update2-iso-preGen9-650.U2.9.6.8.3 (Hewlett Packard Enterprise)

Thanks,

Drew
 
If you can ping between machines don't even bother looking at routing table stuff. I doubt this is an OS issue, but an application issue. At work we get a fair bit of "application issue must be the firewall" tickets. Looking at our firewall logs we see all sort of funky TCP state flag stuff going on, with the application. We tell the application owner to run wireshark and try to determine why their application is behaving that way.
That just said, upgrade your machine to a supported version with the latest security update patches installed.
 
I'm working on upgrading. Had some trouble going from 10.3 to 11.4 so now I'm attempting to go to 10.4 first. Then I'll go to 11.4 and finally to 12.1. Will be interesting to see if this issue goes away. I have my doubts simply because everything has been working fine on 10.3 and I didn't make any changes but we'll see. I'm doing source upgrades so this will take a while but I'll report back when completed.

Thanks,

Drew
 
Updated to 12.1-p10 but my symptom persists. Something seems to be preventing the packets being sent to Apache because I see nothing in either my httpd access or error logs.

The packets do get to my FBSD box and it sends reset packets when I attempt to connect. See the tcpdump output in my first post at the top of this thread.

Thanks,

Drew
 
I am still betting on the VMWare stuff ... I really don't remember well, maybe even VMWare 6.5 caused us lots of network troubles. Can you try to change the network driver of the FreeBSD VM? Or can you upgrade to ESXi 7?
 
The initial tcpdump (very brief as it is) shows your web server replying with a window size of 0. What gives there?
When you run tcpdump are you also looking at the OS and a potential CPU bottleneck?
As anonymous9 says, what gives with a blank flag? Is that a tcpdump peculiarity?
 
rootbert said:
I am still betting on the VMWare stuff ... I really don't remember well, maybe even VMWare 6.5 caused us lots of network troubles. Can you try to change the network driver of the FreeBSD VM? Or can you upgrade to ESXi 7?
Click to expand...
I can't upgrade any higher than 6.5 because it's on an HP ML310 Gen 8 server and HP is not releasing any ESXi images beyond 6.5.

Other network connectivity to the FBSD VM passes just fine including ssh, smtp, dhcp, dns, etc. I'm only having trouble with Apache traffic as far as I know. And it used to work but quit at some point in August. I I do not recall making any changes. Heck, I hardly touch the box (as evidenced by still running 10.3 when I opened the thread :))

I also have two Windows VMs on the same ESXi server. I do not have any network connectivity issues with either of them.

I poked around in the VSwitch config but didn't see anything relevant. Do you have any ideas of what else I can check on the VMWare?

temp.jpg


Thanks,

Drew
 
anonymous9 said:
Could be a bad switch. Could be a bad NIC on VMWare since it has to process the packets. Is the client in your diagram also on VMWare?

Why in your tcpdump is there no flag on the client?
Code: 
Flags , cksum 0x8026 (correct), seq 3837723177

I would expect
Code: 
Flags \[S\],
Click to expand...
My diagram was just an attempt to keep it simple. I actually have multiple clients on my network but it's just one network. I don't have any additional internal networks. Some clients are VMs and others are physical machines. In my initial post, that output was a test from a windows VM while my FBSD box was still on 10.3.

After seeing your question about missing flags, I tested again now that my FBSD box is on 12.1. I tested from both a physical client and a VM client with the same results. I learned that this forum software interprets \[S\] as strikethough text. I added it to the output below as "\[S\]" and suspect it was really there in the initial post.

Physical Client
Code: 
   root@blacklamb pts/0 11:42:53 Sat Oct 31 ~/
# tcpdump port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:43:31.718035 IP NotATablet.mykitchentable.net.12824 > blacklamb.mykitchentable.net.http: Flags \[S\], seq 2946492531, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:43:31.718174 IP blacklamb.mykitchentable.net.http > NotATablet.mykitchentable.net.12824: Flags [R.], seq 0, ack 2946492532, win 0, length 0
11:43:31.967212 IP NotATablet.mykitchentable.net.12825 > blacklamb.mykitchentable.net.http: Flags \[S\], seq 3323795826, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

VM Client
Code: 
   root@blacklamb pts/0 12:38:40 Mon Nov 02 ~/
# tcpdump port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:39:05.675833 IP tagalong-ii.mykitchentable.net.1569 > blacklamb.mykitchentable.net.http: Flags \[S\], seq 3072877358, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
12:39:05.683486 IP blacklamb.mykitchentable.net.http > tagalong-ii.mykitchentable.net.1569: Flags [R.], seq 0, ack 3072877359, win 0, length 0
12:39:05.905447 IP tagalong-ii.mykitchentable.net.1570 > blacklamb.mykitchentable.net.http: Flags \[S\], seq 2973882592, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
12:39:05.908919 IP blacklamb.mykitchentable.net.http > tagalong-ii.mykitchentable.net.1570: Flags [R.], seq 0, ack 2973882593, win 0, length 0

Thanks,

Drew
 
mark_j said:
The initial tcpdump (very brief as it is) shows your web server replying with a window size of 0. What gives there?
When you run tcpdump are you also looking at the OS and a potential CPU bottleneck?
As anonymous9 says, what gives with a blank flag? Is that a tcpdump peculiarity?
Click to expand...
The server is not CPU bound at all. Is there anything else I should check?

temp.jpg


And regarding the missing "S" flag, the forum software interprets "bracket S bracket" as strikethrough text code. I kept wondering why much of my text was strikethrough after posting and then finally figured it out. :D


Thanks,

Drew
 
You must log in or register to reply here.
Back
Top