Solved Help Finding Cause of FBSD Sending RST Packets

anonymous9 said:
I still see nothing in Flags where I would expect to see an S inside the square brackets. And as mark_j mentioned, you'll see the length is 0. Nothing in the packet?
Click to expand...
Sorry. After my initial post, I realized the S flag was missing again and updated my previous post. Seems the forum software interprets "bracket S bracket" as code for strikethrough text.

I don't think there is supposed to be anything in the packet at this point. This is the initial SYN, SYN/ACK, ACK sequence to establish a session. The client sends the SYN and immediately, server sends a RST instead of a SYN/ACK.

Please let me know if my understanding is incorrect.

Thanks,

Drew
 
610Drew said:
The server is not CPU bound at all. Is there anything else I should check?

View attachment 8709

And regarding the missing "S" flag, the forum software interprets "bracket S bracket" as strikethrough text code. I kept wondering why much of my text was strikethrough after posting and then finally figured it out. :D


Thanks,

Drew
Click to expand...
My misunderstanding, I misread the [R] flag, so win = 0 is fine.

Can you move the port the BSD machine's connected to on the switch? Maybe rule out a faulty port? I take it there's just one switch in your network between the http server and the other applications?

Also, if Apache's not co-operating by logging information maybe it's not actually doing anything? Is it possible to truss the process while running tcpdump and querying Apache?
You will see select() trigger and a bunch of other stuff. At least that will tell you Apache is responding. In fact, you don't even have to do that while tcpdump is running, simply verifying it's processing ANYTHING would be a step in the right direction. Yes?

(Yes, gotta love the forum software... :rolleyes:)
 
anonymous9 said:
I would install thttpd and run it to see if you still get a SYN, RST or SYN, SYN/ACK, ACK which is expected. And does the traceroute make it all the way through?
Click to expand...
I'll try your suggestion of installing another web server.

A traceroute makes it just fine. Only one hop because everything is on the same network.


Code: 
   drew@vm pts/0 16:40:39 Mon Nov 02 ~/ 
 $ traceroute blacklamb
traceroute to blacklamb.mykitchentable.net (192.168.1.4), 64 hops max, 40 byte packets
 1  blacklamb (192.168.1.4)  0.527 ms  0.273 ms  0.372 ms
   drew@vm pts/0 16:40:47 Mon Nov 02 ~/ 
 $

Thanks,

Drew
 
610Drew said:
I can't upgrade any higher than 6.5 because it's on an HP ML310 Gen 8 server and HP is not releasing any ESXi images beyond 6.5.
Click to expand...
6.5 is supported for another year (until 2021-11-15), so no problem there just yet, but you have been installing the updates right? ESXi 6.5 is on update 3 now.
 
jem said:
6.5 is supported for another year (until 2021-11-15), so no problem there just yet, but you have been installing the updates right? ESXi 6.5 is on update 3 now.
Click to expand...
I was on update 2. I've moved to update 3. The issue persists. The RST/ACK appears to be coming directly from the VM. If I run tcpdump on the VM, I see the RST/ACK going out. I found an article on how to run a packet capture within ESXi and see the RST/ACK coming from the VM as it enters the virtual switch.

I am at a loss as to why connection attempts to Apache are reset. Any further ideas appreciated.

Thanks,

Drew
 
I would run the tcpdump with -e for the MAC addresses. I think there's another flag to show the directionality of the packet, to confirm if it's being pushed by the VM itself or not.
 
eldaemon said:
I would run the tcpdump with -e for the MAC addresses. I think there's another flag to show the directionality of the packet, to confirm if it's being pushed by the VM itself or not.
Click to expand...
I tried that and it appears to be coming from the VM.
Code: 
 # ifconfig -a
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
        [HIGHLIGHT]ether 00:0c:29:83:3c:f4[/HIGHLIGHT]
        inet 192.168.1.4 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
Code: 
 # tcpdump -eQ out port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:52:04.207352 [HIGHLIGHT]00:0c:29:83:3c:f4[/HIGHLIGHT] (oui Unknown) > 28:c6:3f:a2:e3:1f (oui Unknown), ethertype IPv4 (0x0800), length 54: blacklamb.mykitchentable.net.http > NotATablet.mykitchentable.net.22789: Flags [R.], seq 0, ack 2606353443, win 0, length 0
Thanks,

Drew
 
mark_j said:
Also, if Apache's not co-operating by logging information maybe it's not actually doing anything? Is it possible to truss the process while running tcpdump and querying Apache?
You will see select() trigger and a bunch of other stuff. At least that will tell you Apache is responding. In fact, you don't even have to do that while tcpdump is running, simply verifying it's processing ANYTHING would be a step in the right direction. Yes?
Click to expand...
I tried your "truss" suggestion. It's my understanding that the process owned by root is the initial process and then it spawns child processes as needed with www as the owner.
Code: 
# ps acux | grep http
root     737   0.0  0.1 17164 5892  -  Ss   10:11     0:00.94 httpd
www      741   0.0  0.1 17120 5780  -  I    10:11     0:00.00 httpd
www      742   0.0  0.1 17120 5780  -  I    10:11     0:00.00 httpd
So I ran truss on pid 737 and got constant output like this:
Code: 
# truss -fp 737
  737: wait4(-1,{ EXITED,val=0 },WNOHANG|WUNTRACED,0x0) = 0 (0x0)
  737: getrusage(RUSAGE_SELF,{ u=0.475702,s=0.479913,in=1,out=0 }) = 0 (0x0)
  737: getrusage(RUSAGE_CHILDREN,{ u=0.000000,s=0.000000,in=0,out=0 }) = 0 (0x0)
  737: select(0,0x0,0x0,0x0,{ 1.000000 })        = 0 (0x0)
  737: wait4(-1,{ EXITED,val=0 },WNOHANG|WUNTRACED,0x0) = 0 (0x0)
  737: getrusage(RUSAGE_SELF,{ u=0.475702,s=0.479967,in=1,out=0 }) = 0 (0x0)
  737: getrusage(RUSAGE_CHILDREN,{ u=0.000000,s=0.000000,in=0,out=0 }) = 0 (0x0)
  737: select(0,0x0,0x0,0x0,{ 1.000000 })        = 0 (0x0)
  737: wait4(-1,{ EXITED,val=0 },WNOHANG|WUNTRACED,0x0) = 0 (0x0)
The sequence repeats over and over after the wait4.

Not really understanding what I was looking at, I ran truss on pids 741 and 742. No output at all, even when attempting to connect to the web page.
Code: 
# truss -fp 741
^C
 # truss -fp 742
^C
Thanks,

Drew
 
eldaemon said:
I wonder if there's some sysctl setting. This is very strange.

Maybe paste /etc/sysctl.conf?
Click to expand...
Nothing in it.
Code: 
# cat /etc/sysctl.conf
# $FreeBSD: releng/12.1/sbin/sysctl/sysctl.conf 337624 2018-08-11 13:28:03Z brd $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#


# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
#
Thanks,

Drew
 
610Drew said:
I tried your "truss" suggestion. It's my understanding that the process owned by root is the initial process and then it spawns child processes as needed with www as the owner.
Code: 
# ps acux | grep http
root     737   0.0  0.1 17164 5892  -  Ss   10:11     0:00.94 httpd
www      741   0.0  0.1 17120 5780  -  I    10:11     0:00.00 httpd
www      742   0.0  0.1 17120 5780  -  I    10:11     0:00.00 httpd
So I ran truss on pid 737 and got constant output like this:
Code: 
# truss -fp 737
  737: wait4(-1,{ EXITED,val=0 },WNOHANG|WUNTRACED,0x0) = 0 (0x0)
  737: getrusage(RUSAGE_SELF,{ u=0.475702,s=0.479913,in=1,out=0 }) = 0 (0x0)
  737: getrusage(RUSAGE_CHILDREN,{ u=0.000000,s=0.000000,in=0,out=0 }) = 0 (0x0)
  737: select(0,0x0,0x0,0x0,{ 1.000000 })        = 0 (0x0)
  737: wait4(-1,{ EXITED,val=0 },WNOHANG|WUNTRACED,0x0) = 0 (0x0)
  737: getrusage(RUSAGE_SELF,{ u=0.475702,s=0.479967,in=1,out=0 }) = 0 (0x0)
  737: getrusage(RUSAGE_CHILDREN,{ u=0.000000,s=0.000000,in=0,out=0 }) = 0 (0x0)
  737: select(0,0x0,0x0,0x0,{ 1.000000 })        = 0 (0x0)
  737: wait4(-1,{ EXITED,val=0 },WNOHANG|WUNTRACED,0x0) = 0 (0x0)
The sequence repeats over and over after the wait4.

Not really understanding what I was looking at, I ran truss on pids 741 and 742. No output at all, even when attempting to connect to the web page.
Code: 
# truss -fp 741
^C
# truss -fp 742
^C
Thanks,

Drew
Click to expand...
This sequence means NOTHING is being received by Apache. If you tried it while connecting to the web server, then the packets are not getting through. If you didn't then you need to browse a page on the server while using truss.
 
mark_j said:
This sequence means NOTHING is being received by Apache. If you tried it while connecting to the web server, then the packets are not getting through. If you didn't then you need to browse a page on the server while using truss.
Click to expand...
I did try browsing. Nothing. So how can I figure out what is sending the RST and not allowing the traffic to Apache. I reviewed my process list (below) and see 3 processes (highlighted) that might mean I"m running pf? How can I confirm one way or the other? Is there anything else running that might be blocking packets?

Code: 
 # ps aux
USER     PID  %CPU %MEM   VSZ   RSS TT  STAT STARTED       TIME COMMAND
root      11 200.0  0.0     0    32  -  RNL  Wed10   3499:03.17 [idle]
root       0   0.0  0.0     0   288  -  DLs  Wed10      3:33.99 [kernel]
root       1   0.0  0.0  9964   584  -  ILs  Wed10      0:00.25 /sbin/init --
root       2   0.0  0.0     0    16  -  DL   Wed10      0:00.00 [crypto]
root       3   0.0  0.0     0    16  -  DL   Wed10      0:00.00 [crypto returns 0]
root       4   0.0  0.0     0    16  -  DL   Wed10      0:00.00 [crypto returns 1]
root       5   0.0  0.0     0    32  -  DL   Wed10      0:10.97 [cam]
root       6   0.0  0.0     0    16  -  DL   Wed10      0:00.00 [mpt_recovery0]
root       7   0.0  0.0     0    16  -  DL   Wed10      0:00.00 [fdc0]
root       8   0.0  0.0     0    16  -  DL   Wed10      0:00.00 [sctp_iterator]
root       9   0.0  0.0     0    16  -  DL   Wed10      0:22.70 [rand_harvestq]
root      10   0.0  0.0     0    16  -  DL   Wed10      0:00.00 [audit]
root      12   0.0  0.0     0   720  -  WL   Wed10      1:01.23 [intr]
root      13   0.0  0.0     0    48  -  DL   Wed10      0:00.05 [geom]
root      14   0.0  0.0     0    16  -  DL   Wed10      0:00.00 [sequencer 00]
root      15   0.0  0.0     0    16  -  DL   Wed10      0:00.05 [soaiod1]
root      16   0.0  0.0     0    16  -  DL   Wed10      0:00.05 [soaiod2]
root      17   0.0  0.0     0    16  -  DL   Wed10      0:00.04 [soaiod3]
root      18   0.0  0.0     0    16  -  DL   Wed10      0:00.03 [soaiod4]
root      19   0.0  0.0     0    48  -  DL   Wed10      0:41.56 [pagedaemon]
root      20   0.0  0.0     0    16  -  DL   Wed10      0:00.00 [vmdaemon]
root      21   0.0  0.0     0    64  -  DL   Wed10      0:18.59 [bufdaemon]
root      22   0.0  0.0     0    16  -  DL   Wed10      2:59.10 [syncer]
root      23   0.0  0.0     0    16  -  DL   Wed10      0:01.47 [vnlru]
root     100   0.0  0.0 10828  1576  -  Is   Wed10      0:00.03 adjkerntz -i
root     353   0.0  0.0 10456  1028  -  Ss   Wed10      0:14.98 /sbin/devd
[HIGHLIGHT]root     359   0.0  0.0     0    16  -  DL   Wed10      0:26.89 [pf purge]
root     371   0.0  0.1 12428  2180  -  Is   Wed10      0:00.07 pflogd: [priv] (pflogd)
_pflogd  376   0.0  0.1 12496  2200  -  S    Wed10      0:14.43 pflogd: [running] -s 116 -i pflog0 -f /var/log/pflog (pflogd)[/HIGHLIGHT]
root     507   0.0  0.0 11328  1928  -  Ss   Wed10      0:10.78 /usr/sbin/syslogd -s
root     571   0.0  0.1 16624  3080  -  Is   Wed10      0:00.04 /usr/local/sbin/saslauthd -a pam
root     581   0.0  0.1 16624  3072  -  I    Wed10      0:00.00 /usr/local/sbin/saslauthd -a pam
root     582   0.0  0.1 16624  3072  -  I    Wed10      0:00.00 /usr/local/sbin/saslauthd -a pam
root     583   0.0  0.1 16624  3072  -  I    Wed10      0:00.00 /usr/local/sbin/saslauthd -a pam
root     584   0.0  0.1 16624  3072  -  I    Wed10      0:00.00 /usr/local/sbin/saslauthd -a pam
dhcpd    633   0.0  0.2 23280  8568  -  Is   Wed10      0:08.99 /usr/local/sbin/dhcpd -q -cf /usr/local/etc/dhcpd.conf -lf /var/db/
root     705   0.0  0.2 51060  6424  -  Is   Wed10      0:07.54 /usr/local/libexec/postfix/master -w
postfix  707   0.0  0.2 51132  6516  -  I    Wed10      0:08.31 qmgr -l -t unix -u
root     728   0.0  0.1 19660  4060  -  Is   Wed10      0:00.86 /usr/sbin/sshd
root     737   0.0  0.1 17164  4156  -  Ss   Wed10      0:06.50 /usr/local/sbin/httpd
www      741   0.0  0.1 17120  4044  -  I    Wed10      0:00.00 /usr/local/sbin/httpd
www      742   0.0  0.1 17120  4044  -  I    Wed10      0:00.00 /usr/local/sbin/httpd
root     752   0.0  0.0 11460  1952  -  Is   Wed10      0:30.25 /usr/sbin/cron -s
postfix 1585   0.0  0.2 51304  6752  -  I    17:55      0:00.68 tlsmgr -l -t unix -u
postfix 4826   0.0  0.9 51084 39452  -  I    14:33      0:03.39 pickup -l -t unix -u
root    4941   0.0  0.2 20372  8364  -  Is   15:44      0:05.25 sshd: drew [priv] (sshd)
root    4942   0.0  0.2 20372  8484  -  Is   15:44      0:02.16 sshd: drew [priv] (sshd)
drew    4945   0.0  0.2 20376  8356  -  I    15:44      0:00.25 sshd: drew@pts/1 (sshd)
drew    4946   0.0  0.2 20376  8492  -  S    15:44      0:00.16 sshd: drew@pts/0 (sshd)
root     802   0.0  0.0 10880  1584 v0  Is+  Wed10      0:00.06 /usr/libexec/getty Pc ttyv0
root     803   0.0  0.0 10880  1584 v1  Is+  Wed10      0:00.00 /usr/libexec/getty Pc ttyv1
root     804   0.0  0.0 10880  1584 v2  Is+  Wed10      0:00.00 /usr/libexec/getty Pc ttyv2
root     805   0.0  0.0 10880  1584 v3  Is+  Wed10      0:00.03 /usr/libexec/getty Pc ttyv3
root     806   0.0  0.0 10880  1584 v4  Is+  Wed10      0:00.00 /usr/libexec/getty Pc ttyv4
root     807   0.0  0.0 10880  1584 v5  Is+  Wed10      0:00.00 /usr/libexec/getty Pc ttyv5
root     808   0.0  0.0 10880  1584 v6  Is+  Wed10      0:00.00 /usr/libexec/getty Pc ttyv6
root     809   0.0  0.0 10880  1584 v7  Is+  Wed10      0:00.00 /usr/libexec/getty Pc ttyv7
drew    4948   0.0  0.1 13164  3744  0  Is   15:44      0:01.46 -bash (bash)
root    4959   0.0  0.1 12108  2808  0  I    15:44      0:00.41 su -m
root    4960   0.0  0.1 13220  3796  0  S    15:44      0:00.99 su (bash)
root    4975   0.0  0.1 11644  2460  0  R+   15:46      0:00.00 ps aux
drew    4947   0.0  0.1 13164  3708  1  Is+  15:44      0:01.35 -bash (bash)
Thanks,

Drew
 
Apparently, I'm retarded. My post above got me to thinking so I looked in /etc/rc.conf and found pf was enabled. I'm not sure how or when that happened but disabling it and stopping the running pf processes has solved this issue.

Thank you for all of the help!!!

Cheers,

Drew
 
You must log in or register to reply here.
Back
Top