Can connect in but not out from server

X

xy16644

I have just moved my server over to a new machine. I did this as a test a few days ago and it worked great but now that I have done it for real I am experiencing the most peculiar issue.

I can connect to my server via the usual network services (SSH, IMAP etc) but while logged into the servers console I can't ping anything. Not the default gateway or anything on the internet. I know the network card name changed between the two machines (from bge0 to em0) and I have made the necessary changes in my /etc/rc.conf and /etc/pf.conf files (like I did during the test).

Can someone help me troubleshoot this please. I have exhausted everything I can think of and have looked through the logs but I am obviously overlooking something.

Any ideas? :e

My /etc/rc.conf:
Code: 
ifconfig_em0="inet 192.168.0.200 netmask 255.255.255.0"

My /etc/pf.conf:
Code: 
tcp_services = "{ 22, 25, 80, 161, 162, 443, 587, 993, 3551 }"
udp_services = "{ 161, 162 }"
#table <bruteforce> persist
#block quick from <bruteforce>
#pass inet proto tcp fom any to bge0 port 22 keep state (maxsrc-conn 100, max-src-conn-rate 15/5, \
#overload <bruteforce> flush global)

set skip on lo0
set loginterface em0

pass out quick on em0

block in all
pass in log on em0 proto tcp from any to em0 port $tcp_services
pass in proto icmp from 192.168.0.0/24 to em0
pass in log on em0 proto udp from any to em0 port 6277
pass in log on em0 proto udp from any to em0 port 3551

pass in log on em0 proto udp from 192.168.0.1 to em0 port $udp_services

The issues I am experiencing from this include extremely slow SSH logons and NTP is moaning as it can't connect to the internet and obviously emails cant be delivered.

Very confused!!
 
Try changing the following:

Code: 
pass out [B]log[/B] on em0

And then use tcpdump to do a capture:

[CMD=""]# tcpdump -n -e -ttt -i pflog0[/CMD]

That should give you some hints.
 
Thanks, I'll give that a try! I should also mention that I tried stopping the pf service and that didn't make a difference...weird!
 
gkontos said:
Try changing the following:

Code: 
pass out [B]log[/B] on em0

And then use tcpdump to do a capture:

[CMD=""]# tcpdump -n -e -ttt -i pflog0[/CMD]

That should give you some hints.
Click to expand...

When trying this I get:
Code: 
    192.168.0.10.50860 > 192.168.0.200.443: Flags [S], cksum 0xb709 (correct), seq 1520038632, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
00:00:11.093013 rule 8..16777216/0(match): pass in on em0: (tos 0x0, ttl 128, id 29366, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.10.50861 > 192.168.0.200.443: Flags [S], cksum 0xc4c8 (correct), seq 119814302, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
00:00:01.408055 rule 8..16777216/0(match): pass in on em0: (tos 0x0, ttl 128, id 29375, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.10.50862 > 192.168.0.200.443: Flags [S], cksum 0xe81d (correct), seq 272305201, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
00:00:04.467144 rule 8..16777216/0(match): pass in on em0: (tos 0x0, ttl 128, id 29389, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.10.50863 > 192.168.0.200.443: Flags [S], cksum 0xcb70 (correct), seq 4020914541, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
00:00:33.720556 rule 5..16777216/0(match): pass in on em0: (tos 0x0, ttl 128, id 29613, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.10.50867 > 192.168.0.200.80: Flags [S], cksum 0x69de (correct), seq 1614494678, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
00:00:00.000030 rule 5..16777216/0(match): pass in on em0: (tos 0x0, ttl 128, id 29614, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.10.50866 > 192.168.0.200.80: Flags [S], cksum 0xa06f (correct), seq 1542261140, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
00:11:57.853249 rule 8..16777216/0(match): pass in on em0: (tos 0x0, ttl 128, id 32351, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.10.50950 > 192.168.0.200.443: Flags [S], cksum 0x831d (correct), seq 2749226294, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
00:00:01.316117 rule 8..16777216/0(match): pass in on em0: (tos 0x0, ttl 128, id 32365, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.10.50951 > 192.168.0.200.443: Flags [S], cksum 0x4386 (correct), seq 814845977, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
00:00:12.201579 rule 8..16777216/0(match): pass in on em0: (tos 0x0, ttl 128, id 32375, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.10.50952 > 192.168.0.200.443: Flags [S], cksum 0x7083 (correct), seq 1463303284, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
 
kpa said:
Can you post full outputs of # ifconfig and # netstat -nr.
Click to expand...
Code: 
alpha# netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.0.1        UGS         2      777    em0
127.0.0.1          link#10            UH          0     8983    lo0
192.168.0.0/24     link#1             U           3     9817    em0
192.168.0.200      link#1             UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UH          lo0
fe80::%em0/64                     link#1                        U           em0
fe80::221:5aff:fe6b:3d79%em0      link#1                        UHS         lo0
fe80::%lo0/64                     link#10                       U           lo0
fe80::1%lo0                       link#10                       UHS         lo0
ff01::%em0/32                     fe80::221:5aff:fe6b:3d79%em0  U           em0
ff01::%lo0/32                     ::1                           U           lo0
ff02::%em0/32                     fe80::221:5aff:fe6b:3d79%em0  U           em0
ff02::%lo0/32                     ::1                           U           lo0

alpha# ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
        ether 00:21:5a:6b:3d:79
        inet 192.168.0.200 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 fe80::221:5aff:fe6b:3d79%em0 prefixlen 64 scopeid 0x1
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33200
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

I can't ping 192.168.0.1 from the server but I can ping the servers IP (192.168.0.200) from my laptop.

In /var/run/dmesg.boot:
Code: 
em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x1100-0x111f mem 0xf0180000-0xf019ffff,0xf01a4000-0xf01a4fff i
em0: Using an MSI interrupt
em0: Ethernet address: 00:21:5a:6b:3d:79
 
Try this when you ping your gateway:

[CMD=""]# tcpdump -n -e -ttt -i pflog0 host 192.168.0.200[/CMD]

Assuming 192.168.0.1 is your gateway:

[CMD=""]# tcpdump -n -e -ttt -i pflog0 host 192.168.0.1[/CMD]

Those 2 captures should show if the packets are leaving.
 
gkontos said:
Try this when you ping your gateway:

[CMD=""]# tcpdump -n -e -ttt -i pflog0 host 192.168.0.200[/CMD]

Assuming 192.168.0.1 is your gateway:

[CMD=""]# tcpdump -n -e -ttt -i pflog0 host 192.168.0.1[/CMD]

Those 2 captures should show if the packets are leaving.
Click to expand...

Thanks for the help. I am struggling here :\

I had 3 SSH windows open, one for each of the commands above and a third to run the ping test. It did nothing?

Here is the output:
Code: 
alpha# tcpdump -v -n -e -ttt -i pflog0 host 192.168.0.200
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 65535 bytes

alpha# tcpdump -v -n -e -ttt -i pflog0 host 192.168.0.1
tcpdump: WARNING: pflog0: no IPv4 address assigned
tcpdump: listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 65535 bytes

I have to add the -v option to the command for it to run. I am completely baffled by this!! I can SSH to the server and view Webmin on the server but most other things have no connectivity like DNS, NTP, all email etc. I can't ping anything from the server.

Any other ideas before I pull my hair out? :stud I can't think of anymore log files to check!
 
kpa said:
Do you have pf(4) actually enabled? What do # service pf status and # pfctl -sr report?
Click to expand...

Yes, been using pf for ages. Here is the output:
Code: 
alpha# service pf status
Status: Enabled for 0 days 00:12:10           Debug: Urgent

Interface Stats for em0               IPv4             IPv6
  Bytes In                          215163                0
  Bytes Out                        1090025                0
  Packets In
    Passed                            1356                0
    Blocked                             46                0
  Packets Out
    Passed                            1572                0
    Blocked                              0                0

State Table                          Total             Rate
  current entries                       61
  searches                            2974            4.1/s
  inserts                               94            0.1/s
  removals                              40            0.1/s
Counters
  match                                156            0.2/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s

and:

Code: 
alpha# pfctl -sr
pass out log on em0 all flags S/SA keep state
pass out quick on em0 all flags S/SA keep state
block drop in all
pass in log on em0 inet proto tcp from any to 192.168.0.200 port = ssh flags S/SA keep state
pass in log on em0 inet proto tcp from any to 192.168.0.200 port = smtp flags S/SA keep state
pass in log on em0 inet proto tcp from any to 192.168.0.200 port = http flags S/SA keep state
pass in log on em0 inet proto tcp from any to 192.168.0.200 port = snmp flags S/SA keep state
pass in log on em0 inet proto tcp from any to 192.168.0.200 port = snmptrap flags S/SA keep state
pass in log on em0 inet proto tcp from any to 192.168.0.200 port = https flags S/SA keep state
pass in log on em0 inet proto tcp from any to 192.168.0.200 port = submission flags S/SA keep state
pass in log on em0 inet proto tcp from any to 192.168.0.200 port = imaps flags S/SA keep state
pass in log on em0 inet proto tcp from any to 192.168.0.200 port = 3551 flags S/SA keep state
pass in log on em0 inet6 proto tcp from any to fe80::221:5aff:fe6b:3d79 port = ssh flags S/SA keep state
pass in log on em0 inet6 proto tcp from any to fe80::221:5aff:fe6b:3d79 port = smtp flags S/SA keep state
pass in log on em0 inet6 proto tcp from any to fe80::221:5aff:fe6b:3d79 port = http flags S/SA keep state
pass in log on em0 inet6 proto tcp from any to fe80::221:5aff:fe6b:3d79 port = snmp flags S/SA keep state
pass in log on em0 inet6 proto tcp from any to fe80::221:5aff:fe6b:3d79 port = snmptrap flags S/SA keep state
pass in log on em0 inet6 proto tcp from any to fe80::221:5aff:fe6b:3d79 port = https flags S/SA keep state
pass in log on em0 inet6 proto tcp from any to fe80::221:5aff:fe6b:3d79 port = submission flags S/SA keep state
pass in log on em0 inet6 proto tcp from any to fe80::221:5aff:fe6b:3d79 port = imaps flags S/SA keep state
pass in log on em0 inet6 proto tcp from any to fe80::221:5aff:fe6b:3d79 port = 3551 flags S/SA keep state
pass in inet proto icmp from 192.168.0.0/24 to 192.168.0.200 keep state
pass in log on em0 inet proto udp from 192.168.0.1 to 192.168.0.200 port = snmp keep state
pass in log on em0 inet proto udp from 192.168.0.1 to 192.168.0.200 port = snmptrap keep state
pass in log on em0 inet proto udp from any to 192.168.0.200 port = 6277 keep state
pass in log on em0 inet proto udp from any to 192.168.0.200 port = 3551 keep state
pass in log on em0 inet6 proto udp from any to fe80::221:5aff:fe6b:3d79 port = 6277 keep state
pass in log on em0 inet6 proto udp from any to fe80::221:5aff:fe6b:3d79 port = 3551 keep state
 
I don't see anything really wrong in your rules. I would rearrange them so that there are default deny rules first. Also I would add quick to all pass rules, now the evaluation of packets continues to the last pass rule on every packet.

Code: 
block in all
block out all

pass out quick on em0

pass in quick in em0 proto icmp from em0:network to em0
pass in log quick on  em0 proto tcp from any to em0 port $tcp_services
...
 
Could it be an issue with the network card? Drivers? I experience this issue even if I stop the pf service.

One thing I did before noticing this issue was trying to rename the network card name as follows:
Code: 
ifconfig_em0_name=”nic0″
ifconfig_nic0=”inet 192.168.0.200 netmask 255.255.255.0″

Could this have caused the issues above? I have rolled back this change but it has made no difference.
 
Try any of the following (not all at once though) if they make a difference:

# ifconfig em0 -tso
# ifconfig em0 -tso4
# ifconfig em0 -rxcsum
# ifconfig em0 -txcsum
 
Tried them one after the other and tried to ping the router after each one but with no joy I'm afraid.
 
I see that your NIC is capable of gigabit speed but it's configured only at 100baseTX, bad cable or is your switch only a 100mbit one?
 
I use Ethernet over power plugs (by Devolo). I think they have a 100mb port on them...not gigabit. Been running my server off this plug for 3 years now with no issues.

PS: I just tried another Devolo plug but with no joy.
 
Thanks to everyone for their help but I have rolled back to the original server. I am totally confused by this issue. The original server booted up fine and could ping the router straight away. Here is my network card output if anyones curious:
Code: 
alpha# ifconfig
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
        ether 00:11:85:e5:d6:25
        inet 192.168.0.200 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 fe80::211:85ff:fee5:d625%bge0 prefixlen 64 scopeid 0x1
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33200
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

How frustrating.
 
Heres another one to make you scratch your head and wonder! I connected up the problematic server somewhere else and changed the IP to 192.168.0.210 and I could ping the router (192.168.0.1) straight away!! What?!
 
It might not be a static arp problem but just an arp cache problem. If you switch the new server back to .200 and reboot the router and it starts to work then it's definitely a problem with a cached arp entry for address .200 that makes the router refuse connections from the new mac address until the cached arp entry expires.
 
kpa said:
It might not be a static arp problem but just an arp cache problem. If you switch the new server back to .200 and reboot the router and it starts to work then it's definitely a problem with a cached arp entry for address .200 that makes the router refuse connections from the new mac address until the cached arp entry expires.
Click to expand...

You Sir, are an absolute GENIUS and have fixed my issue!! I deleted the ARP entry for both .200 and .210 and rebooted the new server and I can now ping the router and resolve names on the internet...thank you very VERY much!!
 
You must log in or register to reply here.
Back
Top