ipfw dynamic rule unmatched?

qsecofr

qsecofr

Hi,
Trying to track down a possible problem with my firewall.
Had a replacement DSL modem sent to me by ISP. With this new modem I am suddenly unable to reach a website that I use daily. The name resolves in the DNS to an IP. I can make the connection to target website. It's not blocked by my firewall outbound or inbound. There are no packets denied, and nothing logged to /var/log/security.
Code: 
ipfw -d show
displays the connection from my machine to target IP of the website. Yet the browser is waiting until finally timing out without displaying any content.
Code: 
ping
and
Code: 
traceroute
both timeout.

It doesn't seem like any data is coming back to my machine from this one website. Actually it's at least 2 websites that appear not to work. Murhpys Law says the problem will be with the one most essential site of course. But Im not sure which tools can best help me diagnose where the problem is. Suggestions much appreciated.

I suspect the issue may be my firewall only because I've connected my laptop directly to the DSL modem, which had been configured as a DCHP server. No browser issues.

But the modems current config is for a block of static IPs, no DHCP, firewall, filtering, nothing else. Freebsd 7.1 server with a custom firewall ruleset with NAT that had by all appearances worked fine until the new modem arrived yesterday.

Thanks in advance for suggestions
 
Temporarily disable IPFW to test your theory?

If I'm reading the the ipfw(8) manpages correctly:
# sysctl net.inet.ip.fw.enable = 0

... that should do it.
 
Good idea I thought. I disabled the sysctl variable. Tried to navigate to the website and failed, while other websites still worked. And then I re-enabled the sysctl variable and reloaded the ruleset script. That may prove the problem is not in the firewall. But the problem is still there somewhere.

I had called the ISP and asked them to check their equipment & routes, thinking that maybe the power outage that fried my modem may have damaged a neighborhood switch or card, or maybe their routing tables had an issue. The traceroute always timed out at the first hop, continued on a few hops, and then timed out forever after some hop in maybe Phoenix.

The ISP helpdesk helped re-configure my modem to new settings, which did not match the old modem settings. At that time, with my notebook connected directly to the modem, everything appeared to work. But then the FBSD server couldn't communicate with the modem when I hooked it up to bring up the rest of the 192.168 subnet..

Maybe the modems settings arent right?
Judging by the presence of a dynamic rule it appears the destination website can be reached. Is there a way to identify if that target website is just not responding (or responding too slowly) for some reason? Or to identify if too much packet loss is going on along the route? Or even to know if it is the server? If as above shows it's not the firewall, then I'm not sure what else in the server might be misbehaving.
 
tcpdump results

i tried
Code: 
tcpdump -c 500 -i bge0 -p -w /tmp/dump.txt host {name | dotted-decimal}
After waiting a while I use Crtl-C to end. Not quite sure yet how to interpret the output.

But I'll also try the two ports mentioned. Thanks
 
tcpdump

Any suggested switches on tcp to make the output understandable (by me) ? I do have some output. Though not a lot. But I'm not sure how to interpret it.

tcping and tcptraceroute both confirm the host is open. In fact tcptraceroute makes it all the way to the host, whereas traceroute just times out after a few hops.
 
Try [cmd=]tcpdump -s 0 -pnli bge0 host http://www.that.website[/cmd]

Look for packets with ' S '. The first one is a SYN, which should be answered by the web host with a 'SYN/ACK' (look for 'ack' in the line), and another one from you (a.k.a. the three-way handshake).
 
tcpdump results

results:
Code: 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bge0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:21:19.789256 IP 67.40.255.97.3924 > 162.93.224.80.80: S 2702876872:2702876872(0) win 64240 <mss 1460,nop,nop,sackOK>
10:21:19.854901 IP 162.93.224.80.80 > 67.40.255.97.3924: S 2655703422:2655703422(0) ack 2702876873 win 5840 <mss 1460>
10:21:19.857233 IP 67.40.255.97.3924 > 162.93.224.80.80: . ack 1 win 64240
10:21:19.885239 IP 67.40.255.97.3924 > 162.93.224.80.80: P 1:770(769) ack 1 win 64240
10:21:19.956828 IP 162.93.224.80.80 > 67.40.255.97.3924: . ack 770 win 6921
10:21:19.962247 IP 162.93.224.80.80 > 67.40.255.97.3924: P 1:727(726) ack 770 win 6921
10:21:20.163224 IP 67.40.255.97.3924 > 162.93.224.80.80: . ack 727 win 63514
10:21:20.172258 IP 67.40.255.97.3925 > 162.93.224.80.443: S 2703037080:2703037080(0) win 64240 <mss 1460,nop,nop,sackOK>
10:21:20.238182 IP 162.93.224.80.443 > 67.40.255.97.3925: S 2671922860:2671922860(0) ack 2703037081 win 5840 <mss 1460>
10:21:20.241178 IP 67.40.255.97.3925 > 162.93.224.80.443: . ack 1 win 64240
10:21:20.244181 IP 67.40.255.97.3925 > 162.93.224.80.443: P 1:82(81) ack 1 win 64240
10:21:20.310422 IP 162.93.224.80.443 > 67.40.255.97.3925: . ack 82 win 5840
10:21:34.964198 IP 162.93.224.80.80 > 67.40.255.97.3924: F 727:727(0) ack 770 win 6921
10:21:34.966713 IP 67.40.255.97.3924 > 162.93.224.80.80: . ack 728 win 63514
10:21:38.377291 IP 67.40.255.97.3924 > 162.93.224.80.80: F 770:770(0) ack 728 win 63514
10:21:38.442267 IP 192.168.1.251.3924 > 162.93.224.80.80: . ack 2655704150 win 0
10:21:38.442516 IP 162.93.224.80.80 > 67.40.255.97.3924: . ack 771 win 6921
10:21:38.443299 IP 67.40.255.97.3924 > 162.93.224.80.80: . ack 728 win 63514
10:22:31.985903 IP 67.40.255.97.3925 > 162.93.224.80.443: F 82:82(0) ack 1 win 64240
10:22:32.083194 IP 162.93.224.80.443 > 67.40.255.97.3925: . ack 83 win 5840

in /var/log/messages i see a lot of AYN|ACK and syncache messages. Though they mostly appear to be when another system is connecting to my mail server. A new entry which reads in part
Code: 
collect: I/O error on connection from
is logged from the mailserver of the website im trying to connect to. But I don't see any SYN|ACK error specifically from the website IP.
Code: 
grep '162.93.2' /var/log/messages
only turns up spurious RST errors when their mailserver tries to connect to mine,
 
qsecofr said:
results: ...
Click to expand...

Looks like a normal tcp conversation. Handshake is established, data is pushed, connection is torn down.

What exactly are you seeing client side? Try another web browser?
 
client web browser

.. just hangs.. progress bars seem to indicate receiving some data. but content never displays in browser. Just waits. firefox 3.5 on both FBSD & WinXP. IE6. FBSD Konqueror status bar displays "retrieving 254 B" from host and just waits until I manually stop it basically..

Doesnt seem to matter whether i prefix the URL with http or https. same results. Though i think the website does an automatic redirect to https, but that's not a new behavior either.
 
Since tcptraceroute to port 80 works, try the following: $ telnet webserver.ip.or.name 80 If that connects, try entering get / to pull up the default web page. If that works, then there's an issue with the browser and not the network. (Never underestimate the power of the telnet client.) :)
 
A loose 'get /' will give an error (well, that's a reply too ;))

Type
Code: 
GET / HTTP/1.0
or
HEAD / HTTP/1.0
followed by two enters.

The latter should produce something like this:

Code: 
HTTP/1.1 200 OK
Date: Fri, 16 Oct 2009 23:01:45 GMT
Server: Apache/2.2.13 (FreeBSD) PHP/5.2.11 with Suhosin-Patch mod_scgi/1.12 mod_ssl/2.2.13 OpenSSL/0.9.8k
Last-Modified: Sat, 20 Nov 2004 20:16:24 GMT
ETag: "970-2c-3e9564c23b600"
Accept-Ranges: bytes
Content-Length: 44
Connection: close
Content-Type: text/html

Connection closed by foreign host.
 
telnet responses

telnet to port 80 returns
Code: 
get /
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://www.host.com/public/host/home/welcomep.html">here</a>.</p>
<hr />
<address>IBM_HTTP_Server Server at www.host.com Port 80</address>
</body></html>
Connection closed by foreign host.

telnet to 443 and a get / causes the foreign host to close the connection.

with the HTTP/1.0
Code: 
get / HTTP/1.0

HTTP/1.1 301 Moved Permanently
Date: Fri, 16 Oct 2009 23:15:34 GMT
Server: IBM_HTTP_Server
P3P: CP="CAO CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELi OUR DEL SAMi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA GOV"
Location: https://www.host.com/public/host/home/welcomep.html
Cache-Control: max-age=0
Expires: Fri, 16 Oct 2009 23:15:34 GMT
Vary: Accept-Encoding
Content-Length: 338
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://www.host.com/public/host/home/welcomep.html">here</a>.</p>
<hr />
<address>IBM_HTTP_Server Server at www.host.com Port 80</address>
</body></html>
Connection closed by foreign host.

the head command produced the same results as get.

and then once again trying the HTTP/1.0 qualifier on port 443 causes foreign host to close the connection without transmitting any data - at least none visible to the telnet session.

not sure if Lynx would help with pinpointing the problem, but ill give a go & install it and try that too.

>>

a couple weeks ago i upgraded apache22 and ran into a warning about an accept filter. searching led me to conclude i needed to edit /boot/loader.conf to include
Code: 
accf_http_load="YES"
in my notes i dont see that i rebooted afterward, even though i think i did. I'll try to unset that and reboot.
 
Port 443 expects HTTPS/SSL, not plain HTTP. It should throw an error about that (like '400 Bad Request').
 
telnet to 443

fails even with
Code: 
get / HTTPS/SSL
connection closed by foreign host.

web browser to other ssl-enabled sites seems OK so far. tried a couple. i perceived them to be a bit slower, but thats probably just my imagination. they otherwise worked.

I installed lynx. to yahoo:
Code: 
unable to get local issuer certificate-Continue?

ended up on a page reading
Code: 
If you are seeing this page, your browser settings prevent you from automatically
   redirecting to a new URL.
but again that's lynx and not to my intended target site..

the target site yields a status message of
Code: 
making HTTPS connection to www.host.com
and again waits forever till i kill it like the gui browsers.

I did revert the boot setting for apache back commenting it out. apache started up without warning to my surprise. but no effect on reaching the target host.

watching the dmesg messages during reboot makes me think I should ask about dropping SYN+FIN. potential cause & effect? Any sysctl.conf or kernel options that might drop packets without logging anything?

/etc/sysctl.conf includes
Code: 
security.bsd.see_other_uids=0
net.link.ether.inet.max_age=1200
net.inet.tcp.blackhole=0
net.inet.tcp.delayed_ack=0
net.inet.tcp.sendspace=65535
net.inet.tcp.recvspace=65535
net.inet.udp.recvspace=65535
net.inet.udp.blackhole=0
net.inet.udp.maxdgram=57344
net.inet.icmp.bmcastecho=0
net.inet.icmp.maskrepl=0
net.inet.ip.random_id=1
net.inet.ip.stealth=1
net.inet.ip.redirect=0
net.inet.ip.sourceroute=0
net.inet.ip.accept_sourceroute=0
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535
kern.ipc.somaxconn=8192
kern.ipc.maxsockets=16424
kern.ipc.maxsockbuf=2097152
kern.maxfiles=65535
kern.maxfilesperproc=32768
vfs.usermount=1
at least one of which appears to be invalid judging by the boot messages.

in my kernel config contains the options
Code: 
options         IPFIREWALL
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=100
#options                TCP_DROP_SYNFIN 20090219
options         IPSTEALTH
options         DUMMYNET

even though drop SYNFIN is commented out it appeared in my bootup messages. its not in /var/log/messages but i did read it before logging in, <pause> and <PgUp>. I'm sure I didn't imagine it, but still leads me to wonder if there's kernel options or sysctl settings that may be stomping on this one site without my awareness. Maybe the host changed something, who knows?
 
qsecofr said:
fails even with
Code: 
get / HTTPS/SSL
connection closed by foreign host.
Click to expand...

Heh ;) That's not what I meant. Port 443 doesn't expect the words HTTPS/SSL, but the https protocol (encryption, certificates, etc.).

I have no idea why you think your Apache has any influence on using an external website ..
 
no, didnt mean my apache..

I don't think my apache influences external webservers. The discussion so far has seemingly ruled out my firewall and ruled out the internet. So Im wondering if my kernel has some option setting that's causing my troubles. The accept filter popped into my head as potentially affecting my kernel's behavior. So even if I was doubtful, I decided to remove the setting and just eliminate the possibility completely. Grasping at straws maybe.

I get a lot of messages logged relating to syncache_timeout, syncache_chkrst, SYN|ACK etc.. They haven't yet seemed to be a major problem that I can perceive. Maybe it's the other host's system causing these messages? Maybe it's mine? I did see the mailserver of the company's website generate some of these errors when talking to my mailserver. But not their webserver to my browser.

I just don't know what the issue might be with SSL to their website. Grasping at straws to see if it's got any connection to anything else going on in my system that could provide a clue.
 
disabling firewall

If I disable firewall as above by changing the sysctl setting, does it immediately disable the firewall, or is reboot required? -- just double-checking that I did it right..
 
I don't have IPFW running anywhere any more, but you can set up your own contrived test.

# ipfw -q add 001 deny icmp from any to any out

That should prevent you from pinging external hosts. (Confirm that that's true.)

Next, run the sysctl command. That should make it so you can ping external hosts again. (Confirm that is true as well.)
 
[closed but not solved] firewall ruled out.

Ive tried a few different things to make sure I can completely rule out firewall. And I think it can be ruled out. probably this thread can be closed. maybe open a new one in networking. Still problem persists.
 
You must log in or register to reply here.
Back
Top