IPF IPF Top Destination IP Question

JLAIP

JLAIP

I began using ipf's top (ipfstat -t), but haven't been able to locate a simple legend explaining whether the Source and Destination sections reference IPs that are being blocked or allowed through.

Two questions:
1) Is ipf blocking or allowing these Destination IPs?
2) Why would there be ANY Destination IPs coming from either server when I'm the only login? Other than ntp, there shouldn't be any background processes on the email server that're sending data to external IPs?

The 10.0.0.2 IP is our internal email server. It connects to the Internet through our email server (for security, I'm masking that IP), which also handles firewall duties. Hopefully, the attached diagram makes sense.
 

Attachments

  • firewall.jpg
    firewall.jpg
    244.4 KB · Views: 197
  • firewall2.jpg
    firewall2.jpg
    241.6 KB · Views: 163
  • network layout.jpg
    network layout.jpg
    29.9 KB · Views: 172
To re-state your problem so I know I understand it, the host "web svr" is acting as a web server, email gateway, and firewall for traffic to and from 10.0.0.2, yes? And you're concerned that it shouldn't be allowing inbound connections to 10.0.0.2, nor allowing connections from 10.0.0.2 to external hosts? Can you state on which system(s) firewall.jpg and firewall2.jpg come from? I presume they're both from the web server, as there's a connection from a redacted IP address to 10.0.0.2 on SSH. That would mean that the connection 36.99.136.128,20645 -> 10.0.0.2,80 listed in firewall.jpg shows the web server forwarding traffic from 36.99.136.128 on to 10.0.0.2, right?

I don't use ipf() at all, but judging from the output you shared, it looks like the the last column is a duration field. In the screenshot attachments, the first entry looks to be a long-lived SSH session from the redacted IP address to 10.0.0.2. Does a 119-minute duration make sense for that connection?

For the outbound connections from 10.0.0.2 to external addresses, those are all DNS (port 53). While DNS can be abused, it's pretty typical to allow outbound DNS requests, no? Or should 10.0.0.2 be using only a local DNS server?

For the outbound connections from the web server, port 123 is NTP as you explained, so I should imagine isn't blocked. I had to lookup port 113 in /etc/services, as I'd never heard of the Ident protocol. Do you have hosts using that protocol to identify the owner of incoming TCP connections?

If you're concerned that the web server isn't blocking inbound traffic to 10.0.0.2 properly, I would run sockstat -l on 10.0.0.2 to list all active connections to the services listening on the host, and see how the output compares to ipfstat -t on the web server. If you are running a web service on 10.0.0.2, have you checked its logs for entries showing successful connections from outside your network?
 
darmokandjalad said:
To re-state your problem so I know I understand it, the host "web svr" is acting as a web server, email gateway, and firewall for traffic to and from 10.0.0.2, yes? And you're concerned that it shouldn't be allowing inbound connections to 10.0.0.2, nor allowing connections from 10.0.0.2 to external hosts? Can you state on which system(s) firewall.jpg and firewall2.jpg come from? I presume they're both from the web server, as there's a connection from a redacted IP address to 10.0.0.2 on SSH. That would mean that the connection 36.99.136.128,20645 -> 10.0.0.2,80 listed in firewall.jpg shows the web server forwarding traffic from 36.99.136.128 on to 10.0.0.2, right?

I don't use ipf() at all, but judging from the output you shared, it looks like the the last column is a duration field. In the screenshot attachments, the first entry looks to be a long-lived SSH session from the redacted IP address to 10.0.0.2. Does a 119-minute duration make sense for that connection?

For the outbound connections from 10.0.0.2 to external addresses, those are all DNS (port 53). While DNS can be abused, it's pretty typical to allow outbound DNS requests, no? Or should 10.0.0.2 be using only a local DNS server?

For the outbound connections from the web server, port 123 is NTP as you explained, so I should imagine isn't blocked. I had to lookup port 113 in /etc/services, as I'd never heard of the Ident protocol. Do you have hosts using that protocol to identify the owner of incoming TCP connections?

If you're concerned that the web server isn't blocking inbound traffic to 10.0.0.2 properly, I would run sockstat -l on 10.0.0.2 to list all active connections to the services listening on the host, and see how the output compares to ipfstat -t on the web server. If you are running a web service on 10.0.0.2, have you checked its logs for entries showing successful connections from outside your network?
Click to expand...
My apologies. In my rush, I mis-labeled the two servers. 10.0.0.2 is the web server & it lies behind the email server (x.x.x.x). The email server faces the Internet and handles the firewall. I'm attaching a corrected layout diagram.

I understand port 53 is DNS, but I often see other ports listed in the Destination IP column (e.g., port 80, port 25, port 113, etc.). So I'm concerned with who or what is sending data out from our servers.....especially, when no one but me is logged-in.

The last column is ttl (it's cut off on my screenshots), but I'm primarily interested in what the Destination IP column is indicating? That is, does it indicate data being ALLOWED out to external IP addresses, or does it indicate data that was BLOCKED?? That's really all I want to know. What does the Destination IP column report?
 

Attachments

  • network layout.jpg
    network layout.jpg
    24.7 KB · Views: 149
Destination port 113 is IDENT. It's a really old protocol, usually used in combination with IRC. You should definitely look into that traffic. The rest seems 'normal', I see 25 (mail), 53 (DNS) and 123 (NTP).

And please don't post pictures of text. Just copy/paste the information. Pictures are impossible to quote from.
 
darmokandjalad said:
For the outbound connections from the web server, port 123 is NTP as you explained, so I should imagine isn't blocked. I had to lookup port 113 in /etc/services, as I'd never heard of the Ident protocol. Do you have hosts using that protocol to identify the owner of incoming TCP connections?
Click to expand...
No. Other than outgoing email, there should be nothing being sent from our email server.....especially, when I am the ONLY login and I am NOT sending out any email.
 
SirDice said:
Destination port 113 is IDENT. It's a really old protocol, usually used in combination with IRC. You should definitely look into that traffic. The rest seems 'normal', I see 25 (mail), 53 (DNS) and 123 (NTP).
Click to expand...

Do you know if the IPs listed in the Destination IP column are being allowed or blocked?
 
SirDice said:
I suggest you use tcpdump(1) and actually have a look at the traffic.
Click to expand...
Thanks, I'm watching it now and there's a huge amount of traffic that I don't understand....and probably should be stopped. I don't know how to save this data as a text file and there's so much of it that I think a single file would be too large to post here anyway.

I appreciate that you don't like screenshots, but I think I'm in over my head here. Could I PM four tcpdump screenshots of what I'm seeing and get your opinion? Under the circumstances, I'd rather not post more identifiable info on a public forum if I can avoid it.
 

Attachments

  • firewall2 006.jpg
    firewall2 006.jpg
    159 KB · Views: 153
  • firewall3.jpg
    firewall3.jpg
    123.3 KB · Views: 155
JLAIP said:
I don't know how to save this data as a text file and there's so much of it that I think a single file would be too large to post here anyway.
Click to expand...
I'll give you a few tips. The -w option of tcpdump(1) allows you to write the packet capture to a PCAP file. This file can be loaded in net/wireshark (install wireshark on Windows if you have to, it'll read the PCAP file just fine). That'll make it easier to analyze. The command line tcpdump(1) output takes some experience to understand. You can also add filters, so you're only capturing traffic you're interested in. If you just capture everything you're going to quickly drown in the amount of information you're seeing. Also note that running tcpdump(1) while being SSH'ed into your server might actually capture your SSH session, which gets printed by tcpdump(1), which causes more SSH traffic, which gets captured.... etc. You get the idea.

tcpdump -ni em0 host 1.2.3.4 and host 2.3.4.5 # This will only capture the traffic between hosts 1.2.3.4 and 2.3.4.5
tcpdump -ni em0 udp and port 123 # This would only capture traffic to/from UDP port 123
tcpdump -ni em0 not \(tcp and port 22\) # Capture everything except tcp/22 (ssh) Watch out for the shell when you're going to use (..), most shells will interpret them before sending the arguments to tcpdump(1).

You can create really complex filters to 'zoom' in on the specific traffic you're looking for. Learning to use tcpdump(1) is really invaluable when dealing with firewalls (doesn't matter if it's PF, IPFW or IPFilter). Because it allows you to actually look at what's on the wire, instead of trying to guess what is happening.
 
SirDice said:
I'll give you a few tips. The -w option of tcpdump(1) allows you to write the packet capture to a PCAP file. This file can be loaded in net/wireshark (install wireshark on Windows if you have to, it'll read the PCAP file just fine). That'll make it easier to analyze. The command line tcpdump(1) output takes some experience to understand. You can also add filters, so you're only capturing traffic you're interested in. If you just capture everything you're going to quickly drown in the amount of information you're seeing. Also note that running tcpdump(1) while being SSH'ed into your server might actually capture your SSH session, which gets printed by tcpdump(1), which causes more SSH traffic, which gets captured.... etc. You get the idea.

tcpdump -ni em0 host 1.2.3.4 and host 2.3.4.5 # This will only capture the traffic between hosts 1.2.3.4 and 2.3.4.5
tcpdump -ni em0 udp and port 123 # This would only capture traffic to/from UDP port 123
tcpdump -ni em0 not \(tcp and port 22\) # Capture everything except tcp/22 (ssh) Watch out for the shell when you're going to use (..), most shells will interpret them before sending the arguments to tcpdump(1).

You can create really complex filters to 'zoom' in on the specific traffic you're looking for.

Learning to use tcpdump(1) is really invaluable when dealing with firewalls (doesn't matter if it's PF, IPFW or IPFilter). Because it allows you to actually look at what's on the wire, instead of trying to guess what is happening.
Click to expand...
Thank you for this, but there is way too much data & my concern is NOT with one or two IPs or datastreams. But, rather, the HUGE total number of nearly constant incoming and outgoing data on two servers that only see light [legitimate] use.
If someone knowledgeable could see what I'm seeing at the console screen, it would likely be obvious what is happening....and the remedy simple.

I could really use some help here...
 
The outgoing ident 113 request are from your smtp server to the remote clients when your mail server is trying to lookup they host name. Same goes for the DNS requests for rDNS records and for normal MX resolves during sending of emails.
 
VladiBG said:
The outgoing ident 113 request are from your smtp server to the remote clients when your mail server is trying to lookup they host name. Same goes for the DNS requests for rDNS records and for normal MX resolves during sending of emails.
Click to expand...
My question is: Who are these remote clients?? There were 0 outgoing emails and no users logged in other than me. And I'm seeing similar data transfers on a number of different ports on both local and external IPs.

Again, other than ntp, there should be 0 outgoing activity on the email server and little, if any, on the web server. Yet I'm seeing a nearly constant stream of in- and out-going data to unknown IPs on both servers at times when they should be mostly silent.
 
When you have port 25 open there's many internet scan bots that are connecting to your smtp and your server is trying to resolve they host names. There's nothing to worry about that. If you want to reduce such attempts you can use some IPS like fail2ban to block them for specific time after N number of failure logins.
 
VladiBG said:
When you have port 25 open there's many internet scan bots that are connecting to your smtp and your server is trying to resolve they host names. There's nothing to worry about that. If you want to reduce such attempts you can use some IPS like fail2ban to block them for specific time after N number of failure logins.
Click to expand...
Can I send you a few screenshots of what I'm seeing? If, after seeing them, you don't see a problem, that'll be good enough for me. I just want a knowledgeable second opinion.
 
I am the only user logged-into our email server (address masked) and there is no [legitimate] outgoing email. Does this output from tcpdump look benign to you?
 

Attachments

  • tcpdump.jpg
    tcpdump.jpg
    162.6 KB · Views: 184
JLAIP said:
I began using ipf's top (ipfstat -t), but haven't been able to locate a simple legend explaining whether the Source and Destination sections reference IPs that are being blocked or allowed through.

Two questions:
1) Is ipf blocking or allowing these Destination IPs?
Click to expand...

ipfstat -t lists top state table entries. State table entries are created when "pass" is used with "keep state".

JLAIP said:
2) Why would there be ANY Destination IPs coming from either server when I'm the only login? Other than ntp, there shouldn't be any background processes on the email server that're sending data to external IPs?
Click to expand...

That would depend on your ipf rules.

JLAIP said:
The 10.0.0.2 IP is our internal email server. It connects to the Internet through our email server (for security, I'm masking that IP), which also handles firewall duties. Hopefully, the attached diagram makes sense.
Click to expand...
Again, this depends on your rules, when and where you use "keep state" in your rules.

BTW, you probably don't want to post your rules here but send me a private email (I'm the ipf maintainer in FreeBSD) and I can help understand how your rules effect the state table.

Also, even if you don't use "keep state", NAT automatically creates state table entries when NAT table entries are created. NAT and firewall are handled by different parts of ipfilter (similar to ipfw, pf, and even iptables in Linux). It's the way these kinds of things are designed and coded.

As to why one might want to use state table instead of old-school check the ACK bit in the TCP header (UDP uses a timeout), finding the correct entry in a hash table is much faster than serially comparing each rule for a match.

Commercial firewalls such as Checkpoint do the same, except that using the state table is the only option. There is no old-school way to define Checkpoint rules without the use of the state table. At least open source firewalls allow you that option.
 
cy@ said:
BTW, you probably don't want to post your rules here but send me a private email (I'm the ipf maintainer in FreeBSD) and I can help understand how your rules effect the state table.
Click to expand...
PM sent. Thank you!
 
JLAIP said:
I am the only user logged-into our email server (address masked) and there is no [legitimate] outgoing email. Does this output from tcpdump look benign to you?
Click to expand...
What exactly are your concerns?
Looking at the network traffic you see some DNS, HTTP,HTTPS traffic. It's normal when you have web server. Instead of trying to track what is happening at the network level it's better to monitor the application logs.
 
VladiBG said:
What exactly are your concerns?
Looking at the network traffic you see some DNS, HTTP,HTTPS traffic. It's normal when you have web server. Instead of trying to track what is happening at the network level it's better to monitor the application logs.
Click to expand...
My concern is why there appears to be outgoing data from an email server that is serving no [legitimate] users?
 
You must log in or register to reply here.
Back
Top