Emerging Threats

I haven't used that particular site before but I do use definitions from two of their sources already. Nice that they have pf rules available already, I might rework my firewall scripts.

Curious if anyone else has used that service before.
 
Ah, ok! Do you use a custom made script to keep those block lists up to date? I'm wondering of building such a script, unless there is one out there already created :stud
 
I would be interested in hearing the answer to this as well and if anyone has used Emerging Threats as a quality resource. I have made my own script but updating it is a pain in the butt.
 
Create the following script (assuming /root/tables/emerging-block-ips is where the ip list is locally stored ) :

Code: 
#!/bin/sh
/usr/local/bin/wget -nd -O /root/tables/emerging-block-ips http://www.emergingthreats.net/fwrules/emerging-Block-IPs.txt
/sbin/pfctl -Tl -f /etc/pf.conf

Add the following to your pf.conf:
Code: 
table <emerging-block-ips> persist file "/root/tables/emerging-block-ips"

block in log quick on $ext_if from <emerging-block-ips> } to any
block out log quick on $ext_if from any to <emerging-block-ips> #optional,paranoid-friendly

Last step: cron the script or store it in /etc/rc.local.

and yes I am quite happy with their ip lists.
 
Ctaranotte, hope you don't mind but I modified your script.

Code: 
#!/bin/sh

fetch http://www.emergingthreats.net/fwrules/emerging-Block-IPs.txt -o /root/tables/emerging-block-ips
pfctl -t emerging-block-ips -Tr -f /root/tables/emerging-block-ips

Also, for single NIC servers you can use this.

Code: 
# emerging threats list
table <emerging-block-ips> persist file "/root/tables/emerging-block-ips"
block in log quick on $int from <emerging-block-ips> to any
block out log quick on $int from any to <emerging-block-ips>

The $int is the variable for my interface
 
Use [cmd=]fetch -m[/cmd] or [cmd=]fetch -i[/cmd].

Code: 
/usr/bin/fetch -i /root/tables/emerging-block-ips http://www.emergingthreats.net/fwrules/emerging-Block-IPs.txt
Code: 
/usr/bin/fetch -m -o /root/tables/emerging-block-ips http://www.emergingthreats.net/fwrules/emerging-Block-IPs.txt
 
Here's my new script, thanks to Dutchman! Forgot about fetch's "-M" flag.
Also, this script will work without adding any ports, all the utilities it needs is in the base install.

Code: 
#!/bin/sh

old_time=`ls -lt /root/tables/emerging-Block-IPs.txt | awk '{print $6" "$7" "$8}'`

cd /root/tables
fetch -m http://www.emergingthreats.net/fwrules/emerging-Block-IPs.txt

new_time=`ls -lt /root/tables/emerging-Block-IPs.txt | awk '{print $6" "$7" "$8}'`
if [ "$old_time" != "$new_time" ]
then
        pfctl -t emerging-block-ips -Tr -f /root/tables/emerging-Block-IPs.txt
fi
 
rbelk said:
Ctaranotte, hope you don't mind but I modified your script.
Click to expand...

Be my guest. I however have to add my two cents were released under a modified beerware license based on Poul-Henning Kamp
Code: 
/*
 * ----------------------------------------------------------------------------
 * "THE BEER-WARE LICENSE" (Revision 42a):
 * <phk@FreeBSD.ORG> inspired this file. As long as you retain this notice, you don't try to fix this stuff as it is not broken and you keep your work based on this stuff sweet and simple, you
 * can do whatever you want with this stuff. If we meet some day in this life or in an other, you think
 * this stuff is worth it and you can afford it, you can buy me a beer or any other spirited beverage in return. In any case, I would be glad to buy you one.
 * ----------------------------------------------------------------------------
 */

That being said. While I respect your research on the use of fetch vs wget, you may want to reflect on this line:
Code: 
block out log quick on $ext_if from any to <emerging-block-ips> #optional,paranoid-friendly

This block action would prevent any infected box behind your firewall to connect to the RBN or any bots.

In conjunction with the foregoing, you may also want to use snort to sniff on your internal interface. Emerging Threats offers also frequently updated snort rules.

Let me know if my oinkmaster.conf is of interest.
 
You could just replace all that ls -lt'ing & awk'ing & time stamp comparing with a simple 'md5 -q' on the file and using 'fetch -i' (which will never get an older file, so a different hash means a newer file).

Code: 
#!/bin/sh

old_hash=$( /sbin/md5 -q /root/tables/emerging-Block-IPs.txt )

/usr/bin/fetch -i /root/tables/emerging-Block-IPs.txt http://www.emergingthreats.net/fwrules/emerging-Block-IPs.txt

new_hash=$( /sbin/md5 -q /root/tables/emerging-Block-IPs.txt )

if [ "$old_hash" != "$new_hash" ]
then
        /sbin/pfctl -t emerging-block-ips -Tr -f /root/tables/emerging-Block-IPs.txt
fi

Some error checking (retries, network outage, etc.) should be inserted, but that's for later.
 
ctaranotte said:
Let me know if my oinkmaster.conf is of interest.
Click to expand...

Definitely would be interested. I havent played with Snort in a while but was just planning on building a new box using that. Also working on a Wireshark box to see what the latest advancements in both have been.

Thanks for some great snippets of code.
 
in your oinkmaster.conf (actually oinkmaster.emerging.conf here), you will need at least this:

Code: 
[color="Red"]url = http://www.emergingthreats.net/rules/emerging.rules.tar.gz[/color]

path = /bin:/usr/bin:/usr/local/bin
tmpdir = /tmp/oinkmaster/
update_files = \.rules$|\.config$|\.conf$|\.txt$|\.map$

use_path_checks = 1
skipfile local.rules
skipfile deleted.rules
skipfile snort.conf
skipfile sid-msg.map

You will need also this croned script to refresh sid-msg.map otherwise you will get unidentified alerts:

Code: 
#!/bin/sh
/usr/local/bin/oinkmaster -o  /usr/local/etc/snort/rules/emerging-threads -C /usr/local/etc/oinkmaster.emerging.conf
/bin/rm /usr/local/etc/snort/sid-msg.map
/bin/cat /usr/local/etc/snort/sid-msg.map-sample /usr/local/etc/snort/rules/emerging-threads/emerging-sid-msg.map > /usr/local/etc/snort/sid-msg.map
/usr/local/etc/rc.d/snort restart

As for pf lists, you can download more specific rules if you feel you don't need all of that, have a look on the EmergingThread site.

Whiteshark is great but this is a professional tool for professionally-trained eyes preferably. That means; unless you are a genius like me (just kidding of course mmm... on reflection maybe not), you might find your learning curve to be quite slow.

Now if you want to make a career in security, have a look at Prelude. in addition to a free opportunity to learn French,
Code: 
Prelude is a Universal "Security Information Management" (SIM) system. Prelude collects, normalizes, sorts, aggregates, correlates and reports all security-related events independently of the product brand or license giving rise to such events; Prelude is "agentless".

Have also a look at something like Honeyd.

Just my two cents
 
Re:

I try modify the DutchDaemon's script.

Code: 
#!/bin/sh
#-------------------------------
#original written by DutchDaemon
#-------------------------------
old_hash=$( /sbin/md5 -q /usr/local/etc/emerging-Block-IPs.txt )
new_hash=$old_hash

/usr/bin/fetch -4 -A -i /usr/local/etc/emerging-Block-IPs.txt -o /tmp/pf_emerging_Block_IP.tmp http://www.emergingthreats.net/fwrules/emerging-Block-IPs.txt

if [ $? -eq 0 ] && [ -f /tmp/pf_emerging_Block_IP.tmp ]
then
  mv /tmp/pf_emerging_Block_IP.tmp /usr/local/etc/emerging-Block-IPs.txt
  new_hash=$( /sbin/md5 -q /usr/local/etc/pf_emerging_Block_IP.conf )
fi

if [ "$old_hash" != "$new_hash" ]
then
   /sbin/pfctl -t emerging_block_ips -T replace -f /usr/local/etc/pf_emerging_Block_IP.conf
fi
 
Question.

How many ips can you block like this before you start having performance problems?
 
I have a couple of questions about this site as well that I'm hoping somebody familiar with them can easily answer.

The emerging-Block-IPs.txt file is not shown on their rule list, but it is a valid link. I don't see anything in their rule list that would indicate a replacement.

I was thinking of employing this on my mail server. The Block-IPs.txt file is a list of IP addresses instead of pf/snort rules like the rest of their released files. I do not have snort installed and am interested in using pf to screen things, but the full pf rule-set seems exceedingly large and overkill for what I'd be using it for. Would using the Block-IPs.txt file to block IP addresses (as in DutchDaemon's script above) be a prudent alternative? Are there any down-sides to doing something like this? I've poked around their site for quite a while and cannot find anything related to the rate of false-positives/miscategorized sites. (In fact, there seems to be very little general information for somebody unfamiliar with the service; the 'new user guide' seems to assume a certain level of familiarity and is focused on people using snort.) I also have the same concern voiced by feralape - what is the performance hit taken by blocking so many IPs in what appears to be an unfiltered list?

I'd like to protect my server as much as possible and this sounds like a good option, but I do not want to impair legitimate functionality while doing so.
 
The PF <table> can be populated a large amount of IP addresses from a valid formatted text file to a specified <table> id by command pfctl.

You can assign this large volume of IP addresses with only one <table> id to your own filter rules, nat rules or rdr rules. Certainly, I use emerging threats IP list in my own blocked rules.

By combining <table> feature and the emergingthreat_block_ip.txt, you can block huge IP addresses with a few rules. May be just only one rule.

You may read this: OpenBSD PF :table section

example:

In /etc/pf.conf
Code: 
#create empty table in memory
<et_block_ip> persist

# in filter section block all ip in <et_block_ip>
block in on $ext_if from <et_block_ip>

After download The text file, you can populate all IPs to the empty <et_block_ip> table by command pfctl.
Code: 
su root
pfctl -t et_block_ip -T replace -f /tmp/emerging-Block-IPs.txt

Now you just use only one PF filter to block all IP addresses in the text file.

How to sync the up-to-dated copy of emerging-Block-IPs.txt to PF <table> is the main subject of this thread.
 
feralape said:
Question.

How many ips can you block like this before you start having performance problems?
Click to expand...

Haven't test this before.

The tables are stored in RAM, and they require very little memory. 10 MB should be more than enough to hold tens of thousands of IPs.
 
I've enabled this block list as a table in my pf firewall. It was as easy as adding the following 4 lines to my pf.conf:

Code: 
# Block IPs classed as threats by EmergingThreats.org
table <emerging-threats> persist file "/usr/local/etc/IPBlocks/EmergingThreats"
block in log quick on $ext_if from <emerging-threats> to any
block out log quick on $ext_if from any to <emerging-threats>


I then used DutchDaemon's shell script posted above as inspiration to create a perl script. I wrote it to be generic, so any file in the system can be kept updated with a network copy using this script. As posted, it does pretty much the same thing as DD's script, but is easily customizable via variables. It has the following features:


Use any destination directory.

Use any file name.

Keep a number of backup files, deleting the oldest ones as needed.

A debug mode, which will spit out the commands issued as they're executed.

The ability to specify what temporary file name to use for the download.

Prevent re-downloading of the file if it hasn't changed. (This is done via the -m parameter to fetch. I kept getting errors using -i and -i isn't in the man page for fetch on my 7.2 system.)

Tokens are used for the standard UNIX commands to allow for non-standard installs.

An arbitrary command can be specified to be run afterward, but is only executed if the update occurs. (If the file hasn't changed, it's not run.) This can be used to refresh whatever daemon it's being used for.




Script to follow...
 
Code: 
#!/usr/bin/perl

###############################################################################
#                                                                             #
# Perl script to keep the a file on your system up to date with one from the  #
# Internet.  Originally written and intended to update the IP Block List from #
# EmergingThreats.org and refresh the pf table with the newest version.       #
#                                                                             #
# Transformed into a geneic utility on 2009-11-16 to work with any file on    #
# the system and run a given command afterwards to refresh whatever daemon    #
# needs to be.                                                                #
#                                                                             #
# Usually run from cron, but no reason it can't be run interactively.         #
#                                                                             #
###############################################################################
#                                                                             #
# Ruler's Common-Sense License:                                               #
#                                                                             #
#   You may use this script however you want to, but I don't warrant it to    #
#   be good for anything in particular, though it happens to work well for    #
#   me.  (I hate putting BS like this in, but I hate more being sued.)  If    #
#   you use this script, you must keep this license and credit to me in it    #
#   in the form of this block, even if you modify it for your own use.  If    #
#   you want to send me money for it, fantastic!  Send me a private message   #
#   on the freebsd.org forums and I'll give you my PayPal address. :-)  Even  #
#   just a simple 'thank you' would be nice.  If not, that's fine too.  All   #
#   hate mail/spam is sent directly to /dev/null                              #
#                                                       - Jim, AKA Ruler2112  #
#                                                                             #
###############################################################################
#                                                                             #
# History:                                                                    #
#                                                                             #
#   2009-08-26 by DutchDaemon     Wrote a shell script that inspired this.    #
#   2009-11-16 by Ruler2112       Wrote as a perl script as generic utility   #
#                                 to keep a file on the system synchronized   #
#                                 with one on the network.                    #
#   2009-11-18 by Ruler2112       Released on freebsd.org forums.             #
#                                                                             #
###############################################################################

use strict;

###############################################################################
#                      Variable Declaration Section                           #
# Set these variables to customize this script's behavior.                    #
###############################################################################

# Storage Directory
#   Where to keep the file or files downloaded.
my $StorageDirectory = "/usr/local/etc/IPBlocks";

# FileName
#   What file name to use for the file downloaded.
my $FileName = "EmergingThreats";

# HowManyBackups
#   The system will rotate and keep this many of the most recent files.
#   A setting of 0 will result in no old copies being kept.
my $HowManyBackups = 7;

# DebugMode
#   If this value is not set to zero, all commands executed will be printed.
my $DebugMode = 0;

# FileURL
#   The URL where the file to download exists.
my $FileURL = "http://www.emergingthreats.net/fwrules/emerging-Block-IPs.txt";

# TempFile
#   Where the downloaded file is kept while being downloaded and compared to
#   the most recent in StorageDirectory.
my $TempFile = "/tmp/Emerging-Threats-Block-IPs.txt";

# PreventRedownload
#   If non-zero, the system will copy the file to the temporary location in
#   order to try to prevent re-downloading the file needlessly.  If a large
#   file is used that you are fairly certain has changed since the last 
#   download, this option may be of use to you.  Usually, a non-zero setting
#   is best to save bandwidth at the cost of copying the last downloaded file.
my $PreventRedownload = 1;

# Location_*
#   Where the * files are located on your system.  Shouldn't change for BSD,
#   but included for ease of porting.
my $Location_cp = "/bin/cp";
my $Location_fetch = "/usr/bin/fetch";
my $Location_md5 = "/sbin/md5";
my $Location_mkdir = "/bin/mkdir";
my $Location_mv = "/bin/mv";
my $Location_rm = "/bin/rm";

# PostCommand
#   A command that will be executed after the file is updated.
#   It is only executed if the file downloaded has changed from the previous
#   version of the file downloaded.  Use care when setting this!
my $PostCommand = "/sbin/pfctl -t emerging-threats -Tr -f '$StorageDirectory/$FileName'";


###############################################################################
# !!! WARNING !!! WARNING !!! WARNING !!! WARNING !!! WARNING !!! WARNING !!! #
###############################################################################
#                    This is the Beginning of the Script                      #
#                                                                             #
# Do not change anything below this line unless you know what you're doing!   #
###############################################################################

my ($existinghash, $downloadedhash, $fetchstatus);
my ($workstr, $counter);

# Disable output buffering if in debug mode
if($DebugMode)
  {
  $| = 1;
  }

# Create the storage directory if it does not already exist
if(! -d "$StorageDirectory" )
  {
  Execute("$Location_mkdir '$StorageDirectory'");
  }

# Remove temp file if it still exists
if(-e "$TempFile")
  {
  Execute("$Location_rm -f '$TempFile'");
  }

# Perform the following only if we already have at least one copy of the file
$workstr = "";
if( -r "$StorageDirectory/$FileName")
  {
  # Calculate MD5 hash of the file we have
  $existinghash = Execute("$Location_md5 -q '$StorageDirectory/$FileName'");
  chomp($existinghash);
  if($PreventRedownload)
    {
    # Copy the newest file we have to the temp file to prevent re-downloading
    Execute("$Location_cp -p '$StorageDirectory/$FileName' '$TempFile'");
    $workstr = "-m";
    }
  }
else
  {
  $existinghash = "blah";
  }

# Download new file
$fetchstatus = Execute("$Location_fetch $workstr -q -o '$TempFile' $FileURL");

# If download failed, print error message and exit
if($fetchstatus > 0)
  {
  print "Download of new $FileName FAILED from $FileURL with error code $fetchstatus!\n";
  exit;
  }

# Calculate MD5 hash of the file just downloaded
$downloadedhash = Execute("$Location_md5 -q '$TempFile'");
chomp($downloadedhash);

# If hashes are different, we have a new file
if($downloadedhash ne $existinghash)
  {
  if($HowManyBackups > 0)
    {
    # Rotate files, keeping as many as configured in HowManyBackups
    if(-e "$StorageDirectory/$FileName.$HowManyBackups")
      {
      Execute("$Location_rm -f '$StorageDirectory/$FileName.$HowManyBackups'");
      }
    for($counter = $HowManyBackups - 1; $counter > 0; $counter--)
      {
      $workstr = $counter + 1;
      if(-e "$StorageDirectory/$FileName.$counter")
        {
        Execute("$Location_mv '$StorageDirectory/$FileName.$counter' '$StorageDirectory/$FileName.$workstr'");
        }
      }
    if(-e "$StorageDirectory/$FileName")
      {
      Execute("$Location_mv '$StorageDirectory/$FileName' '$StorageDirectory/$FileName.1'");
      }
    }
  else
    {
    # Just remove the old file - we don't need no stinking backups!
    if(-e "$StorageDirectory/$FileName")
      {
      Execute("$Location_rm -f '$StorageDirectory/$FileName'");
      }
    }

  # Move newly downloaded file into place and execute command after update
  Execute("$Location_mv '$TempFile' '$StorageDirectory/$FileName'");
  Execute("$PostCommand");
  }

# Remove temp file if it still exists
if(-e "$TempFile")
  {
  Execute("$Location_rm -f '$TempFile'");
  }

# Done
exit;


# Modularized command execution to allow for easy debugging
sub Execute()
  {
  my $command = shift;
  my($retval);
  if($DebugMode)
    {
    print "Running->$command<-\n";
    }
  $retval = `$command`;
  return $retval;
  }
 
Heads Up - EmergingThreats fwrules URL change again!

Below is the update to my script. Also to add the correct entries into PF, consult this post.

Code: 
#!/bin/sh

# ----------------------------------------------------------------------------
# "THE BEER-WARE LICENSE" (Revision 42a):
# <phk@FreeBSD.ORG> inspired this file. As long as you retain this notice, you don't try to fix this stuff as it is not broken and you keep your work based on this stuff sweet and simple, you
# can do whatever you want with this stuff. If we meet some day in this life or in an other, you think
# this stuff is worth it and you can afford it, you can buy me a beer or any other spirited beverage in return. In any case, I would be glad to buy you one.
# ----------------------------------------------------------------------------

# -- Maximum time, in seconds, to wait before aborting an HTTP connection.
HTTP_TIMEOUT=60

cd /root/tables

old_hash=$( /sbin/md5 -q /root/tables/emerging-Block-IPs.txt )

/usr/bin/fetch -m /root/tables/emerging-Block-IPs.txt http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt
# http://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt

new_hash=$( /sbin/md5 -q /root/tables/emerging-Block-IPs.txt )

if [ "$old_hash" != "$new_hash" ]
then   
	( /sbin/pfctl -t emerging-block-ips -Tr -f /root/tables/emerging-Block-IPs.txt ; \
	echo ">>> emerging-Block-IPs.txt has been updated, and PF was restarted!" ) | mail -s "emerging-Block-IPs.txt report" root
else
	echo ">>> emerging-Block-IPs.txt has not been updated, PF was not restarted!"
fi
 
You must log in or register to reply here.
Back
Top