HowTo: Monitor Network Traffic with Netflow/NfDump/NfSen on FreeBSD

woodsb02

woodsb02

Many system administrators will be familiar with system monitoring tools such as net-mgmt/nagios, net-mgmt/collectd5, net-mgmt/cacti or net/vnstat. Whilst these tools can monitor and record network traffic statistics, they do not record details of source and destination IP addresses or TCP/UDP port numbers for the network traffic. Without this information, troubleshooting a spike in network traffic is left to reviewing log files or asking "what happened at that point in time?".

This is where NetFlow can help. Originally introduced by Cisco, it allows source/destination network traffic statistics ("flow records") to be stored and analysed. The NetFlow Wikipedia article provides further information about NetFlow.

A typical NetFlow setup consists of 3 main components:
  • Flow exporter: The device which the network traffic is going through will generate source/destination network traffic statistics and export these flow records to a flow collector over a network connection. This is typically a Cisco switch, however this HowTo guide will explain how to do this from any FreeBSD computer using the netgraph(4) subsystem built into the kernel.
  • Flow collector: A device which receives the flow records and stores them for future analysis. This HowTo guide will explain how to use the nfcapd daemon from the net-mgmt/nfdump port to perform this task in FreeBSD. Whilst this would typically be done on a central flow collection server, it could also be on the same FreeBSD computer as the exporter.
  • Analysis application: Analyses the stored flow data, often presenting it to the user as time-based graphs, a table of the top 10 network flows, or by email when an automatic threshold is breached. This HowTo guide will explain how to use the net-mgmt/nfsen FreeBSD port to provide a graphical web based frontend to view the flow data, or the net-mgmt/nfdump port to query the data at the terminal of a FreeBSD computer.

HowTo continued in next post...
 
CONFIGURE IN KERNEL NETFLOW EXPORT WITH netgraph(4)

If you haven't heard of netgraph(4) before, you should take the time to read the All About Netgraph article by Archie Cobbs. In summary, netgraph(4) is an in-kernel networking subsystem that allows the system administrator to make online changes to the interconnections ("edges") between the different components of the FreeBSD network stack ("nodes"). An edge connects 2 nodes using 2 hooks, one on each node. Data packets flow bidirectionally along the edges from node to node. When a node receives a data packet, it performs some processing on it, and then (usually) forwards it to another node.

FreeBSD's netgraph(4) implementation includes the ng_netflow(4) node that can be inserted in between the ng_ether(4) node lower hook (raw Ethernet device) and the ng_ether(4) node upper hook (upper protocol layers connected to the kernel). The following code will do this for a network interface named igb0:

Code: 
# kldload netgraph ng_netflow ng_ksocket ng_ether   // Load the necessary kernel modules

# /usr/sbin/ngctl -f- <<-SEQ                        // Run the ngctl(8) program, using the lines below as netgraph commands
    mkpeer igb0: netflow lower iface0               // Create new ng_netgraph(4) node, connecting its iface0 hook to the lower hook of the existing ng_ether(4) node igb0
    name igb0:lower netflow                         // Name the newly created ng_netflow(4) node which is connected to the igb0 node's lower hook "netflow"
    connect igb0: netflow: upper out0               // Connect the out0 hook of the netflow node to the upper hook of the igb0 node
    mkpeer netflow: ksocket export inet/dgram/udp   // Create new ng_ksocket(4) node, connecting its inet/dgram/udp hook to the export hook of the existing netflow node
    msg netflow: setconfig {iface=0 conf=7}         // Configure the netflow iface0 hook to export flow data for traffic flowing in either direction (default is ingress only)
    msg netflow:export connect inet/127.0.0.1:4444  // Configure the ksocket UDP hook to send flow data to IP address 127.0.0.1 on UDP port 4444
SEQ

These commands must be run each time the computer is booted. This can most easily be achieved by creating a new /usr/local/etc/rc.d/ng_netflow rc script based on this GitHub link. For further control over what network traffic is included in the netflow export data, the /usr/local/etc/rc.d/ng_netflow_ipfw rc script attached to PR 134660 allow the IPFW firewall to be used to filter traffic that will be exported.

HowTo continued in next post...
 
CONFIGURE NETFLOW STORAGE AND DISPLAY WITH NFDUMP AND NfSen

The net-mgmt/nfdump and net-mgmt/nfsen tools are best described by their respective websites:

The components which are used by this HowTo guide include:
  • nfcapd (part of the net-mgmt/nfdump port): Netflow capture daemon. Reads the netflow data from the network and stores the data into files. Automatically rotate files every n minutes. ( typically ever 5 min ) nfcapd reads netflow v5, v7 and v9 flows transparently. You need one nfcapd process for each netflow stream.
  • nfdump (part of the net-mgmt/nfdump port): Netflow dump. Reads the netflow data from the files stored by nfcapd. It's syntax is similar to tcpdump. If you like tcpdump you will like nfdump. Displays netflow data and can create lots of top N statistics of flows IP addresses, ports etc ordered by whatever order you like.
  • nfsen (part of the net-mgmt/nfsen port): Graphical web based frontend for navigating and displaying netflow data with RRD graphs. Also includes Perl scripts to start/stop the nfcapd daemon and interpret the data retrieved by nfdump.

The FreeBSD net-mgmt/nfsen port includes an rc script for managing the nfsen service, which in turns starts the nfcapd daemon. Whilst the net-mgmt/nfdump port can be used without the net-mgmt/nfsen port, it does not include an rc script, leaving it to the user to start the nfcapd service.

Installing and configuring both NFDUMP and NfSen can be achieved with the following steps:
  1. Install the net-mgmt/nfsen port (this will automatically install the net-mgmt/nfdump port as a dependency).

    Code: 
    # pkg install nfsen

  2. Edit the configuration file /usr/local/etc/nfsen.conf to suit your needs. At a minimum, the %sources section at line 162 must be edited to suit the netflow data being collected:

    Code: 
    sources = (
    'freebsd'    => { 'port' => '4444', 'col' => '#0000ff', 'type' => 'netflow' },
);

    Note that by default NfSen saves flow data to /usr/local/var/nfsen. Steps 3-5 explain how to change this to a different location (/zstore/nfsen in this example). If the default location is OK for you, skip to step 6.

    If you need to change the default location, refer to the /usr/local/etc/nfsen.conf example at this GitHub link. Only 2 changes are really required:

    Code: 
    $PIDDIR="/var/run/nfsen";  // Use the standard system directory for the PID and BSD sockets files
$VARDIR="/zstore/nfsen";   // Storage directory for flow data and nfsen profiles

  3. Move the PIDDIR created by the port installation from the default location to the new location.

    Code: 
    # mv /usr/local/var/nfsen/run /var/run/nfsen

  4. Move the storage directories created by the port installation from the default location to the new location.

    Code: 
    # mv /usr/local/var/nfsen /zstore/nfsen

  5. Edit the /usr/local/etc/rc.d/nfsen rc script to update the VARDIR location in the nfsen_start_precmd() function on lines 34 and 50.

    Code: 
    nfsen_start_precmd()
{
        # This replicates functionality provided by
        # the perl install script that we're bypassing.
        # It also is better because you don't have to reinstall
        # to get this file back
        if [ ! -e /zstore/nfsen/profiles-stat/live/profile.dat ]; then
                time=$(/bin/date +%s);
                timeu=$(/bin/expr $time - 300);
                /usr/bin/printf "name = live
                group = .
                tbegin = $time
                tcreate = $time
                tstart = $time
                tend = $time
                updated = $timeu
                expire = 0
                maxsize = 0
                size = 0
                type = 0
                locked = 0
                status = OK
                version = 130" | /usr/bin/tr -d '\011' > /zstore/nfsen/profiles-stat/live/profile.dat;
        fi
}


  6. Ensure the user and group being used to run nfcapd and nfsen have write access to the PIDDIR and VARDIR directories.

    Code: 
    // If using the default location
# chown -R www:www /usr/local/var/nfsen

// If using a non-default location (per steps 3-5)
# chown -R www:www /var/run/nfsen
# chown -R www:www /zstore/nfsen

  7. Add the NfSen service to /etc/rc.conf:

    Code: 
    nfsen_enable="YES"

  8. Attempt to start NfSen for the first time to create the profile.dat file.

    Code: 
    # service nfsen start

  9. Configure NfSen to accept the Netflow sources specified in /usr/local/etc/nfsen.conf:

    Code: 
    # nfsen reconfig

  10. Start the NfSen data collection service:

    Code: 
    # service nfsen start

  11. Check that the nfcapd, nfsend and nfsend-comm processes are running:

    Code: 
    % ps aux | grep /usr/local/bin/nf
www      79240   0.3  0.2 162128  28424  -  Ss   12:47pm      1:10.31 /usr/local/bin/perl /usr/local/bin/nfsend
www      79238   0.0  0.0  33220   3044  -  S    12:47pm      0:01.69 /usr/local/bin/nfcapd -w -D -p 4444 -u www -g www -B 200000 -S 1 -P /var/run/nfsen/p4444.pid -z -I freenas -l
www      79241   0.0  0.2 155544  25780  -  Ss   12:47pm      0:03.32 /usr/local/bin/nfsend-comm (perl)
woodsb02 83355   0.0  0.0  18804   2304  1  S+    9:35pm      0:00.00 grep /usr/local/bin/nf


  12. Check the log files for errors:

    Code: 
    #tail /var/log/messages

  13. After 10 minutes or so, check that Netflow data is being saved in the VARDIR directory:

    Code: 
    % ls -lah /zstore/nfsen/profiles-data/live/freenas/2014/12/28/
total 7031
drwxr-xr-x  2 www  www   109B 28 Dec 21:35 .
drwxr-xr-x  3 www  www     3B 28 Dec 12:41 ..
-rw-r--r--  1 www  www    18K 28 Dec 12:45 nfcapd.201412281240
-rw-r--r--  1 www  www    17K 28 Dec 12:50 nfcapd.201412281245

  14. Having waited the 10 minutes for some data to be collected, list the top 10 flows ordered by bytes:

    Code: 
    % nfdump -R /zstore/nfsen/profiles-data/live/freenas/ -o extended -s record/bytes
Aggregated flows 208132
Top 10 flows ordered by bytes:
Date first seen          Duration Proto      Src IP Addr:Port          Dst IP Addr:Port   Flags Tos  Packets    Bytes      pps      bps    Bpp Flows
2014-12-28 19:01:59.381  2306.145 TCP        81.209.35.6:64539 ->      192.168.1.5:51413 .AP.SF   0   215400  271.9 M       93   943051   1262     2
2014-12-28 19:57:29.946  1707.000 TCP        192.168.1.5:51413 ->  121.131.109.103:52085 .AP.S.   0    80435  117.4 M       47   550066   1459     1
2014-12-28 19:31:21.314  4884.446 UDP        192.168.1.5:51413 ->   220.237.23.116:24367 ......   0    78863  110.1 M       16   180256   1395     3
2014-12-28 19:21:10.526  1155.000 TCP      80.229.17.226:64990 ->      192.168.1.5:51413 .AP.SF   0    71342  103.9 M       61   719415   1455     1
2014-12-28 19:12:13.526  1692.000 TCP      79.161.91.136:45984 ->      192.168.1.5:51413 .AP.SF   0    51944   75.6 M       30   357290   1454     1
2014-12-28 19:05:08.061   468.000 TCP     217.164.255.37:41354 ->      192.168.1.5:51413 .APRS.   0    47782   69.6 M      102    1.2 M   1457     1
2014-12-28 19:16:01.526  1464.000 TCP     122.107.99.144:61637 ->      192.168.1.5:51413 .AP.SF   0    40439   44.4 M       27   242709   1098     1
2014-12-28 19:57:12.469  1062.000 TCP        192.168.1.5:51413 ->   121.217.108.67:55559 .AP.SF   0    27428   39.9 M       25   300719   1455     1
2014-12-28 19:27:32.526   773.000 TCP      107.184.39.95:59243 ->      192.168.1.5:51413 .AP.SF   0    23260   33.3 M       30   345118   1433     1
2014-12-28 19:05:39.833   929.000 TCP     121.99.173.211:57438 ->      192.168.1.5:51413 .AP.S.   0    24628   30.3 M       26   261020   1230     1
Summary: total flows: 401435, total bytes: 3971465381, total packets: 6350558, avg bps: 924514, avg pps: 184, avg bpp: 625
Time window: 2014-12-28 12:12:13 - 2014-12-28 21:44:59
Total flows processed: 401435, Blocks skipped: 0, Bytes read: 22493440
Sys: 0.347s flows/second: 1156506.6  Wall: 0.339s flows/second: 1181772.2

HowTo continued in next post...
 
CONFIGURE NGINX WEB SERVER TO DISPLAY NFSEN

The NfSen graphical web frontend requires a web server running with PHP support running on the same machine as the net-mgmt/nfsen software. This HowTo guide explains how to do this with www/nginx, but it could equally be done with www/apache2.

  1. Install Nginx and PHP:

    Code: 
    # pkg install nginx php5 php5-extensions

  2. Configure PHP by editing the /usr/local/etc/php.ini file, by initially copying the /usr/local/etc/php.ini-production file:

    Code: 
    # cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini

  3. Configure Nginx by editing the /usr/local/etc/nginx/nginx.conf file. A basic example file is shown below, with the important parts being the definition or root and the handling of .php files:

    Code: 
    user www www;
worker_processes 4; 

events {
        worker_connections 1024;
} 

http {
       include mime.types;
       default_type application/octet-stream;
       sendfile on;
       keepalive_timeout 65;

       server {
              listen 80;
              server_name localhost; 
              root /usr/local/www;

              #error_page 404 /404.html;
              # redirect server error pages to the static page /50x.html
              #
              error_page 500 502 503 504 /50x.html;
              location = /50x.html {
                     root /usr/local/www/nginx-dist;
              } 

              # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
              #
              location ~ \.php$ {
                     fastcgi_pass 127.0.0.1:9000;
                     try_files $uri =404;
                     fastcgi_index index.php;
                     fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                     include fastcgi_params;
              } 

              # deny access to .htaccess files, if Apache's document root
              # concurs with nginx's one
              #
              location ~ /\.ht {
                     deny all;
              }
       }
}

  4. Add the Nginx and PHP services to /etc/rc.conf:

    Code: 
    nginx_enable="YES"
php_fpm_enable="YES"

  5. Start the Nginx and PHP services:

    Code: 
    # service php-fpm start
# service nginx start

  6. Test the webserver from another computer on the network, by using a web browser to navigate to the URL http://YOUR.FREEBSD.IP.ADDRESS/nfsen/nfsen.php

That's everything - enjoy your network traffic monitoring with NfSen! Note that the web page often shows an error "Frontend - Backend version missmatch!", but this can be safely ignored with no loss of functionality.
 
You must log in or register to reply here.
Back
Top