Other Compiling Suricata to work with DPDK

HackTheMa/K · Jun 26, 2024

As the cliche goes, long time lurker...
Until getting into problem that can't seem to get past, even with some GPT advisory.

I am attempting to experiment using Suricata (v7.0.5) with DPDK, but keep running into hangups. The latest method seemed promising which was starting with a Virtual Machine of a clean FreeBSD latest version (v14), and installing DPDK first by means of compiling from the Ports Tree, then the same for Suricata. DPDK compiled successfully. The problem is with compiling Suricata:
- I have edited the Makefile file to allow for the enabling of DPDK (--enable-dpdk).
- When attempting to compile, an error occurs that there is no "libnuma" available, and makes references for how to install with Linux Distros. This is the headache as when to compile with enabling DPDK, the configure script checks for numa. Without enabling for DPDK, there appears to be no issues.
-- I then have tried installing the linux-c7-numctl-libs, along with configuring for use Linux support compatibility. This does not resolve the issue.
- I have modified the suricata configure.ac file to bypass the testing of the presence of numa. This looks promising until requiring numa.h file associated with DPDK.
-- Having edited that out, then run into another error of an undeclared identifier of 'UNIT16_WIDTH' which I assume is in relation due not including the numa.h file.

Outside of compiling, have tried to install in combinations of both DPDK and Suricata via pkg, and along with possible options (there is an option for Netmap, but not for DPDK) but that has failed as well.
Perhaps I am over thinking this, in that I just need to enter in the DPDK configuration into the Suricata,yml file, but that did not work either, unless I did that wrong.
Or missing some environmental variables?

Or is it that in order to compile Suricata with DPDK, it has to be done in Linux and then ported over into FreeBSD, as I see a number of references in the configure files with Linux but little with FreeBSD?

So I am at stopping point currently and need some advice. I am willing to document the process so others can perform the same procedure.

cracauer@ · Jun 26, 2024

You will have no choice but to hack it up not requiring libnuma.

HackTheMa/K · Jun 27, 2024

I was thinking that would be the case, but hoping that perhaps not.
Ok then, made some edits to files and had completed the compiling and now to test it out.

Four files modified (/usr/ports/security/suricata:
1 - Makefile: Added --enable-dpdk

Code:

CONFIGURE_ARGS+=    --disable-gccmarch-native \
                    --enable-bundled-htp \
                    --enable-dpdk \
                    --enable-gccprotect

2 - configure.ac: Committed out references to numa and changed one line:

Code:

  # DPDK support
    enable_dpdk_bond_pmd="no"
    AC_ARG_ENABLE(dpdk,
            AS_HELP_STRING([--enable-dpdk], [Enable DPDK support [default=no]]),
                        [enable_dpdk=$enableval],[enable_dpdk=no])
    AS_IF([test "x$enable_dpdk" = "xyes"], [
# Committed out below
#         AC_CHECK_LIB(numa, numa_available,, [numa_found="no"])
#         if test "$numa_found" = "no"; then
#            echo
#            echo "  ERROR! libnuma not found by pkg-config, go get it"
#            echo "      from http://github.com/numactl/numactl or your distribution:"
#            echo "          Ubuntu: apt-get install libnuma-dev"
#            echo "          Fedora: dnf install numactl-devel"
#            echo "          CentOS/RHEL: yum install numactl-devel"
#            echo
#            exit 1
#        fi

Changed the LIBS line to:

Code:

LIBS="${LIBS} `pkg-config --libs libdpdk`"

In the /usr/ports/security/suricata/work/suricata-7.0.5/src folder:
3 - source-dpdk.c: Add lines to deal with the undeclared UINT16_WIDTH and remove reference to numa.h

Code:

#include "util-privs.h"
#include "action-globals.h"
// Modified Code Here
#include <stdint.h>

#inndef UINT16_WIDTH
#define UINT16_WIDTH 16
#endif
// End Modified Code
#ifndef HAVE_DPDK

Find "#include <numa.h>" and commit it out.

Code:

// #include <numa.h>

4 - util-affinity.c: Remove reference of "no member __bits" in "struct ThreadsAffinityType" error
Committed out: #define _THREAD_AFFINITY

Code:

//#define _THREAD_AFFINITY

Committed out with CPU_ZERO:

Code:

uint16_t UtilAffinityCpusOverlap(ThreadsAffinityType *taf1, ThreadsAffinityType *taf2)
{
// Committed out the following
//    ThreadsAffinityType tmptaf;
//    CPU_ZERO(&tmptaf);
//    SCMutexInit(&tmptaf.taf_mutex, NULL);

    cpu_set_t tmpcset;

    SCMutexLock(&taf1->taf_mutex);
    SCMutexLock(&taf2->taf_mutex);
    CPU_AND(&tmpcset, &taf1->cpu_set, &taf2->cpu_set);
    SCMutexUnlock(&taf2->taf_mutex);
    SCMutexUnlock(&taf1->taf_mutex);

    for (int i = UtilCpuGetNumProcessorsOnline(); i >= 0; i--)
        if (CPU_ISSET(i, &tmpcset))
            return 1;
    return 0;
}

HackTheMa/K · Jul 2, 2024

Was able to compile and run DPDK with Suricata and below is the steps to do so, for anyone else interested. This was done directly on a server (hardware).

Perform a clean install of FreeBSD (v14).
During the installation process, for System Components, included Ports and Src.
The rest of the installation process were of defaults (ex: Keymap-Default, Auto (ZFS), DHCP, etc)
Added no other users, just root and no further items (Security was not important here, for a clean machine and to minimize potential problems)
After installation, reboot and log in.

Perform Configuration and Updates

Perform Updates:

Code:

pkg install updates
pkg install upgrades
pkg install nano

Enable SSH for Root:
Code:
```
nano /etc/ssh/sshd_config
```
Find the following line:
Code:
```
#PermitRootLogin no
```
Uncomment it by removing the hash (#) symbol and change it's value to yes like below:
Code:
```
PermitRootLogin yes
```
CTRL-O (Write) and CTRL-X (Exit)
Restart SSH:
Code:
```
/etc/rc.d/sshd restart
```
To check for the IP, type:
Code:
```
ifconfig
```
Open up PUTTY or SSH to log into the FreeBSD machine.

To compile from the Ports Compile DPDK

DPDK is required to be installed first before Suricata. There are two methods, by installing from the Package Manager or compiling from Ports. Compiling from Ports is recommended as this will help with DPDK being more suited to the platform being used.

To compile from the Ports, at the terminal type:
Code:
```
cd /usr/ports/net/dpdk
make install
```
During the compiling, there will be some prompts, use to the following:

Code:

Ports Collection:        Default
Py39-pyrlftools-0.30:    Default
Ninja-1.11.1,2:            Uncheck Docs
Pkgconf02.2.0,1:        Uncheck Docs
Libpcap-1.10.4,2:        Default
Flex-2.6.4_3:            Uncheck Docs
Help2man-1.49.3_1:        Default
Gettext-tools-0.22.5:    Uncheck Docs
Libtextstyle-0.22.5:    Uncheck Docs
Gmake-4.4.1:            Default
Textinfo-7.1.2,1:        Default
P5-Locale-libintl-1.33:    Default
Bison-3.8.2_1,1:        Uncheck Docs
M4-1.4.19_1,1:            Default
Yasm-1.3.0_1:            Default
Autoconf-2.72:            Default
Automake-1.16.5_1:        Uncheck Docs

Configure DPDK:
1. Determine which NICs to use:
  Code:
```
pciconf -lv | grep -B3 'class=0x02 '
```
2. Look for the interfaces, such as ixl0, ixl1, igb0, igb1, etc
3. Make note of:
  1. The first column constitutes three components:
  2. Device name: igb0
  3. Unit name: pci0
  4. Selector (Bus : Device : Function): 5:0:0
  5. Just the Selectors are needed.
  6. Example (ixl0 and ixl1): 3:0:0,3:0:1
4. Edit the Loader.conf file
  Code:
```
nano /boot/loader.conf
```
  1. Add these lines at the end:
    Code:
    # DPDK: Reserve One 1G block of contiguous memory using contigmem driver hw.contigmem.num_buffers=1 hw.contigmem.buffer_size=1073741824 contigmem_load="YES" # Identify NIC devices for DPDK apps to use and load nic_uio driver: hw.nic_uio.bdfs="3:0:0,3:0:1" nic_uio_load="YES"
5. CTRL-O (Write) and CTRL-X (Exit)
6. Reboot
SSH or Log back in again.

Verify Contimem Variables:

Code:

sysctl -a hw.contigmem

Example output:

Code:

hw.contigmem.physaddr.0: 5368709120
hw.contigmem.num_references: 0
hw.contigmem.buffer_size: 1073741824
hw.contigmem.num_buffers: 1

After the reboot, the NICs that were used for DPDK are now listed as nic_uio#.

Example:

Code:

pciconf -l | more

Code:

...
nic_uio0@pci0:3:0:0: class=0x020000 rev=...
nic_uio1@pci0:3:0:1: class=0x020000 rev=...
igb0@pci0:5:0:0: class=0x020000 rev=...
igb1@pci0:5:0:1: class=0x020000 rev=...
...

Test DPDK with Helloworld Applcation

Code:

cp -r /usr/local/share/dpdk/examples/helloworld .
cd helloworld/
gmake
cc -O3 -I/usr/local/include -include rte_config.h -D__BSD_VISIBLE main.c -o build/helloworld-shared -L/usr/local/lib -lrte_bpf -lrte_pipeline -lrte_table -lrte_port -lrte_fib -lrte_ipsec -lrte_stack -lrte_security -lrte_sched -lrte_reorder -lrte_rib -lrte_rcu -lrte_rawdev -lrte_pdump -lrte_member -lrte_lpm -lrte_latencystats -lrte_jobstats -lrte_ip_frag -lrte_gso -lrte_gro -lrte_eventdev -lrte_efd -lrte_distributor -lrte_cryptodev -lrte_compressdev -lrte_cfgfile -lrte_bitratestats -lrte_bbdev -lrte_acl -lrte_timer -lrte_hash -lrte_metrics -lrte_cmdline -lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_ring -lrte_eal -lrte_kvargs

Code:

ln -sf helloworld-shared build/helloworld
./build/helloworld -l 0-3

Output:

Code:

EAL: Sysctl reports 12 cpus
EAL: Detected CPU lcores: 12
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Contigmem driver has 1 buffers, each of size 1GB
EAL: Mapped memory segment 0 @ 0x1180000000: physaddr:0x140000000, len 1073741824
EAL: Probe PCI driver: net_ixgbe (8086:15ad) device: 0000:03:00.0 (socket -1)
EAL: Probe PCI driver: net_ixgbe (8086:15ad) device: 0000:03:00.1 (socket -1)
EAL: 0000:05:00.0 not managed by UIO driver, skipping
EAL: 0000:05:00.1 not managed by UIO driver, skipping
hello from core 1
hello from core 2
hello from core 3
hello from core 0

To compile from the Ports Compile Suricata

Suricata needs to be compiled to allow use of DPDK. However, Suricata in FreeBSD is not coded properly, which relies still with Linux. The follow steps show what is needed to prepare to compile Suricata.

Modified Suricata Makefile file:

Code:

nano /usr/ports/security/suricata/Makefile

Look or Search (CTRL W) for: CONFIGURE_ARGS

Add this line of “--enable-dpdk \” so that CONFIGURE_ARGS looks like the following:

Code:

CONFIGURE_ARGS+=    --disable-gccmarch-native \
                    --enable-bundled-htp \
                    --enable-dpdk \
                    --enable-gccprotect

CTRL-O (Write) and CTRL-X (Exit)

Execute Make:
Code:
```
make
```
1. Suricata-7.0.5 Port Window, select:
  GEOIP
  HYPERSCAN
  IPFW
  NETMAP
  NSS
  PORTS_PCAP
  PYTHON
  Hit Enter for OK
2. Rust01.77.0 Port Window, select:
  SOURCES
  WASM
  Hit Enter for OK
3. Cmake0core-3.28.3 Port Window, select:
  CPACK
  Hit Enter for OK
4. Rhash01.4.4_1 Port Window, select:
  NLS
  Hit Enter for OK
5. Curl-8.7.1 Port Window, select:
  Defaults
  Hit Enter for OK
6. Libssh2-1.11.0_1,3 Port Window, select:
  Defaults
  Hit Enter for OK
7. Libpsi-0.21.5 Port Window, select:
  Defaults
  Hit Enter for OK
  Long wait
8. Py39-yaml-6.0.1 Port Window, select:
  Defaults
9. Hit Enter for OK
  Libnet-1.3,1 Port Window, select:
  BPF
  Examples
  Hit Enter for OK
10. Hyperscan-5.4.0 Port Window, select:
  Defaults
  Hit Enter for OK
11. Rage1-6.10_1 Port Window, select:
  Defaults
  Hit Enter for OK

Error Occurs:

Code:

checking for numa_available in -lnuma... no
ERROR! libnuma not found by pkg-config, go get it
...
*** Error code 1

The error occurs due to checking for Numa Library files which are not present in FreeBSD, but are in Lunix.

Modify the configure.ac file to commit out references to Numa.

Code:

nano work/suricata-7.0.5/configure.ac

Look or Search (CTRL-W) for “DPDK”. Modify the lines as shown below”

Code:

# DPDK supportenable_dpdk_bond_pmd="no"
AC_ARG_ENABLE(dpdk,
AS_HELP_STRING([--enable-dpdk], [Enable DPDK support [default=no]]),
[enable_dpdk=$enableval],[enable_dpdk=no])
AS_IF([test "x$enable_dpdk" = "xyes"], [
    # Committed out below
    # AC_CHECK_LIB(numa, numa_available,, [numa_found="no"])
    # if test "$numa_found" = "no"; then
        # echo
        # echo " ERROR! libnuma not found by pkg-config, go get it"
        # echo " from http://github.com/numactl/numactl or your distribution:"
        # echo " Ubuntu: apt-get install libnuma-dev"
        # echo " Fedora: dnf install numactl-devel"
        # echo " CentOS/RHEL: yum install numactl-devel"
        # echo
        # exit 1
    # fi

Scroll down and look for the following line:

Code:

LIBS="${LIBS} -Wl,-R,`pkg-config --libs-only-L libdpdk | cut -c 3-` -lnuma `pkg-config --libs libdpdk`"

Change to:

Code:

LIBS="${LIBS} `pkg-config --libs libdpdk`"

CTRL-O (Write) and CTRL-X (Exit)

Modify the source-dpdk.c file to commit out references to Numa.
Code:
```
nano work/suricata-7.0.5/src/source.dpdk.c
```
1. Scroll down to the line of:
  Code:
```
#include "action-globals.h"
```
2. Add the following lines afterward:
  Code:
```
// Modified DPDK Code Here
#include <stdint.h>

#ifndef UINT16_WIDTH
#define UINT16_WIDTH 16
#endif
// End Modified DPDK Code
```
3. Look or Search (CTRL-W) for: “numa.h”
4. Find the line of:
  Code:
```
#include <numa.h>
```
5. Change to:
  Code:
```
// #include <numa.h>
```
6. CTRL-O (Write) and CTRL-X (Exit)
Modify the util-affinity.c file to commit out references to CPU_ZERO:
Code:
```
nano work/suricata-7.0.5/src/util-affinity.c
```
1. Scroll down to the line of: #define _THREAD_AFFINITY
2. Change the line to:
  Code:
```
// #define _THREAD_AFFINITY
```
3. Look or Search (CTRL-W) for: “DPDK”
4. Scroll down to the line of “uint16_t UtilAffinityCpusOverlap(ThreadsAffinityType *taf1, ThreadsAffinityType *taf2)”
5. Commit the following lines of:
  Code:
```
ThreadsAffinityType tmptaf;
CPU_ZERO(&tmptaf);
SCMutexInit(&tmptaf.taf_mutex, NULL);
```
6. Change to:
  Code:
```
// ThreadsAffinityType tmptaf;
// CPU_ZERO(&tmptaf);
// SCMutexInit(&tmptaf.taf_mutex, NULL);
```
7. CTRL-O (Write) and CTRL-X (Exit)
Recompile again, by typing:
Code:
```
make
```
Now install, by typing:
Code:
```
make install
```

Configure Suricata to use DPDK

In order for Suricata to use DPDK, the suricata.yaml file needs to be edited. But before doing so, the Network Interfaces need to be changed into another format.
Currently they are referred to as 3:0:0 and 3:0:1 (in this example). They will need to be into the format of 0000/NN:00.X.
Thus, they become in this case: 0000/03:00.0 and 0000/03:00.1.

Edit the suricata.yaml file.
Code:
```
nano /usr/local/etc/suricata/suricata.yaml
```
1. Look or Search (CTRL-W) for: "dpdk"
2. Add or modify the line with "allow" to allow for the interface(s) to be used.
  Code:
```
dpdk:  eal-params:
    proc-type: primary
    allow: ["0000:03:00.0", "0000:03:00.1"]
```
3. Then go down a couple of lines and modify the line begining with "-interface:" to include one (or the only one) of the interfaces:
  Code:
```
 interfaces:    
  - interface: 0000:03:00.0 # PCIe address of the NIC port
```
4. Leave all other settings as is.
5. If to add in another interface, then scroll down to the section begining with "-interface: default".
6. Copy the entire section of the -interface: default and paste it above as a new section.
7. Then edit the word "default" with the identifier of the other network interfaces (ex: 0000/03:00.1).
8. It should now look like the following:
  Code:
```
    - interface: 0000:03:00.1
      threads: 1
      promisc: true
      multicast: true
      checksum-checks: true
      checksum-checks-offload: true
      mtu: 1500
      rss-hash-functions: auto
      mempool-size: 65535
      mempool-cache-size: 257
      rx-descriptors: 1024
      tx-descriptors: 1024
      copy-mode: none
      copy-iface: none

    - interface: default
      threads: auto
      promisc: true
      multicast: true
      checksum-checks: true
      checksum-checks-offload: true
      mtu: 1500
      rss-hash-functions: auto
      mempool-size: 65535
      mempool-cache-size: 257
      rx-descriptors: 1024
      tx-descriptors: 1024
      copy-mode: none
      copy-iface: none
```
9. Now to allow for CPU Cores/Threads to be used.
10. Scroll down or Search (CTRL W) for "set-cpu-affinity".
11. Change the value from "no" to "yes"
  Code:
```
 set-cpu-affinity: yes
```
12. Then look afterwards for "worker-cpu-set".
13. To avoid allocating all CPU Cores/Threads, change from "all" to the designated Core/Threads. For example. To use one thread per interface, [ 2, 4].
  Code:
```
- worker-cpu-set:
    # cpu: [ "all" ]
    cpu: [ 2,4 ]
```
14. CTRL-O (Write) and CTRL-X (Exit)

Run Suricata

To run Suricata with DPDK, type:

Code:

suricata --dpdk

HackTheMa/K · Jul 2, 2024

Have a few more items to deal with,:

Enable use of IPS Mode (requires a recompile from the looks of it using; "--enable-nfqueue \" in the CONFIGURE_ARGS section of the Makefile.
Numa Node remains at 1 and can't seem to change it unless to recompile the OS in relation to MAXMEMDOM?
DPDK NICs do not have IP Addressing, unless through an application; however, there are compiling errors with those other applications (ex: Internet Protocol (IP) Pipeline application).

I have read up on a porting of VPP (which allows for IP Addressing), with the latest article from the 11th of June, which looks like it is about ready, but have seen no codes or procedures of how to test out.
So going to stop here, until VPP becomes available.
Open to anyother thoughts or recommendations.

Other Compiling Suricata to work with DPDK

HackTheMa/K

cracauer@

HackTheMa/K

HackTheMa/K

Perform Configuration and Updates​

To compile from the Ports Compile DPDK​

To compile from the Ports Compile Suricata​

Configure Suricata to use DPDK​

Run Suricata​

HackTheMa/K

Perform Configuration and Updates

To compile from the Ports Compile DPDK

To compile from the Ports Compile Suricata

Configure Suricata to use DPDK

Run Suricata