Solved Sending syslog event outside of Jail

patpro

patpro

Hello,

I'm running a server (FreeBSD 10.1 x64) with a jail inside. The host runs splunk forwarder so that logs from various source are forwarded to remote splunk indexers.

host ifconfig:

Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 00:50:56:b8:f4:22
    inet 192.168.143.219 netmask 0xffffff00 broadcast 192.168.143.255
    inet 192.168.143.174 netmask 0xffffffff broadcast 192.168.143.174
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.174 netmask 0xffffffff
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160

jail ifconfig:

Code: 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
    ether 00:50:56:b8:f4:22
    inet 192.168.143.174 netmask 0xffffffff broadcast 192.168.143.174
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.174 netmask 0xffffffff
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160

jail syslog.conf ends with:

Code: 
!*
*.*                        @127.0.0.174:1234

syslogd runs with default settings (just -s).

And Splunk forwarder listens to *:1234

In jail, I have a warning when I restart syslogd:

Code: 
# service syslogd restart
Stopping syslogd.
Waiting for PIDS: 91938, 91938.
ln: /dev/log: Operation not permitted
Starting syslogd.

Unfortunately, logs from jail's syslog won't go to splunk, and I can't figure out why...
I'm quite lost here. Any hint?
 
From the host system, what does tcpdump -n -i lo1 port 1234 show? Do a logger test inside the jail. That should result in a log going to the port shown for Splunk.
 
mjollnir said:
Do you have in your host's /etc/sysctl.conf:
Code: 
security.jail.allow_raw_sockets=1
Click to expand...
No, but it's not useful here: I can successfully send syslog events to a remote server (outside of the box), and I can send syslog events to my syslog forwarder on the host but not always. It appears to depend on service start order or something else I don't understand. Sometimes it works, but most of the time it won't.
 
junovitch said:
From the host system, what does tcpdump -n -i lo1 port 1234 show? Do a logger test inside the jail. That should result in a log going to the port shown for Splunk.
Click to expand...
tcpdump sees nothing on lo1 but on lo0 it sees syslog events. I still have to fight for this, as it appears not to work out of the box after a reboot of the host. As I've written above I think it's related to service start order.
I think I'll rely on a remote splunk forwarder (out of host), because it works out of the box. Simpler -> more reliable.

thanks
 
Have you considered using the host's cron job to fetch the logs from jail directories and send it to remote server? This will make your job much easier.
 
Oh no, the job is really easier with splunk forwarder, especially because I have a full deployment (~350 servers). The lazy option would be to tell the splunk forwarder to monitor Jail's /var/log from the host and forward files changes to splunk indexers. But that's not so great either. Best option: syslogd sends Jail's events to splunk forwarder on host, second best option: syslogd sends Jail's events to remote splunk forwarder outside of host (this is working great).
 
Sorry for that late reply: it could be a loopback oddity, but I'm pretty sure now it's not such a good idea to send logs to Jail's IP address even though the splunk forwarder on the host listens to this address. Unfortunately I can't do more tests now, the server is in production.
I could have tried to send logs to the public IP address of the host, but it would have needed some work in pf.conf. I've solved my problem sending logs to a remote splunk forwarder server. Thanks
 
You must log in or register to reply here.
Back
Top