(I've read other threads in the forum, but they didn't help me figure out this.)
Since a few days my server has regularly logged many messages like the following ones every 20 minutes:
There, xx.xx.xx.100 is the IP address of my server and yy:yy:yy:yy:yy:yy is the MAC address of the physical interface which the jail is bridged to (see below), while 02:0e:b6:59:43:0a is the MAC of the epair120a interface (see below). I have checked for IP conflicts (found none), and I've turned off the DHCP server on the xx.xx.xx.0 subnet for good measure (no clients were using DHCP anyway, afaict).
This has suddenly started without an apparent reason: I can't find any clue in the system log (messages have appeared ten days ago at noon, and the previous recorded log message was from the day before, and totally unrelated: I didn't touch the system in between).
I've noticed that if I remove the jail and I create it again, a message like the following is logged almost immediately:
Why is that on a different interface (epair124b)? That doesn't make much sense to me: the updated MAC is the one of epair120b inside the jail.
I have a dozen other jails configured similarly to this one and attached to the same bridge, but only this one triggers those messages. FWIW, it runs Prometheus. AFAICS, I don't have any connectivity issues or other networking problems anywhere, and all my services are up and running just fine. What could be causing those messages?
Some more details:
Since a few days my server has regularly logged many messages like the following ones every 20 minutes:
Code:
Nov 1 16:24:58 myserver kernel: arp: xx.xx.xx.100 moved from 02:0e:b6:59:43:0a to yy:yy:yy:yy:yy:yy on epair120b
Nov 1 16:44:58 myserver kernel: arp: xx.xx.xx.100 moved from yy:yy:yy:yy:yy:yy to 02:0e:b6:59:43:0a on epair120b
Nov 1 17:04:58 myserver syslogd: last message repeated 1 times
Nov 1 17:24:58 myserver syslogd: last message repeated 1 times
Nov 1 17:44:59 myserver kernel: arp: xx.xx.xx.100 moved from 02:0e:b6:59:43:0a to yy:yy:yy:yy:yy:yy on epair120b
Nov 1 18:04:59 myserver kernel: arp: xx.xx.xx.100 moved from yy:yy:yy:yy:yy:yy to 02:0e:b6:59:43:0a on epair120b
Nov 1 18:24:59 myserver kernel: arp: xx.xx.xx.100 moved from yy:yy:yy:yy:yy:yy to 02:0e:b6:59:43:0a on epair120b
Nov 1 18:44:59 myserver kernel: arp: xx.xx.xx.100 moved from 02:0e:b6:59:43:0a to yy:yy:yy:yy:yy:yy on epair120b
Nov 1 19:04:59 myserver kernel: arp: xx.xx.xx.100 moved from yy:yy:yy:yy:yy:yy to 02:0e:b6:59:43:0a on epair120bThere, xx.xx.xx.100 is the IP address of my server and yy:yy:yy:yy:yy:yy is the MAC address of the physical interface which the jail is bridged to (see below), while 02:0e:b6:59:43:0a is the MAC of the epair120a interface (see below). I have checked for IP conflicts (found none), and I've turned off the DHCP server on the xx.xx.xx.0 subnet for good measure (no clients were using DHCP anyway, afaict).
This has suddenly started without an apparent reason: I can't find any clue in the system log (messages have appeared ten days ago at noon, and the previous recorded log message was from the day before, and totally unrelated: I didn't touch the system in between).
I've noticed that if I remove the jail and I create it again, a message like the following is logged almost immediately:
Code:
Nov 1 20:59:59 myserver kernel: arp: xx.xx.xx.120 moved from 02:ec:5d:e5:c5:0b to 02:34:30:29:62:0b on epair124bWhy is that on a different interface (epair124b)? That doesn't make much sense to me: the updated MAC is the one of epair120b inside the jail.
I have a dozen other jails configured similarly to this one and attached to the same bridge, but only this one triggers those messages. FWIW, it runs Prometheus. AFAICS, I don't have any connectivity issues or other networking problems anywhere, and all my services are up and running just fine. What could be causing those messages?
Some more details:
Code:
# ifconfig -j prometheus
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x13
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=1000141<UP,RUNNING,PROMISC,LOWER_UP> metric 0 mtu 33152
options=0
groups: pflog
epair120b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:0e:b6:59:43:0b
inet xx.xx.xx.120 netmask 0xffffff00 broadcast xx.xx.xx.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
# ifconfig
jail0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,HWSTATS,MEXTPG>
ether yy:yy:yy:yy:yy:yy
inet xx:xx:xx.100 netmask 0xffffff00 broadcast xx.xx.xx.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
iot0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,HWSTATS,MEXTPG>
ether yy:yy:yy:yy:yy:zz
inet xx.xx.ww.100 netmask 0xffffff00 broadcast xx.xx.ww.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
jail0bridge: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=0
ether 58:9c:fc:10:1b:22
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
[…]
member: epair120a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 17 priority 128 path cost 2000
member: epair123a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 81 priority 128 path cost 2000
member: epair121a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 77 priority 128 path cost 2000
member: jail0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 55
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
iot0bridge: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=0
ether 58:9c:fc:00:2f:00
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair1010a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 73 priority 128 path cost 2000
member: iot0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 2000000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
pflog0: flags=1000141<UP,RUNNING,PROMISC,LOWER_UP> metric 0 mtu 33152
options=0
groups: pflog
epair121a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: jail:grafana
options=8<VLAN_MTU>
ether 02:6c:7f:30:5d:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair123a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: jail:something
options=8<VLAN_MTU>
ether 02:e4:f0:f9:5c:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair1010a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: jail:somethingelse
options=8<VLAN_MTU>
ether 02:79:29:96:1b:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair120a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: jail:prometheus
options=8<VLAN_MTU>
ether 02:0e:b6:59:43:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
[...]
# ps -J prometheus -ax
PID TT STAT TIME COMMAND
5266 - IsJ 0:01.02 /usr/sbin/syslogd -ss
22957 - IsJ 0:00.02 daemon: prometheus[23828] (daemon)
23828 - IJ 10:38.05 /usr/local/bin/prometheus --config.file=/usr/local/etc/prometheus.yml --web.console.templates=/usr/local/share/prometheus/consoles --web.console.libraries=/usr/local/share
26794 - IsJ 0:00.00 daemon: /usr/bin/env[27186] (daemon)
27186 - IJ 1:30.65 /usr/local/bin/node_exporter --web.listen-address=:9100 --collector.textfile.directory=/var/tmp/node_exporter --web.config.file=/usr/local/etc/node_exporter_config.yml
29440 - IsJ 0:00.00 daemon: blackbox_exporter[29668] (daemon)
29668 - IJ 31:43.10 /usr/local/bin/blackbox_exporter --web.listen-address=0.0.0.0:9115 --config.file=/usr/local/etc/blackbox_exporter.yml
36118 - IsJ 0:02.10 /usr/sbin/cron -J 60 -s
62577 - IsJ 0:00.00 pflogd: [priv] (pflogd)
62753 - SJ 0:17.63 pflogd: [running] -s 116 -i pflog0 -f /var/log/pflog (pflogd)
# cat /etc/jail.conf
$prefix = "/usr/local/jails";
$mount_dir = "${prefix}/mount";
# STARTUP/LOGGING
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_console_${name}.log";
# PERMISSIONS
allow.raw_sockets;
exec.clean;
mount.devfs;
securelevel = 2; # See security(7) and jail(8)
devfs_ruleset = 4; # For standard jails. See jail(8) and /etc/defaults/devfs.rules
enforce_statfs = 2; # Default, but let's be explicit
# HOSTNAME/PATH
host.hostname = "${name}";
path = "${mount_dir}/${name}";
# NETWORK
interface = jail0;
# MOUNT
mount.fstab = "${mount_dir}/${name}.fstab";
.include "/etc/jail.conf.d/*.conf";
# cat /etc/jail.d/prometheus.conf
prometheus {
$id = "120";
# NETWORKS/INTERFACES
$ip = "xx.xx.xx.${id}/24";
$gateway = "xx.xx.xx.1";
$bridge = "jail0bridge";
$epair = "epair${id}";
$epair_bridge = "${epair}a";
$epair_jail = "${epair}b";
devfs_ruleset = 14;
# VNET/VIMAGE
vnet;
vnet.interface = "${epair_jail}";
# ADD TO bridge INTERFACE
exec.prestart = "ifconfig ${epair} create up";
exec.prestart += "ifconfig ${epair_bridge} up descr jail:${name}";
exec.prestart += "ifconfig ${bridge} addm ${epair_bridge} up";
exec.start += "ifconfig ${epair_jail} ${ip} up";
exec.start += "route add default ${gateway}";
exec.poststop = "ifconfig ${bridge} deletem ${epair_bridge}";
exec.poststop += "ifconfig ${epair_bridge} destroy";
}
# cat /etc/devfs.rules
[lifepillar_vnet=14]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add include $devfsrules_jail_vnet
add path 'bpf*' unhide