Before the following bootup, /var/db/dhclient.leases.vtnet0 was removed, I also ran
Attention to these lines in particular:
DHCPACK from 0.0.0.0? Isn't that saying this host is acknowledging its own DHCP request? What's going on here?
And then after bootup...
I'm unable to replicate that "DHCPACK" after I've booted up, and obviously I'm unable to ping the default gateway or anything else, and have nothing in the ARP cache for anything either. A packet capture shows the ARP requests going out of the interface, but cannot see any other traffic.
I've tested all of the above on a CentOS box on the same platform (same VPS provider, same network) and I'm having no such problems - this is why I say this looks like an OS issue - correct MAC's have also been verified since I'm aware that the VPS provider implements layer 2 access control in this way, since I had to troubleshoot this before, though I remember seeing layer 2 traffic on the network even with a bad MAC before, which I'm unable to see now. The network configurations of this interface and a couple of others as well as VNET interfaces WERE all working fine on this host, I'm not sure what's happened here. Any advice?
arp -d -a and route flush.
Code:
Hypervisor: Origin = "KVMKVMKVM" [96/125]
real memory = 4294967296 (4096 MB)
avail memory = 4100857856 (3910 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <BOCHS BXPCAPIC>
random device not loaded; using insecure entropy
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 1.1> irqs 0-23 on motherboard
random: <Software, Yarrow> initialized
module_register_init: MOD_LOAD (vesa, 0xffffffff80d584f0, 0) error 19
kbd1 at kbdmux0
acpi0: <BOCHS BXPCRSDT> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 100000000 Hz quality 950
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc0c0-0xc0cf at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 1.3 (no driver attached)
virtio_pci0: <VirtIO PCI Balloon adapter> port 0xc080-0xc09f irq 10 at device 2.0 on pci0
vtballoon0: <VirtIO Balloon Adapter> on virtio_pci0
virtio_pci1: <VirtIO PCI Entropy adapter> port 0xc0a0-0xc0bf irq 11 at device 3.0 on pci0
virtio_pci2: <VirtIO PCI SCSI adapter> port 0xc000-0xc03f mem 0xfebc0000-0xfebc0fff irq 11 at device 4.0 on pci0
vtscsi0: <VirtIO SCSI Adapter> on virtio_pci2
virtio_pci3: <VirtIO PCI Network adapter> port 0xc040-0xc07f mem 0xfebc1000-0xfebc1fff irq 10 at device 5.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci3
vtnet0: Ethernet address: 00:16:3e:fe:69:c9
vtnet0: Ethernet address: 00:16:3e:fe:69:c9 [61/125]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
orm0: <ISA Option ROM> at iomem 0xed800-0xeffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: MDA <16 virtual consoles, flags=0x100>
vga0: <Generic ISA VGA> at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff on isa0
attimer0: <AT timer> at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
fdc0: No FDOUT register!
ppc0: cannot reserve I/O port range
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
da0 at vtscsi0 bus 0 scbus2 target 0 lun 0
da0: <gandi.ne QEMU HARDDISK 2.2.> Removable Direct Access SPC-3 SCSI device
da0: 4294967.295MB/s transfers
da0: Command Queueing enabled
da0: 25600MB (52428800 512 byte sectors: 255H 63S/T 3263C)
da1 at vtscsi0 bus 0 scbus2 target 1 lun 0
da1: <gandi.ne QEMU HARDDISK 2.2.> Removable Direct Access SPC-3 SCSI device
da1: 4294967.295MB/s transfers
da1: Command Queueing enabled
random: unblocking device. [24/125]
Timecounter "TSC-low" frequency 1300032396 Hz quality 800
Trying to mount root from zfs:zroot/ROOT/default []...
Setting hostuuid: 2313bdc1-444e-11e5-806c-00163e3e25b6.
Setting hostid: 0x4eb3b91f.
Entropy harvesting: interrupts ethernet point_to_point swi.
Starting file system checks:
Mounting local file systems:.
Writing entropy file:.
Setting hostname: ALCATRAZ.
vtnet0: link state changed to UP
Starting Network: lo0 vtnet0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:16:3e:fe:69:c9
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T <full-duplex>
status: active
Starting devd.
Starting dhclient.
DHCPREQUEST on vtnet0 to 255.255.255.255 port 67
DHCPACK from 0.0.0.0
bound to 9.8.7.14 -- renewal in 2147483647 seconds.
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
32-bit compatibility ldconfig path: /usr/lib32
Creating and/or trimming log files.
Starting syslogd.
No core dumps found.
Clearing /tmp (X related).
Updating motd:.
Mounting late file systems:.
Configuring syscons: blanktime.
Starting sendmail_submit.
Starting sendmail_msp_queue.
Starting cron.
sysctl: unknown oid 'net.link.bridge.pfil_onlyip' at line 12: No such file or directory
sysctl: unknown oid 'net.link.bridge.pfil_bridge' at line 13: No such file or directory
sysctl: unknown oid 'net.link.bridge.pfil_member' at line 14: No such file or directory
Starting background file system checks in 60 seconds.
Starting syslogd.
No core dumps found.
Clearing /tmp (X related).
Updating motd:.
Mounting late file systems:.
Configuring syscons: blanktime.
Starting sendmail_submit.
Starting sendmail_msp_queue.
Starting cron.
sysctl: unknown oid 'net.link.bridge.pfil_onlyip' at line 12: No such file or directory
sysctl: unknown oid 'net.link.bridge.pfil_bridge' at line 13: No such file or directory
sysctl: unknown oid 'net.link.bridge.pfil_member' at line 14: No such file or directory
Starting background file system checks in 60 seconds.Attention to these lines in particular:
Code:
Starting dhclient.
DHCPREQUEST on vtnet0 to 255.255.255.255 port 67
DHCPACK from 0.0.0.0
bound to 9.8.7.14 -- renewal in 2147483647 seconds.
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1DHCPACK from 0.0.0.0? Isn't that saying this host is acknowledging its own DHCP request? What's going on here?
And then after bootup...
Code:
netstat -r
Internet:
Destination Gateway Flags Netif Expire
default 9.8.7.254 UGS vtnet0
localhost link#2 UH lo0
9.8.7.0 link#1 U vtnet0
9.8.7.14 link#1 UHS lo0
Internet:
Destination Gateway Flags Netif Expire
default 9.8.7.254 UGS vtnet0
localhost link#2 UH lo0
9.8.7.0 link#1 U vtnet0
9.8.7.14 link#1 UHS lo0
root@ALCATRAZ:~ # ping 9.8.7.254
PING 9.8.7.254 (9.8.7.254): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
root@ALCATRAZ:~ # rm /var/db/dhclient.leases.vtnet0
root@ALCATRAZ:~ # service netif restart
Stopping dhclient.
Aug 29 04:58:08 ALCATRAZ dhclient[674]: connection closed
Aug 29 04:58:08 ALCATRAZ dhclient[674]: exiting.
Stopping Network: lo0 vtnet0.
lo0: flags=8048<LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:16:3e:fe:69:c9
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T <full-duplex>
status: active
Starting Network: lo0 vtnet0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:16:3e:fe:69:c9
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet 10Gbase-T <full-duplex>
status: active
root@ALCATRAZ:~ # dhclient vtnet0
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 7
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 8
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 7
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 15
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 12
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 12
No DHCPOFFERS received.
No working leases in persistent database - sleeping.
root@ALCATRAZ:~ # route add default 9.8.7.254
route: writing to routing socket: Network is unreachable
add net default: gateway 9.8.7.254 fib 0: Network is unreachable
root@ALCATRAZ:~ # ping 9.8.7.254
PING 9.8.7.254 (9.8.7.254): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
^C
--- 9.8.7.254 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
root@ALCATRAZ:~ # arp -a
root@ALCATRAZ:~ # arp -a
root@ALCATRAZ:~ # arp -a
root@ALCATRAZ:~ # arp -aI'm unable to replicate that "DHCPACK" after I've booted up, and obviously I'm unable to ping the default gateway or anything else, and have nothing in the ARP cache for anything either. A packet capture shows the ARP requests going out of the interface, but cannot see any other traffic.
I've tested all of the above on a CentOS box on the same platform (same VPS provider, same network) and I'm having no such problems - this is why I say this looks like an OS issue - correct MAC's have also been verified since I'm aware that the VPS provider implements layer 2 access control in this way, since I had to troubleshoot this before, though I remember seeing layer 2 traffic on the network even with a bad MAC before, which I'm unable to see now. The network configurations of this interface and a couple of others as well as VNET interfaces WERE all working fine on this host, I'm not sure what's happened here. Any advice?