ena interface down and up repeatedly

Phishfry · Nov 22, 2018

Here is the AWS bug report regarding your problem:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225791

I see you commented there.
How about this for a patch until FreeBSD 12 hits:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229432#c3

I modified this for your interface:

As workaround, you can try to add to /etc/dhclient.conf

Code:

interface "ena0" {
  supersede interface-mtu 0;

}

So dhcpdump was good for seeing that dhcp option 26 is used on your network.

Jaehak Lee · Nov 22, 2018

Here are my configurations

/etc/rc.conf : removed mongod, postgresql, redis, nfs_server

Code:

# cat /etc/rc.conf
hostname="db-20"

ec2_configinit_enable=YES
ec2_fetchkey_enable=YES
ec2_ephemeralswap_enable=YES
ec2_loghostkey_enable=YES
firstboot_freebsd_update_enable=YES
firstboot_pkgs_enable=YES
growfs_enable="YES"
#ifconfig_DEFAULT="SYNCDHCP accept_rtadv"
ifconfig_DEFAULT="SYNCDHCP"
sshd_enable="YES"
firstboot_pkgs_list="awscli"
ipv6_activate_all_interfaces="YES"
dhclient_program="/usr/local/sbin/dual-dhclient"

#ifconfig_ena0="inet 10.1.20.20 netmask 255.255.255.0 mtu 1500"
#defaultrouter="10.1.20.1"

# user config start

# zfs
zfs_enable="YES"

# ntpdate
ntpdate_enable="YES"
#ntpdate_hosts="asia.pool.ntp.org"
ntpd_enable="YES"
ntpd_leapfile_sources="https://www.ietf.org/timezones/data/leap-seconds.list"

/boot/loader.conf

Code:

# cat /boot/loader.conf
autoboot_delay="-1"
beastie_disable="YES"
boot_multicons="YES"
hw.broken_txfifo="1"
if_ena_load="YES"

# ZFS
zfs_load="YES"
vfs.zfs.arc_meta_limit="1094967296"
vfs.zfs.arc_max="4294967296"
#vfs.zfs.arc_max="2147483648"

/var/db/dhclient.leases.ena0

Code:

# cat /var/db/dhclient.leases.ena0
lease {
  interface "ena0";
  fixed-address 10.1.20.20;
  option subnet-mask 255.255.255.0;
  option routers 10.1.20.1;
  option domain-name-servers 10.1.0.2;
  option host-name "ip-10-1-20-20";
  option domain-name "ap-northeast-1.compute.internal";
  option interface-mtu 9001;
  option broadcast-address 10.1.20.255;
  option dhcp-lease-time 3600;
  option dhcp-message-type 5;
  option dhcp-server-identifier 10.1.20.1;
  renew 4 2018/11/22 04:54:31;
  rebind 4 2018/11/22 05:17:01;
  expire 4 2018/11/22 05:24:31;
}
lease {
  interface "ena0";
  fixed-address 10.1.20.20;
  option subnet-mask 255.255.255.0;
  option routers 10.1.20.1;
  option domain-name-servers 10.1.0.2;
  option host-name "ip-10-1-20-20";
  option domain-name "ap-northeast-1.compute.internal";
  option interface-mtu 9001;
  option broadcast-address 10.1.20.255;
  option dhcp-lease-time 3600;
  option dhcp-message-type 5;
  option dhcp-server-identifier 10.1.20.1;
  renew 4 2018/11/22 05:24:31;
  rebind 4 2018/11/22 05:47:01;
  expire 4 2018/11/22 05:54:31;
}

And tcpdump result

Code:

# tcpdump -nnvXSs 0 -i ena0 '((port 67 or port 68) and (udp[38:4] = 0x064d4b64e186))'
tcpdump: listening on ena0, link-type EN10MB (Ethernet), capture size 262144 bytes



14:24:31.628441 IP (tos 0x10, ttl 128, id 59552, offset 0, flags [none], proto UDP (17), length 328, bad cksum 0 (->14de)!)
    10.1.20.20.68 > 10.1.20.1.67: BOOTP/DHCP, Request from 06:4d:4b:64:e1:86, length 300, xid 0x1a53c74d, Flags [none]
          Client-IP 10.1.20.20
          Client-Ethernet-Address 06:4d:4b:64:e1:86
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Request
            Client-ID Option 61, length 7: ether 06:4d:4b:64:e1:86
            Hostname Option 12, length 5: "db-20"
            Parameter-Request Option 55, length 10:
              Subnet-Mask, BR, Time-Zone, Classless-Static-Route
              Default-Gateway, Domain-Name, Domain-Name-Server, Hostname
              Option 119, MTU
        0x0000:  4510 0148 e8a0 0000 8011 0000 0a01 1414  E..H............
        0x0010:  0a01 1401 0044 0043 0134 ba48 0101 0600  .....D.C.4.H....
        0x0020:  1a53 c74d 0000 0000 0a01 1414 0000 0000  .S.M............
        0x0030:  0000 0000 0000 0000 064d 4b64 e186 0000  .........MKd....
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0090:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00f0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0100:  0000 0000 0000 0000 6382 5363 3501 033d  ........c.Sc5..=
        0x0110:  0701 064d 4b64 e186 0c05 6462 2d32 3037  ...MKd....db-207
        0x0120:  0a01 1c02 7903 0f06 0c77 1aff 0000 0000  ....y....w......
        0x0130:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0140:  0000 0000 0000 0000                      ........
14:24:31.628504 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 576)
    10.1.20.1.67 > 10.1.20.20.68: BOOTP/DHCP, Reply, length 548, xid 0x1a53c74d, Flags [none]
          Your-IP 10.1.20.20
          Client-Ethernet-Address 06:4d:4b:64:e1:86
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: ACK
            Server-ID Option 54, length 4: 10.1.20.1
            Lease-Time Option 51, length 4: 3600
            Subnet-Mask Option 1, length 4: 255.255.255.0
            BR Option 28, length 4: 10.1.20.255
            Domain-Name Option 15, length 31: "ap-northeast-1.compute.internal"
            Domain-Name-Server Option 6, length 4: 10.1.0.2
            Hostname Option 12, length 13: "ip-10-1-20-20"
            MTU Option 26, length 2: 9001
            Default-Gateway Option 3, length 4: 10.1.20.1
        0x0000:  4500 0240 0000 4000 ff11 3d96 0a01 1401  E..@..@...=.....
        0x0010:  0a01 1414 0043 0044 022c 0000 0201 0600  .....C.D.,......
        0x0020:  1a53 c74d 0000 0000 0000 0000 0a01 1414  .S.M............
        0x0030:  0000 0000 0000 0000 064d 4b64 e186 0000  .........MKd....
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0060:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0070:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0080:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0090:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x00f0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0100:  0000 0000 0000 0000 6382 5363 3501 0536  ........c.Sc5..6
        0x0110:  040a 0114 0133 0400 000e 1001 04ff ffff  .....3..........
        0x0120:  001c 040a 0114 ff0f 1f61 702d 6e6f 7274  .........ap-nort
        0x0130:  6865 6173 742d 312e 636f 6d70 7574 652e  heast-1.compute.
        0x0140:  696e 7465 726e 616c 0604 0a01 0002 0c0d  internal........
        0x0150:  6970 2d31 302d 312d 3230 2d32 301a 0223  ip-10-1-20-20..#
        0x0160:  2903 040a 0114 01ff 0000 0000 0000 0000  )...............
        0x0170:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0180:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0190:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x01f0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0200:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0210:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0220:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0230:  0000 0000 0000 0000 0000 0000 0000 0000  ................
^C
2 packets captured
787 packets received by filter
0 packets dropped by kernel

Phishfry · Nov 22, 2018

Sorry I deleted the request for more info.
It is now obvious to me what you need. Add the above MTU fix to /etc/dhclient.conf until FreeBSD 12 comes out.

Phishfry · Nov 22, 2018

The 30 minute time-frame for the flapping also matches the other posters so I am sure that this is the issue.
Try this:
/etc/dhclient.conf

Code:

interface "ena0" {
  supersede interface-mtu 0;
}

Jaehak Lee · Nov 22, 2018

Phishfry said:
The 30 minute time-frame for the flapping also matches the other posters so I am sure that this is the issue.
Try this:
/etc/dhclient.conf

Code:

interface "ena0" { supersede interface-mtu 0; }

Thank you Phishfry.
It worked for me.
ena0 interface is not going down any more.

Code:

 % ifconfig ena0
ena0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=422<TXCSUM,JUMBO_MTU,LRO>
        ether 06:4d:4b:64:e1:86
        hwaddr 06:4d:4b:64:e1:86
        inet6 fe80::44d:4bff:fe64:e186%ena0 prefixlen 64 scopeid 0x1
        inet 10.1.20.20 netmask 0xffffff00 broadcast 10.1.20.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active

% tail -n 100 /var/log/messages|grep ena
Nov 22 15:19:29 db-20 kernel: ena0: <ENA adapter> mem 0xfebd4000-0xfebd7fff,0xfe800000-0xfe8fffff,0xfebc0000-0xfebcffff at device 5.0 on pci0
Nov 22 15:19:29 db-20 kernel: ena0: Elastic Network Adapter (ENA)ena v0.7.0
Nov 22 15:19:29 db-20 kernel: ena0: initalize 4 io queues
Nov 22 15:19:29 db-20 kernel: ena0: Ethernet address: 06:4d:4b:64:e1:86
Nov 22 15:19:29 db-20 kernel: ena0: Allocated msix_entries, vectors (cnt: 5)
Nov 22 15:19:29 db-20 kernel: ena0: link is UP
Nov 22 15:19:29 db-20 kernel: ena0: link state changed to UP
Nov 22 15:19:29 db-20 kernel: ena0: device is going UP
Nov 22 15:19:29 db-20 kernel: ena0: queue 0 - cpu 0
Nov 22 15:19:29 db-20 kernel: ena0: queue 1 - cpu 1
Nov 22 15:19:29 db-20 kernel: ena0: queue 2 - cpu 2
Nov 22 15:19:29 db-20 kernel: ena0: queue 3 - cpu 3

Thanks.

Phishfry · Nov 22, 2018

When you go to upgrade to FreeBSD 12 you might want to pull that setting out and let the driver handle the MTU's.

Jaehak Lee · Nov 22, 2018

Yes, I got what you mean.
Thank a lot.

ena interface down and up repeatedly

Phishfry

Jaehak Lee

Phishfry

Phishfry

Jaehak Lee

Phishfry

Jaehak Lee