Solved Server won't resolve and times out a lot

The Alchemist · Aug 13, 2024

Context
I have been running FreeBSD on my server PC for a long while now with jails (Iocage).
Since a few months, I moved to a different place that has a router with IPV6 and adjusted my server to use that as well.

After a power outage / upgrade to FreeBSD 13.3-RELEASE-p3, my server has a lot of issues with resolving connections.
Incoming connections work as a charm, but retrieving stuff from (mostly) https://plex.tv is horrible.

Jails

Code:

# iocage list
+-----+-----------+-------+--------------+-----------------------+
| JID |   NAME    | STATE |   RELEASE    |          IP4          |
+=====+===========+=======+==============+=======================+
| 4   | apache    | up    | 13.3-RELEASE | 172.16.10.2           |
+-----+-----------+-------+--------------+-----------------------+
| 3   | databases | up    | 13.3-RELEASE | 172.16.10.6           |
+-----+-----------+-------+--------------+-----------------------+
| 1   | nginx     | up    | 13.3-RELEASE | 172.16.10.5           |
+-----+-----------+-------+--------------+-----------------------+
| 6   | plex      | up    | 13.3-RELEASE | 172.16.10.1,127.0.0.1 |
+-----+-----------+-------+--------------+-----------------------+
| 5   | squid     | up    | 13.3-RELEASE | 172.16.10.3           |
+-----+-----------+-------+--------------+-----------------------+
| 2   | unifi     | up    | 13.3-RELEASE | 172.16.10.8           |
+-----+-----------+-------+--------------+-----------------------+

rc.conf

Code:

hostname="virtualalchemy.nl"
ifconfig_em0="DHCP"
sshd_enable="YES"
dumpdev="AUTO"
zfs_enable="YES"
defaultrouter="192.168.0.1"
# ifconfig_em0="inet 192.168.0.137 netmask 255.255.255.0"
ifconfig_em0="DHCP"
ifconfig_em0_ipv6="inet6 accept_rtadv"
ezjail_enable="NO"
cloned_interfaces="lo1"
ipv4_addrs_lo1="172.16.10.0-9/29"
iocage_enable="YES"
pf_enable="YES"
smartd_enable="YES"
smartd_enable="YES"
# ipv6_enable="YES"
# ipv6_activate_all_interfaces="YES"

Setting up IPV6 was only working without activating the last 2 lines for some reason...
resolv.conf

Code:

# cat /etc/resolv.conf
# Generated by resolvconf
search home
nameserver 1.0.0.1
nameserver 1.1.1.1
nameserver 2606:4700:4700::1001
nameserver 2606:4700:4700::1111

Code:

# ifconfig
em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether d0:50:99:9b:27:27
        inet6 fe80::d250:99ff:fe9b:2727%em0 prefixlen 64 scopeid 0x1
        inet6 fd4c:4b32:3330:0:d250:99ff:fe9b:2727 prefixlen 64 autoconf
        inet6 2a02:a456:204c:0:d250:99ff:fe9b:2727 prefixlen 64 autoconf
        inet 192.168.0.123 netmask 0xffffff00 broadcast 192.168.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 172.16.10.0 netmask 0xfffffff8
        inet 172.16.10.4 netmask 0xffffffff
        inet 172.16.10.7 netmask 0xffffffff
        inet 172.16.10.9 netmask 0xffffffff
        inet 172.16.10.5 netmask 0xffffffff
        inet 172.16.10.8 netmask 0xffffffff
        inet 172.16.10.6 netmask 0xffffffff
        inet 172.16.10.2 netmask 0xffffffff
        inet 172.16.10.3 netmask 0xffffffff
        inet 172.16.10.1 netmask 0xffffffff
        inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

pf.conf

Code:

# cat /etc/pf.conf
# Public IP address
IP_PUB="192.168.0.123"
TCP_PASS = "{ 22 }"

# Packet normalization
scrub in all

# Allow outbound connections from within the jails
nat on em0 from lo1:network to any -> (em0)

# Plex + qbittorent
rdr on em0 proto tcp from any to $IP_PUB port 32400 -> 172.16.10.1
rdr on em0 proto tcp from any to $IP_PUB port 8181 -> 172.16.10.1
rdr on em0 proto tcp from any to $IP_PUB port 56000 -> 172.16.10.1
rdr on em0 proto tcp from any to $IP_PUB port 25025 -> 172.16.10.1


# Allow ssh
pass out proto tcp to port $TCP_PASS keep state

*I removed the lines from the jails that are not interesting for this issue.

netstat

netstat -ra (TAKES AGES)

Code:

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.0.254      UGS         em0
default            192.168.0.1        UGS         em0
localhost          link#2             UH          lo0
172.16.10.0        link#3             UH          lo1
172.16.10.1        link#3             UH          lo1
172.16.10.2        link#3             UH          lo1
172.16.10.3        link#3             UH          lo1
172.16.10.4        link#3             UH          lo1
172.16.10.5        link#3             UH          lo1
172.16.10.6        link#3             UH          lo1
172.16.10.7        link#3             UH          lo1
172.16.10.8        link#3             UH          lo1
172.16.10.9        link#3             UH          lo1
192.168.0.0/24     link#1             U           em0
192.168.0.123      link#1             UHS         lo0

Internet6:
Destination        Gateway            Flags     Netif Expire
::/96              localhost          URS         lo0
default            fe80::ce58:30ff:fe UG          em0
localhost          link#2             UHS         lo0
::ffff:0.0.0.0/96  localhost          URS         lo0
2a02:a456:204c::/6 link#1             U           em0
2a02-a456-204c-0-d link#1             UHS         lo0
fd4c:4b32:3330::/6 link#1             U           em0
fd4c:4b32:3330:0:d link#1             UHS         lo0
fe80::/10          localhost          URS         lo0
fe80::%em0/64      link#1             U           em0
fe80::d250:99ff:fe link#1             UHS         lo0
fe80::%lo0/64      link#2             U           lo0
fe80::1%lo0        link#2             UHS         lo0
fe80::%lo1/64      link#3             U           lo1
fe80::1%lo1        link#3             UHS         lo0
ff02::/16          localhost          URS         lo0

netstat in plex jail:

Code:

root@plex:~ # netstat -ra
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
localhost          link#2             UH          lo0
172.16.10.1        link#3             UH          lo1

More
After noticing that Plex could not retrieve metadata, I started searching for solutions for the issue.
After a lot of time, I tried to simply do a wget on the URLs that threw an 408 connection timeout.

Code:

root@plex:~ # wget [URL]https://plex.tv[/URL]
--2024-08-13 20:24:28--  [URL]https://plex.tv/[/URL]
Resolving plex.tv (plex.tv)... 52.18.158.99, 54.154.120.177
Connecting to plex.tv (plex.tv)|52.18.158.99|:443... failed: Operation timed out.
Connecting to plex.tv (plex.tv)|54.154.120.177|:443...

Drilling also takes a long while or just throws network errors, weirdly drilling has intervals of 5 second delays:

Code:

root@plex:~ # drill @1.1.1.1 plex.tv
Error: error sending query: Could not send or receive, because of network error
root@plex:~ # drill @1.1.1.1 plex.tv
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 9724
;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; plex.tv.     IN      A

;; ANSWER SECTION:
plex.tv.        33      IN      A       54.154.120.177
plex.tv.        33      IN      A       52.18.158.99

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 5051 msec
;; SERVER: 1.1.1.1
;; WHEN: Tue Aug 13 20:26:54 2024
;; MSG SIZE  rcvd: 57
root@plex:~ # drill @1.1.1.1 plex.tv
Error: error sending query: Could not send or receive, because of network error
root@plex:~ # drill @1.1.1.1 plex.tv
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 33672
;; flags: qr rd ra ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; plex.tv.     IN      A

;; ANSWER SECTION:
plex.tv.        18      IN      A       52.18.158.99
plex.tv.        18      IN      A       54.154.120.177

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 5 msec
;; SERVER: 1.1.1.1
;; WHEN: Tue Aug 13 20:27:40 2024
;; MSG SIZE  rcvd: 57

Above are 4 drills:

fail
5 + something seconds
fail
directly

Conclusion
I have a second server connected to the same switch + router, that works like a charm.
I probably have some messed up configuration, but hope you can see what I am doing wrong.

Thank you in advance for your time and help!

SirDice · Aug 13, 2024

The Alchemist said:

Code:

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.0.254      UGS         em0
default            192.168.0.1        UGS         em0

You have two default gateways to different IP addresses?

The Alchemist · Aug 13, 2024

SirDice said:
You have two default gateways to different IP addresses?

I noticed that too, but have NO clue where this is coming from.

I did have a unifi network before, and now I don't. Only using the switch ATM.

Any way to figure out where this originates from?

SirDice · Aug 13, 2024

Don't know where it comes from, but I'm guessing it's part of the problem. defaultrouter in rc.conf looks good and suggests it should only be 192.168.0.1.

The Alchemist said:
Any way to figure out where this originates from?

Rebooting and see if it comes back? If it's back after the reboot it must be something that's configured somewhere.

The Alchemist · Aug 13, 2024

SirDice said:
Don't know where it comes from, but I'm guessing it's part of the problem. defaultrouter in rc.conf looks good and suggests it should only be 192.168.0.1.

Rebooting and see if it comes back? If it's back after the reboot it must be something that's configured somewhere.

Yeah I tried that already, sadly...

I did get an idea though, by grepping the IP address and found nothing in /etc, but found this when grepping in root (/):

/var/log/daemon.log:Aug 13 20:13:43 virtualalchemy dhclient[754]: DHCPREQUEST on em0 to 192.168.0.254 port 67
/var/log/daemon.log:Aug 13 20:13:43 virtualalchemy dhclient[754]: DHCPACK from 192.168.0.254
/var/log/messages:Apr 1 21:52:47 virtualalchemy dhclient[7731]: New Routers (em0): 192.168.0.254
/var/log/messages:Apr 7 16:27:19 virtualalchemy dhclient[81162]: New Routers (em0): 192.168.0.254
/var/log/messages:Jul 4 11:46:42 virtualalchemy dhclient[24189]: New Routers (em0): 192.168.0.254
/var/db/dhclient.leases.em0: next-server 192.168.0.254;
/var/db/dhclient.leases.em0: option routers 192.168.0.254;
/var/db/dhclient.leases.em0: option dhcp-server-identifier 192.168.0.254;

Noooooo idea where this is coming from, sigh...

Some Anon Guy · Aug 13, 2024

Your DHCP server is handing out 192.168.0.254 as the gateway, and you have

Code:

defaultrouter="192.168.0.1"

set in your /etc/rc.conf

The Alchemist · Aug 13, 2024

Aaaaah damn, changed that, testing it right now! Rebooting

SirDice · Aug 13, 2024

Right. Only set defaultrouter if you're using static addresses.

The Alchemist · Aug 13, 2024

Some Anon Guy said:
Your DHCP server is handing out 192.168.0.254 as the gateway, and you have

Code:

defaultrouter="192.168.0.1"

set in your /etc/rc.conf

WAAAUW, changing that fixed it!

Thank you so much

The Alchemist · Aug 14, 2024

SirDice said:
Right. Only set defaultrouter if you're using static addresses.

Aaaah perfect, good to know, the current router does not support static IPs, thats why I removed the static IP address line.
Didn't know about that the defaultrouter setting is for static only

Solved Server won't resolve and times out a lot

The Alchemist

SirDice

Administrator

The Alchemist

SirDice

Administrator

The Alchemist

Some Anon Guy

The Alchemist

SirDice

Administrator

The Alchemist

The Alchemist