fetch/wget hangs on download

l33tname · Apr 23, 2023

Fetch regularly takes a long time to deliver content and often completely timeout the connection.
I notices that poudriere builds often fail to complete because some ports can not be fetched.
(Different ports this is just one example)

Code:

=> Attempting to fetch https://files.pythonhosted.org/packages/source/c/calver/calver-2022.6.26.tar.gz
fetch: transfer timed out

When fetching the url i can reproduce this issue, sometimes it is slow sometimes it completely blocks.

Code:

time fetch -A -v "https://files.pythonhosted.org/packages/source/c/calver/calver-2022.6.26.tar.gz"
resolving server address: files.pythonhosted.org:443
SSL options: 82004854
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-AES128-GCM-SHA256
Certificate subject: /CN=*.pythonhosted.org
Certificate issuer: /C=BE/O=GlobalSign nv-sa/CN=GlobalSign Atlas R3 DV TLS CA 2022 Q3
requesting https://files.pythonhosted.org/packages/source/c/calver/calver-2022.6.26.tar.gz
fetch: https://files.pythonhosted.org/packages/source/c/calver/calver-2022.6.26.tar.gz: Moved Temporarily
fetch -A -v   0.01s user 0.00s system 0% cpu 3.662 total

I suspected DNS but switching to only google dns (2001:4860:4860::8888) did not solve much

Code:

17:11:21.249178 IP6 X:X:X:786::20.33395 > dns.google.domain: 30806+ A? files.pythonhosted.org. (40)
17:11:21.265794 IP6 dns.google.domain > X:X:X:786::20.33395: 30806 5/0/0 CNAME dualstack.r.ssl.global.fastly.net., A 151.101.1.63, A 151.101.65.63, A 151.101.129.63, A 151.101.193.63 (151)
17:11:21.265837 IP6 X:X:X:786::20.16073 > dns.google.domain: 24685+ AAAA? files.pythonhosted.org. (40)
17:11:21.279706 IP6 dns.google.domain > X:X:X:786::20.16073: 24685 5/0/0 CNAME dualstack.r.ssl.global.fastly.net., AAAA 2a04:4e42::319, AAAA 2a04:4e42:200::319, AAAA 2a04:4e42:400::319, AAAA 2a04:4e42:600::319 (199)

https traffic

Code:

17:22:58.823559 IP6 X:X:X:786::20.52642 > 2a04:4e42::319.https: Flags [S], seq 3075915642, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 257907271 ecr 0], length 0
17:22:58.837023 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [S.], seq 70373961, ack 3075915643, win 65535, options [mss 1440,sackOK,TS val 617740724 ecr 257907271,nop,wscale 9], length 0
17:22:58.837033 IP6 X:X:X:786::20.52642 > 2a04:4e42::319.https: Flags [.], ack 1, win 1035, options [nop,nop,TS val 257907285 ecr 617740724], length 0
17:22:58.844331 IP6 X:X:X:786::20.52642 > 2a04:4e42::319.https: Flags [P.], seq 1:518, ack 1, win 1035, options [nop,nop,TS val 257907292 ecr 617740724], length 517
17:22:58.857497 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [.], ack 518, win 281, options [nop,nop,TS val 617740746 ecr 257907292], length 0
17:22:58.861177 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [P.], seq 2857:3247, ack 518, win 281, options [nop,nop,TS val 617740747 ecr 257907292], length 390
17:22:58.861181 IP6 X:X:X:786::20.52642 > 2a04:4e42::319.https: Flags [.], ack 1, win 1035, options [nop,nop,TS val 257907309 ecr 617740746,nop,nop,sack 1 {2857:3247}], length 0
17:23:02.349207 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [.], seq 1:1209, ack 518, win 281, options [nop,nop,TS val 617744232 ecr 257907309], length 1208
17:23:02.349227 IP6 X:X:X:786::20.52642 > 2a04:4e42::319.https: Flags [.], ack 1209, win 1017, options [nop,nop,TS val 257910797 ecr 617744232,nop,nop,sack 1 {2857:3247}], length 0
17:23:02.367810 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [.], seq 1209:1429, ack 518, win 281, options [nop,nop,TS val 617744250 ecr 257910797], length 220
17:23:02.367814 IP6 X:X:X:786::20.52642 > 2a04:4e42::319.https: Flags [.], ack 1429, win 1032, options [nop,nop,TS val 257910816 ecr 617744250,nop,nop,sack 1 {2857:3247}], length 0
17:23:02.368129 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [.], seq 1429:2637, ack 518, win 281, options [nop,nop,TS val 617744250 ecr 257910797], length 1208
17:23:02.368131 IP6 X:X:X:786::20.52642 > 2a04:4e42::319.https: Flags [.], ack 2637, win 1017, options [nop,nop,TS val 257910816 ecr 617744250,nop,nop,sack 1 {2857:3247}], length 0
17:23:02.385210 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [.], seq 2637:2857, ack 518, win 281, options [nop,nop,TS val 617744268 ecr 257910816], length 220
17:23:02.385214 IP6 X:X:X:786::20.52642 > 2a04:4e42::319.https: Flags [.], ack 3247, win 1026, options [nop,nop,TS val 257910833 ecr 617744268], length 0
17:23:02.385879 IP6 X:X:X:786::20.52642 > 2a04:4e42::319.https: Flags [P.], seq 518:611, ack 3247, win 1035, options [nop,nop,TS val 257910834 ecr 617744268], length 93
17:23:02.427515 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [.], ack 611, win 281, options [nop,nop,TS val 617744310 ecr 257910834], length 0
17:23:02.427795 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [P.], seq 3247:3298, ack 611, win 281, options [nop,nop,TS val 617744310 ecr 257910834], length 51
17:23:02.428100 IP6 X:X:X:786::20.52642 > 2a04:4e42::319.https: Flags [P.], seq 611:1090, ack 3298, win 1035, options [nop,nop,TS val 257910876 ecr 617744310], length 479
17:23:02.448472 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [.], ack 1090, win 284, options [nop,nop,TS val 617744331 ecr 257910876], length 0
17:23:02.450313 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [P.], seq 3298:4067, ack 1090, win 284, options [nop,nop,TS val 617744333 ecr 257910876], length 769
17:23:02.450385 IP6 X:X:X:786::20.52642 > 2a04:4e42::319.https: Flags [P.], seq 1090:1121, ack 4067, win 1035, options [nop,nop,TS val 257910898 ecr 617744333], length 31
17:23:02.450912 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [P.], seq 4067:4098, ack 1090, win 284, options [nop,nop,TS val 617744334 ecr 257910876], length 31
17:23:02.450913 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [F.], seq 4098, ack 1090, win 284, options [nop,nop,TS val 617744334 ecr 257910876], length 0
17:23:02.450915 IP6 X:X:X:786::20.52642 > 2a04:4e42::319.https: Flags [.], ack 4099, win 1035, options [nop,nop,TS val 257910899 ecr 617744334], length 0
17:23:02.451006 IP6 X:X:X:786::20.52642 > 2a04:4e42::319.https: Flags [F.], seq 1121, ack 4099, win 1035, options [nop,nop,TS val 257910899 ecr 617744334], length 0
17:23:02.464969 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [R], seq 70378060, win 0, length 0
17:23:02.464977 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [R], seq 70378060, win 0, length 0
17:23:02.467635 IP6 2a04:4e42::319.https > X:X:X:786::20.52642: Flags [R], seq 70378028, win 0, length 0

Similar hosts in this network do not suffer this issue.

DNS & HTTPS for a case where it hangs for ever

Code:

17:35:24.534422 IP6 X:X:X:786::20.30393 > dns.google.domain: 35028+ A? files.pythonhosted.org. (40)
17:35:24.534693 IP6 X:X:X:786::20.34910 > dns.google.domain: 27579+ PTR? 0.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.6.8.7.0.6.2.0.0.0.7.4.0.1.0.0.2.ip6.arpa. (90)
17:35:24.550041 IP6 dns.google.domain > X:X:X:786::20.30393: 35028 5/0/0 CNAME dualstack.r.ssl.global.fastly.net., A 151.101.1.63, A 151.101.65.63, A 151.101.129.63, A 151.101.193.63 (151)
17:35:24.550080 IP6 X:X:X:786::20.10117 > dns.google.domain: 14804+ AAAA? files.pythonhosted.org. (40)
17:35:24.563721 IP6 dns.google.domain > X:X:X:786::20.10117: 14804 5/0/0 CNAME dualstack.r.ssl.global.fastly.net., AAAA 2a04:4e42::319, AAAA 2a04:4e42:200::319, AAAA 2a04:4e42:400::319, AAAA 2a04:4e42:600::319 (199)
17:35:24.563811 IP6 X:X:X:786::20.18461 > 2a04:4e42::319.https: Flags [S], seq 3654686065, win 65535, options [mss 1440,nop,wscale 6,sackOK,TS val 1341886755 ecr 0], length 0
17:35:24.564756 IP6 dns.google.domain > X:X:X:786::20.34910: 27579 NXDomain 0/1/0 (147)
17:35:24.564853 IP6 X:X:X:786::20.25679 > dns.google.domain: 46600+ PTR? 8.8.8.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.6.8.4.0.6.8.4.1.0.0.2.ip6.arpa. (90)
17:35:24.573613 IP6 dns.google.domain > X:X:X:786::20.25679: 46600 1/0/0 PTR dns.google. (114)
17:35:24.573735 IP6 X:X:X:786::20.38779 > dns.google.domain: 45452+ PTR? 9.1.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.4.e.4.4.0.a.2.ip6.arpa. (90)
17:35:24.576935 IP6 2a04:4e42::319.https > X:X:X:786::20.18461: Flags [S.], seq 1577346110, ack 3654686066, win 65535, options [mss 1440,sackOK,TS val 1224999737 ecr 1341886755,nop,wscale 9], length 0
17:35:24.576942 IP6 X:X:X:786::20.18461 > 2a04:4e42::319.https: Flags [.], ack 1, win 1035, options [nop,nop,TS val 1341886768 ecr 1224999737], length 0
17:35:24.584322 IP6 dns.google.domain > X:X:X:786::20.38779: 45452 NXDomain 0/1/0 (150)
17:35:24.584383 IP6 X:X:X:786::20.18461 > 2a04:4e42::319.https: Flags [P.], seq 1:518, ack 1, win 1035, options [nop,nop,TS val 1341886775 ecr 1224999737], length 517
17:35:24.596216 IP6 2a04:4e42::319.https > X:X:X:786::20.18461: Flags [.], ack 518, win 281, options [nop,nop,TS val 1224999756 ecr 1341886775], length 0
17:35:24.600127 IP6 2a04:4e42::319.https > X:X:X:786::20.18461: Flags [P.], seq 2857:3247, ack 518, win 281, options [nop,nop,TS val 1224999758 ecr 1341886775], length 390
17:35:24.600131 IP6 X:X:X:786::20.18461 > 2a04:4e42::319.https: Flags [.], ack 1, win 1035, options [nop,nop,TS val 1341886791 ecr 1224999756,nop,nop,sack 1 {2857:3247}], length 0

Any idea what is happening here?

Alain De Vos · Apr 23, 2023

You could try,

Code:

time wget www.google.com

yuripv79 · Apr 23, 2023

l33tname said:
Similar hosts in this network do not suffer this issue.

Are those hosts using IPv6 as well?

l33tname · Apr 23, 2023

Same issue with wget and yes all my host are dual stack IPv6/v4.
It might be an IPv6 tunnel issue, since i don't have native v6 and it is provided via HE tunnel. (https://tunnelbroker.net/)

yuripv79 · Apr 23, 2023

l33tname said:
all my host are dual stack IPv6/v4

No, I mean can you check that the hosts that don't see the issue are indeed using IPv6 for those transfers? That could rule out this being IPv6 (hence tunnel) specific.

covacat · Apr 23, 2023

try to run some kind of mss-fix on the router (pf, ipfw can do that i think)

fetch/wget hangs on download

l33tname

Alain De Vos

yuripv79

l33tname

yuripv79

covacat