jails Can no longer update jails using iocage

E

Echoxxzz

Over the last few days, I have been unable to update my jails using iocage.

I've managed to trace the problem down to a few lines of Python code in the ioc_fetch.py module:

Code: 
            f = "https://cgit.freebsd.org/src/plain" \
                "/usr.sbin/freebsd-update/freebsd-update.sh"

            tmp = tempfile.NamedTemporaryFile(delete=False)
            with urllib.request.urlopen(f) as fbsd_update:
                tmp.write(fbsd_update.read())
            tmp.close()
            os.chmod(tmp.name, 0o755)
            fetch_name = tmp.name

When the above code is run more often than not, the temp file that is created does not contain the freebsd-update.sh shell script, but instead it's an HTML file that has something to do with Anubis (sample attached), which in turn crashes iocage.

It doesn't happen all the time, it's really hit and miss, but what's strange is that I can always download the file using curl.

I even wrote a Python script that uses the above code to download that shell script 100 times, and it also fails, but a similar shell script using curl to download the freebsd-update.sh file 100 times never fails.

I don't know where to report this problem.
It's definitely not a problem with iocage since I can duplicate the problem using a simple Python test script.
Is a problem with the urllib.request() method, or is it a problem with the cgit.FreeBSD.org host?

Since curl always works, it leads me to believe it's more of a problem with the urllib.request() method.

I have tested this on FreeBSD 14.4-p1 and 14.4-p4, and it occurs on both builds.
I have also tried using three different ISPs to rule out some kind of ISP caching issue.
 

Attachments

Echoxxzz said:
is it a problem with the cgit.FreeBSD.org host
Click to expand...
I don't know, but would suspect this is more likely - for some traffic that host wants to pass you through the Anubis bot-checker.

Something (http headers?) makes the bot-checker kick in for the Python request, but whatever curl does differently appears to be enough to stop the Anubis check.
 
I updated all of my iocage jails last week to the latest patch levels. I only had a problem with one 15.0 host that is running 13.5, 14.4 and 15.0 jails, it needed a pkg bootstrap -f on the 13.5 jail. All of my jail hosts are 15.0 and have had config changed to assign IP to the bridge and not the parent network interface.
 
Got this tip from Google AI:

It seems that urllib.request.urlopen often downloads bot detection pages (aka Anubis) because it defaults to a Python-specific User-Agent string, which many websites immediately identify as a bot and block. To fix this, you must spoof a browser User-Agent header in your request, as detailed in the Python documentation for HTTP headers. [1, 2, 3, 4]
Click to expand...
I will fix the Python code and see if this tip works. If it does I will created a bug report with patch.
 
Using a custom User-Agent header did not work.
I used SirDice's suggestion and changed the code to download the file from github mirror.
After over 1000 tests it hasn't failed once using github.
I will open a ticket and see if it gets accepted.
 
You must log in or register to reply here.
Back
Top