cannot portsnap: metadata is corrupt

[fluca1978]

Hi all,

I'm often having a "metadata is corrupt error" while trying to upgrade my ports tree. I try removing the /var/db/portsnap/ directory content and do it again but often without success. I'm behind an http proxy, if it matters, and this kind of error happens quite often to me on different installations. Is it normal? I mean, what should be the ratio of such failure? Today I tried three times, all of them without success. I guess this is not a remote site problem but one of mine.

# portsnap fetch extract                          
Looking up portsnap.FreeBSD.org mirrors... 9 mirrors found.                                
Fetching snapshot tag from geodns-1.portsnap.freebsd.org... done.                          
Fetching snapshot metadata... done.                                                        
Updating from Thu Jun 28 00:07:47 UTC 2012 to Thu Jun 28 10:57:54 UTC 2012.                
Fetching 4 metadata patches. done.                                                         
Applying metadata patches... done.                                                         
Fetching 4 metadata files... 
/usr/sbin/portsnap: cannot open 
578739397c3f39d9e47adc5c2aec26d8a19f3f5e51f910cebad18465e71ea0c7.gz: 
No such file or directory                           
metadata is corrupt.

 

[fluca1978]

It could be just a lucky try, but after another three failures today, I unset the http_proxy variable and now the ports tree is being extracted.

 

[fluca1978]

I can confirm that unsetting http_proxy makes the portsnap(1) working all the times so far. I've took a look at the source code of /usr/sbin/portsnap I found that the http_proxy is used to generate a random seed for mirror selection:

# Generate a random seed for use in picking mirrors.  If HTTP_PROXY
# is set, this will be used to generate the seed; otherwise, the seed
# will be random.                                        
        if [ -n "${HTTP_PROXY}${http_proxy}" ]; then
                RANDVALUE=`sha256 -qs "${HTTP_PROXY}${http_proxy}" |
                    tr -d 'a-f' |                                                          
                    cut -c 1-9
        else
                RANDVALUE=`jot -r 1 0 999999999`
        fi

But the problem occurs here:

       lam -s "f/" - -s ".gz" < filelist |
            xargs ${XARGST} ${PHTTPGET} ${SERVERNAME} 2>${QUIETREDIR}            

        while read Y; do
                if [ `gunzip -c < ${Y}.gz | ${SHA256} -q` = ${Y} ]; then
                        mv ${Y}.gz files/${Y}.gz
                else
                        echo "metadata is corrupt."

                        return 1
                fi
        done < filelist

In particular removing the 2> I was able to see the error: the http_proxy variable I set was in the form http://username:password@proxy_host:proxy_port and seems it is not the right form since lam(1) cannot interpret it correctly. Am I setting the variable wrong?

 

[DiverGuy]

This may -or not- be helpful to others, and actually may apply to the above posters as well:

I maintain several freebsd FreeBSD servers and recently switched from cvsup to portsnap.

After switching to portsnap I was receiving the following (similar to above) error:

# portsnap fetch update
Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.
Fetching snapshot tag from your-org.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Mon Dec 10 13:21:53 CST 2012 to Mon Dec 10 13:31:35 CST 2012.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 1 metadata files... /usr/sbin/portsnap: cannot open 
44fe4cfa820c8c14ee93dcb3d6706bb5e9f9fd1b81f99f38c1543c70dba1f4ac.gz: No such file or directory
metadata is corrupt.

Note:

I lock down my servers fairly securely with firewall rules and grant outbound 80 on a limited basis. In particular I grant www stick-group access to fetch and wget and have firewall allow outbound 80 for gid www

The challenge was, part of portsnap uses fetch, so that part worked correctly. And, it seems that part of portsnap update does -not- use fetch.

This resulted in a long diagnostic effort due to the red herring effect of partial functionality

Ultimately I started logging firewall blocks and realized something was amiss.

Adding global allow outbound to port 80 uid root resolved the problem.

Hopefully this will save someone else some effort.