problem fetching ports tree

I'm getting errors while trying to install the ports tree using # portsnap:

Code:
# portsnap fetch extract
portsnap: Directory does not exist or is not writable: /var/db/portsnap
bsdmag# rm /var/db/portsnap 
bsdmag# rm /usr/ports 
bsdmag# mkdir /var/db/portsnap
bsdmag# portsnap fetch extract
Looking up portsnap.FreeBSD.org mirrors... 5 mirrors found.
Fetching public key from portsnap5.FreeBSD.org... done.
Fetching snapshot tag from portsnap5.FreeBSD.org... done.
Fetching snapshot metadata... done.
Fetching snapshot generated at Thu Jan 12 00:06:43 UTC 2012:

67204aa8591a39a8121c6c2012129ee50551d7f6065175100% of   64 MB  168 kBps 00m00s
Extracting snapshot... done.
Verifying snapshot integrity... done.
Fetching snapshot tag from portsnap5.FreeBSD.org... done.
Fetching snapshot metadata... done.
Updating from Thu Jan 12 00:06:43 UTC 2012 to Thu Jan 12 15:05:00 UTC 2012.
Fetching 4 metadata patches. done.
Applying metadata patches... done.
Fetching 4 metadata files... 

/usr/sbin/portsnap: cannot open a1d3ca465757c636dd8295698fe05e7d3fc4943204e504fee6cd5b1665ba0753.gz: 
No such file or directory
metadata is corrupt.

I've already tried to remove and recreate /var/db/portsnap but without any success. My system is:

Code:
# uname -a
FreeBSD bsdluk 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Fri Feb 18 02:24:46 UTC 2011     
root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386

Any idea on what is going wrong here?
 
I'm experiencing the same problem again. I've tried to do a portsnap(8) which failed due to metadata corrupt, I then recreated the /var/db/portsnap folder and I then tried also to do a new fetch (instead of an update). The result is always the same:

Code:
19714ed1c2980d1e1a2c44d292a7ccfd84f400a83211f9100% of   67 MB  177 kBps 00m00s
Extracting snapshot... done.
Verifying snapshot integrity... 
done.
Fetching snapshot tag from geodns-1.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Tue May 22 02:12:15 CEST 2012 to Tue May 22 10:45:23 CEST 2012.
Fetching 4 metadata patches. done.
Applying metadata patches... done.
Fetching 4 metadata files... /usr/sbin/portsnap: 
cannot open 1e7ee6890d8372822978351df474d31be0c7e77f30a5055ccad013360479f9fd.gz: 
No such file or directory

The only thing that is different in my environment is that I'm behind a proxy, that portsnap gets from the http_proxy environment variable.
I've tried today and yesterday.
Any idea?
 
I'm still fighting with this problem, and now, due to some changes of the proxy I'm behind, again I cannot download the ports update. This time the error is different and is related to phttpget:

Code:
# portsnap fetch extract
Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.                               
Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done.                    
Fetching snapshot metadata... done.                                                       
Updating from Mon Nov  5 00:03:52 UTC 2012 to Thu Nov  8 11:39:28 UTC 2012.               
Fetching 4 metadata patches. done.                                                        
Applying metadata patches... done.                                                        
[B]Fetching 4 metadata files... phttpget: host = fluca, port = pwd@192.168.1.7:8080: servname not supported for ai_socktype                [/B]                                     
Processing file ee2cf31cffdb8da7ecb6db2dd8d36ed4823fce7cffbb09b55c2754247961a812          
/usr/sbin/portsnap: cannot open ee2cf31cffdb8da7ecb6db2dd8d36ed4823fce7cffbb09b55c2754247961a812.gz: No such file or directory                                                      
metadata is corrupt.                                                                      
/usr/sbin/portsnap: cannot open ee2cf31cffdb8da7ecb6db2dd8d36ed4823fce7cffbb09b55c2754247961a812.gz: No such file or directory                                                      
SHA256 =  e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

The wrong line is the one where my proxy authentication information is used as a port (i.e., I've got an env variable http_proxy set as http://username:password@proxyIP:proxyPort - being those shown not my real credentials). So phttpget is not working properly with proxy authentication.
I found this old PR but it mentions also a proxy option for phttpget that seems to me it is no more present.

Anybody has a suggestion on how to solve the problem?
 
I've fixed it for my machines making a change to portsnap: now it uses net/wget to download the stuff instead of phttpget, and since wget can handle the http_proxy variable well, the process works. Of course, I had to change also the logic that built the files to download as in the following patch. I've named this portsnap_wget on my system, and maybe this can be useful for someone else.

Code:
268c268,270
< PHTTPGET=/usr/libexec/phttpget
---
> PHTTPGET=/usr/local/bin/wget
>
> echo "WARNING: this version of portsnap is usign $PHTTPGET as Web-HTTP-Get program!"
687,690c689,694
< tr '|' '-' < patchlist |
< lam -s "tp/" - -s ".gz" |
< xargs ${XARGST} ${PHTTPGET} ${SERVERNAME} \
< 2>${STATSREDIR} | fetch_progress
---
> # fetch metadata files using wget behind a proxy
> for current_file in `tr '|' '-' < patchlist | lam -s "tp/" - -s ".gz"`
> do
> ${PHTTPGET} ${SERVERNAME}/${current_file} > /dev/null 2>&1
> done
>
721,724c725,732
< echo ${NDEBUG} "metadata files... "
< lam -s "f/" - -s ".gz" < filelist |
< xargs ${XARGST} ${PHTTPGET} ${SERVERNAME} \
< 2>${QUIETREDIR}
---
> echo ${NDEBUG} "metadata files..."
>
> # fetch metadata files using wget behind a proxy
> for current_file in `cat filelist`
> do
> ${PHTTPGET} ${SERVERNAME}/f/${current_file}.gz > /dev/null 2>&1
> done
>
763,765c771,776
< tr '|' '-' < patchlist | lam -s "bp/" - |
< xargs ${XARGST} ${PHTTPGET} ${SERVERNAME} \
< 2>${STATSREDIR} | fetch_progress
---
>
> # fetch metadata files using wget behind a proxy
> for current_file in `tr '|' '-' < patchlist | lam -s "bp/" - -s ".gz"`
> do
> ${PHTTPGET} ${SERVERNAME}/${current_file} > /dev/null 2>&1
> done
794,796c805,809
< lam -s "f/" - -s ".gz" < filelist |
< xargs ${XARGST} ${PHTTPGET} ${SERVERNAME} \
< 2>${QUIETREDIR}
---
> # fetch metadata files using wget behind a proxy
> for current_file in `lam -s "f/" - -s ".gz" < filelist`
> do
> ${PHTTPGET} ${SERVERNAME}/${current_file} > /dev/null 2>&1
> done

I was wondering if it is useful to change the adoption of phttpget into portsnap, since this problem could happen to others. In short, the problem was that phttpget does not handle correctly an http_proxy of the form
Code:
http://user:password@proxy:port
considering the :password@proxy as the port specification.
 
Make it a configurable option in /etc/portsnap.conf and ask on the mailing lists (current, stable) if this change could be committed.
 
Good afternoon,

Could guide me on how to apply this patch?
I'm having some doubts:

1) In which of these files 'portsnap' Should I apply the patch?

Code:
/etc/portsnap.conf
/usr/sbin/portsnap
/usr/share/examples/etc/portsnap.conf
/usr/share/man/man5/portsnap.conf.5.gz
/usr/share/man/man8/portsnap.8.gz
/usr/src/etc/portsnap.conf
/usr/src/share/man/man5/portsnap.conf.5
/usr/src/usr.sbin/portsnap
/usr/src/usr.sbin/portsnap/Makefile
/usr/src/usr.sbin/portsnap/Makefile.inc
/usr/src/usr.sbin/portsnap/make_index
/usr/src/usr.sbin/portsnap/make_index/Makefile
/usr/src/usr.sbin/portsnap/make_index/make_index.c
/usr/src/usr.sbin/portsnap/phttpget
/usr/src/usr.sbin/portsnap/phttpget/Makefile
/usr/src/usr.sbin/portsnap/phttpget/phttpget.c
/usr/src/usr.sbin/portsnap/portsnap
/usr/src/usr.sbin/portsnap/portsnap/Makefile
/usr/src/usr.sbin/portsnap/portsnap/portsnap.8
/usr/src/usr.sbin/portsnap/portsnap/portsnap.sh

2) The command syntax would be this one:

# patch -p1 < portsnap.patch

Like I said, I do not know if the patch is to be applied in portsnap which is inside the directory /usr/sbin or whether it is to be applied to portsnap which is in one of the directories shown above.

Thank you in advance for any help.

hugs
 
Back
Top