Python portupgrade woes

rambetter · Sep 13, 2010

I have 3 FreeBSD computers, and all 3 had the port lang/python25 installed. I now csup'ed the port sources and portupgrade'd all 3 computers. Two of the computers chose to upgrade Python 2.5 to 2.6 automagically, and the third didn't. On the FreeBSD 7.1-i386 computer that didn't upgrade to 2.6 for some reason:

Code:

root@daffy# pkg_info -Qao | grep pytho
python25-2.5.5_1:lang/python25
root@daffy# pkg_info -R python25-2.5.5_1
Information for python25-2.5.5_1:

Required by:
apache-ant-1.7.1
glib-2.24.2
irssi-0.8.15_2
jdk-1.5.0.16p9_6,1
libXaw-1.0.7,1
libXp-1.0.0,1
libXpm-3.5.7
libXrandr-1.3.0
maven2-2.2.1
open-motif-2.2.3_6
qt4-corelib-4.6.3
qt4-dbus-4.6.3
qt4-gui-4.6.3
qt4-network-4.6.3
qt4-sql-4.6.3
qt4-sqlite-plugin-4.6.3
qt4-xml-4.6.3
xcb-proto-1.6

On the FreeBSD 7.1-i386 computer where the automagic upgrade from Python 2.5 to Python 2.6 seems to have gone well:

Code:

root@speedy# pkg_info -Qao | grep pyth
python26-2.6.6:lang/python26
root@speedy# pkg_info -R python26-2.6.6
Information for python26-2.6.6:

Required by:
apache-ant-1.7.1
jdk-1.5.0.16p9_6,1
libXaw-1.0.7,1
libXp-1.0.0,1
libXpm-3.5.7
open-motif-2.2.3_6
xcb-proto-1.6

On the FreeBSD 8.0-amd64 computer where portupgrade did upgrade from Python 2.5 to Python 2.6, but things don't seem to be so well now:

Code:

root@porky# pkg_info -Qao | grep pyth
python26-2.6.6:lang/python26
root@porky# pkg_info -R python26-2.6.6
Information for python26-2.6.6:

Required by:
xcb-proto-1.6


root@porky# pkg_info -Qao | grep apache-ant
apache-ant-1.7.1:devel/apache-ant
root@porky# pkg_info -Qao | grep jdk
jdk-1.5.0.16p9_6,1:java/jdk15
root@porky# pkg_info -Qao | grep motif
open-motif-2.2.3_6:x11-toolkits/open-motif

Are the dependencies in my package database on porky broken now? :r
I would expect to see Python required by a lot of packages, like on my other 2 computers.

DutchDaemon · Sep 13, 2010

I don't understand how 'the automatic upgrade' from 2.5 to 2.6 could have taken place. These Python versions can co-exist, so unless you specifically instruct portupgrade (using the -o flag) to move to a different version, Python 2.6 will not replace Python 2.5, but exist alongside it. Chances are that on your 'porky' machine most installed ports are still expecting Python 2.5. Try:

[cmd=]cd /usr/ports/x11-toolkits/open-motif && make all-depends-list[/cmd]

I'm pretty sure you'll find /usr/ports/lang/python25 instead of /usr/ports/lang/python26.

I would pkg_delete 2.6 and install 2.5 before advancing further.

rambetter · Sep 13, 2010

Hrm actually no [unfortunately].

Code:

nlandys@porky# cd /usr/ports/x11-toolkits/open-motif 
nlandys@porky# make all-depends-list | grep -i pyth
/usr/ports/lang/python26
nlandys@porky# pkg_info -Qao | grep motif
open-motif-2.2.3_6:x11-toolkits/open-motif
nlandys@porky# pkg_info -R open-motif-2.2.3_6
Information for open-motif-2.2.3_6:


nlandys@porky# pkg_info -Qao | grep -i pyth
python26-2.6.6:lang/python26
nlandys@porky#

Yeah, here is my procedure for upgrading ports, and I do this same procedure every time:

Code:

csup ports-supfile
cd /usr/ports
less UPDATING # Read to find out about possible problems when upgrading
make fetchindex
pkgdb -F
portupgrade -ai

So, are the ports that depend on Python on porky broken?
And why did Python 2.5 get upgraded to 2.6 on 2 of the three machines only?
It seems that on daffy, open-motif still depends on Python 2.5 and not 2.6, even though I just csup'ed.

Code:

root@daffy# csup ports-supfile 
Connected to 208.83.20.166
Updating collection ports-all/cvs
Finished successfully
root@daffy# cd /usr/ports/
root@daffy# make fetchindex
/usr/ports/INDEX-7.bz2                        100% of 1422 kB 2790 kBps
root@daffy# pkgdb -F
--->  Checking the package registry database
root@daffy# cd x11-toolkits/open-motif/
root@daffy# make all-depends-list | grep -i pyth
/usr/ports/lang/python25

This is the machine where the upgrade didn't happen (from Python 2.5 to 2.6) and where the dependencies seem to be intact. Why is the dependency for open-motif w.r.t. Python different on daffy and porky?

Do I need to address the dependency issues on porky, and how? I guess I could rebuild all ports or at least those that depend on Python. Is there a convenient command to do this?

BTW thanks for all you help Daemon it's great that you're helping everyone out.

ProFTP · Sep 13, 2010

SUPER script:
http://forums.freebsd.org/showthread.php?t=9232&highlight=super
http://forums.freebsd.org/showthread.php?t=5658
http://www.lissyara.su/articles/freebsd/tuning/fast_port_upgrade/
http://unixforum.org.ua/index.php?topic=25706.0

rambetter · Sep 13, 2010

Sorry, I don't speak Russian.

rambetter · Sep 14, 2010

Code:

pkg_info -Qao > current-packages

for f in `cat current-packages`; do
  PKG_NAME=`echo $f | tr ':' ' ' | awk '{ print $1 }'`
  PKG_PATH=`echo $f | tr ':' ' ' | awk '{ print $2 }'`
  cd "/usr/ports/${PKG_PATH}"
  if make all-depends-list | grep -q "lang/python"; then
    echo $PKG_NAME
  fi
  cd - > /dev/null
done > rebuild-these-packages

portupgrade -fi `cat rebuild-these-packages`

This is what I'm in the process of doing. I imagine there is a flag on portupgrade to do this, but I guess I'm lazy to fully understand the man page. :\

rambetter · Sep 14, 2010

OK I think that the different dependencies are having to do with FreeBSD 8.0 amd64 vs. FreeBSD 7.1 i386. I removed Python and rebuilt all the ports that build-depended on Python, and nothing has changed. I guess maybe 7.1 doesn't distinguish between build dependencies and runtime dependencies, and 8.0 does? Just a guess.

DutchDaemon · Sep 14, 2010

I made the change from 7/32 to 8/32 to 8/64 a while ago, and I have seen no change in behaviour, and Python 2.5 and 2.6 have never traded places without my consent. I'm currently running 2.6 (I think I did a 'formal' 2.5 -> 2.6 upgrade), and everything is just fine. I have not seen any change to how pkg_info and dependencies in general are handled and/or listed (in the 7/32, 8/32 and 8/64 arenas).

ProFTP · Sep 14, 2010

Code:

man portupgrade

Code:

portupgrade -rRf lang/python25

or

Code:

portupgrade -rRf python25

or (N - new port)

Code:

portupgrade -rRfN lang/python25

-rRf - 20-40 hours

Full Upgrade:

1)
/etc/make.conf

Code:

WITCH=BATCH=yes
BATCH=yes

2)

Code:

cd /usr/ports/sysutils/portupgrade && make && make install && make clean

and

Code:

pkgdb -aF

or fix db: pkgdb -fu

3)

minimal script:
(or most likely better: portupgrade -byrf )

Code:

#!/usr/bin/perl

$nn = 0;

while (1) {

    $nn++;

    open( OPERN, "portversion |" );

    my @all2 = <OPERN>;

    close OPERN;

    my @all;

    for ( $i = 0 ; $i < @all2 ; $i++ ) {

        my ( $pp, $st ) = split( / /, $all2[$i], 2 );
        if ( $st =~ '<' ) {

            push @all, $pp;    # $all[$i] = $pp;

        }

    }

    exit if ( !$all[0] || $nn > 2 );

    while (1) {

        last if !$all[0];

        print "$all[0]\n";

        logsave( get_time(), $all[0] );

        system("portupgrade -rf $all[0]");  
      # system("portupgrade -Rf $all[0]");

      # Ð¿ÐµÑ€Ð²Ñ‹Ð¹ Ð¿Ð¾Ñ€Ñ‚ Ð¿Ð¾Ð¿Ñ€Ð¾Ð±Ð¾Ð²Ð°Ñ‚ÑŒ Ð¾Ð±Ð½Ð¾Ð²Ð¸Ñ‚ÑŒ Ð²Ð´Ð¾Ð»ÑŒ Ð¸ Ð¿Ð¾Ð¿ÐµÑ€ÐµÐº
      # (Ð²Ñ‹ÑˆÐµ system Ð½ÑƒÐ¶Ð½Ð¾ Ð·Ð°ÐºÐ¾Ð¼Ð¼ÐµÐ½Ñ‚Ð¸Ñ€Ð¾Ð²Ð°Ñ‚ÑŒ)
      # if ($nn == 1) {
      #  system("portupgrade -rRf $all[0]");
      #  } else {
      #  system("portupgrade -rf $all[0]");
      #  }

   #  # experimentally: 
   #  # use "Enter"
   #    $timeout = '800';
   #    use IO::Select;  
   #     my $select = IO::Select->new;
   #   # for(@array) 
   #     open my $pipe, "| portupgrade -rf $all[0]";
   #     $select->add($pipe);
   #   # }
   #    my @waiters = $select->can_write($timeout);
   #    print $_ "\x0a" for @waiters; 


        logsave( get_time(), $all[0] );

        shift @all;

        my @all = old(@all);

    }

}

sub old {

    my @all = @_;

    open( OPEN2, "portversion |" );

    my @all_all = <OPEN2>;

    close OPEN2;

    my @old;
    my @no_old;

    foreach my $p (@all_all) {

        my ( $pname, $status ) = split( / /, $p, 2 );

        if ( $status =~ '<' ) {

            push @old, $pname;

        }
        else {
            push @no_old, $pname;
        }
    }

    my %seen;
    @seen{@all} = ();
    delete @seen{@no_old};
    return keys %seen;

}

sub get_time {
    my ( $sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst ) =
      localtime(time);
    $mon++;
    $year += 1900;
    if ( $mday < 10 ) { $mday = "0$mday"; }
    if ( $mon < 10 )  { $mon  = "0$mon"; }
    if ( $min < 10 )  { $min  = "0$min"; }
    my $date        = "$mday $mon $year";
    my $time        = "$hour:$min:$sec";
    my $cur_all_day = $mday + $mon * 30 + $year * 365;
    my $radate      = "$year-$mon-$mday $hour:$min:$sec";
    return $radate;
}


sub logsave {
    my ( $time, $ports ) = @_;
    my $logfile;
    $logfile = "\n time:  $time \n  ports: $ports \n\n";
    system("touch /var/log/portupgrade.log");
    open( DB2, "/var/log/portupgrade.log" ) || die "Cannot open file: $!";
    my @base = <DB2>;
    close(DB2);
    open( DB, ">/var/log/portupgrade.log" ) || die "Cannot open file : $!";
    print DB @base;
    print DB $logfile;
    close(DB);
}

Python portupgrade woes

rambetter

DutchDaemon

Administrator

rambetter

ProFTP

rambetter

rambetter

rambetter

DutchDaemon

Administrator

ProFTP