Inconsistencies Between pkg <arg>, pkg_* and portmaster

rtwingfield

rtwingfield

I cannot understand the inconsistent nuances between pkgng and portmaster. With apologies, this is a long and complicated post (I actually have over forty typed pages of examples, test results, and notes accumulated over a period of five man-days. Very frustrating!)

NOTICE:
pkgng is installed on this system
portmaster is installed on this system
portupgrade is not installed on this system.​

Yes . . .I have read the man pages and Handbook, et al.

pkgng is the next generation replacement for the traditional FreeBSD package management tools, offering many features that make dealing with binary packages faster and easier.

pkgng is not a replacement for port management tools like ports-mgmt/ portmaster or ports-mgmt/ portupgrade. These tools can be used to install third-party software from both binary packages and the Ports Collection, while pkgng installs only binary packages.

To ensure that the FreeBSD Ports Collection registers new software with pkgng, and not the traditional packages format, FreeBSD versions earlier than 10.X require this line,
Code: 
WITH_PKGNG= yes
in /etc/make.conf:

Once pkgng is installed, the package database must be converted from the traditional format to the new format by running the command. pkg2ng.​

Unfortunately, there is not much information in the Handbook, Chapter 5.6.3.1. Upgrading Ports Using Portmaster (see http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports-using.html) regarding specific examples of how to use the application.

There are two commands that can generate lists of installed ports and packages:
pkg_info
pkg info​

Additionally, ls listings of the /var/db/pkg and /var/db/ports databases do not agree with the lists generated by pkg_info and pkg info commands.

(From the man pages) the output of PKG_INFO(1) is either extracted from package files named on the command line, or from already installed package information in /var/db/pkg/<pkg-name>.

There are apparently three databases effecting the results:

/var/db/pkg
/var/db/ports
/usr/ports/packages​


/usr/ports/packages is the default directory pointed to by the PACKAGES variable of PORTMASTER(8) and is used to store new and backup packages. The content of this database seems to be volatile or transient.

I'm not sure how the /var/db/ports database is maintained or updated, but I do know that to check for stale entries in /var/db/ports, # portmaster -v --check-port-dbdir apparently tries to clean up the database. For example:

Ran make deinstall in /usr/ports/lang/perl5.14
. . .still has corresponding sub-dir in /var/db/ports that contains an options file. Why?
Running portmaster -v --check-port-dbdir prompted to delete the db for perl5.14.

Note that two versions existed for python:

===>>> Checking lang_python27: Ok
===>>> Checking lang_python33: Ok

python33 is the latest version. How to remove python27? (Used make deinstall.)
Again, had to run portmaster -v --check-port-dbdir to remove from /var/db/ports.​



The portmaster --delete-packages argument sounds dangerous. If there is . . .what is the correct syntax to instruct portmaster to delete only one specific port? If portmaster can delete only one specific port, will it update the /var/db/ports database with having to run portmaster -v --check-port-dbdir? (Every time I've tried to delete a specific single port with portmaster, it launches into a (re)install of the port.) The only way I've been able to delete an installed port is to make deinstall, and apparently the make process does not update the /var/db/ports database (or others uniformily).



Regardless, I have produced a four-column report listing the output of ls /var/db/pkg, ls /var/db/ports, pkg info, and pkg_info. None of them agree with each of the others! This document is a 56K PDF and this forum will not let me upload it, so here is a URL to the same: http://archaxis.net/htdocs/RTWingfield/htdocs/FreeBSD/ports-pkgs/FreeBSD_Port_and_Package_Management_Lists.pdf.

If you look at (relative) page 15 of 19, you will see four columns:
ls /var/db/pkg (360 total) -- ls /var/db/ports (84 total) -- pkg info (376 total) -- pkg_info (358 total)
(The numbers are the number of rows or objects in each column.)

MySQL Example: Neither the mysql-server or mysql-client is listed in the ls /var/db/ports column.
The versions, 5.6.14, are correct for both client and server in the ls /var/db/pkg and pkg_info columns; however, the pkg info column identifies the mysql-server-5.5.33 version! Where did it get this???


Continuing re. Perl: I've tried to install perl5.18 via portmaster, but it seems to insist on reinstalling perl5.14.

Only perl5.14 is listed in the ls /var/db/pkg and pkg_info columns; doesn't appear in either the ls /var/db/ports or pkg info columns. I've just restarted (~5:20 PM) using portmaster -i /usr/ports/lang/perl5.18. This will take a while. It will be interesting to see what changes. See my followup post.


Final Example re. PHP5: This has been a real "bear"! I'm going to try a "remake" using portmaster instead of make deinstall and make build install clean. Currently, the only php5-* objects show up in the pkg info column as "extensions". I will start this once the Perl compiles and (hopefully) installation finish. It will be interesting to see what changes.

It is very annoying that the number and versions in the example columns do not balance.

Why are they so different?

What I suspect: Some make files (when run from a make command) do not consistently update the databases as per pkgng and portmaster hopefully do.

How to synchronize the results?
 
perl5.18 portmaster results

Perhaps this test results will illustrate why this is so frustrating!

Just completed portmaster -i /usr/ports/lang/perl5.18

The results are as follows:

ls /var/db/pkg
perl5.14-5.14.4_2 . . .why is this still here?
perl5.18-5.18.1_2 . . .this was added as a result of the portmaster operations.

ls /var/db/ports
lang_perl5.18 . . .this was added as a result of the portmaster operations.

pkg info
perl5.18-5.18.1_2 . . .this was added as a result of the portmaster operations.

pkg_info
perl5.14-5.14.4_2 . . .why is this still pointing to perl5.14-5.14.4_2?
​

Ran the following, but Perl reported as "corrupt" . . .why?:

# pkg_version -l "<"|less
Code: 
alsa-plugins                        <
bash                                <
bison                               <
ca_root_nss                         <
cairo                               <
desktop-file-utils                  <
dialog4ports                        <
en-freebsd-doc                      <
gettext                             <
gmp                                 <
hal                                 <
libSM                               <
libX11                              <
libXaw                              <
libXmu                              <
libXpm                              <
libXrandr                           <
libXv                               <
libgsf                              <
libidn                              <
libltdl                             <
libvpx                              <
m4                                  <
nspr                                <
nss                                 <
pciids                              <
pkg_version: the package info for package 'perl5.18-5.18.1_2' is corrupt
pixman                              <
webp                                <
xf86-video-intel                    <
xf86-video-vesa                     <
xinput                              <
xkeyboard-config                    <
xorg-server                         <
xterm

Ran the following, but NO mention of Perl
(lines numbers provided by vi)

# portmaster -L --index-only | egrep '(ew|ort) version|total install' > newv_available.txt

Code: 
      1         ===>>> New version available: dialog4ports-0.1.5_2
      2         ===>>> New version available: en-freebsd-doc-42697,1
      3         ===>>> New version available: libvpx-1.2.0
      4         ===>>> New version available: ca_root_nss-3.15.2_1
      5         ===>>> New version available: gmp-5.1.3
      6         ===>>> New version available: m4-1.4.17,1
      7         ===>>> New version available: nspr-4.10.1
      8         ===>>> New version available: pciids-20131023
      9         ===>>> New version available: pixman-0.30.2
     10         ===>>> New version available: cairo-1.10.2_6,2
     11         ===>>> New version available: desktop-file-utils-0.22_1
     12         ===>>> New version available: gettext-0.18.3.1
     13         ===>>> New version available: hal-0.5.14_21
     14         ===>>> New version available: libSM-1.2.2,1
     15         ===>>> New version available: libXaw-1.0.12,2
     16         ===>>> New version available: libXmu-1.1.2,1
     17         ===>>> New version available: libXrandr-1.4.2
     18         ===>>> New version available: libXv-1.0.10,1
     19         ===>>> New version available: libgsf-1.14.28
     20         ===>>> New version available: libidn-1.28_1
     21         ===>>> New version available: php5-gd-5.4.21
     22         ===>>> New version available: php5-pdo-5.4.21
     23         ===>>> New version available: php5-xml-5.4.21
     24         ===>>> New version available: webp-0.3.1_1
     25         ===>>> New version available: xf86-video-intel-2.7.1_5
     26         ===>>> New version available: xf86-video-vesa-2.3.3
     27         ===>>> New version available: xinput-1.6.1
     28         ===>>> New version available: xkeyboard-config-2.9_1
     29         ===>>> New version available: xorg-server-1.7.7_11,1
     30         ===>>> New version available: xterm-297
     31         ===>>> New version available: alsa-plugins-1.0.27_1
     32         ===>>> New version available: bash-4.2.45
     33         ===>>> New version available: bison-2.7.1,1
     34         ===>>> New version available: mysql55-server-5.5.34
     35         ===>>> New version available: nss-3.15.2
     36         ===>>> New version available: php5-filter-5.4.21
     37         ===>>> New version available: php5-json-5.4.21
     38         ===>>> New version available: php5-mbstring-5.4.21
     39         ===>>> New version available: php5-mcrypt-5.4.21
     40         ===>>> New version available: php5-mysql-5.4.21
     41         ===>>> New version available: php5-mysqli-5.4.21
     42         ===>>> New version available: php5-openssl-5.4.21
     43         ===>>> New version available: php5-pdo_sqlite-5.4.21
     44         ===>>> New version available: php5-posix-5.4.21
     45         ===>>> New version available: php5-session-5.4.21
     46         ===>>> New version available: php5-simplexml-5.4.21
     47         ===>>> New version available: php5-snmp-5.4.21
     48         ===>>> New version available: php5-sqlite3-5.4.21
     49         ===>>> New version available: php5-tokenizer-5.4.21
     50         ===>>> New version available: php5-xmlreader-5.4.21
     51         ===>>> New version available: php5-xmlwriter-5.4.21
     52         ===>>> New version available: php5-zip-5.4.21
     53         ===>>> New version available: php5-zlib-5.4.21
     54 ===>>> 377 total installed ports
     55         ===>>> 53 have new versions available
 
portmaster is for upgrading ports. pkg is a binary package manager that keeps track of installed packages and can (optionally) install binary packages. Think of it as the database that tracks which packages are installed. It is a replacement for the old pkg_install programs.

After switching to pkg, stop using the old pkg_install programs. The data they use is no longer updated after the switchover.

There are equivalents to all the pkg_* commands, they are just subcommands given after pkg. For instance, the old pkg_info is pkg info.
 
Hello Warren,

With sincere thanks for your reply, I have read your blog post, http://www.wonkity.com/~wblock/docs/html/portupgrade.html#_updating_the_ports_collection, if I may quote myself (from FreeBSD Handbook The FreeBSD Documentation Project, Revision: 42953, Copyright © 1995-2013 The FreeBSD Documentation Project, Chapter 5.5. Using pkgng for Binary Package Management http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/pkgng-intro.html, ):
pkgng is not a replacement for port management tools like ports-mgmt/ portmaster or ports-mgmt/ portupgrade. These tools can be used to install third-party software from both binary packages and the Ports Collection, while pkgng installs only binary packages.
Click to expand...
I understand this; however, once a reader drills down into the lexicon, it seems that many of the "port management" tools start to refer to functions as " pkg-this" or " pkg-that"; very confusing. IMHO, there should be a clear distinction between pkg delete <args> and portmaster delete <args>. The fact that some of these portmaster operations are executed as pkg-whatever contributes to (my) perception that the commands should only pertain to installed packages.

Arguably (I suppose) . . .once a port is downloaded, compiled, and installed, then the executables are essentially a package.

Regarding you comment,
After switching to pkg, stop using the old pkg_install programs.
Click to expand...
I do not (never have) use pkg_install. I am an old programmer and prefer to compile in-place native code for the platform's OS.

your comment:
There are equivalents to all the pkg_* commands, they are just subcommands given after pkg. For instance, the old pkg_info is pkg info.
Click to expand...
. . .so should the recommendation be to not "trust" the pkg_info command? (I'm concluding . . .apparently so.)

. . .moving on:

pkd info lists mysql-server-5.5.33 when in fact, the version installed and executable is/are:
# mysqladmin -V

mysqladmin Ver 8.42 Distrib 5.6.14, for FreeBSD9.1 on i386

# mysql -V

mysql Ver 14.14 Distrib 5.6.14, for FreeBSD9.1 (i386) using 5.2​


Note that pkg_info "got it right"; however, pkg info still displays mysql-server-5.5.33.

QUESTION: Once portmaster and pkgng are installed, should one ever simply execute a command line make build install clean scenario? I'm still concerned how reliable legacy make files are regarding this new database scenario.
 
@wblock@ is trying to tell you that pkg_info should not be used at all after the conversion to PKGNG packages, the old and the new package tools do not use the same database and you're guaranteed to see inconsistencies if you try to use both.

QUESTION: Once portmaster and pkgng are installed, should one ever simply execute a command line make build install clean scenario? I'm still concerned how reliable legacy make files are regarding this new database scenario.
Click to expand...

No, when you are using PKGNG packages the ports Makefiles are not used any differently than before. All packages are still built using the same ports(7) infrastructure as before, only the packaging and package registration part has changed to use the new pkg(8) tools. Stop thinking that PKGNG has somehow magically replaced everything in the ports system when it really hasn't.
 
Last edited by a moderator:
rtwingfield said:
I understand this; however, once a reader drills down into the lexicon, it seems that many of the "port management" tools start to refer to functions as " pkg-this" or " pkg-that"; very confusing. IMHO, there should be a clear distinction between pkg delete <args> and portmaster delete <args>. The fact that some of these portmaster operations are executed as pkg-whatever contributes to (my) perception that the commands should only pertain to installed packages.

Arguably (I suppose) . . .once a port is downloaded, compiled, and installed, then the executables are essentially a package.
Click to expand...

Not essentially, it is a package. A port builds into a package, which is then installed. Information is kept on where the files are installed and what it depends on. That information in handled by either the old pkg_* programs or the new pkg.

I do not (never have) use pkg_install. I am an old programmer and prefer to compile in-place native code for the platform's OS.
Click to expand...

I meant to say pkg_add. All of those individual programs with the names starting with pkg_ are part of the old system, replaced by pkg.

so should the recommendation be to not "trust" the pkg_info command? (I'm concluding . . .apparently so.
Click to expand...

More than that: the information given by pkg_info is coming from a database that is no longer being used (text files in /var/db/pkg). When you switch to pkg, it keeps all the information in a SQLite table. At that point, looking at the old information is, at best, misleading. It does not apply any more. It's an old, stale copy of information.

QUESTION: Once portmaster and pkgng are installed, should one ever simply execute a command line make build install clean scenario?
Click to expand...

Yes. The ports system builds a package, and pkg records the installation in the database.

I'm still concerned how reliable legacy make files are regarding this new database scenario.
Click to expand...

Port builds, installs, deinstalls, and other operations are the same. It's just the database layer underneath that has been rewritten. To use a database metaphor, the old pkg_* programs were MySQL, and the new pkg system is PostgreSQL. The end user (portmaster, or manual port builds or installs) does not see a difference, only the database engine has changed.
 
This was in response to kpa's comments:

. . .then why does pkg info insist on displaying that mysql-server-5.5.33 is installed, when in fact, mysql56-server-5.6.14 is the current version, ergo:

Code: 
# mysqladmin -V
mysqladmin  Ver 8.42 Distrib 5.6.14, for FreeBSD9.1 on i386

# mysql -V
mysql  Ver 14.14 Distrib 5.6.14, for FreeBSD9.1 (i386) using  5.2
 
That one is harder to answer. Was the conversion to pkg done completely, importing the old package data to the new database? Maybe a port was installed that depended on the old version of MySQL and installed it because the pkg database did not show that the newer version was already installed.
 
Re: Warren Block's comment:
More than that: the information given by pkg_info is coming from a database that is no longer being used (text files in /var/db/pkg). When you switch to pkg, it keeps all the information in a SQLite table. At that point, looking at the old information is, at best, misleading. It does not apply any more. It's an old, stale copy of information.
Click to expand...

I've found no reference to this nuance in any man or Handbook article. How would you know this?

Regardless, entries in /var/db/pkg appear to be updated.

I would prefer that only one scenario be enforced, i.e., installed. For example, if pkg is installed, then all legacy systems should be removed, waxed, deep-six'ed, ;) . . .along with all the pkg_* commands.

/var/db/pkg should be deleted.

This is way too confusing.
 
wblock@ said:
That one is harder to answer. Was the conversion to pkg done completely, importing the old package data to the new database? Maybe a port was installed that depended on the old version of MySQL and installed it because the pkg database did not show that the newer version was already installed.
Click to expand...

. . .well, I did run pkg2ng based on the following documentation:

To ensure that the FreeBSD Ports Collection registers new software with pkgng, and not the traditional packages format, FreeBSD versions earlier than 10.X require this line,
Code: 
WITH_PKGNG= yes
, in /etc/make.conf:

Once pkgng is installed, the package database must be converted from the traditional format to the new format by running the command. pkg2ng.​

Regarding port dependencies, I was working with Perl and PHP . . .both huge applications with (I'm sure) many dependencies.

. . .so I assume that pkg2ng converted /var/db/pkg to the MySQLite db? (I haven't looked for it.)
 
wblock@ said:
That one is harder to answer. Was the conversion to pkg done completely, importing the old package data to the new database? Maybe a port was installed that depended on the old version of MySQL and installed it because the pkg database did not show that the newer version was already installed.
Click to expand...

. . .perhaps an installed version of phpMyAdmin, Perl or PHP. A lot of inter-dependencies.
 
Would it not be sane to simply have pkg_ commands removed (or otherwise hobbled to simply report "You need to use the PKGNG tools!" or similar once the package database is migrated to PKGNG?

If the commands are going to be at best misleading, and possibly dangerous, they should be disabled (or at least require an over-ride switch), in my opinion.
 
rtwingfield said:
Regardless, entries in /var/db/pkg appear to be updated.
Click to expand...
I may be wrong, but in a system that is fully upgraded to use the new pkg tool, there shouldn't be any of the old directories with the +COMMENT, +CONTENTS, etc. files - whether updated or not.
You should only find SQLite databases in there:
ls /var/db/pkg
Code: 
local.sqlite
local.sqlite-journal
repo-packagesite.sqlite

I used pkg2ng months ago, so I don't remember whether it clears the old database or not. So, again, I may be wrong.

When I want to use the new tool, I just remove everything myself and start from scratch:
# rm -R /usr/local
# rm -R /var/db/pkg
# pkg



throAU said:
Would it not be sane to simply have pkg_ commands removed (or otherwise hobbled to simply report "You need to use the PKGNG tools!" or similar once the package database is migrated to PKGNG?
Click to expand...
If you want to start using the new pkg tool, then actually *using* it makes perfect sense. That's the whole point, no?
 
No, it's a fair point. The old programs should not be left around after the conversion, it gives the user an option to make an unnecessary mistake. But they are part of the base system, so it's different than just being a port that can be uninstalled. It might be possible to have the pkg_* programs detect pkgng and refuse to run, which would be an improvement. Anyone want to enter a PR?
 
So, have the pkg_ tools just look for a sqlite database and toss up a "I hope you know what you are doing :)" warning if it's there? Something like this perhaps?

Code: 
#include <stdio.h>
#include <sys/stat.h>
#include <fcntl.h>

struct stat buffer;
int main()
{
if(stat("/var/db/pkg/local.sqlite", &buffer) != 1)
        printf("WARNING: Found /var/db/pkg/local.sqlite\nUse of pkg_ tools not recommended when pkgng is in use\n");
}
 
Yes, that was a very quick and ugly example. I can poke at it a bit this weekend to make it a legit patch and submit a PR. It does bring a question to mind though, since 10.0-RELEASE will be on PKGNG tools will the 9.3-RELEASE that follows be as well? Or will the 9 branch stay with old pkg_ tools until EOL?
 
There may be a common place to patch all of the old pkg_* programs. I don't know what the plans are for FreeBSD 8 and 9.
 
You must log in or register to reply here.
Back
Top