[ports][GIT][portsnap] Warning about INDEX

Zirias

Daemon

Reaction score: 1,346
Messages: 2,370

In the context of ports moving to GIT, as well as (expected) deprecation of portsnap, I've seen some questions regarding INDEX, which was automatically fetched by portsnap.

Right now, I learned the following in an IRC channel full of devs:
Code:
< XXXXXX> INDEX is currently dangerously misleading, that needs to be fixed first
< XXXXXX> I looked into fixing it but given the dependencies between INDEX and portsnap it made sense to wait for portsnap to die first
< Zirias> I never use it anyways. But some workflows need it…
< Zirias> XXXXXX: what exactly is misleading? does it affect a locally created one?
< XXXXXX> currently the dependency info in it can be outright wrong due to failure to interpret flavors
< Zirias> uh!
< XXXXXX> it assumes it can take a port origin without flavor (i.e. cat/port rather than cat/port@flavor) and translate that to a package name
< XXXXXX> which is wrong
< Zirias> so it's essentially broken since flavors are supported? :o
< XXXXXX> yup
(anonymized the handle here and removed timestamps)

I just think anyone should be aware and think twice about using something that relies on INDEX.
 

trev

Aspiring Daemon

Reaction score: 276
Messages: 994

I can't say that I've ever had a problem with INDEX when using portsnap nor now when using gitup. My flavoured ports continue to be updated using portupgrade.

Is there any actual manifestation of the issue?
 

PMc

Daemon

Reaction score: 627
Messages: 1,286

The INDEX was always broken, since ports have options. It should list the dependencies for each port - but as soon as you change an option for some port, the dependencies may change, and then the INDEX (specifically the one that is downloaded) is no longer accurate.

I know only portupgrade that would use the INDEX: And an inaccurate INDEX leads to some prereqs built by portupgrade, others (that are missing in the INDEX) built directly by make. Those built by portupgrade will respect the variables configured in /usr/local/etc/pkgtools.conf, those built by make will not know them.

Now with flavors the chaos might be a little more imminent - and that one might even be fixable, while the essential problem that the INDEX is never really accurate as soon as options bring in other dependencies - that is not fixable.
 

bagas

Well-Known Member

Reaction score: 7
Messages: 258

How then to be?
If I need to update the software on the server.
Earlier I just looked at pkg version | grep '<' .
If everything is ok then portupgrade -arR .
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,346
Messages: 2,370

The INDEX was always broken, since ports have options. It should list the dependencies for each port - but as soon as you change an option for some port, the dependencies may change, and then the INDEX (specifically the one that is downloaded) is no longer accurate.
A locally created INDEX doesn't have this problem unless you change options after creating it, but

Now with flavors the chaos might be a little more imminent
From what I was told (see the IRC log snippet), it's not a little more imminent but outright broken. If flavors are involved, creation of the index doesn't set them when looking for dependencies, so you will get wrong dependencies (probably based on the default flavor, even if a different one is used).
 

Argentum

Well-Known Member

Reaction score: 195
Messages: 424

In the context of ports moving to GIT, as well as (expected) deprecation of portsnap, I've seen some questions regarding INDEX, which was automatically fetched by portsnap.
Noticed

Code:
[Reading data from pkg(8) ... - 1484 packages found - done]
Fetching the ports index ... /usr/bin/env  fetch -am -o /usr/ports/INDEX-13.bz2 https://www.FreeBSD.org/ports/INDEX-13.bz2
/usr/ports/INDEX-13.bz2                        77% of 2271 kB 1815 kBps    01s
fetch: /usr/ports/INDEX-13.bz2 appears to be truncated: 1802911/2326023 bytes
*** Error code 1

Stop.
make: stopped in /usr/ports
failed to fetch INDEX!

Building index locally.
 

mickey

Aspiring Daemon

Reaction score: 302
Messages: 747

Is there any compelling reason not to go completely without the INDEX file then? What tools do actually use it? So far I can tell that pkg version -I uses it, but is also able to use the ports tree or the repository catalog instead. Also ports-mgmt/portmaster has options to make use of the index file at some point, but this doesn't seem mandatory either.

In the past I have used make fetchindex without thinking too much about where that index is actually used. But in light of the facts that the index seems flawed in several ways it's probably time to question it's usefulness.
Fetching the ports index ... /usr/bin/env fetch -am -o /usr/ports/INDEX-13.bz2 https://www.FreeBSD.org/ports/INDEX-13.bz2 /usr/ports/INDEX-13.bz2 77% of 2271 kB 1815 kBps 01s fetch: /usr/ports/INDEX-13.bz2 appears to be truncated: 1802911/2326023 bytes *** Error code 1
This is also one problem I've seen occuring one too many times when fetching the index. I think I saw some discussion about it and that it's probably related to some sort of HTTP accelerator in use on the website, but can't say for sure. Fact is, this has been happening for years.
 
OP
Zirias

Zirias

Daemon

Reaction score: 1,346
Messages: 2,370

Is there any compelling reason not to go completely without the INDEX file then?
IMHO, no. I never used it, I build my local pkg repository with ports-mgmt/poudriere, and pkg gets dependency information from the repository. I don't think INDEX is all too useful for other workflows either. It definitely isn't unless the flavors problem is fixed, and then, you have to create it locally if you changed any port options, default versions etc and you have to re-create it on any change or ports update. I'll keep going without it ;)
 

mickey

Aspiring Daemon

Reaction score: 302
Messages: 747

It definitely isn't unless the flavors problem is fixed, and then, you have to create it locally if you changed any port options, default versions etc and you have to re-create it on any change or ports update. I'll keep going without it
Eh no, no, nonono. So not doing this on a daily basis for each machine (as they have different port options), which also wouldn't work cause the ports tree is a read-only NFSv4 mount. INDEX no more 🥶
 

PMc

Daemon

Reaction score: 627
Messages: 1,286

A locally created INDEX doesn't have this problem unless you change options after creating it,
That's correct - but the offering to change the options usually appears when make starts to build some port. You could run make config-recursive beforehand to get all the options together first.

Later I found out that even when all dependencies are properly found and configured, the ports themselves may look for already installed software and then configure themselves differently on what is already present, consequentially again changing their dependencies.
So I stopped to use portupgrade for creating the depencency-lists, created my own dependency list, and built the ports in a limited chroot where only the identified prereqs are installed.

Then I figured that under these circumstances there is no need to use portupgrade at all.

But there are still ports doing really crappy things. For instance, this construct was in editors/emacs not so long ago:

Code:
# has graphics/ImageMagick been compiled with OPENMP?
.if ${PORT_OPTIONS:MMAGICK} && ${:!${GREP} -sc " \-fopenmp " ${LOCALBASE}/libdata/pkgconfig/ImageMagick.pc || true!} == "1"
USES+=        compiler:openmp
.endif
This is the makefile that wants to build emacs. Now what does this do: it searches for the graphics/ImageMagick port (which is supposed to be already installed), reads the configuration options of that libary from the /usr/local/libdata/pkgconfig/ImageMagick.pc file - and then decides on which compiler to require as a prereq for building emacs!

You cannot capture that in an INDEX file - and it is bogus in any way, because: only after make reads this makefile, it will decide to build+install ImageMagick as a prereq.

What we would need, is. to have all the ports' options kept in some kind of database where other ports can do a lookup. (It does already exist under /var/db/ports, it is just not suitable for arbitrary querying.)

Is there any compelling reason not to go completely without the INDEX file then?
No. Lately I forgot to configure a proper path where to write it, so it isn't written anymore. But since I already ceased to use portupgrade, there is nothing present that would require the INDEX.
 

bagas

Well-Known Member

Reaction score: 7
Messages: 258

But since I already ceased to use portupgrade, there is nothing present that would require the INDEX.
How do you update the software on the server?
portupgrade is convenient, what's the best way to update the software?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,636
Messages: 38,004

Is there any compelling reason not to go completely without the INDEX file then? What tools do actually use it?
If I remember correctly port-mgmt/portupgrade but that port has quite a number of other issues too, I stopped using it a really long time ago. When ports-mgmt/portmaster was available I switched over. Then ports-mgmt/poudriere came around and used that. Never looked back since.

I like my current setup with only one machine building software (using ports-mgmt/poudriere) and all other machines installing with pkg from that repository, exposed by www/nginx.
Same here. It's so convenient, having a single server set up to build a bunch of different repositories for different versions. Then simply use pkg(8) on all my other machines and updates are a breeze.
 

mickey

Aspiring Daemon

Reaction score: 302
Messages: 747

No. Lately I forgot to configure a proper path where to write it, so it isn't written anymore. But since I already ceased to use portupgrade, there is nothing present that would require the INDEX.
portupgrade ... wait what? ... already? Better late than never I guess 🥶

My index is gone, portmaster masters updating without it (no sorcery involved) and pkg version automatically switches to using the ports tree if no index was found, but I usually dont bother using it, as it just takes additional time without adding anything useful:
 

PMc

Daemon

Reaction score: 627
Messages: 1,286

How do you update the software on the server?
portupgrade is convenient, what's the best way to update the software?
There is no "best" way - it depends on your site layout and requirements/preferences. portupgrade is convenient if you just have a few machines, build (some) ports locally and need to keep track (or at least some track) on the dependencies.
If you need anything more big or more structured/reliable, then I would recommend: build all ports on one machine, bring them to the target machines (by whatever means, NFS, http, zfs-send, as appropriate), and create some custom config in /usr/local/etc/pkg/repos/ to install from your own repo.
 

bagas

Well-Known Member

Reaction score: 7
Messages: 258

Removed old ports.
Got new ports with git.
I create an index file.

cd /usr/ports/ && make index
...
--- describe.x11-toolkits ---
--- describe.x11-wm ---
make_index: /usr/ports/databases/p5-AnyEvent-CouchDB: no entry for /usr/ports/databases/couchdb2
Done.

Update soft.
portupgrade -arR
[Reading data from pkg(8) ... - 398 packages found - done]
[Updating the portsdb <format:bdb_btree> in /usr/ports ... - 0 port entries found ..... done]
[missing key: virtual_categories] [Updating the portsdb <format:bdb_btree> in /usr/ports ... - 0 port entries found ..... done]
missing key: virtual_categories: Cannot read the portsdb!
database file error
 

bagas

Well-Known Member

Reaction score: 7
Messages: 258

downloaded the base index.
make fetchindex
pkg version | grep "couchdb2"
I do not have such an installed couchdb2 port.
 

bagas

Well-Known Member

Reaction score: 7
Messages: 258

if I delete /usr/ports/databases/p5-AnyEvent-CouchDB now, then the next time the ports are updated, it will appear again.
 

jmos

Active Member

Reaction score: 176
Messages: 243

So far I can tell that pkg version -I uses it

You mustn't use the "-I" option: The INDEX file is used (if present) by default! Anyone uses portsnap has/had the INDEX, and anyone switched to gitup lost it. So expect many people noticing that a pkg version -vl '<' (looking for non up to date packages) slowed extremely down.
 

mickey

Aspiring Daemon

Reaction score: 302
Messages: 747

You mustn't use the "-I" option: The INDEX file is used (if present) by default!
Personally I never use pkg version at all cause I don't want to know what needs updating, I want to get the update done. Therefore I just run portmaster and if there is nothing to update it probably took about the same time as running pkg version.
 

chrcol

Well-Known Member

Reaction score: 51
Messages: 471

why the need to anonymize names and timestamps, even reveal the channel/server.
 

bagas

Well-Known Member

Reaction score: 7
Messages: 258

You mustn't use the "-I" option: The INDEX file is used (if present) by default! Anyone uses portsnap has/had the INDEX, and anyone switched to gitup lost it. So expect many people noticing that a pkg version -vl '<' (looking for non up to date packages) slowed extremely down.
What to do then?
Before updating the software on the server, I want to know what will be updated.
 

bagas

Well-Known Member

Reaction score: 7
Messages: 258

Had to drop the local index(make index) and download the base index (make fetchindex).
The oddities of switching to git!
 
Top