make index for ports

[jotawski]

Hi,

I got this message when I did a make index for my ports tree

[dell] ~# cd /usr/ports/
[dell] /usr/ports# make index
Generating INDEX-8 - please wait..Warning: Duplicate INDEX entry: openldap-sasl-client-2.4.26
 Done.
[dell] /usr/ports#

I do not understand and how to get rid of this kind of warning.

Thanks in advance for any answers.

 

[YZMSQ]

Maybe this will help ?

 

[jotawski]

Maybe this will help ?

Really thanks for your helps and great hints. I have two openldap in my ports tree. Both of them are in net/, they are openldap-sasl-client and openldap-client. They point to the same server openldap24-server. They give the same package name too.

Is this a bug in ports tree?

Anyway, many thanks indeed for your time.

 

[DutchDaemon]

Update your ports tree and run [cmd=]make fetchindex[/cmd]

 

[jotawski]

Update your ports tree and run [cmd=""]make fetchindex[/cmd]

Once I did a make fetchindex I also got

% pkg_version -v -smouse

....PKGNAME not found in mousepad/Makefile

when I want to check for up to date of a few ports afterwards. But not all of them gave this warning so that I revert back in doing make index and ignore the warning.

Thank you for your hints.

 

[DutchDaemon]

I think your ports tree is not in pristine condition. You should never see errors like the ones you describe. I would really urge you to run a [cmd=]rm /var/db/portsnap/tag && portsnap fetch extract[/cmd]

This should give you a brand-new and complete ports tree, and an up-to-date INDEX file.

 

[jotawski]

I think your ports tree is not in pristine condition. You should never see errors like the ones you describe. I would really urge you to run a [cmd=]rm /var/db/portsnap/tag && portsnap fetch extract[/cmd]

This should give you a brand-new and complete ports tree, and an up-to-date INDEX file.

There is no tag in my /var/db/portsnap/ but I am now fetching.

 

[DutchDaemon]

No tag. That would suggest you either never updated your ports tree, or didn't use portsnap to do that. Maybe you were using csup or cvsup. Switching to portsnap is advisable (don't forget to take the 'ports' stuff out of csup/cvsup if you're switching over).

Once the portsnap fetch extract is done you can keep the ports tree up-to-date with portsnap fetch update, or use my script to automate the process.

 

[jotawski]

No tag. That would suggest you either never updated your ports tree, or didn't use portsnap to do that. Maybe you were using csup or cvsup. Switching to portsnap is advisable (don't forget to take the 'ports' stuff out of csup/cvsup if you're switching over).

Once the portsnap fetch extract is done you can keep the ports tree up-to-date with portsnap fetch update, or use my script to automate the process.

Yes, this is my new machine. I always use my old faithful cvsup with all supfiles from /usr/share/examples/cvsup and look for the feastest host via fastest_cvsup and pass the first one to the command, for example of ports
[cmd=""]# cvsup -g -L 2 -h host /usr/share/examples/cvsup/ports-supfile[/cmd]

Once again really thanks for all your helps and hints.

 

[SirDice]

I got this message when I did a make index for my ports tree

[dell] ~# cd /usr/ports/
[dell] /usr/ports# make index
Generating INDEX-8 - please wait..Warning: Duplicate INDEX entry: openldap-sasl-client-2.4.26
 Done.
[dell] /usr/ports#

I do not understand and how to get rid of this kind of warning.

FWIW for the past decade or so I've seen similar duplicate entry messages come up from time to time (I always make index myself). You can safely ignore them. It's just a slight error and it's usually corrected after a short while. Even with the error the INDEX file will be good to use.

 

[rwv37]

Resurrecting ancient post... does anyone have any idea how to fix this now that portsnap and cvsup have been retired? I use gitup, and have been having this problem for... a couple months, I guess? On mldonkey-core-3.1.7.2, which I don't have installed. It seems not to be causing any problems, but I'd still like to fix whatever issue is behind it. Thanks.

 

[zirias@]

• Issues creating a ports index are always (subtle?) errors in individual ports, so "resurrecting ancient post" isn't the best idea, it's most likely a completely different issue.
• The portsnap utility fetched a pre-created index. The index depends on port options and similar, so as long as you're not building with ALL default settings, it has always been (slightly) wrong.
• If "it doesn't cause problems", do you actually know why you're creating an index at all? If you don't have an actual need for it, there's nothing to fix either ?‍ (I personally never used it)

 

[rwv37]

• Issues creating a ports index are always (subtle?) errors in individual ports, so "resurrecting ancient post" isn't the best idea, it's most likely a completely different issue.

Well, the symptom is exactly as described here, except for the port itself. And I've encountered this post several times over the course of the past months as one of the top hits while searching (or possibly the top hit). Given that the suggestions in it are defunct, I thought it would be good to try to get some up-to-date suggestions so as to hopefully stop wasting people's time when they search for this type of issue and discover this now-essentially-useless post.

• The portsnap utility fetched a pre-created index. The index depends on port options and similar, so as long as you're not building with ALL default settings, it has always been (slightly) wrong.

I'm not sure that I know what you mean by "always" here, but it definitely hasn't always been happening on this machine for me. It's been happening for months.

• If "it doesn't cause problems", do you actually know why you're creating an index at all? If you don't have an actual need for it, there's nothing to fix either ?‍ (I personally never used it)

I didn't say it doesn't cause problems. I said it seems not to be causing any problems. I do not know that it's causing problems, but I also do not know that it doesn't, nor do I know that it won't. Since it doesn't seem to be causing problems, this is not a terribly high priority for me -- like I said, it's been happening for months, and this is the first time in all that time that I've mentioned it to any other people -- but all things equal, I would rather get it fixed than leave it as is.

As for actually knowing why I am creating an index at all, yes, I do:

When I migrated from portsnap (in September of last year), I started by using git pull instead. I then started having some minor issues based on the fact that git pull did not automatically update the index (as had always been done by portsnap). In October, I switched to gitup ports instead, hoping that it would resolve the issues; unfortunately, it did not. So I then looked into how to build the index myself, found make index, and started doing that as a matter of course after every gitup ports.

One such minor issue that I remember specifically is that ports-mgmt/psearch started having difficulties (I forget whether it gave incorrect output or stopped working entirely or what).

Another (but to be clear, for this one, I'm pretty sure that the lack of make index was the cause of this issue, but at this point I'm no longer 100% sure) was that I started frequently getting emails from my automated build system incorrectly telling me that I had some out-of-date ports installed: Part of my system does a pkg version -v | grep -v up-to-date, and emails me if it that results in anything. Since the lack of an updated index meant that the grep would now detect not just installed versions that were older than what was in the index, but also installed versions that were newer than what was in the index, I started getting these frequent spurious emails.

 

[zirias@]

I'm not sure that I know what you mean by "always" here, but it definitely hasn't always been happening on this machine for me. It's been happening for months.

Again, index depends on all sorts of ports configuration (like individual port options). It lists dependencies, and these often depend on selected options. An index fetched by portsnap matches all default options, so as soon as you configure something, it will be "wrong" somewhere.

The only way to get a fully correct ports index is, as soon as you do configure something, building it yourself. Still, you can always fetch the pre-built index, portsnap isn't required for that, it's as simple as running make fetchindex in the root of your ports tree.

One such minor issue that I remember specifically is that ports-mgmt/psearch started having difficulties (I forget whether it gave incorrect output or stopped working entirely or what).

That's indeed a tool that uses the index. As does pkg-version for locally built and installed ports (which I personally would never recommend, instead building your own package repository with poudriere and installing from there is much more robust).

 

[rwv37]

I do build my own package repository and install from there (though with synth, not poudriere). I have virtually everything automated, and run overnight while I'm sleeping, pulling the latest source, making the index, having synth determine whether there's anything it should rebuild, having synth rebuild whatever it determines should be rebuilt, and updating my repository with the newly compiled stuff. The only thing I don't have done for me automatically overnight is actually updating my system from the repository, which takes very little time to do manually and which I feel more comfortable doing manually just in case something goes wrong or whatever. pkg version is very useful for me with regards to that - like I said, I automatically get emailed if (and only if) the result of all this stuff is that synth has built new things for me, and pkg version is what enables that.

I appreciate that you're trying to help, but would it be possible to please stick to the question of whether or not anyone knows how to resolve (or how to further investigate) this sort of warning, rather than trying to convince me to update installed software in your preferred manner rather than in my preferred manner, which has been working great for me for years? In case it's not clear, I'm not trying to be snarky or anything here. Thanks.

 

[SirDice]

As does pkg-version for locally built and installed ports

Minor nitpick, pkg-version(8) doesn't care if ports have been locally built from ports or not. All it does is compare versions from different sources.

     The database of available packages and versions to compare against the
     installed packages may be chosen by specifying one of -P, -R or -I or by
     setting VERSION_SOURCE in pkg.conf(5).  If not specified then the ports
     index file will be used if it exists (-I).  Otherwise, should a ports
     tree exist that will be used to compare versions (-P).  Failing either of
     those two choices, the repository catalogue will be used (-R).

     -I [index], --index [index]
                 Use index file for determining if a package is out of date.
                 If no index file name is specified, uses the default index
                 file.  This is the default, if the index file exists.

     -P, --ports
                 Use ports for determining if a package is out of date.  This
                 is the default if the index file is not present and a ports
                 tree exists.  The tree used can be overridden by PORTSDIR,
                 see pkg.conf(5) for more information.

     -R, --remote
                 Use repository catalogue for determining if a package is out
                 of date.  This is the default if neither the ports index nor
                 the ports tree exists.

I personally always explicitly use -R because I want to compare against my "remote" repository, I usually don't care in what state the local ports tree is (it might be out of date on that system).

I'm not that familiar with Synth but I presume it doesn't use or care about INDEX, poudriere certainly doesn't.

 

[richardtoohey2]

An index fetched by portsnap matches all default options, so as soon as you configure something, it will be "wrong" somewhere

I have a few ports where I don’t use the defaults and I can’t recall seeing this described behaviour.

But portsnap on the way out and I’m slowly getting used to gitup and not using the -I flag on pkg version.

Related post (maybe?):

readline 8.2.10

FreeBSD 13.2-RELEASE-p8 devel/readline was installed a few days ago with no errors. However I'm now being informed that ... readline-8.2.10 > succeeds index (index has 8.2.:[.1]}) and to me that index looks corrupted. This isn't a major issue but any ideas what's gone...

forums.freebsd.org

 

[zirias@]

I have a few ports where I don’t use the defaults and I can’t recall seeing this described behaviour.

There's actually no behavior described here regarding using a "slightly wrong" index.

What's described here is a failure to build the index, which seems to be triggered by custom port configuration somehow (so there's definitely something wrong in some port, but it's "hidden" with all-default options). It should always be possible to build an index locally, which then takes all your local configuration into account.

Please don't confuse these. As I said, you can always fetch the pre-built index. It won't be 100% correct unless you build your ports with all-default options.

 

[Tieks]

anyone knows how to resolve (or how to further investigate) this sort of warning

Two reasons for such warnings that I know of. First, these errors may incidentally come in with a regular update of the ports tree. These will disappear soon enough with a later update. I haven't seen such a thing in a long time. Second reason is some error in you ports tree. These usually won't go away with a later update, like the one with your mldonkey-core. Donkeys tend to be stubborn.

In such a case, compare the contents of the port entry with a known good one, for instance https://ports.freebsd.org/cgi/ports.cgi and correct the contents. If it's a port that you don't use at all, you could just delete that entire port entry. If you have reason to believe there may be more of these problems in you ports tree, consider downloading a fresh copy of the ports tree. First, remove everything in /usr/ports. Then for latest, use git clone https://git.FreeBSD.org/ports.git /usr/ports. For quarterly, use git clone https://git.FreeBSD.org/ports.git -b 2024Q2 /usr/ports.

A download will give you the standard index /usr/ports/INDEX-NN, which is the same file that make fetchindex will download. That will be okay in most cases. If you altered many port configuration options and therefore have many entries in /etc/make.conf and /var/db/ports, it is best to create the index yourself with make index so that it reflects the changes you made. Once /usr/ports/INDEX-NN is up-to-date, file /usr/ports/INDEX-NN.db (=the faster database version of INDEX-NN) needs to be updated too. Most pkg utils will do that automatically, you can do it with portsdb -fu.

 

[SirDice]

consider downloading a fresh copy of the ports tree.

If I get weird issues with my ports tree(s) that's usually the first thing I do. Just nuke the tree and clone a fresh copy. I still mess around with git quite a lot, haven't quite gotten the hang of merging, rebasing, cherry-picking, etc. yet. I often end up with a completely broken working copy

In any case, I would try to forgo make index (and remove any existing INDEX-* files) and see if Synth complains about it. INDEX is a bit of a relic from the past, nothing in the ports system actually relies on it any more.

 

[T-Daemon]

rwv37

In any case, I would try to forgo make index (and remove any existing INDEX-* files) and see if Synth complains about it.

Infrequent ports-mgmt/synth user here, synth does't use ports INDEX, it creates its own index file. Not the same format though.

/var/cache/synth/LiveSystem-index

accessibility/accerciser
accessibility/at-spi2-core
accessibility/atkmm
accessibility/caribou
accessibility/darkman
...
databases/py-sqlite3@py39
databases/py-sqlite3@py27
databases/py-sqlite3@py38
databases/py-sqlite3@py310
databases/py-sqlite3@py311

synth(1)

     <profile>-index   The introduction of flavors within FreeBSD ports
                       destroyed the 1:1 relationship between port directories
                       and packages.  Now any port can generated any number of
                       packages.  This requires scanning every port in the
                       tree to determine the true origin range.  This is a
                       time consuming task, so Synth caches the scan results
                       and reuses it as long as all ports files are older than
                       the generated index.  When Synth detects the index is
                       out-of-date or if it doesn't exist, it is automatically
                       generated.  One index exists for each profile, with the
                       default profile index normally located at
                       /var/cache/synth/LiveSystem-index

 

[rwv37]

I think I might know what's going on, but I am not familiar enough with the ins and outs of the ports system, make, etc. to say for sure, so I'd appreciate if anyone could please tell me whether or not I'm on the right track here:

I found that the message is coming from within /usr/ports/Tools/make_index, and it happens based on the "name" having already been handled (or some such thing - it's a Perl script and I am also not familiar with Perl, lol). From there I tried to figure out where this "name" was coming from. Unfortunately I haven't had much sleep and so I've lost track of exactly where I found this information or exactly what it said, but I gathered that it's coming from within the make stuff for each individual port, and that if there are two or more ports that have the same base name (not sure that "base" is the proper term here), they are supposed to be distinguished from each other by "PKGNAMEPREFIX" and "PKGNAMESUFFIX" (and maybe other things too, I dunno).

So, I looked for mldonkey ports. I found three: net-p2p/mldonkey, net-p2p/mldonkey-core, and net-p2p/mldonkey-gui. The Makefiles for the -core and -gui ones are pretty short; each just sets a few variables (PKGNAMEPREFIX and PKGNAMESUFFIX are not among them) and then includes the Makefile for the main mldonkey port.

I searched mldonkey/Makefile for PKGNAME. It doesn't contain any references (at least not obvious, direct ones) to PKGNAMEPREFIX, but it does contain a couple to PKGNAMESUFFIX:

. if !${PORT_OPTIONS:MGUI}
PKGNAMESUFFIX=    -core

(...)

. if !${PORT_OPTIONS:MCORE}
PKGNAMESUFFIX=    -gui

Given that the message complains about a duplicate of "mldonkey-core", I am guessing that both mldonkey-core and mldonkey-gui are hitting that .if !$(PORT_OPTIONS:MGUI), and therefore setting PKGNAMESUFFIX to -core.

Unfortunately, due to my lack of knowledge and/or sleep, I haven't been able to figure out where this PORT_OPTIONS:MGUI thing (or, more specifically, the lack thereof) is coming from, but I do know this: One of the things I do in my make.conf is OPTIONS_UNSET+=GUI.

I'm guessing that that somehow causes PORT_OPTIONS:MGUI to be false, regardless of whether or not the actual package being analyzed is mldonkey-core or mldonkey-gui.

Does this make sense? Should I bring it up to the maintainer(s) of the various mldonkey ports? Or am I doing something wrong? GUI has been one of the things that I've intentionally turned off in make.conf on a whole bunch of FreeBSD machines for literally decades, and as far as I can tell doing so has been fine for all that time, but maybe I shouldn't be doing that? Or should be doing it differently somehow?

Thanks for any help.