Solved 'poudriere bulk' not finding updates to the tree

You might want to fix this:
Code: 
[00:00:00] Warning: Using '-' in a SETNAME is not recommended as it causes ambiguities with parsing the build name of kde-default-base-options
 
SirDice said:
You might want to fix this:
Code: 
[00:00:00] Warning: Using '-' in a SETNAME is not recommended as it causes ambiguities with parsing the build name of kde-default-base-options
Click to expand...
I actually noticed that, too. I'm a little reluctant to completely rebuild a successfully built base (It did take my processor 28 hours to compile that, not including a needed reboot when the whole thing was about 60% done).
--
If I do have to re-do the whole thing from scratch, I would move the base-options to base_options. Or is there a way to do the renaming w/o redoing the whole thing from scratch?
--
Think that would help the null-mounting issues?
 
Looks like I misread the CUSTOMIZATION section in poudriere(8) for the -z option to pass to poudriere-bulk(8) 😩 . SirDice really hit the head on the nail when he pointed out in post #26 what to fix.
--
After a week of thinking (and wrestling with git), I realized that /usr/local/etc/poudriere.d/options and the -z option for custom stuff doesn't fit my scenario - I can simply use /var/db/ports, because I just enable as many options as I can in the make config anyway. Right now, I am recompiling (Poduriere is not complaining about not finding KF5 5.82 vs 5.85), and facing the prospect of tracking down the correct directories. This is easier than fighting with SSL certs and git.
 
Dammit... even with fixing the options, I don't seem to be getting anywhere. Poudriere still looks for KF5 5.82. I guess it's back to fighting with git.
 
Update: git work-around completed. But even that is not helping:
Code: 
# poudriere ports -l
default   git+file 2021-12-11 10:52:55 /usr/local/poudriere/ports/default/
('remote' repo that I manually manage is at /usr/ports/poudriere_tree)
--
# poudriere ports -u -p default works fine, git fishes out my updates correctly. But:
--
# poudriere bulk -f new_list.txt -j kde (note, no -z option, not needed for me) is not working, Poudriere still tries to look for KF5 5.82. But, running # poudriere bulk -f new_list.txt -j kde -vv -p default produced something interesting. A partial dump of Typescript follows:
Code: 
...
Line 4: =>> [0;0mDebug:[0;0m Reading /usr/local/etc/poudriere.conf[0;0m
...
Line 14: [00:00:00] Loading MOVED for /usr/local/poudriere/data/.m/kde-default/ref/usr/ports
...
Line 20: [1;33m[00:00:01] [0;0m[1;33mWarning: [0;0m([0;36mdevel/kf5-kio[0;0m): [1;34m[0;0m[1;34mDebug:[0;0m deps_fetch_vars: discovered devel/kf5-kio is kf5-kio-5.82.0[0;0m[0;0m
...
# more output follows, 1.7 MB worth of text.

Code: 
# cat /usr/ports/devel/kf5-kio/distinfo
TIMESTAMP = 1620404276
SHA256 (KDE/frameworks/5.82.0/kio-5.82.0.tar.xz) = cb970d0550054fd0bb0957de010b9ea73b2d89e8df725c0d8344320e82473144
SIZE (KDE/frameworks/5.82.0/kio-5.82.0.tar.xz) = 3330028
# cat /usr/local/poudriere/ports/default/devel/kf5-kio/distinfo
TIMESTAMP = 1628252758
SHA256 (KDE/frameworks/5.85.0/kio-5.85.0.tar.xz) = e4d58a7d2ae6530c4b676fe7f7e4583a86436c68861eb971bf91af9efead19f6
SIZE (KDE/frameworks/5.85.0/kio-5.85.0.tar.xz) = 3200300

It seems like poudriere-bulk(8) is looking in the wrong place... mounting the base /usr/ports tree, rather than /usr/local/poudriere/ports/default/ tree.

So, any ideas on how to fix that? I'd appreciate any help.

Edit: I tried to create a different tree, named 'freebsd_kde', but that did not help, same results as before.
 
Code: 
/usr/local/bin/poudriere ports -k -d                       -p default
/usr/local/bin/poudriere ports -c -m null -M /usr/local/poudriere/ports/default    -p default
/usr/local/bin/poudriere ports -l
 
Alain De Vos said:
Code: 
/usr/local/bin/poudriere ports -k -d                       -p default
/usr/local/bin/poudriere ports -c -m null -M /usr/local/poudriere/ports/default    -p default
/usr/local/bin/poudriere ports -l
Click to expand...

OK, tried that. No difference at all. Issue remains.
 
😩 Even with git finally working right for me ( poudriere ports -c and poudriere ports -u producing the expected results (Plasma bumping from 5.21 to 5.22.5, and KF5 from 5.82 to 5.85), original problem still remains! I can see changed files on disk ( cat distinfo shows the correct version) but poudriere bulk still thinks I haven't done a thing!

Trying a fresh jail and a different name for the ports tree did absolutely nothing!
Original poudriere bulk:
Code: 
poudriere bulk -f new_list.txt -j kde -p freebsd_kde -z options
Modified poudriere bulk:
Code: 
poudriere bulk -f new_list.txt -j kde_0 -p my_kde -z options

The modified version takes longer to compile (duh, that's to be expected), but in the end fails exactly the same as the original.
 
What exactly are the options?

astyle said:
poudriere bulk
Click to expand...

Repeat using option -v then share a transcript. Include the commands that precede the bulk run.

Also, please share the content of new_list.txt, and outputs from the commands below.
  1. poudriere ports -l
  2. poudriere jail -i -j kde
  3. poudriere jail -i -j kde_0
  4. pkg info -x poudriere ; freebsd-version -kru ; uname -aKU ; pkg -vv | grep -e url -e enabled
 
grahamperrin said:
What exactly are the options?



Repeat using option -v then share a transcript. Include the commands that precede the bulk run.

Also, please share the content of new_list.txt, and outputs from the commands below.
  1. poudriere ports -l
  2. poudriere jail -i -j kde
  3. poudriere jail -i -j kde_0
  4. pkg info -x poudriere ; freebsd-version -kru ; uname -aKU ; pkg -vv | grep -e url -e enabled
Click to expand...

The options have been prepared as suggested in the poudriere(8) manpage's CUSTOMIZATION section:
As a starter, you may want to copy an existing /var/db/ports/ to
/usr/local/etc/poudriere.d/options.
Click to expand...

Contents of new_list.txt : Just one line, x11-toolkits/kf5-attica. Its only KF5 dependency is devel/kf5-extra-cmake-modules.

I haven't upgraded Poudriere since starting this thread, so it's still at 3.3.7_1. The rest of the info will come in about 10 hours, when I will have access to my rig.
 
grahamperrin said:
and outputs from the commands below.
  1. poudriere ports -l
  2. poudriere jail -i -j kde
  3. poudriere jail -i -j kde_0
  4. pkg info -x poudriere ; freebsd-version -kru ; uname -aKU ; pkg -vv | grep -e url -e enabled
Click to expand...
Code: 
# poudriere ports -l
my_kde    git+file 2022-02-10 15:54:54 /usr/myports
# poudriere ports -u -p my_kde
[00:00:00] Updating portstree "my_kde" with git+file... done
# poudriere jail -i -j kde
Jail name:         kde
Jail version:      13.0-RELEASE-p5
Jail arch:         amd64
Jail method:       http
Jail mount:        /usr/local/poudriere/jails/kde
Jail fs:           zroot/poudriere/jails/kde
Jail updated:      2022-01-09 17:21:45
root@beastie:/home/alex # poudriere jail -i -j kde_0
Jail name:         kde_0
Jail version:      13.0-RELEASE-p7
Jail arch:         amd64
Jail method:       http
Jail mount:        /usr/local/poudriere/jails/kde_0
Jail fs:           zroot/poudriere/jails/kde_0
Jail updated:      2022-02-09 20:28:08
# pkg info -x poudriere ; freebsd-version -kru ; uname -aKU ; pkg -vv | grep -e url -e enabled
poudriere-3.3.7_1
13.0-RELEASE
13.0-RELEASE
13.0-RELEASE
FreeBSD beastie.localdomain 13.0-RELEASE FreeBSD 13.0-RELEASE #0: Fri Jul 16 18:48:13 PDT 2021     astyle@beastie.localhost.localdomain:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64 1300139 1300139
    url             : "https://pkg.beastie.localdomain/",
    enabled         : yes,
I'm not using poudriere-devel because I have not seen any information that says "using poudriere-devel as opposed to plain poudriere will make a difference" .

I de-registered the freebsd_kde tree, since Poudriere could not detect any changes to it (even though the changes are clearly there! That's why only the my_kde tree is still there from poudriere ports -l

As for transcripts of the poudriere bulk run with the -v option - that's coming in another post soon.
 
OK, here's the transcript that represents the output of the command: (I changed new_list.txt to list devel/kf5-extra-cmake-modules, because that one doesn't have any KF5 dependencies.
poudriere bulk -v -f new_list.txt -j kde_0 -p my_kde -z options.

BTW, I don't mind re-building, I just run a job overnight while I sleep :p
 

Attachments

  • typescript.txt
    170.5 KB · Views: 83
Something's telling me that this might help:
Code: 
 # zfs list | grep poudriere
zroot/poudriere                               18.4G   163G       96K  /zroot/poudriere
zroot/poudriere/data                          14.0G   163G      144K  /usr/local/poudriere/data
zroot/poudriere/data/.m                        128K   163G      128K  /usr/local/poudriere/data/.m
zroot/poudriere/data/cache                    33.9M   163G     33.9M  /usr/local/poudriere/data/cache
zroot/poudriere/data/images                     96K   163G       96K  /usr/local/poudriere/data/images
zroot/poudriere/data/logs                      265M   163G      265M  /usr/local/poudriere/data/logs
zroot/poudriere/data/packages                 13.7G   163G     13.7G  /usr/local/poudriere/data/packages
zroot/poudriere/data/wrkdirs                    96K   163G       96K  /usr/local/poudriere/data/wrkdirs
zroot/poudriere/jails                         2.76G   163G       96K  /zroot/poudriere/jails
zroot/poudriere/jails/kde                     1.38G   163G     1.38G  /usr/local/poudriere/jails/kde
zroot/poudriere/jails/kde_0                   1.38G   163G     1.38G  /usr/local/poudriere/jails/kde_0
zroot/poudriere/ports                         1.63G   163G       96K  /zroot/poudriere/ports
zroot/poudriere/ports/default                   96K   163G       96K  /usr/local/poudriere/ports/default/
zroot/poudriere/ports/freebsd_kde              873M   163G      873M  /usr/local/poudriere/ports/freebsd_kde/
zroot/poudriere/ports/my_kde                   798M   163G      798M  /usr/myports
A fresh listing.
 
From the attached typescript.txt file in post #39, I think it's those 10 lines where poudriere messes up and thinks I still have KF5 5.82, rather than KF5 5.85... it should be possible to increase the verbosity of poudriere bulk output by specifying the -v option twice, so maybe I'll go with that. 😩
Code: 
[00:00:04] Checking for stale cache files
[00:00:04] Checking packages for incremental rebuild needs
[00:00:05] Checking packages for missing dependencies
[00:00:05] Deleting stale symlinks... done
[00:00:05] Deleting empty directories... done
[00:00:05] Cleaning the build queue
[00:00:05] Sanity checking build queue
[00:00:05] Processing PRIORITY_BOOST
[00:00:05] Balancing pool
[00:00:05] Recording filesystem state for prepkg... done
 
Tried specifying the -vv option for poudriere bulk... turns out Poudriere is getting confused much earlier:
Code: 
[00:00:01] Debug: queueing devel/kf5-extra-cmake-modules into gatherqueue (rdep=listed)
[00:00:01] Warning: (devel/kf5-extra-cmake-modules): Debug: gather_port_vars_port (devel/kf5-extra-cmake-modules): LOOKUP
[00:00:01] Warning: (devel/kf5-extra-cmake-modules): Debug: deps_fetch_vars: discovered devel/kf5-extra-cmake-modules is kf5-extra-cmake-modules-5.82.0
The distinfo file says the version should be 5.85, so what gives???
 
Renamed the thread to 'poudriere bulk' not finding updates to the tree. Post #43 describes the issue. I tried passing the -m null and -m git+file to poudriere ports -c tree creation, but no go, same result that only shows up at compilation step. 😩
 
Maybe jails is my next bullfight / adventure on this road. Figure out what poudriere bulk tries to do inside a jail, line it up, and follow along, hopefully I will discover something useful. Stuff like null-mounting, evaluating deps, etc. Getting updates into a jail seems problematic... 😩 gotta figure out why.
 
grahamperrin said:
Without me looking in depth:



– maybe think about the state of the source (from which the tree is updated).
Click to expand...
Well, when I do this:
Code: 
# poudriere ports -l
my_kde    git+file 2022-02-10 15:54:54 /usr/myports
# poudriere ports -u -p my_kde
[00:00:00] Updating portstree "my_kde" with git+file... done
# cat /usr/myports/devel/kf5-extra-cmake-modules/distinfo
TIMESTAMP = 1628252745
SHA256 (KDE/frameworks/5.85.0/extra-cmake-modules-5.85.0.tar.xz) = 7a4209c3b113dc50250920186a2d30b71870e11ebb92a700a611b423ce6b6634
SIZE (KDE/frameworks/5.85.0/extra-cmake-modules-5.85.0.tar.xz) = 318048

That tells me that the update went fine - until I run poudriere bulk -f new_list.txt -j kde_0 -p my_kde -z options. Even git log looks OK outside of the jail.

Or is there something to check in git to make sure the changes actually make it into the jail and are visible to Poudriere? I am a git newbie, so if there are some things I can check for, I'd appreciate some pointers.
 
The update is done, but what about the source from which it updates?

I mean, git+file, I'm not familiar with that combination, where exactly is the source?
 
grahamperrin said:
The update is done, but what about the source from which it updates?

I mean, git+file, I'm not familiar with that combination, where exactly is the source?
Click to expand...
git+file - in my case, the source is a bare git tree at
file:///usr/local/www/git/kde_ports.git/ . <--- This is what is specified in poudriere.conf. It's a bare repo to which I publish my modifications with a git push from /usr/local/poudriere/ports/freebsd_kde/. I thought I got git working OK as per this thread:
forums.freebsd.org

Solved - Git pull not pulling in updates!

Discussion: I set up a manually-managed 'Remote' git repo at https://git.beastie.localdomain/freebsd_kde/.git . Corresponding localhost directory is: /usr/local/poudriere/ports/freebsd_kde/ I can use that HTTP URL to clone it: # pwd /home/astyle # git clone...
forums.freebsd.org forums.freebsd.org
Edit: I actually tried to go with git+https first, but ended up going with git+file because git did not like self-signed certs that I set up.
Edit 2: Now that I think about it, I may have gotten lost in filepaths and the trees when I was setting things up. Like where I should do my modifications/updates, where to publish my changes, and where to compile. I don't like the idea of going back and starting again from scratch, but I may have to bite the bullet at some point. 😩
 
Ahh... after 2 more weeks of fighting the issue, I re-started the setup on a different SSD:
  • I tried ports-mgmt/poudriere-devel this time around (from ports dated around June of 2021, so version is 3.99. .... something.) It compiled just fine.
  • /usr/home/astyle/my_ports/ <--- Manually-managed git repo. A complete tree where I do my changes (updating just the ports I want, leave everything else alone).
  • file:///usr/local/www/git/ports.git/ . <--- This is what is specified for $GIT_PORTSURL in poudriere.conf. It's a bare repo to which I publish my modifications with a git push from /usr/home/astyle/my_ports/
  • Created a new tree with the following command: # poudriere ports -c -m git+file -B master -p freebsd_kde. That created a tree at the expected location (/usr/local/poudriere/ports/freebsd_kde/), and a corresponding ZFS dataset.
  • Created a new jail with the following command: # poudriere jail -c -v 13.0-RELEASE -a amd64 -j kde_0. That created a jail at the expected location (/usr/local/poudriere/jails/kde_0/), and a corresponding ZFS dataset.
  • Initially, a run of poudriere bulk is a success.
So far, everything is going as documented in official documentation (manpages, Handbook, and even Porter's Handbook). BUT:
(To demonstrate the issue, I will use portsnap, it will make it a bit easier to follow along)
Code: 
# pwd
/usr/home/astyle/my_ports/
# cat devel/kf5-kio/distinfo
TIMESTAMP = 1620404276
SHA256 (KDE/frameworks/5.82.0/kio-5.82.0.tar.xz) = cb970d0550054fd0bb0957de010b9ea73b2d89e8df725c0d8344320e82473144
SIZE (KDE/frameworks/5.82.0/kio-5.82.0.tar.xz) = 3330028
# portsnap extract -p /usr/home/astyle/my_ports/ devel/kf5-kio/
# cat devel/kf5-kio/distinfo
TIMESTAMP = 1628252758
SHA256 (KDE/frameworks/5.85.0/kio-5.85.0.tar.xz) = e4d58a7d2ae6530c4b676fe7f7e4583a86436c68861eb971bf91af9efead19f6
SIZE (KDE/frameworks/5.85.0/kio-5.85.0.tar.xz) = 3200300
# git status
(shows that my edits are visible, but need to be staged for commit)
# git add --all
(Stages everything I edited)
# git commit -m 'Commit 1, KF5 5.82 -> 5.83, Plasma5 5.21.5 -> 5.22.5'
# git push /usr/local/www/git/ports.git/ master
# cd /usr/local/www/git/ports.git/
# git log
(My commit entry shows up, among other output)
# cat devel/kf5-kio/distinfo
TIMESTAMP = 1628252758
SHA256 (KDE/frameworks/5.85.0/kio-5.85.0.tar.xz) = e4d58a7d2ae6530c4b676fe7f7e4583a86436c68861eb971bf91af9efead19f6
SIZE (KDE/frameworks/5.85.0/kio-5.85.0.tar.xz) = 3200300
(So far, so good)
# cd /usr/home/astyle
# poudriere ports -u -p freebsd_kde
(Message of success, "ports -u" is really a front-end to "git pull /usr/local/www/git/ports.git/ master")
# cd /usr/local/poudriere/ports/freebsd_kde/
# git log
(My commit entry shows up, among other output, looks correct)
# cat devel/kf5-kio/distinfo
TIMESTAMP = 1628252758
SHA256 (KDE/frameworks/5.85.0/kio-5.85.0.tar.xz) = e4d58a7d2ae6530c4b676fe7f7e4583a86436c68861eb971bf91af9efead19f6
SIZE (KDE/frameworks/5.85.0/kio-5.85.0.tar.xz) = 3200300
(So far, so good)

But, after all that, running # poudriere bulk makes absolutely no difference! My updates are not visible. The output is exactly the same as the transcript I attached in post #39 Even paying attention to directories is not helping. Maybe I messed up on git???
 
Just got back home and examined the results of compiling the same stuff with ports-mgmt/synth. Synth actually failed in the exact same way, (failed to see my updates) but logs from /var/log/synth/ proved informative. Doing # cat /var/log/synth/devel___kf5-extra-cmake-modules.log | less showed me something interesting:
Code: 
--------------------------------------------------
--  PLIST_SUB
--------------------------------------------------
...
...
KDE_APPLICATIONS_VERSION="21.04.1"
KDE_FRAMEWORKS_VERSION="5.82.0"
KDE_PLASMA_VERSION="5.21.5"
...
...
Turns out that KF5 version and Plasma 5 version are hard-coded somewhere, and this is why my compilations kept failing! Poudriere logs will probably show something similar, but in a different way. Man, that was one totally roundabout way to discover that I did not update everything I needed to.
--
I'm sure that if I tried to work with the whole tree, rather than just a list of KDE ports, I wouldn't have this issue. Now I gotta figure out the specifics of how KDE was worked into the tree in the first place. 😩
 
You must log in or register to reply here.
Back
Top