Solved 'poudriere bulk' not finding updates to the tree

astyle

astyle

Discussion:

I set up a tree at /usr/ports/poudriere_tree. Just setting everything up for the first time, everything worked fine.
Code: 
 # poudriere ports -l
default   null   2021-11-23 17:10:39 /usr/ports/poudriere_tree
When I was compiling packages for the first time, Plasma was at v. 5.21.5, and KF5 was at version 5.82.0, and it was successful.
--
Trouble came in when I tried to update the default tree. I ran portsnap extract -p /usr/ports/poudriere_tree on a carefully curated list of Plasma5 and KF5 ports. It looks like it worked fine:
Code: 
 # cat /usr/ports/poudriere_tree/devel/kf5-kio/distinfo
TIMESTAMP = 1628252758
SHA256 (KDE/frameworks/5.85.0/kio-5.85.0.tar.xz) = e4d58a7d2ae6530c4b676fe7f7e4583a86436c68861eb971bf91af9efead19f6
SIZE (KDE/frameworks/5.85.0/kio-5.85.0.tar.xz) = 3200300
Frameworks was supposed to get bumped up from 5.82.0 to 5.85.0. But Poudriere ignored this update. If I went with straight make in the port's directory, it would happily build the correct version. But that's not what's happening:
# poudriere bulk -f poud_kf5_plasma_sorted.txt -j kde -z base-options // partial output follows:
Code: 
[00:00:08] Building 63 packages using 8 builders
[00:00:08] Starting/Cloning builders
[00:00:09] Hit CTRL+t at any time to see build progress and stats
[00:00:09] [01] [00:00:00] Building devel/kf5-kio | kf5-kio-5.82.0
Same story with Plasma5 (should be getting bumped up from 5.21.5 to 5.22.5, but not happening)
--
I tried deleting the ports tree with poudriere ports -d default and re-creating it.
--
I'm re-using the jail named kde that I made for my very first compilation. And poudriere bulk takes my carefully curated lists just fine.
Code: 
# poudriere version
3.3.7_1

Not sure where I messed up with updating the ports tree or null-mounting it. Or maybe I messed up elsewhere altogether? If more info is needed, I can provide it. Any suggestions or ideas would be greatly appreciated. I would like to avoid re-compiling the entire base, if possible. But if not, I guess I'll bite the bullet.
 
I will probably make the switch to git at some point. Right now, my concern is that files in the tree /usr/ports/poudriere_tree seem to have been updated correctly, but Poudriere is unable to 'see' the changes.
 
Maybe try something like,
Code: 
/usr/local/bin/poudriere ports -k -d  -p $MYPORTS
/usr/local/bin/poudriere ports -c -m null -M /usr/ports/poudriere_tree  -p $MYPORTS
 
Tried reading the Porter's Handbook section on Poudriere.... specifically, Section 10.6.5, Setting Up Ports Trees for use with Poudriere. That section says,
These commands create tank/poudriere/ports/default, mount it on /poudriere/ports/default, and populate it using Git, portsnap(8), or Subversion. Afterward it is included in the list of known ports trees.
Click to expand...
.
Code: 
# zfs list | grep poudriere
zroot/poudriere                               7.61G   149G       96K  /zroot/poudriere
zroot/poudriere/data                          6.22G   149G      144K  /usr/local/poudriere/data
zroot/poudriere/data/.m                         96K   149G       96K  /usr/local/poudriere/data/.m
zroot/poudriere/data/cache                    15.1M   149G     15.1M  /usr/local/poudriere/data/cache
zroot/poudriere/data/images                     96K   149G       96K  /usr/local/poudriere/data/images
zroot/poudriere/data/logs                     88.8M   149G     88.8M  /usr/local/poudriere/data/logs
zroot/poudriere/data/packages                 6.12G   149G     6.12G  /usr/local/poudriere/data/packages
zroot/poudriere/data/wrkdirs                    96K   149G       96K  /usr/local/poudriere/data/wrkdirs
zroot/poudriere/jails                         1.38G   149G       96K  /zroot/poudriere/jails
zroot/poudriere/jails/kde                     1.38G   149G     1.38G  /usr/local/poudriere/jails/kde
and
Code: 
# zfs get canmount | grep  -v @ | grep poudriere
zroot/poudriere                                                           canmount  on        default
zroot/poudriere/data                                                      canmount  on        default
zroot/poudriere/data/.m                                                   canmount  on        default
zroot/poudriere/data/cache                                                canmount  on        default
zroot/poudriere/data/images                                               canmount  on        default
zroot/poudriere/data/logs                                                 canmount  on        default
zroot/poudriere/data/packages                                             canmount  on        default
zroot/poudriere/data/wrkdirs                                              canmount  on        default
zroot/poudriere/jails                                                     canmount  on        default
zroot/poudriere/jails/kde                                                 canmount  on        default
Among my datasets (in the zroot pool), I don't see a zroot/poudriere/ports/default... Could that be the issue? If I try to mount that dataset to the path of the tree I want to use (/usr/ports/poudriere_tree)... Am I on the right track, or asking for trouble?

Or am I stuck recompiling everything if I try to move away from using 'default' as the name for my tree? What other pitfalls should I know about?
 
I never looked much at the jails as created by poudriere, as they are mainly poudrieres internal working.
What count's are the arguments of the poudriere commands & poudriere.conf, e.g.,
Code: 
/usr/local/bin/poudriere jail -d   -j $JAIL
/usr/local/bin/poudriere jail -c -a amd64 -v $REL -j $JAIL
/usr/local/bin/poudriere jail -u -j $JAIL
/usr/local/bin/poudriere ports -k -d  -p $PORTS
/usr/local/bin/poudriere ports -c -m null -M /usr/ports    -p $PORTS
/usr/local/bin/poudriere options  -j $JAIL -p $PORTS -f $PACKAGES
nice -n 20 /usr/local/bin/poudriere bulk  -j $JAIL -p $PORTS -f $PACKAGES
 
For me, the whole point of setting up Poudriere is so that it pays attention to the updates, and does what's needed (a LONG, finely tuned list of chores) when the updates do happen. And my hurdle here is that it's not able to 'see' the updated files correctly. That means I didn't set up shop correctly, so the command-line arguments don't work quite as expected. 😩 Or maybe there's something I didn't understand correctly. Should I have been setting up shop differently?
 
do a rm -fR & give git a try ?
Code: 
rm -r /usr/ports/* /usr/ports/.??* 
cd /usr/ports/ 
git clone --branch 2021Q4 https://git.FreeBSD.org/ports.git /usr/ports 
mkdir /usr/ports/distfiles 
mkdir /usr/ports/packages
rm -r /usr/ports2/* /usr/ports2/.??* 
mkdir /usr/ports2 
cd /usr/ports2/ 
git clone --branch main https://git.FreeBSD.org/ports.git /usr/ports2
&
Code: 
cd /usr/ports 
/usr/local/bin/git pull --rebase --autostash 
cd /usr/ports2 
/usr/local/bin/git pull --rebase --autostash
 
I guess my issue with git is that it works with the entire ports tree, and I haven't yet learned how to use git to reliably cherry-pick and manage just the list of ports I want. Besides, if I update just one port by hand on the host, Poudriere should be able to pick up on the changes - and it's not doing that.
--
I like the idea of git, I know how powerful it can be - but it can do too much at once, and it will be another time-consuming train wreck to sift through. I'm reviewing some decisions I made for automating the updating of KDE - and realizing that some decisions, either I'm stuck with them, or it would be very time-consuming to rework them. I don't exactly have unlimited disk space - just a 256 GB SSD with ZFS on it. However, a partial listing of my zroot pool shows this:
Code: 
# zfs list
NAME                                           USED  AVAIL     REFER  MOUNTPOINT
zroot                                         76.5G   152G       96K  /zroot
I'd rather not re-create EVERYTHING from scratch here (albeit with an updated ports tree).
--
So, my priority is to see why Poudriere is not picking up on the changes to the tree (no matter the method used to update the tree).
 
grahamperrin said:
Please attach a log file and/or full transcript of the run of poudriere.
Click to expand...
Here you go...
Code: 
root@beastie:/var/db/portsnap/help/lists # cat /usr/local/poudriere/ports/default/devel/kf5-kio/distinfo
TIMESTAMP = 1628252758
SHA256 (KDE/frameworks/5.85.0/kio-5.85.0.tar.xz) = e4d58a7d2ae6530c4b676fe7f7e4583a86436c68861eb971bf91af9efead19f6
SIZE (KDE/frameworks/5.85.0/kio-5.85.0.tar.xz) = 3200300
root@beastie:/var/db/portsnap/help/lists # poudriere ports -l
default   null   2021-12-01 19:57:08 /usr/local/poudriere/ports/default
root@beastie:/var/db/portsnap/help/lists # cat new_list.txt
devel/kf5-kio
root@beastie:/var/db/portsnap/help/lists # poudriere bulk -f new_list.txt -j kde -z base-options
[00:00:00] Warning: Using '-' in a SETNAME is not recommended as it causes ambiguities with parsing the build name of kde-default-base-options
[00:00:00] Creating the reference jail... done
[00:00:00] Mounting system devices for kde-default-base-options
[00:00:00] Mounting ports/packages/distfiles
[00:00:00] Using packages from previously failed build: /usr/local/poudriere/data/packages/kde-default-base-options/.building
[00:00:00] Mounting packages from: /usr/local/poudriere/data/packages/kde-default-base-options
/etc/resolv.conf -> /usr/local/poudriere/data/.m/kde-default-base-options/ref/etc/resolv.conf
[00:00:00] Starting jail kde-default-base-options
[00:00:00] Will build as nobody: (65534:65534)
[00:00:00] Logs: /usr/local/poudriere/data/logs/bulk/kde-default-base-options/2021-12-01_19h59m24s
[00:00:00] Loading MOVED for /usr/local/poudriere/data/.m/kde-default-base-options/ref/usr/ports
[00:00:01] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:01] Gathering ports metadata
[00:00:03] Calculating ports order and dependencies
[00:00:04] Sanity checking the repository
[00:00:04] Checking packages for incremental rebuild needs
[00:00:06] Deleting stale symlinks... done
[00:00:06] Deleting empty directories... done
[00:00:06] Cleaning the build queue
[00:00:06] Sanity checking build queue
[00:00:06] Processing PRIORITY_BOOST
[00:00:06] Balancing pool
[00:00:06] Recording filesystem state for prepkg... done
[00:00:06] Building 1 packages using 1 builders
[00:00:06] Starting/Cloning builders
[00:00:06] Hit CTRL+t at any time to see build progress and stats
[00:00:06] [01] [00:00:00] Building devel/kf5-kio | kf5-kio-5.82.0
[00:00:08] [01] [00:00:02] Finished devel/kf5-kio | kf5-kio-5.82.0: Failed: checksum
[00:00:08] Stopping 1 builders
[00:00:08] No package built, but repository needs to be created
[00:00:08] Creating pkg repository
Creating repository in /tmp/packages: 100%
Packing files for repository: 100%
[00:00:21] Warning: Not committing packages to repository as failures were encountered
[00:00:21] Failed ports: devel/kf5-kio:checksum
[kde-default-base-options] [2021-12-01_19h59m24s] [pkgrepo:] Queued: 1  Built: 0  Failed: 1  Skipped: 0  Ignored: 0  Tobuild: 0   Time: 00:00:20
[00:00:21] Logs: /usr/local/poudriere/data/logs/bulk/kde-default-base-options/2021-12-01_19h59m24s
[00:00:21] Cleaning up
[00:00:21] Unmounting file systems
root@beastie:/var/db/portsnap/help/lists #

I realize the tree in this post is different than in the OP - that's a result of trying to move updated the tree to a different path on the disk and trying to re-import. Made no difference.
Still trying to build KF5 v. 5.82, even though it's supposed to be v. 5.85... Any help would be appreciated.
 
Sorry, I missed that in #12. Speed reading for a header (compare with what's below) but I guess that non -devel poudriere is quite different.

Assuming git in your case, what's the result from this?

git status /usr/local/poudriere/ports/default


Code: 
% poudriere ports -l
PORTSTREE   METHOD    TIMESTAMP           PATH
default     git+https 2021-12-02 05:05:09 /usr/local/poudriere/ports/default
portoverlay -                             /usr/local/poudriere/ports/portoverlay
%
 
Actually, I was trying to do a null-mount, because I maintain my tree manually. Git was going to be my next step if I can't get null-mount to work. If you re-read my OP, the manual method of updating the tree did update the files properly, but Poudriere is not picking up on the changes. No matter what I tried (while still sticking to the null-mount method), there's no difference in the output.
 
😩 Sigh... Even manually creating proper ZFS datasets (zroot/poudriere/ports/ and zroot/poudriere/ports/default) and mounting the second one to /usr/local/poudriere/ports/default as per Porter's Handbook section on Poudriere - that did not make a difference. Looks like my next step is to tell Git that my tree at /usr/ports/poudriere_tree is a remote git repo, while /usr/local/poudriere/ports/default is the local tree/repo for Poudriere's use. Maybe then Poudriere can be tricked into 'seeing' the updates I make. Awkward, and not immediately obvious, I know, but probably what it takes to line everything up.
 
You can google for the keywords "git,pull,rebase,autostash".
Git is not simple, but neither compilated, don't be afraid to use it.
 
Alain De Vos said:
You can google for the keywords "git,pull,rebase,autostash".
Git is not simple, but neither compilated, don't be afraid to use it.
Click to expand...
Reading the Git book right now. No, nothing complicated, but more details to line up so that things work the way I want them to, reliably. I'd just rather not fly completely blind. 'Keep It Simple, Stupid' is a saying I picked up in college.
 
3.3.7_1
It's imperative to have a good procedure to update your /usr/ports sources i.e. get them all in sync with quarterly/main , this without touching the files you modified manually.
 
Same as what
Alain De Vos said:
3.3.7_1
It's imperative to have a good procedure to update your /usr/ports sources i.e. get them all in sync with quarterly/main , this without touching the files you modified manually.
Click to expand...
Your version is the same as what I quoted in OP. Which makes issues on my end quite the head-scratcher.
 
You must log in or register to reply here.
Back
Top