Putting POUDRIERE_DATA and ccache on NFS

tuaris

tuaris

To set the location of the resulting packages that Poudriere will build to a NFS mount, I only need to change POUDRIERE_DATA in /usr/local/etc/poudriere.conf:

Code: 
# The directory where the jail will store the packages and logs
# by default a zfs filesystem will be created and set to
# ${BASEFS}/data
#
#POUDRIERE_DATA=${BASEFS}/data

Does that cause any conflicts if I am using ZFS?
What about putting ccache on NFS (to share with multiple Poudriere hosts), would that be a bad idea?
 
tuaris said:
Does that cause any conflicts if I am using ZFS?
Click to expand...
Poudriere creates zfs datasets below that POUDRIERE_DATA directory (its jails are spawned under $POUDRIERE_DATA/.m). So if that's NFS you need to use NO_ZFS=yes. Probably not the result you want.
 
tuaris said:
… What about putting ccache on NFS (to share with multiple Poudriere hosts), …
Click to expand...

Gut feeling (I haven't toyed with NFS for more than a decade): you might lose some of the performance benefits that can be realised with a local ccache on fast(er) storage.

Code: 
% grep TMPFS /usr/local/etc/poudriere.conf | grep -v \#
USE_TMPFS=all
% grep CCACHE /usr/local/etc/poudriere.conf | grep -v \#
CCACHE_DIR=/var/cache/ccache
% bectl list | grep NR
n250650-ef396441ceb-a NR     /          65.6G 2021-11-14 00:24
% zfs get compression,compressratio august/ROOT/n250650-ef396441ceb-a
NAME                               PROPERTY       VALUE           SOURCE
august/ROOT/n250650-ef396441ceb-a  compression    zstd            inherited from august
august/ROOT/n250650-ef396441ceb-a  compressratio  4.25x           -
% geom disk list ada0
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 1000204886016 (932G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e4
   descr: HGST HTS721010A9E630
   lunid: 5000cca8c8f669d2
   ident: JR1000D33VPSBE
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

% zpool status -v august
  pool: august
 state: ONLINE
  scan: scrub repaired 0B in 02:45:23 with 0 errors on Thu Sep 16 05:48:48 2021
config:

        NAME                STATE     READ WRITE CKSUM
        august              ONLINE       0     0     0
          ada0p3.eli        ONLINE       0     0     0
        cache
          gpt/cache-august  ONLINE       0     0     0
          gpt/duracell      ONLINE       0     0     0

errors: No known data errors
% lsblk
DEVICE         MAJ:MIN SIZE TYPE                                          LABEL MOUNT
ada0             0:124 932G GPT                                               - -
  ada0p1         0:126 260M efi                                    gpt/efiboot0 -
  <FREE>         -:-   1.0M -                                                 - -
  ada0p2         0:128  16G freebsd-swap                              gpt/swap0 SWAP
  ada0p2.eli     2:62   16G freebsd-swap                                      - SWAP
  ada0p3         0:130 915G freebsd-zfs                                gpt/zfs0 <ZFS>
  ada0p3.eli     0:136 915G -                                                 - -
  <FREE>         -:-   708K -                                                 - -
da0              0:161  29G GPT                                               - -
  da0p1          0:162  29G freebsd-zfs                        gpt/cache-august <ZFS>
da1              0:209 466G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da1p1          0:210 466G freebsd-zfs                           gpt/Transcend <ZFS>
da2              0:211  15G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da2p1          0:214  15G freebsd-zfs                            gpt/duracell <ZFS>
da3              0:212  14G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da3p1          0:215  14G freebsd-zfs                     gpt/cache-transcend <ZFS>
% geom disk list da0
Geom name: da0
Providers:
1. Name: da0
   Mediasize: 30943995904 (29G)
   Sectorsize: 512
   Mode: r1w1e3
   descr: Kingston DataTraveler 3.0
   ident: E0D55EA1C84FF390A9500FDA
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

% geom disk list da2
Geom name: da2
Providers:
1. Name: da2
   Mediasize: 16567500800 (15G)
   Sectorsize: 512
   Mode: r1w1e3
   descr: Generic Flash Disk
   ident: YTAI0012080818233298
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

%

That's not to suggest that a 7,200 RPM disk + L2ARC can be faster (for me) than what you might achieve with NFS. Just food for thought.

Also, the USE_TMPFS=all is probably over-ambitious, just rarely, but I do like to push things.
 
grahamperrin said:
Gut feeling (I haven't toyed with NFS for more than a decade): you might lose some of the performance benefits that can be realised with a local ccache on fast(er) storage.
Click to expand...
You are probably right on the performance hit. I forgot that building stuff is IO bound.

SirDice said:
Poudriere creates zfs datasets below that POUDRIERE_DATA directory (its jails are spawned under $POUDRIERE_DATA/.m). So if that's NFS you need to use NO_ZFS=yes. Probably not the result you want.
Click to expand...
Yep that was my concern and was afraid to try it on my current system. I recently had some down time and spun up a test VM and do some experimenting.

In /usr/local/etc/poudriere.conf I have:
Code: 
ZPOOL=storage

### NO ZFS
# To not use ZFS, define NO_ZFS=yes
#NO_ZFS=yes

# root of the poudriere zfs filesystem, by default /poudriere
# ZROOTFS=/poudriere

# The directory where poudriere will store jails and ports
BASEFS=/usr/local/poudriere

# The directory where the jail will store the packages and logs
# by default a zfs filesystem will be created and set to
# ${BASEFS}/data
#
POUDRIERE_DATA=/mnt/data

DISTFILES_CACHE=/storage/distfiles

I create the ZFS pool and mount NFS

Code: 
root@freebsd:~ # zpool create storage da1
root@freebsd:~ # zfs create storage/distfiles
root@freebsd:~ # mount -t nfs charon:/mnt/internal/PoudriereData /mnt/data

root@freebsd:~ # df -h
Filesystem                            Size    Used   Avail Capacity  Mounted on
/dev/da0p2                             18G    2.4G     14G    15%    /
devfs                                 1.0K    1.0K      0B   100%    /dev
/dev/da0p1                            256M    1.7M    254M     1%    /boot/efi
storage                                19G     24K     19G     0%    /storage
storage/distfiles                      17G    3.8M     17G     0%    /storage/distfiles
charon:/mnt/internal/PoudriereData    1.0T     96K    1.0T     0%    /mnt/data

Then I create a jail, ports tree

Code: 
root@freebsd:~ # poudriere jail -c -j 13amd64 -v 13.0-RELEASE
...

root@freebsd:~ # zfs list
NAME                              USED  AVAIL     REFER  MOUNTPOINT
storage                          1.12G  17.8G       24K  /storage
storage/distfiles                3.78M  17.5G     3.78M  /storage/distfiles
storage/poudriere                1.12G  17.8G       24K  /storage/poudriere
storage/poudriere/jails          1.12G  17.8G       24K  /storage/poudriere/jails
storage/poudriere/jails/13amd64  1.12G  17.8G     1.12G  /usr/local/poudriere/jails/13amd64

root@freebsd:~ # poudriere ports -c

[00:00:00] Creating default fs at /usr/local/poudriere/ports/default... done
[00:00:00] Cloning the ports tree... done

Then kick off a bulk build.

Code: 
root@freebsd:~ # poudriere bulk -j 13amd64 ports-mgmt/poudriere
[00:00:00] Creating the reference jail... done
[00:00:00] Mounting system devices for 13amd64-default
[00:00:00] Mounting ports/packages/distfiles
[00:00:00] Using packages from previously failed build: /mnt/data/packages/13amd64-default/.building
[00:00:00] Mounting packages from: /mnt/data/packages/13amd64-default
/etc/resolv.conf -> /mnt/data/.m/13amd64-default/ref/etc/resolv.conf
[00:00:00] Starting jail 13amd64-default
[00:00:00] Will build as nobody: (65534:65534)
[00:00:01] Logs: /mnt/data/logs/bulk/13amd64-default/2021-11-17_00h54m49s
[00:00:01] WWW: http://pkg.morante.net/poudriere/build.html?mastername=13amd64-default&build=2021-11-17_00h54m49s
[00:00:01] Loading MOVED for /mnt/data/.m/13amd64-default/ref/usr/ports
[00:00:03] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:03] Gathering ports metadata
[00:00:03] Calculating ports order and dependencies
[00:00:03] pkg package missing, skipping sanity
[00:00:03] Skipping incremental rebuild and repository sanity checks
[00:00:04] Cleaning the build queue
[00:00:04] Sanity checking build queue
[00:00:04] Processing PRIORITY_BOOST
[00:00:04] Balancing pool
[00:00:04] Recording filesystem state for prepkg... done
[00:00:05] Building 6 packages using 2 builders
[00:00:05] Starting/Cloning builders
[00:00:06] Hit CTRL+t at any time to see build progress and stats
[00:00:06] [01] [00:00:00] Building ports-mgmt/pkg | pkg-1.17.5
[00:02:30] [01] [00:02:24] Finished ports-mgmt/pkg | pkg-1.17.5: Success
[00:02:30] [02] [00:00:00] Building misc/freebsd-release-manifests | freebsd-release-manifests-20211111
[00:02:30] [01] [00:00:00] Building lang/perl5.32 | perl5-5.32.1_1
[00:02:34] [02] [00:00:04] Finished misc/freebsd-release-manifests | freebsd-release-manifests-20211111: Success
[00:02:34] [02] [00:00:00] Building ports-mgmt/dialog4ports | dialog4ports-0.1.6_1
[00:02:41] [02] [00:00:07] Finished ports-mgmt/dialog4ports | dialog4ports-0.1.6_1: Success
[00:11:00] [01] [00:08:30] Finished lang/perl5.32 | perl5-5.32.1_1: Success
[00:11:01] [01] [00:00:00] Building security/ca_root_nss | ca_root_nss-3.71
[00:11:15] [01] [00:00:14] Finished security/ca_root_nss | ca_root_nss-3.71: Success
[00:11:16] [01] [00:00:00] Building ports-mgmt/poudriere | poudriere-3.3.7_1
[00:11:43] [01] [00:00:27] Finished ports-mgmt/poudriere | poudriere-3.3.7_1: Success
[00:11:44] Stopping 2 builders
[00:11:44] Creating pkg repository

During the build this is what zfs list prints out

Code: 
root@freebsd:~ # zfs list
NAME                                             USED  AVAIL     REFER  MOUNTPOINT
storage                                         1.41G  17.5G       24K  /storage
storage/distfiles                               3.78M  17.5G     3.78M  /storage/distfiles
storage/poudriere                               1.40G  17.5G       24K  /storage/poudriere
storage/poudriere/jails                         1.13G  17.5G       24K  /storage/poudriere/jails
storage/poudriere/jails/13amd64                 1.12G  17.5G     1.12G  /usr/local/poudriere/jails/13amd64
storage/poudriere/jails/13amd64-default-ref     4.40M  17.5G     1.12G  /mnt/data/.m/13amd64-default/ref
storage/poudriere/jails/13amd64-default-ref/01  3.89M  17.5G     1.13G  /mnt/data/.m/13amd64-default/01
storage/poudriere/jails/13amd64-default-ref/02   121K  17.5G     1.12G  /mnt/data/.m/13amd64-default/02
storage/poudriere/ports                          282M  17.5G       24K  /storage/poudriere/ports
storage/poudriere/ports/default                  282M  17.5G      282M  /usr/local/poudriere/ports/default

This is what df -h shows:

Code: 
root@freebsd:~ # df -h
Filesystem                                        Size    Used   Avail Capacity  Mounted on
/dev/da0p2                                         18G    2.5G     14G    15%    /
devfs                                             1.0K    1.0K      0B   100%    /dev
/dev/da0p1                                        256M    1.7M    254M     1%    /boot/efi
storage                                            17G     24K     17G     0%    /storage
charon:/mnt/internal/PoudriereData                1.0T    1.1M    1.0T     0%    /mnt/data
storage/poudriere                                  17G     24K     17G     0%    /storage/poudriere
storage/poudriere/jails                            17G     24K     17G     0%    /storage/poudriere/jails
storage/poudriere/jails/13amd64                    19G    1.1G     17G     6%    /usr/local/poudriere/jails/13amd64
storage/poudriere/ports                            17G     24K     17G     0%    /storage/poudriere/ports
storage/poudriere/ports/default                    18G    282M     17G     2%    /usr/local/poudriere/ports/default
storage/distfiles                                  17G    3.8M     17G     0%    /storage/distfiles
storage/poudriere/jails/13amd64-default-ref        19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/ref
/usr/local/poudriere/jails/13amd64/rescue          19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/ref/rescue
/usr/local/poudriere/jails/13amd64/usr/share       19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/ref/usr/share
/usr/local/poudriere/jails/13amd64/usr/tests       19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/ref/usr/tests
/usr/local/poudriere/jails/13amd64/usr/lib32       19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/ref/usr/lib32
/usr/local/poudriere/jails/13amd64/usr/src         19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/ref/usr/src
devfs                                             1.0K    1.0K      0B   100%    /mnt/data/.m/13amd64-default/ref/dev
fdescfs                                           1.0K    1.0K      0B   100%    /mnt/data/.m/13amd64-default/ref/dev/fd
procfs                                            4.0K    4.0K      0B   100%    /mnt/data/.m/13amd64-default/ref/proc
linprocfs                                         4.0K    4.0K      0B   100%    /mnt/data/.m/13amd64-default/ref/compat/linux/proc
tmpfs                                             1.0G    308K    1.0G     0%    /mnt/data/.m/13amd64-default/ref/.p
tmpfs                                             1.7G    4.0K    1.7G     0%    /mnt/data/.m/13amd64-default/ref/wrkdirs
/usr/local/poudriere/ports/default                 18G    282M     17G     2%    /mnt/data/.m/13amd64-default/ref/usr/ports
/mnt/data/packages/13amd64-default/.building      1.0T    1.1M    1.0T     0%    /mnt/data/.m/13amd64-default/ref/packages
/storage/distfiles                                 17G    3.8M     17G     0%    /mnt/data/.m/13amd64-default/ref/distfiles
tmpfs                                             1.7G    4.0K    1.7G     0%    /mnt/data/.m/13amd64-default/ref/var/db/ports
storage/poudriere/jails/13amd64-default-ref/01     19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/01
storage/poudriere/jails/13amd64-default-ref/02     19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/02
/mnt/data/.m/13amd64-default/ref/rescue            19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/01/rescue
/mnt/data/.m/13amd64-default/ref/usr/share         19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/01/usr/share
/mnt/data/.m/13amd64-default/ref/usr/tests         19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/01/usr/tests
/mnt/data/.m/13amd64-default/ref/usr/lib32         19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/01/usr/lib32
/mnt/data/.m/13amd64-default/ref/usr/src           19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/01/usr/src
devfs                                             1.0K    1.0K      0B   100%    /mnt/data/.m/13amd64-default/01/dev
fdescfs                                           1.0K    1.0K      0B   100%    /mnt/data/.m/13amd64-default/01/dev/fd
procfs                                            4.0K    4.0K      0B   100%    /mnt/data/.m/13amd64-default/01/proc
linprocfs                                         4.0K    4.0K      0B   100%    /mnt/data/.m/13amd64-default/01/compat/linux/proc
tmpfs                                             1.0G    4.0K    1.0G     0%    /mnt/data/.m/13amd64-default/01/.p
tmpfs                                             1.7G     28M    1.7G     2%    /mnt/data/.m/13amd64-default/01/wrkdirs
/usr/local/poudriere/ports/default                 18G    282M     17G     2%    /mnt/data/.m/13amd64-default/01/usr/ports
/mnt/data/packages/13amd64-default/.building      1.0T    1.1M    1.0T     0%    /mnt/data/.m/13amd64-default/01/packages
/storage/distfiles                                 17G    3.8M     17G     0%    /mnt/data/.m/13amd64-default/01/distfiles
/mnt/data/.m/13amd64-default/ref/var/db/ports     1.7G    4.0K    1.7G     0%    /mnt/data/.m/13amd64-default/01/var/db/ports
/mnt/data/.m/13amd64-default/ref/rescue            19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/02/rescue
/mnt/data/.m/13amd64-default/ref/usr/share         19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/02/usr/share
/mnt/data/.m/13amd64-default/ref/usr/tests         19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/02/usr/tests
/mnt/data/.m/13amd64-default/ref/usr/lib32         19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/02/usr/lib32
/mnt/data/.m/13amd64-default/ref/usr/src           19G    1.1G     17G     6%    /mnt/data/.m/13amd64-default/02/usr/src
devfs                                             1.0K    1.0K      0B   100%    /mnt/data/.m/13amd64-default/02/dev
fdescfs                                           1.0K    1.0K      0B   100%    /mnt/data/.m/13amd64-default/02/dev/fd
procfs                                            4.0K    4.0K      0B   100%    /mnt/data/.m/13amd64-default/02/proc
linprocfs                                         4.0K    4.0K      0B   100%    /mnt/data/.m/13amd64-default/02/compat/linux/proc
tmpfs                                             1.0G    4.0K    1.0G     0%    /mnt/data/.m/13amd64-default/02/.p
tmpfs                                             1.7G    4.0K    1.7G     0%    /mnt/data/.m/13amd64-default/02/wrkdirs
/usr/local/poudriere/ports/default                 18G    282M     17G     2%    /mnt/data/.m/13amd64-default/02/usr/ports
/mnt/data/packages/13amd64-default/.building      1.0T    1.1M    1.0T     0%    /mnt/data/.m/13amd64-default/02/packages
/storage/distfiles                                 17G    3.8M     17G     0%    /mnt/data/.m/13amd64-default/02/distfiles
/mnt/data/.m/13amd64-default/ref/var/db/ports     1.7G    4.0K    1.7G     0%    /mnt/data/.m/13amd64-default/02/var/db/ports

After the build is complete I re-check it all and it looks like everything is fine

Code: 
root@freebsd:~ # df -h
Filesystem                            Size    Used   Avail Capacity  Mounted on
/dev/da0p2                             18G    2.5G     14G    15%    /
devfs                                 1.0K    1.0K      0B   100%    /dev
/dev/da0p1                            256M    1.7M    254M     1%    /boot/efi
storage                                17G     24K     17G     0%    /storage
charon:/mnt/internal/PoudriereData    1.0T     25M    1.0T     0%    /mnt/data
storage/poudriere                      17G     24K     17G     0%    /storage/poudriere
storage/poudriere/jails                17G     24K     17G     0%    /storage/poudriere/jails
storage/poudriere/jails/13amd64        19G    1.1G     17G     6%    /usr/local/poudriere/jails/13amd64
storage/poudriere/ports                17G     24K     17G     0%    /storage/poudriere/ports
storage/poudriere/ports/default        18G    282M     17G     2%    /usr/local/poudriere/ports/default
storage/distfiles                      17G    100M     17G     1%    /storage/distfiles

root@freebsd:~ # zfs list
NAME                              USED  AVAIL     REFER  MOUNTPOINT
storage                          1.50G  17.4G       24K  /storage
storage/distfiles                 100M  17.4G      100M  /storage/distfiles
storage/poudriere                1.40G  17.4G       24K  /storage/poudriere
storage/poudriere/jails          1.12G  17.4G       24K  /storage/poudriere/jails
storage/poudriere/jails/13amd64  1.12G  17.4G     1.12G  /usr/local/poudriere/jails/13amd64
storage/poudriere/ports           282M  17.4G       24K  /storage/poudriere/ports
storage/poudriere/ports/default   282M  17.4G      282M  /usr/local/poudriere/ports/default

Also looks like the data directory has the items it needs

Code: 
root@freebsd:~ # ls -plarths /mnt/data/
total 7
4 drwxr-xr-x  3 root  wheel   512B Nov 17 00:39 ../
1 drwxr-xr-x  3 root  wheel     3B Nov 17 00:52 .m/
1 drwxr-xr-x  3 root  wheel     3B Nov 17 00:52 packages/
1 drwxr-xr-x  3 root  wheel     3B Nov 17 00:52 logs/
1 drwxrwxrwx  6 root  wheel     6B Nov 17 00:52 ./
1 drwxr-xr-x  3 root  wheel     3B Nov 17 00:52 cache/

root@freebsd:~ # ls -plarths /mnt/data/packages/13amd64-default/
total 23
1 drwxr-xr-x  3 root  wheel     3B Nov 17 00:52 ../
1 lrwxr-xr-x  1 root  wheel    20B Nov 17 01:06 .jailversion -> .latest/.jailversion
1 lrwxr-xr-x  1 root  wheel    23B Nov 17 01:06 packagesite.pkg -> .latest/packagesite.pkg
1 lrwxr-xr-x  1 root  wheel    23B Nov 17 01:06 packagesite.txz -> .latest/packagesite.txz
1 lrwxr-xr-x  1 root  wheel    11B Nov 17 01:06 All -> .latest/All
1 lrwxr-xr-x  1 root  wheel    17B Nov 17 01:06 meta.conf -> .latest/meta.conf
1 lrwxr-xr-x  1 root  wheel    18B Nov 17 01:06 .buildname -> .latest/.buildname
1 lrwxr-xr-x  1 root  wheel    14B Nov 17 01:06 Latest -> .latest/Latest
1 lrwxr-xr-x  1 root  wheel    16B Nov 17 01:06 meta.txz -> .latest/meta.txz
1 lrwxr-xr-x  1 root  wheel    16B Nov 17 01:06 meta.pkg -> .latest/meta.pkg
1 lrwxr-xr-x  1 root  wheel    16B Nov 17 01:06 .latest -> .real_1637129195
9 drwxr-xr-x  3 root  wheel    13B Nov 17 01:06 ./
9 drwxr-xr-x  4 root  wheel    11B Nov 17 01:06 .real_1637129195/
 
You must log in or register to reply here.
Back
Top