Poudriere Bulk Build Slower on One Server

tuaris

Active Member

Reaction score: 22
Messages: 185

I have two servers with nearly identical CPU's (Dual Xeon E5530 @2.40GHz and Dual Xeon E5540 @2.53GHz). The E5540 is slightly "beefier" with 64 GB of RAM.

I would expect Poudriere to preform the same or better on the E5540, but I am observing the opposite. The difference is significant. I currently have a build for the entire ports tree running. The E5540 has built 4011 packages in 43 hours and the E5530 has so far built 19,591 packages in 57 hours.

Here's some additional details on the hardware and configuration:

E5540
  • HP DL180 G6 (The FreeBSD version isn't a variable since the same performance difference was observed with 12.2)
  • 64 GB of RAM
  • /var/run/dmesg.boot
  • HP Smart Array RAID controller with a RAID5 configuration resulting in a single logical volume.
Code:
# gpart show
=>        34  3906918765  da0  GPT  (1.8T)
          34        1024    1  freebsd-boot  (512K)
        1058   419430400    2  freebsd-ufs  (200G)
   419431458  3355443200    3  freebsd-zfs  (1.6T)
  3774874658   132044140    4  freebsd-swap  (63G)
  3906918798           1       - free -  (512B)

# zpool status
  pool: poudriere
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 02:27:01 with 0 errors on Sun Feb  7 01:16:21 2021
config:

        NAME        STATE     READ WRITE CKSUM
        poudriere   ONLINE       0     0     0
          da0p3     ONLINE       0     0     0

E5530
  • IBM HS22 Blade
  • 48 GB of RAM
  • /var/run/dmesg.boot
  • 4 SAS drives in a JBOD configuration. RAID is done with ZFS.
Code:
# gpart show
=>      40  62499920  da0  GPT  (30G)
        40    409600    1  efi  (200M)
    409640  58310656    2  freebsd-ufs  (28G)
  58720296   3125248    3  freebsd-swap  (1.5G)
  61845544    654416       - free -  (320M)

# zpool status
  pool: storage
state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0

Poudriere is configured (mostly) the same on both systems (E5540, E5530). One difference is that the E5530 doesn't have ccache. The E5540 saw a slight improvement when ccache was enabled.
I can't imagine that ZFS has that much of a performance advantage? There must be something else that's slowing down Poudriere on the HP server.
 

Jose

Aspiring Daemon

Reaction score: 751
Messages: 876

I can't imagine that ZFS has that much of a performance advantage?
Yes, it has:
The NO_ZFS=yes is the culprit. This causes a huge amount of I/O when setting up (and tearing down) the jails. This results in significant slowdowns even on the dual 6 core Xeons I have at work.
 
Last edited:

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 10,968
Messages: 36,896

Yep. Instead of simply cloning a clean jail to set up a new build (which barely uses any time and I/O access) it has to unpack a clean jail using cpio(1). And it has to do this for each port it builds. All that I/O and time adds up. Especially if you push your I/O to the limit by having a bunch of new jails created at the same time.
 
Top