ZFS ZFS Capabilities question

Hi

I thinking of ways I can use ZFS to improve storage and system performance.

Two thinks I am interested in accounting for:
- If I mirror across drives of different read/write speeds, in brief, what effects can I expect this to have and are the any configuration options available to ensure it reads from the faster drive by default.
- If in a mirrored pool a disk becomes temporarily unavailable but later become available, is ZFS capable of synchronizing, with data that has been written to the drive remaining online been written to mirror drive when that becomes available again (even after a system reboot), e.g., if the second drive was a partition on a networked disk image (which is precisely what I am considering doing).

I realize GlusterFS is another option but I want to see is reliably possible simply with ZFS. That said, I would potentially consider going into the ZFS source later and making some updates if I find areas in which I would like to see it improved. For now though, I just want a rough idea of what to expect till I have time to investigate further.
 
- If I mirror across drives of different read/write speeds, in brief, what effects can I expect this to have and are the any configuration options available to ensure it reads from the faster drive by default.
I don't know what ZFS does for reading. The standard solution is to watch the queue depth on both drives, and send new IOs to the drive with the lower queue depth. This works great for intense workloads. For light workloads (where the queue depth is zero or one), this technique doesn't help. For that the file system needs to have history information on the latency per disk, and I don't know whether ZFS does that or not.

For writes: The standard solution is to wait for both writes to finish before returning control to the caller. Which means that writes are at least as slow as the slower of the two drives, perhaps slower (due to fluctuations, and/or due to shared bottlenecks). In theory, it is possible to write only to the faster drive initially, and only copy to the slower drive asynchronously (a technique that was known as AF-RAID when it was invented). I know of no production system that does that.

If in a mirrored pool a disk becomes temporarily unavailable but later become available, is ZFS capable of synchronizing, with data that has been written to the drive remaining online been written to mirror drive when that becomes available again (even after a system reboot),
Yes, when the second drive shows up, only the data that has been changed will be resilvered.

That said, I would potentially consider going into the ZFS source later and making some updates if I find areas in which I would like to see it improved.
Start by reading the ZFS chapter of the "black dragon book" (Design & Implementation of FreeBSD, current edition, by Kirk McKusick et al.). Then contact Kirk, and attend one of his "ZFS internals" classes, or get the class notes.
 
Last edited:
Hi raplhbsz.

Thankyou for that, I will certainly be reading that book. Particularly appreciate the directions on books to read. The BSD kernel is complicated and some resource pointers are definitely appreciated.
 
… ways I can use ZFS to improve storage and system performance. …

Use a cache device.

My current low-spec setup – Kingston DataTraveler G4 cache devices:

Code:
% zpool iostat -v
                          capacity     operations     bandwidth
pool                    alloc   free   read  write   read  write
----------------------  -----  -----  -----  -----  -----  -----
Transcend                308G   156G      0      0    328    231
  gpt/Transcend          308G   156G      0      0    328    231
cache                       -      -      -      -      -      -
  gpt/cache-transcend   14.3G   135M      0      0    159      1
----------------------  -----  -----  -----  -----  -----  -----
copperbowl               243G   205G     11     11   291K   249K
  ada0p4.eli             243G   205G     11     11   291K   249K
cache                       -      -      -      -      -      -
  gpt/cache-copperbowl  28.3G  1.04G     27      0   225K  87.1K
----------------------  -----  -----  -----  -----  -----  -----
% lsblk
DEVICE         MAJ:MIN SIZE TYPE                                          LABEL MOUNT
ada0             0:127 466G GPT                                               - -
  ada0p1         0:129 200M efi                                    gpt/efiboot0 -
  ada0p2         0:131 512K freebsd-boot                           gpt/gptboot0 -
  <FREE>         -:-   492K -                                                 - -
  ada0p3         0:133  16G freebsd-swap                              gpt/swap0 SWAP
  ada0p3.eli     1:44   16G freebsd-swap                                      - SWAP
  ada0p4         0:135 450G freebsd-zfs                                gpt/zfs0 <ZFS>
  ada0p4.eli     0:147 450G zfs                                               - -
  <FREE>         -:-   4.0K -                                                 - -
cd0              0:137   0B -                                                 - -
cd1              0:181  10M cd9660                    iso9660/ONEPLUS%20DRIVERS -
da0              0:197  29G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da0p1          0:199  29G freebsd-zfs                    gpt/cache-copperbowl <ZFS>
da1              0:223 466G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da1p1          0:224 466G freebsd-zfs                           gpt/Transcend <ZFS>
da2              1:59   14G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da2p1          1:60   14G freebsd-zfs                     gpt/cache-transcend <ZFS>
%

Low spec, but copperbowl is much faster with the simple USB flash drive helping the 7200 rpm disk at ada0.
 
This is pretty much repeating what ralphbsz is saying:

Mirrors:
Reads are completed as soon as one device responds, typically will be the one with the lowest read latency
Writes are completed as soon as all devices in the mirror report complete, typically will track the device with the highest write latency.
Synchronizing:
That's what mirrors do, so yes they should automatically resync (resilver).

You state:
"...e.g., if the second drive was a partition on a networked disk image (which is precisely what I am considering doing)."

Interesting, sounds like you're trying to make a backup "easy". Something like an iSCSI drive/device participating with a local physical device in a mirror? If so then definitely look into what grahamperrin is talking about with cache devices.
 
As was already stated, mirroring over drives with different speeds is handled nicely. But, drives with seriously different speeds (in the order of a magnitude) or generally different behaviour (like SSD+spinning) are giving a rather ugly and erratic behaviour, as this is not what it was designed for.
So I would not recommend to share different architectures into a mirror. If you want to do so, do some testing first and see if the behaviour is acceptable for you.
 
As was already stated, mirroring over drives with different speeds is handled nicely. But, drives with seriously different speeds (in the order of a magnitude) or generally different behaviour (like SSD+spinning) are giving a rather ugly and erratic behaviour, as this is not what it was designed for.
So I would not recommend to share different architectures into a mirror. If you want to do so, do some testing first and see if the behaviour is acceptable for you.
Are you able to elaborate more in regards to erratic behaviour?
 
Sadly. no. I was investigating a problem with gmirror - that did defintely write erroneous data when mixing ATA+SATA or ATA+SCSI or SCSI+SATA. So I compared against ZFS where the error did not happen, and recognized that nevertheless I did not like the performance. But my notes do not cover details about that. (That was in 2011, with Release 7.3)

I'd suggest You just try it out.
 
As was already stated, mirroring over drives with different speeds is handled nicely. But, drives with seriously different speeds (in the order of a magnitude) or generally different behaviour (like SSD+spinning) are giving a rather ugly and erratic behaviour, as this is not what it was designed for.
So I would not recommend to share different architectures into a mirror. If you want to do so, do some testing first and see if the behaviour is acceptable for you.
That was inspirational. I dug out my spare single SSD installation and an old 500GB rotating disk. Now I am going to mirror these and let you know what happens.
Screenshot from 2021-08-27 20-44-43.png


(I kissed a girl and I liked it
...
It felt so wrong
It felt so right)
 
That was inspirational. I dug out my spare single SSD installation and an old 500GB rotating disk. Now I am going to mirror these and let you know what happens.
My activity log follows:

Code:
root@Argentum ~# gpart show
=>       40  976770976  ada0  GPT  (466G)
         40     409600     1  efi  (200M)
     409640  976361376     3  freebsd-zfs  (466G)

=>       40  234441568  ada1  GPT  (112G)
         40     409600     1  efi  (200M)
     409640   16779224     4  freebsd-zfs  (8.0G)
   17188864  217251840     3  freebsd-zfs  (104G)
  234440704        904        - free -  (452K)

# ada1 is boot drive, ada0p3 is a fresh zfs partition.

root@Argentum ~# zpool status; zpool iostat -v
  pool: flash
 state: ONLINE
  scan: scrub repaired 0B in 00:26:37 with 0 errors on Tue Apr 13 11:53:12 2021
config:

        NAME        STATE     READ WRITE CKSUM
        flash       ONLINE       0     0     0
          ada1p3    ONLINE       0     0     0

errors: No known data errors
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
flash       93.2G  9.76G     20      6   673K   165K
  ada1p3    93.2G  9.76G     20      6   673K   165K
----------  -----  -----  -----  -----  -----  -----

## And now!!!
## ... the bomb is ticking ...

root@Argentum ~# zpool attach flash ada1p3 ada0p3
root@Argentum ~# zpool status; zpool iostat -v
  pool: flash
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Aug 27 19:52:20 2021
        16.8G scanned at 957M/s, 408K issued at 22.7K/s, 93.2G total
        0B resilvered, 0.00% done, no estimated completion time
config:

        NAME        STATE     READ WRITE CKSUM
        flash       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0

errors: No known data errors
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
flash       93.2G  9.77G     53      7  1.09M   224K
  mirror    93.2G  9.77G  1.90K    263  39.3M  7.92M
    ada1p3      -      -     53      6  1.09M   187K
    ada0p3      -      -      0     25  1.30K  1.33M
----------  -----  -----  -----  -----  -----  -----

### the drive light is on...

root@Argentum ~# zpool status; zpool iostat -v
  pool: flash
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Aug 27 19:52:20 2021
        16.8G scanned at 957M/s, 408K issued at 22.7K/s, 93.2G total
        0B resilvered, 0.00% done, no estimated completion time
config:

        NAME        STATE     READ WRITE CKSUM
        flash       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0

errors: No known data errors
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
flash       93.2G  9.77G     53      7  1.09M   224K
  mirror    93.2G  9.77G  1.90K    263  39.3M  7.92M
    ada1p3      -      -     53      6  1.09M   187K
    ada0p3      -      -      0     25  1.30K  1.33M
----------  -----  -----  -----  -----  -----  -----

# .. after some time it is still resilvering. The desktop keeps running as usual. I am also running a disk intensive command during resilver

root@Argentum ~# gitup ports -v 1; portversion -vl "<"

root@Argentum ~# zpool status; zpool iostat -v
  pool: flash
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Aug 27 19:52:20 2021
        93.2G scanned at 84.2M/s, 32.8G issued at 29.6M/s, 93.2G total
        33.9G resilvered, 35.18% done, 00:34:49 to go
config:

        NAME        STATE     READ WRITE CKSUM
        flash       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0  (resilvering)

errors: No known data errors
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
flash       93.3G  9.71G    269    250  20.2M  19.7M
  mirror    93.3G  9.71G    424    393  31.8M  30.9M
    ada1p3      -      -    269      4  20.2M   135K
    ada0p3      -      -      0    385     21  30.7M
----------  -----  -----  -----  -----  -----  -----

root@Argentum ~# zpool history -l|grep "08-27"
2021-08-27.19:52:24 zpool attach flash ada1p3 ada0p3 [user 0 (root) on Argentum:freebsd]

root@Argentum ~# zpool history -li|grep "08-27"
2021-08-27.19:52:20 [txg:215761] scan setup func=2 mintxg=3 maxtxg=215761 [on Argentum]
2021-08-27.19:52:24 [txg:215763] vdev attach attach vdev=/dev/ada0p3 to vdev=/dev/ada1p3 [on Argentum]
2021-08-27.19:52:24 zpool attach flash ada1p3 ada0p3 [user 0 (root) on Argentum:freebsd]

# almost ready

root@Argentum ~# zpool status; zpool iostat -v
  pool: flash
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Aug 27 19:52:20 2021
        93.2G scanned at 28.9M/s, 93.1G issued at 28.9M/s, 93.2G total
        95.8G resilvered, 99.84% done, 00:00:05 to go
config:

        NAME        STATE     READ WRITE CKSUM
        flash       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0  (resilvering)

errors: No known data errors
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
flash       93.3G  9.71G    300    313  25.1M  25.1M
  mirror    93.3G  9.71G    359    375  30.1M  30.0M
    ada1p3      -      -    300      5  25.1M   142K
    ada0p3      -      -      0    368      7  29.9M
----------  -----  -----  -----  -----  -----  -----

# ... and COOL! Now I have done something that you should never try  at home

root@Argentum ~# zpool status; zpool iostat -v
  pool: flash
 state: ONLINE
  scan: resilvered 96.0G in 00:55:08 with 0 errors on Fri Aug 27 20:47:28 2021
config:

        NAME        STATE     READ WRITE CKSUM
        flash       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0

errors: No known data errors
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
flash       93.3G  9.71G    294    308  24.7M  24.6M
  mirror    93.3G  9.71G    351    367  29.4M  29.3M
    ada1p3      -      -    294      5  24.7M   140K
    ada0p3      -      -      0    360      7  29.2M
----------  -----  -----  -----  -----  -----  -----

# after reboot and login to MATE desktop

root@Argentum ~# zpool status; zpool iostat -v
  pool: flash
 state: ONLINE
  scan: resilvered 96.0G in 00:55:08 with 0 errors on Fri Aug 27 20:47:28 2021
config:

        NAME        STATE     READ WRITE CKSUM
        flash       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0

errors: No known data errors
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
flash       93.3G  9.69G    120     24  3.88M   344K
  mirror    93.3G  9.69G    120     24  3.88M   344K
    ada1p3      -      -    112     13  3.53M   172K
    ada0p3      -      -      7     11   365K   172K
----------  -----  -----  -----  -----  -----  -----

# As expected - most of the reads are going to SSD (ada1) and write speed is limited by HDD (ada0). In regular use does not feel any different. I can assume there is a write penalty
# compared to SSD only, but in regular desktop use it does not feel any different.

# and after some time upgrading a bigger port, we can see that there are almost no reads going to HDD:

root@Argentum ~# zpool status; zpool iostat -v
  pool: flash
 state: ONLINE
  scan: resilvered 96.0G in 00:55:08 with 0 errors on Fri Aug 27 20:47:28 2021
config:

        NAME        STATE     READ WRITE CKSUM
        flash       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0

errors: No known data errors
              capacity     operations     bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
flash       93.3G  9.68G    142     18  1.02M   358K
  mirror    93.3G  9.68G    142     18  1.02M   358K
    ada1p3      -      -    141     10  1005K   179K
    ada0p3      -      -      1      7  35.7K   179K
----------  -----  -----  -----  -----  -----  -----

I will let you know how this ends and how it is working...

And I need a good test to evaluate the result. Perhaps I will measure a build time on that mirror, then disconnect SSD and HDD and run the same build again...
 
Just FYI, if you try to mirror drives with two different speeds, the whole thing will operate at the speed of the slower drive. You can also throw in the data transfer rate of the cable used to connect the two drives. If it's slower than either drive, that cable will bottleneck the entire operation.
 
Just FYI, if you try to mirror drives with two different speeds, the whole thing will operate at the speed of the slower drive. You can also throw in the data transfer rate of the cable used to connect the two drives. If it's slower than either drive, that cable will bottleneck the entire operation.
Seems that with SATA (depending on controller) this is not the problem. The channels operate simultaneously and the throughput adds up in small system (CPU to controller is much faster than single SATA line). I will do the deeper testing later, but my experiment this evening shows that:
  1. Nothing bad happens when you mirror HDD and SSD;
  2. If the original system is SSD, you loose only write performance if you mirror with HDD, but this might be a reasonable budget option for data safety;
  3. If the original system is HDD and you mirror it with SSD, that will give an enormous boost in read performance on budget. The write speed remains the same.
  4. The best and most expensive solution would be to mirror two SSD-s.
P.S. I am talking about small systems.
 
SSD's are actually becoming cheaper these days... A 256-GB SSD cost $100 back in 2017, but only $30 in 2021. I checked my Amazon receipts, so I would know.
I hope so, but still they are not catching up. HDD-s are also moving ahead with a high pace. I have 2TB HDD-s in one of my desktop systems, and these animals are only EUR50 a piece. 2TB SSD prices start from EUR180 on the local hunting grounds.
 
[*]If the original system is SSD, you loose only write performance if you mirror with HDD, but this might be a reasonable budget option for data safety;
[*]If the original system is HDD and you mirror it with SSD, that will give an enormous boost in read performance on budget. The write speed remains the same.
There is a nasty little detail. Clearly, as you have said, the write performance can be no better than the write performance of the slower of the two, since for any write to finish, it has to write on both.

Where it gets tricky is reads. Read performance can be exactly halfway between the two (which means much slower than SSD alone), or it can be nearly identical to the faster one, or it can be a little better than the faster one. Which one it is depends on both the workload, and how balancing the reads between the two is implemented. Concrete example: To make life easier, I will describe the problem in terms of latency. You have a HDD that takes 10ms to perform a read IO, and an SSD that takes 1ms to perform a read IO. Let's start with a workload that at any given time has at most one read in progress, never two. Now let's assume that the file system does not remember which of the two disks is fast and which is slow. Then, when a new read request comes in, it will send the request to one of the disks, either using round-robin (toggle back and forth) or at random. In this case, half the IOs will take 1ms, the other half will take 10ms, on average they will take 5.5ms, and you will get ~180 IOs done per second. Much worse than SSD only (which was able to do 1000 per second), and only a factor of ~2 better than HDD alone.

Now let's make our file system a little smarter. Let's say it keeps long term average performance statistics. It will obviously send all IOs to the faster disk, which does the in 1ms a piece. This is the scenario where the mixed disk is as fast as the faster one. In reality, this doesn't quite work, because the latencies are not fixed numbers, but have wide variability: occasionally the SSD takes 3 or 4ms, while occasionally the HDD can get an IO done in 0.5ms. So using it's performance statistics, the file system will perhaps send 90% of the IOs to the SSD and 10% to the HDD, giving an average performance of 1.9ms, or ~520 per second: a little slower than SSD alone.

To get even better performance, you need a more intense workload: You need to have many IOs outstanding on the disk at the same time. Say there are dozens of IOs being started simultaneously by the applications. The file system will take the first IO that occurs and send it to the SSD, because the SSD is faster. It will take the second IO and also send it to the SSD: The file system expects the first one to take 1ms, so the second one will get done at 2ms, which is still better then sending the second IO to the HDD. This argument remains true until the SSD has 10 IOs pending on it, so the expected latency is 10ms. At this point, the 11th IO should get sent to the HDD, since there it will take "only" 10ms, while on the SSD it would take 11. If you think this through, the file system will exactly balance the latency on the two devices. This means per second the SSD will do 1000 IOs, the HDD will do 100, for a total of 1100 per second, meaning an average latency of ~0.9ms.

What do we learn so far? That putting an SSD and an HDD into a mirror pair may help or hurt read performance, depending on exactly how the file system is implemented and what the workload characteristics are. And that the difference can be a factor of ~6 in my silly trivial example, from 180 IOs per second to 1100.

Now try the real world: Some reads are short, some are long. SSDs typically have superb latency (so they're superb for short reads), but only good throughput (so they're OK for long reads). Some reads are random (which really helps SSDs, they love random IO since they have no seek time), some are sequential (which really helps disks, on sequential IO they can do 200 MB/s for very few $$$). The workload may be highly parallel, with many IOs outstanding: that helps HDDs, whose efficiency increases significantly (factor of 2 or 3) when they have a big IO queue to work on, but it does not help SSDs.

My personal and professional answer: Only mix slow and fast disks if you are willing to tolerate really bizarre performance. It might be good, it might be horrible. Whatever it is, do not complain to me.
 
My personal and professional answer: Only mix slow and fast disks if you are willing to tolerate really bizarre performance. It might be good, it might be horrible. Whatever it is, do not complain to me.
I ran this experiment on my desk just out of curiosity. I will run some performance tests and let you know about the results. And the use-case here is a small desktop.
 
  • Like
Reactions: mer
I ran this experiment on my desk just out of curiosity. I will run some performance tests and let you know about the results. And the use-case here is a small desktop.
Now, I have finished this experiment (mirroring SSD with HDD) and putting both drives safely back in my drawer.

My personal conclusions:
  1. It is working and nothing serious or extraordinary happens;
  2. The experiment gave me another working and booting copy of my spare system on a HDD lying useless on my shelf;
  3. First, for testing I ran time make -j4 buildkernel on that mirror, then on an HDD only and SSD only.
    1. There was no big difference. I assume that building kernel is so much CPU dependent that ZFS speed did not make a big difference;
    2. The build times were 15.78 mins (HDD + SSD mirror), 16.85 mins (HDD only) and 15.33 mins (SSD only)
  4. Then I ran benchmarks/bonnie++ on all of these configurations. Will attach the results, but this seems to be difficult to evaluate;
  5. I ran also an "organoleptic" test on all these configurations without any numerical measurements - starting applications, browsing web, downloading e/mail etc.
    1. The HDD + SSD mirror felt almost the same as single SSD configuration;
    2. HDD only configuration felt a bit slower in certain tasks.
The bonnie++ results:

Code:
# SSD only

Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
Argentum     31120M  112k  99  682m  80  450m  74  434k  99  1.2g  99 +++++ +++
Latency             50248us     256ms    1065ms   21477us     444us   71066us
Version  1.98       ------Sequential Create------ --------Random Create--------
Argentum            -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 29725.271317  81 +++++ +++ 13814.084712  58 30584.740331  80 +++++ +++ 17935.332283  81
Latency              2289us      97us     392ms    2307us      41us   15149us
1.98,1.98,Argentum,1,1630185489,31120M,,8192,5,112,99,698698,80,460658,74,434,99,1251813,99,+++++,+++,16,,,,,29725,81,+++++,+++,13814,58,30584,80,+++++,+++,17935,81,50248us,256ms,1065ms,21477us,444us,71066us,2289us,97us,392ms,2307us,41us,15149us

# HDD + SSD mirror

Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
Argentum     31120M  112k  98  170m  19  174m  28  454k  99  1.3g  99  7880 395
Latency             48063us   10681ms    1527ms   19295us     369us     651ms
Version  1.98       ------Sequential Create------ --------Random Create--------
Argentum            -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 13862.589612  36 +++++ +++ 1597.721693   6 24726.236073  65 +++++ +++ 1169.420582   5
Latency               684ms      98us     583ms    2136us      89us     754ms
1.98,1.98,Argentum,1,1630183550,31120M,,8192,5,112,98,174586,19,177932,28,454,99,1316332,99,7880,395,16,,,,,13862,36,+++++,+++,1597,6,24726,65,+++++,+++,1169,5,48063us,10681ms,1527ms,19295us,369us,651ms,684ms,98us,583ms,2136us,89us,754ms

# HDD only

Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
Argentum     31120M  112k  98  226m  26  194m  32  426k  99  1.2g  99  5867 291
Latency             49908us    1400ms    1451ms   21059us     370us    1021ms
Version  1.98       ------Sequential Create------ --------Random Create--------
Argentum            -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 11651.591159  30 +++++ +++ 1347.271255   5 11630.183863  32 +++++ +++ 1600.102637   7
Latency               875ms     121us     579ms     345ms     140us     605ms
1.98,1.98,Argentum,1,1630191058,31120M,,8192,5,112,98,231442,26,198666,32,426,99,1250281,99,5867,291,16,,,,,11651,30,+++++,+++,1347,5,11630,32,+++++,+++,1600,7,49908us,1400ms,1451ms,21059us,370us,1021ms,875ms,121us,579ms,345ms,140us,605ms
 
… I assume that building kernel is so much CPU dependent that ZFS speed did not make a big difference; …

Someone once told me that whilst there's value in setting a number of jobs for make buildworld and make buildkernel, there's no value in having more than one job for the corresponding installations. The installations are relatively file system-intensive.
 
Last edited:
Someone once told me that whilst there's value in setting a number of jobs for make buildworld and make installkernel, there's no value in having more than one job for the corresponding installations. The installations are relatively file system-intensive.
With make -jn buildkernel the 'n' certainly has a big impact but yes, make installkernel is probably more disk intensive. It is pretty fast anyway. In my little experiment I did not want to intstall the kernel. Just rebuilt GENERIC several times over.
 
… whilst there's value in setting a number of jobs for make buildworld and make installkernel, …

A mistake there, should have been buildkernel. I'll correct the original post.



Incidentally, in recent months I always use multiple jobs for installation.

Durations greatly reduced, for an HDD in a circa 2015 notebook. Example below, YMMV.

Code:
% uname -KU
1500003 1500001
% uname -a
FreeBSD mowa219-gjp4-8570p-freebsd 15.0-CURRENT FreeBSD 15.0-CURRENT #3 main-n266317-f5b3e686292b-dirty: Thu Nov  9 12:49:29 GMT 2023     grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64
% grep -C 2 completed\ on /var/log/buildworld.log

--------------------------------------------------------------
>>> World build completed on Thu Nov  9 10:31:19 GMT 2023
>>> World built in 3208 seconds, ncpu: 8, make -j4
--------------------------------------------------------------
% grep -C 2 completed\ on /var/log/buildkernel.log
     4356.16 real      2766.18 user       363.33 sys
--------------------------------------------------------------
>>> Kernel build for GENERIC completed on Thu Nov  9 11:43:59 GMT 2023
--------------------------------------------------------------
--------------------------------------------------------------
--
     4002.48 real      2817.89 user       367.08 sys
--------------------------------------------------------------
>>> Kernel build for GENERIC-NODEBUG completed on Thu Nov  9 12:50:48 GMT 2023
--------------------------------------------------------------
>>> Kernel(s)  GENERIC GENERIC-NODEBUG built in 8369 seconds, ncpu: 8, make -j4
% grep -C 1 completed\ on /var/log/installkernel.log
--------------------------------------------------------------
>>> Installing kernel GENERIC completed on Thu Nov  9 20:22:06 GMT 2023
--------------------------------------------------------------
--
--------------------------------------------------------------
>>> Installing kernel GENERIC-NODEBUG completed on Thu Nov  9 20:23:48 GMT 2023
--------------------------------------------------------------
% grep -C 2 everything\ completed\ on /var/log/installworld.log
       28.98 real        16.25 user        12.06 sys
--------------------------------------------------------------
>>> Installing everything completed on Fri Nov 10 00:36:46 GMT 2023
--------------------------------------------------------------
      287.91 real       111.51 user       171.26 sys
%
 
Back
Top