confusion about dd conv=sync ?

[j77h]

The handbook's "Writing an Image File to USB" provides this command:

# dd if=FreeBSD-13.1-RELEASE-amd64-memstick.img of=/dev/da0 bs=1M conv=sync

I thought conv=sync was a typo, or BSD-specific, so on Linux I used conv=fsync instead.

NomadBSD nudged me in that direction by giving conv=fsync in its command for use on Linux.

But after more investigation I found that :
"conv=sync" means the same on both FreeBSD and Linux;
the example at the end of the man page says "padding the end with zeros to a 1MiB boundary".

Now I'm wondering if it would be helpul to add "status=progress oflag=fsync" to the documentation as follows:

# dd if=FreeBSD-13.1-RELEASE-amd64-memstick.img of=/dev/da0 bs=1M status=progress oflag=fsync conv=sync

and explain that :

• "conv=sync" simply pads the final block with NUL bytes;
• "oflag=fsync" ensures that dd does not return you to the prompt until it has finished writing to the physical UFD.

The documentation already says "To force all writes to complete, use sync(8)",
but it seems more convenient to have "status=progress oflag=fsync" in the dd command instead.

But I wonder why they didn't use simply "conv=fsync,osync",
as it seems that would do the same job, according to the dd man page
(which isn't really clear enough, I'm still puzzled by it).
This is a condensed extract:

    conv=...
        fsync    Perform an fsync(2) on the output file before closing it.
        sync     Pad every input block with NUL to the input buffer size.
        osync    Pad the final output block to the full output block size.

Without fsync, dd will return when it has finished writing only to the buffer in RAM,
and you won't know when writing to the UFD has finished if it doesn't have a LED (some don't).

Didn't see an easy way to suggest documentation changes,
so posting here, where grahamperrin@ might see it.

 

[YYY__]

FYI:

Didn't see an easy way to suggest documentation changes,

FreeBSD Mailing lists: subscription for freebsd-doc

sync(2)

The sync() system call forces a write of dirty (modified) buffers in
the block buffer cache out to disk. The kernel keeps this information
in core to reduce the number of disk I/O transfers required by the sys-
tem. As information in the cache is lost after a system crash, a
sync() system call is issued frequently by the kernel process syncer(4)
(about every 30 seconds).
The fsync(2) system call may be used to synchronize individual file de-
scriptor attributes.
BUGS
The sync() system call may return before the buffers are completely
flushed.

syncer(4)

The syncer kernel process helps protect the integrity of disk volumes
by flushing volatile cached file system data to disk.

The kernel places all vnode(9)'s in a number of queues. The syncer
process works through the queues in a round-robin fashion, usually pro-
cessing one queue per second. For each vnode(9) on that queue, the
syncer process forces a write out to disk of its dirty buffers.

The usual delay between the time buffers are dirtied and the time they
are synced is controlled by the following sysctl(8) tunable variables:

Variable Default Description
kern.filedelay 30 time to delay syncing files
kern.dirdelay 29 time to delay syncing directories
kern.metadelay 28 time to delay syncing metadata

⚙ D21422 Add oflag=fsync and oflag=sync capability to dd

reviews.freebsd.org

SUMMARY
Sets the O_FSYNC flag on the output file. oflag=fsync and oflag=sync are synonyms just as O_FSYNC and O_SYNC are synonyms. This functionality is intended to improve portability of dd commands in the ZFS test suite.

Sponsored by: iXsytems, Inc.

Sync Linux version:

According to the standard specification (e.g., POSIX.1-2001),
sync() schedules the writes, but may return before the actual
writing is done. However Linux waits for I/O completions, and
thus sync() or syncfs() provide the same guarantees as fsync()
called on every file in the system or filesystem respectively.
BUGS
Persistence guarantees vary per system. See the system calls
below for more details.

Understanding the Linux Kernel, 3rd Edition

The sync( ), fsync( ), and fdatasync( ) System Calls In this section, we examine briefly the three system calls available to user applications to flush dirty buffers to disk: … - Selection from Understanding the Linux Kernel, 3rd Edition [Book]

www.oreilly.com

sync( )
Allows a process to flush all dirty buffers to disk
fsync( )
Allows a process to flush all blocks that belong to a specific open file to disk

 

[smithi]

The documentation already says "To force all writes to complete, use sync(8)",

i.e. # dd <parameters> && sync

but it seems more convenient to have "status=progress oflag=fsync" in the dd command instead.

'status=progress' writes a line every second which is a great way to slow a big dd down ..

I feel adding oflag= to the dd command to make memsticks would likely increase confusion, when fsync can be added to the conv= options to apparently the same effect.

But I wonder why they didn't use simply "conv=fsync,osync",
as it seems that would do the same job, according to the dd man page

'conv=sync,fsync' should do, but not osync, see below.

(which isn't really clear enough, I'm still puzzled by it).
This is a condensed extract:

    conv=...
        fsync    Perform an fsync(2) on the output file before closing it.
        sync     Pad every input block with NUL to the input buffer size.
        osync    Pad the final output block to the full output block size.

Without fsync, dd will return when it has finished writing only to the buffer in RAM,

I've never heard of that being an issue, has anyone? But anyway:

"osync Pad the final output block to the full output block size. [...] This option is incompatible with use of the bs=n block size specification."

Just 'conv=sync,fsync' should be the least astonishing change?

 

[j77h]

Thanks for the feedback.
I was a bit unwell that day, and was annoyed by the lack of clarity about `dd`.

The bottom line was that I thought 'conv=sync' was a typo.
I had previously replaced it with 'conv=fsync' and the UFD worked fine.
The bit about "use sync(8)" is further down the page,
and a bit puzzling to a relative noob like me (not into shell that much).
If " && sync" had been on the end of the dd command I would have got it.

There's a subtle difference between these:
oflag fsync Set the O_FSYNC flag on the output file ...
conv fsync Perform an fsync(2) on the output file before closing it.
Which one is better for knowing when writing to a UFD has finished?

And this looks like it could pad every block with space or NUL:
conv sync Pad every input block to the input buffer size.
which is why I suggested
conv osync Pad the final output block
but by then I'd forgotten it was "incompatible with bs".

I've used dd many times over the years,
simply by cut and pasting a command as given.

Now that I try to understand it, it's too confusing:
+ same options under oflag and conv, but subtly different;
+ do the subtle differences make any practical difference when writing a UFD;
+ conv sync and osync mean "pad", not sync in the usual sense;
+ why pad all input blocks? for optical disks i think, but
+ will it pad all my UFD output blocks? (It didn't, but I didn't know til I tried it.)

I do understand the points made in your replies,
but I still think that `dd` needs to be explained better,
and the 2 '*sync' flags that mean "pad" should be renamed.
Not expecting it to happen, as it's been this way for years
and maybe no-one else mentioned being confused by it.

 

[YYY__]

I put some comments that I later deleted, because they were just assumptions and I want to avoid misunderstandings. These are too technical things that I don't want to address.

I guess specifying a block size and filling the block helps the log be complete and write immediately to/from the cache, without waiting for another block, kind of like saying the read and write is synchronous. Additionally, syncer runs sync from time to time. I have observed that the cadence between dd completion is lower on FreeBSD than on Linux. As you said, in Linux you can find dd finished and the USB continue writing, which is the beginning of this whole discussion.

dd()

The dd utility copies the standard input to the standard output. Input
data is read and written in 512-byte blocks. If input reads are short,
input from multiple reads are aggregated to form the output block.
When finished, dd displays the number of complete and partial input and
output blocks and truncated input records to the standard error output.

 

[Andriy]

Writing to a device is always synchronous on FreeBSD.

 

[smithi]

Writing to a device is always synchronous on FreeBSD.

Thanks for clearing this up. I kept thinking we'd have heard something, and surely release notes advice for making installation memsticks would have said, if this was an issue.

 

[j77h]

Here's what happens on Linux (Manjaro).
The prompt string is the date+time.

[231121 15:37:42]# dd if='FreeBSD-14.0-RELEASE-amd64-dvd1.iso' of=/dev/sde bs=1m status=progress conv=fsync,sync
dd: invalid number: '1m'
[231121 15:37:51]#

Try again with big M.

[231121 15:38:05]# dd if='FreeBSD-14.0-RELEASE-amd64-dvd1.iso' of=/dev/sde bs=1M status=progress conv=fsync,sync
4530896896 bytes (4.5 GB, 4.2 GiB) copied, 26 s, 174 MB/s 4541382656 bytes (4.5 GB, 4.2 GiB) copied, 26.8613 s, 169 MB/s

At this point (after only 26s) dd had finished writing its progress.
The stick's LED continued flashing for nearly 10 minutes, then

4330+1 records in
4331+0 records out
4541382656 bytes (4.5 GB, 4.2 GiB) copied, 559.534 s, 8.1 MB/s
[231121 15:47:42]#

The stick's LED stopped flashing about 5 seconds later.

So 'status=progress' is not much use, but it doesn't slow things down either.

Without 'conv=fsync', the final message and return to prompt
would also have happened at 26s, IIRC from previous use of dd (on Linux),
and with a stick with no LED, you wouldn't know when the writing had finished.

After this, I was able to boot from the USB stick.
The doc page says the DVD iso has to be written to an optical drive,
which is one more reason to try sending a Documentation bug report.

 

[smithi]

Here's what happens on Linux (Manjaro).
The prompt string is the date+time.

It's already been made clear, perhaps elsewhere, that assumptions about FreeBSD's dd(1) from comparison with Linux' version is fraught and likely incorrect.

Case in point #1: 'm' works fine on FreeBSD ...

Try again with big M.

[231121 15:38:05]# dd if='FreeBSD-14.0-RELEASE-amd64-dvd1.iso' of=/dev/sde bs=1M status=progress conv=fsync,sync
4530896896 bytes (4.5 GB, 4.2 GiB) copied, 26 s, 174 MB/s 4541382656 bytes (4.5 GB, 4.2 GiB) copied, 26.8613 s, 169 MB/s

At this point (after only 26s) dd had finished writing its progress.
The stick's LED continued flashing for nearly 10 minutes, then

4330+1 records in
4331+0 records out
4541382656 bytes (4.5 GB, 4.2 GiB) copied, 559.534 s, 8.1 MB/s
[231121 15:47:42]#

The stick's LED stopped flashing about 5 seconds later.

Ok, so now try it on FreeBSD with standard parameters (plus time(1) for timing comparison) :

# time dd if=$dvd1 of=/dev/da0 bs=1m conv=sync

Noting that conv=sync has nothing to do with file write synchronisation, but padding final block with zeroes to 1m if needed, and that:

Writing to a device is always synchronous on FreeBSD.

So 'status=progress' is not much use, but it doesn't slow things down either.

On Linux. Compare with and without for above on FreeBSD.

Without 'conv=fsync', the final message and return to prompt
would also have happened at 26s, IIRC from previous use of dd (on Linux),
and with a stick with no LED, you wouldn't know when the writing had finished.

On Linux, but not FreeBSD. It's good advice for those having to use Linux to write sticks.

After this, I was able to boot from the USB stick.
The doc page says the DVD iso has to be written to an optical drive,
which is one more reason to try sending a Documentation bug report.

You could try, but it's only been relatively recent (before 12.3 and 13.1, I think) that these ISOs were made hybrid for dual use, and then only for amd64.

The release announcements are where they were introduced and definitive, and unlikely to suggest an incorrect method for making installation media on FreeBSD.

 

[j77h]

When someone who is not running FreeBSD (and may be unfamiliar with it)
uses the handbook to help them prepare to install FreeBSD for the first time,
they will need a dd command (or other method) for their current OS.

Wouldn't it be useful to regard that handbook page
as primarily for people who are not running FreeBSD,
and include commands suitable for the other main OSes?

(Sorry for late reply - the forum is not emailing notifications.)

EDIT:

Examples from https://nomadbsd.org/download.html

macOS
# dd if=nomadbsd-x.y.z.img of=/dev/rdiskX bs=1m
Linux
# dd if=nomadbsd-x.y.z.img of=/dev/sdX bs=1M conv=fsync

He does not give "conv=sync" for Linux, Mac, and the other 3 BSDs,
so it seems like it's not really necessary.

 

[smithi]

When someone who is not running FreeBSD (and may be unfamiliar with it)
uses the handbook to help them prepare to install FreeBSD for the first time,
they will need a dd command (or other method) for their current OS.

Wouldn't it be useful to regard that handbook page
as primarily for people who are not running FreeBSD,
and include commands suitable for the other main OSes?

You could put that argument when you submit your doc PR, but you'll likely need to draft alternative wording.

Personally I think adding more complication unhelpful, but a paragraph suggesting maybe adding '&& sync' to that dd command for Linux and Mac should be sufficient.

Certainly NomadBSD docs are specific to self, and that's fine.

EDIT:
Examples from https://nomadbsd.org/download.html

macOS
# dd if=nomadbsd-x.y.z.img of=/dev/rdiskX bs=1m
Linux
# dd if=nomadbsd-x.y.z.img of=/dev/sdX bs=1M conv=fsync

He does not give "conv=sync" for Linux, Mac, and the other 3 BSDs,
so it seems like it's not really necessary.

Once again, it's not necessary specifically for NomadBSD!

Please resist extrapolating specific derivatives to the general case for FreeBSD. From Nomad's changelog:

• The partition alignment has been changed to 1M to improve the write speed on flash drives.

... which makes conv=sync not necessary on dd with bs=1m !

 

[Cath O'Deray]

From the first answer to linux - What does dd conv=sync,noerror do? - Super User, it seems to me that:

• conv=sync without noerror will allow dd to stop if an error is encountered.

 

[smithi]

From the first answer to linux - What does dd conv=sync,noerror do? - Super User, it seems to me that:

• conv=sync without noerror will allow dd to stop if an error is encountered.

I can't speak about Linux - and saw nothing clearly definitive at your first link anyway - but that's true for FreeBSD.

In the general case - and specifically dd'ing .iso or .img to installation memsticks - we surely want to stop on error.

Conversely, noerror is useful where there are bad blocks that are output as zeroes but other valuable data remains.

[edit: I can see no utility in messing with dd's options or manpage at this point; it's too old and convoluted enough that 'retraining' its users would be a fool's errand]