Different Checksum after writing to thumbdrive

L

lawg

Hello,

I have downloaded the Freebsd-10.1-amd64-uefi-memstick.img and did a sha256 checksum which return the same checksum from the FreeBSD index.

However when I write to a thumbdrive - dd if=FreeBSD-10.1-RELEASE-amd64-uefi-memstick.img of=/dev/disk4 bs=64k - and do a - openssl dgst -sha256 /dev/disk4 - it returns a different checksum.

Can the thumbdrive be use to install FreeBSD? If not any suggestion as to how to get it right?

NOTE:
Downloaded the DVD1...iso but the checksum, when applied to the DVD returns a different checksum, something to do with the padding inserted by the burn/write program.

Thank you.
 
I'll use some random numbers here to explain the principle.

You downloaded FreeBSD...img. Say for fun it is 2,000,000,000 bytes long. That size may or may not be a multiple of 512 bytes, and it may or may not be a multiple of 64KiB long. You checksum that file, meaning you let SHA256 take the "checksum" of exactly those 2 billion bytes, and you get answer "X".

Then you copy this file to your thumbdrive. The capacity of that thumbdrive is larger than 2 billion bytes (we hope, otherwise you'd have a big problem). Say for example that it is an 8 gig thumbdrive (I say that simply because the one I have in my pocket full of music for the car happens to be 8 gig big), and say that for example the thumb drive has a real capacity of 2^30 bytes (meaning 8 GiB, not decimal 8GB). That thumbdrive has a certain sector size (say for fun 512 bytes), and you copied onto it using 64 KiB writes (that's the option to the dd command you gave).

Problem 1: The size of the image might not have been an exact multiple of the sector size of the thumbdrive, nor an exact multiple of the IO size used in dd. We could now argue exactly what will happen if you write incomplete sectors (and whether they get padded with zero or random data), and whether dd does one smaller IO at the end, pads the last IO (with what?), or silently drops the last partial IO (I happen to know the answer to the dd question). But as you can see, writing the file may already have changed the last little bit at the end. We could be optimistic and assume that the ...img input file had a size that was exactly divisible by both the sector size of your thumbdrive and by 64KiB. Probably true.

Problem 2: The second checksum command checksummed the whole thumb drive, not the first 2 billion bytes. Look at the command you issued: openssl ... /dev/disk4. So you calculated the checksum of the first 2 billion bytes (which was probably correct), then perhaps of the very last block or sector which was padded/truncated/lost, and then of about another 6 GiB worth of random stuff. The random stuff at the end is the main cause of the invalid checksum.

Here's what I would do. First, look at the exact size of the image file. Is it a multiple of 512, of 2K, of 4K, or of 64 KiB? Then check the sector size of your thumbdrive, and make sure the size of the image file is really a multiple of it. Then copy the image file to the thumbdrive using a block size that is guaranteed to be a multiple of the hardware sector size, but a divisor of the image file size. Like this, you will only write whole sectors, but not have to deal with partial blocks at the end. Should the size of the image file not allow this, then you have a serious problem.

If you want to afterwards verify that the write succeeded, you need to make sure you only read the area actually written while checksumming. Here is one way to do it: Calculate the exact number of dd blocks that the the size of the image file corresponds to. In our example above, that would be 2,000,000,000 / 65536 (which is obviously a silly example, since 2 billion is not divisible by 64K). Say that you have calculated that the image file is exactly 123456 blocks long. Then execute the checksum command as follows:
dd if=/dev/disk4 bs=64k count=123456 | openssl dgst -sha256
and it will spit out the checksum of the area on the disk that actually corresponds to the image file.

Good luck!
 
Thank you for that reply ralphbsz.

That being said, the method you recommended will possibly give you the same checksum if you ignore the padding and the software used.

However, does that mean that it is not possible to get the same checksum when write/burn from an img/iso file to a DVD/CD/Thumbdrive?

I got
Code: 
gptboot: backup GPT header checksum mismatch
when the thumbdrive booted and my desktop system was unable to connect to the internet after the installation. Something went sideways during the install due to perhaps my harddisk, having trouble(got S.M.A.R.T. failure from disk utility program), or perhaps it has something to do with the mismatch checksum. So I got new 500G hard disk and will try again.

Does this means that I am going to have problems with FreeBSD after installation? Or installing any program for that matter.





 
lawg said:
However, does that mean that it is not possible to get the same checksum when write/burn from an img/iso file to a DVD/CD/Thumbdrive?
Click to expand...

Personally, I don't bother verifying any install media with checksums. That's because all "media" (including network transmissions) is already internally protected with mechanisms similar to checksum (ECC on disk and CD/DVD drives, and on flash media, CRCs and friends on networks). The main thing that these checksums protect are software defects and main memory errors: the latter are rare, and the former should be non-existing. What I do however do: After burning a CD or writing a memory stick, I put it in a computer, and read it cover to cover (without looking at the content). It's not uncommon to get read errors there.

Having said that: Assuming that the image file is a sensible size (namely a multiple of the sector size of the media), the checksums SHOULD match, if you read only the part of the CD/DVD/Thumbdrive that was actually written. On a burned CD/DVD, the readable area is exactly the size that is written, so there reading the whole media should give the correct checksum without even counting sectors.

I got
Code: 
gptboot: backup GPT header checksum mismatch
when the thumbdrive booted
Click to expand...


I'm not sure what that message means. The GPT layout has two copies of the partition table, the second one at the end of the media. It could be that the backup copy (at the end of the media) is corrupted or not present. This could happen if the image file is not the correct length (got truncated in transit somehow), but a simple check of the size or checksum should catch that, so this is not the problem. It could also be that the image file was created without a valid backup GPT table at the end, but I would consider that to be a bug.
and my desktop system was unable to connect to the internet after the installation.
Click to expand...
But did the installation otherwise proceed as normal? I would think that if the installation works normally, and puts a copy of FreeBSD on your root disk, that the problem with the missing internet connection would not be a problem with the install media, but with some configuration.

Something went sideways during the install due to perhaps my harddisk, having trouble(got S.M.A.R.T. failure from disk utility program), or perhaps it has something to do with the mismatch checksum. So I got new 500G hard disk and will try again.
Click to expand...

If you already have a SMART error on the boot disk, it's pretty pointless to proceed. The system you install won't live very long if the disk is admitting that it is nearing the end of its life.

Does this means that I am going to have problems with FreeBSD after installation? Or installing any program for that matter.
Click to expand...
If your boot disk is dying or broken, the system is not going to be much fun. I refer to it as "CTD", which is a nasty acronym paramedics use to describe a patient who is "Circling The Drain"; the next stage is FTD = Fixing To Die.

I was going to propose that your first step should be finding a better disk, but you're already doing that.

The second step needs to be finding someone who has tried copying the memory stick install image to a thumbdrive successfully, and have them tell us whether they got that "GPT backup table mismatch" error too. If they didn't, then something is still wrong with the way you write your thumbdrive.
 
Hmm, I think it is perhaps is the way I write the thumbdrive. Going by the check list, sha256() index matches that of the Freebsd-10.1-RELEASE-amd64-uefi-memstick.img downloaded.

Got the following when installing:-

Code: 
.
.
GEOM: da0: the secondary GPT table is corrupt or invalid.
GEOM: da0: using the primary only -- recovery suggested.
.
.

Still end up unable to access the internet, problem with the network auto detect system. But that is another matter and not for this thread.

Thank you for your patience and reply:)
 
You must log in or register to reply here.
Back
Top