External USB disk, shows "corrupt or invalid GPT detected"

PageDir

PageDir

I have two hard disks, both of them were installed with FreeBSD 9.
Put one of them into laptop, it can boot up,
but just can't recognize the other one connected via USB HDD enclosure.

Code: 
mass0: <IT-CEO IT-CEO USB Device, class 0/0, rev 2.00/1.14, addr 2> on usbus7
umass0:  SCSI over Bulk-Only; quirks = 0x0100
umass0:3:0:-1: Attached to scbus3
da0 at umass-sim0 bus 0 scbus3 target 0 lun 0
da0: <IT-CEO USB2.0SATADevice 1.14> Fixed Direct Access SCSI-4 device 
da0: 40.000MB/s transfers
da0: 238475MB (488397167 512 byte sectors: 255H 63S/T 30401C)
GEOM: da0: corrupt or invalid GPT detected.
GEOM: da0: GPT rejected -- may not be recoverable.

I have checked it with "dd if=/dev/da0 of=./data bs=10m count=1",
the pmbr are same, at least the first gpt table is correct.

And I found that the error message occurs only if both gpt tables are corrupt
which is not possible, because if I put one of the disks into laptop
it boots up without error.

Code: 
 839     /* Fail if we haven't got any good tables at all. */
 840     if (table->state[GPT_ELT_PRITBL] != GPT_STATE_OK &&
 841         table->state[GPT_ELT_SECTBL] != GPT_STATE_OK) {
 842         printf("GEOM: %s: corrupt or invalid GPT detected.\n",
 843             pp->name);
 844         printf("GEOM: %s: GPT rejected -- may not be recoverable.\n",
 845             pp->name);
 846         return (EINVAL);
 847     }

Any ideas?
 
There is a backup GPT table at the end of the disk. Erasing that could give this error. Your comparison did not check the backup tables. gptboot can still boot if it finds the primary GPT table, although it will give a warning. See gpart(8) about the recover option. Please make a backup before trying it.
 
Oh, some thing really strange:

When connected throught USB HDD enclosure,
I checked the 2nd gpt table with "dd skip=",
and found out that the last 512b is missing. So the gpt header is lost.

But when the disk is installed into laptop,
"dd skip=" again, the last 512b turns back!

It's also proved by reports of diskinfo:

In laptop:
Code: 
diskinfo -v ada0
ada0
        512             # sectorsize
        500107862016    # mediasize in bytes (465G)
        976773168       # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        969021          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        J2110051J2S4NA  # Disk ident.

In USB HDD enclosure:
Code: 
diskinfo -v da0
da0
        512             # sectorsize
        500107861504    # mediasize in bytes (465G)
        976773167       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        60801           # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        4A323131303035314A325334        # Disk ident.

500107862016-500107861504=512

Both two disks are in such situtation!
One is Hitachi(500G), the other is WD25000(250G).

Is this a bug of my USB HDD enclosure?
If so, any way to avoid checking 2nd gpt table?
(actually 1st gpt table is fine, geom just can't 'see' 2nd gpt table's last 512 bytes)
 
BTW:
when connected via usb, /dev/da0
the disk size reported by smartctl is correct.

WD25000 in USB HDD enclosure::

Code: 
smartctl -d sat -a /dev/da0
User Capacity:    250,059,350,016 bytes [250 GB]

Code: 
# diskinfo -v da0
da0
        512             # sectorsize
        250059349504    # mediasize in bytes (232G)
        488397167       # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        30401           # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        57442D57585230453939585A        # Disk ident.

250,059,350,016 - 250059349504 = 512
 
If the enclosure has RAID functions, it may want the last block of a drive for RAID metadata. Or it may just be an off-by-one error in the enclosure firmware.
 
That's still serious problem because the size of the disk should match exactly regardless of the method of connection. I think there's a PR about this kind of problem with the USB mass media support in FreeBSD, just can't find it right now.
 
kpa said:
That's still serious problem because the size of the disk should match exactly regardless of the method of connection. I think there's a PR about this kind of problem with the USB mass media support in FreeBSD, just can't find it right now.
Click to expand...

So, maybe I should ask on mailing list?
 
[SOLVED] corrupt or invalid GPT detected

I have been encountering what looks to be this exact same problem, and I found a rather interesting solution.

It appears to me that the problem arises when one uses gpart to set up a GPT partition table on a drive, but specifically ** while it is connected directly to the motherboard, e.g. via an internal SATA port (and perhaps also when connected direct via an internal PATA port and perhaps also when connected via an external eSATA port) **. Then, you will only see the problem if you take that same drive and install it into just ** certain kinds ** of external USB enclosures. (A Patroit Gauntlet 2 USB 3.0 2.5" external enclosure exhibited the problem, however the exact same drive installed into a Rosewill external USB 2.0 3.5" enclosure did not.)

diskinfo appears to report different "media" sizes for the same drive depending on whether it is connected via SATA/eSATA/PATA or USB. I'm not yet entirely sure what to make of that, and I need to do more research.

The bizarre and totally unexpected solution that I found for this problem was to take the drive, once it has been installed into an external USB enclosure (and once it is thus exhibiting the problem) and plug the thing in (via USB) to some Windows 7 machine. (I haven't checked, but I suspect that this trick may also work on at least some other versions of Windows as well.)

Plugging the drive into a Windows system (magically?) "cures" the problem. Once you have done this, you can take the drive (in its USB enclosure) back to your FreeBSD system and plug it in again and you will no longer get the "corrupt or invalid GPT detected" error messages and the thing will then work just fine.

I have not yet done any research to try to find out what Windows might be doing to fix the problem, however I suspect that it is taking the god GPT from the beginning of the drive, accepting that as valid, and then just re-writing it again at the (new?) "end" of the drive... which is apparently enough to make the FreeBSD GEOM subsystem happy. But this is all just speculation on my part. I have no real data to support any of that. I just know that the trick of plugging the thing into Windows, briefly, seems to "cure" the problem.
 
A comparison of the GPTs before and after Windows does its... magic, let's say... could be interesting.
 
I agree that detailed analysis of the contents of blocks, both at the start and at the "end" of a drive, before and after Windows performs its "repair" of the GPT information would be useful to the FreeBSD GEOM developers. I however am not one of those, and barely know my GPT from a hole in the ground, so I have decided to just file one or more appropriate PRs on this matter, and then let the people who actually know something about this stuff decide what to do next.

Additionally however, since I have both a "spare" 2.5" drive and also a "spare" one of these Patriot Gauntlet 2 USB 3.0 2.5" external enclosures, if any legitimate FreeBSD GEOM developer should find either of both of these things helpful, as they try to rectify the problem, I will volunteer to loan these items to any such developer, for a period of three weeks, with shipping costs in both directions paid at my expense. (However just for some developer within the U.S.A. Shipping elsewhere is too expensive and might create VAT hassles.)

Please note that when I say "spare", I only mean "for the moment". I bought these things intending to use them, and still plan to do so, so I'd need them back after the loan.
 
Yep, only *some* external USB enclosures implment this weirdness

I went back, just to double check myself, and made sure what I was saying before was correct.

I started again from scratch. I dd'ed several mebibytes of zeros to the start of a drive. Then I took that drive and wired it directly (SATA) to one of my systems, and then did a clean fresh minimal FreeBSD install on it, using GPT partitioning. That all went well and the drive booted afterwards, no problem.

Then I extracted he drive, put it into my Rosewill RX-35-AT-SC USB 2.0 3.5" external (powered) enclosure and then wired that up to a system I have here running 9.1-RELEASE. The system in question then properly saw and recognized the drive and its GPT partitions, no problem.

Then, not even having mounted anything, or done anything else, I carefully extracted the drive and stuck it into one of my Patriot Gauntlet 2 external (unpowered) USB 3.0 enclosures and wider that up to the exact same system. And sure 'nuff... BLAMO! I got the message about alleged "corrupted" GPT again.

For the record, lsusb says about the Rosewill enclosure:

Code: 
Bus /dev/usb Device /dev/ugen2.3: ID 152d:2336 JMicron Technology Corp. / JMicron USA Technology Corp. Hard Disk Drive

whereas lsusb says about the Patriot:

Code: 
Bus /dev/usb Device /dev/ugen2.3: ID 11b0:6518 ATECH FLASH TECHNOLOGY

I'm just providing all this info in the interests of full disclosure. Maybe with the help of Google I can help the next poor schmuck to avoid tearing his hair out over this.
 
wblock@ said:
Just dd(1) the first and last 34 blocks on the drive to files, from before and after.
Click to expand...

Yes, and then what?

Sorry Warren, my plate already overfloweth. I got critical stuff that I've had to let slip already on account of this rubbish.

In addition to this, I also still have to find out why plugging and unplugging external USB drives from my shiny new USB 3.0 PCIe card causes the kernel to completely freeze on occasion (frequently, actually), for no apparently good reason. I still owe somebody else more work on that.

And ah, oh yea... the rest of my life? Oh well. To heck with that!
 
I'm trying not to be annoyed by this entire ridiculous problem, but so far it's not working.

So let me see if I understand this... There is a copy of the GPT at the start of the drive and another duplicate (backup?) copy at the end of the drive? Yes? And if the two don't match then Windows is smart enough to figure out which one is the garbled one, and then replaces that one with the good one and all is well again, whereas FreeBSD sees that the two don't match, and immediately throws up its hands and gives up, thereby rendering the entire drive unusable. Is that about the size of it?

This isn't going to win any awards for user friendliness on FreeBSD's part.
 
FreeBSD follows the GPT standard to the exact letter, that is the safest thing to do in a serious server business. What windows does is irrelevant here.
 
FreeBSD follows the GPT standard to the exact letter
Click to expand...

I can hear a faint echo of my long gone mother, asking me as a child "So, if Johnny jumped off a bridge, you would too?"

Translation: This is one of the absolutely lamest excuses for blatantly broke, non-working software I've ever heard.

Meanwhile, in entirely unrelated news...

@wblock@, Did you know your mail server is down?

Code: 
% telnet wonkity.com 25
Trying 67.158.26.137...
telnet: connect to address 67.158.26.137: Operation timed out
telnet: Unable to connect to remote host
 
Last edited by a moderator:
How the hell do you prepare for all the possible POS broken pieces of hardware that are only designed to work on windows on an OS like FreeBSD? The answer is that you don't in general. You stick to the standards and in case something is so broken it's not worth trying to come up with a work around you tell the users to get something else that works.

I've heard this talk "It has to be user friendly even if it means spending endless hours trying to fix broken crap", I won't hear it anymore and more and more of the IT world is getting fed up with it as well.
 
You said that "FreeBSD follows the GPT standard to the exact letter..."

Having personally served on three different standards committees in my lifetime, I really do appreciate the importance of standards. I do however have a pertinent question relating to your assertion...

Could you please cite for me (or better yet provide a link to) the specific passage or section of the GPT "standard" that specifically requires an operating system which implements the standard to choke and refuse to do anything at all useful when and if the GPT at the start of the drive and/or the GPT at the end of the drive are found to be "corrupted"?

I really would like to peruse and study that specific section of whatever "standard" you are referring to.

I would also like to like to read the "standard" document in question, if for no other reason then at least to see its justification for keeping two different copies of the GPT on each drive, despite the fact that (as you claim) one cannot be used as an emergency backup for the other, e.g. in times of trouble.

(My apologies. I really should have asked these questions earlier.)
 
Warren,

I changed my mind and gathered the data that you felt would be "interesting". I hope that you will grab it now and analyze it. THAT is something that I really do not have time for, personally. Sorry.

The raw dd files can be grabbed here:

ftp://ftp.tristatelogic.com/pub/wb/

There are four files there, the first 1 MiB of the drive (installed into the external USB enclosure) both before and after windows "cured" or "healed' the GPT stuff, and also, a before and after dump of the last part of the drive from that last 1 Mib boundary on the drive until dd decided to call it quits. This is equal to 4 Kib * 173.

This is all just raw binary data obtained via dd.

Now that I took the trouble to get the data, I hope that either you will look at it or else that you will get someone else who knows something about GPT to look at it. As I've already said, it is pointless for me to look at it, because I don't know a GPT from a hole in the ground.
 
Warren, I almost forgot...

Regarding the data files that I put on my ftp server, those came from a drive that had been set up as follows.

All I did was to boot to/from a FreeBSD memory stick image and then ran the "Live" version of FreeBSD. Using that (and gpart) I initialized a GPT scheme on the test drive (while it was connected via SATA) and then just created a single 1G GPT partition on the test drive (again, while it was connected via SATA). That's it, period.

Then, after that, I did a clean shutdown, popped the drive out, popped it into the external Patriot USB enclosure and then plugged that into a different FreeBSD (9.1-RELEASE) system which, of course, and as expected, immediately logged a GPT "corrupted" error to /var/log/messages. Then I plugged the thing in briefly to a Windows 7 machine, and then removed it and plugged in back into the FreeBSD system, which, as I have already reported now found the drive to be just perfectly peachy.

I suppose that all this information might be helpful for you to know, e.g. if you are going to try to look at the before and after GPTs.

Now at least you know what the GPT should contain, i.e. just a single 1 Gib partition.
 
I'm interested in this, but have some other things going on with a higher priority (like the email problem).

So far, it still looks to me like the Patriot USB enclosure is at fault. My guess is still that it either takes that last block for itself, or has an off-by-one error in the firmware. I'd also guess that Windows helpfully rewrites the backup GPT at what is now seen as the end of the drive. The interesting part is how FreeBSD might see that as valid. (And also whether any of this is correct, seeing as it's all based on guesswork.)

The Wikipedia page on GPT is pretty good. AFAIR, the standard is part of the UEFI standard, and last time I looked it required a registration to get that standard.
 
The kern.geom.part.check_integrity sysctl might allow ignoring a corrupted backup GPT. Whether that's a good idea is a judgement call.
 
The kern.geom.part.check_integrity sysctl might allow ignoring a corrupted backup GPT. Whether that's a good idea is a judgement call.
Click to expand...

Given the choice between having the kernel ignore the error, or having the kernel just magically fix everything up, the way that Windows appears to do, I'll take the latter, thank you.

P.S. I made some meager effort to try to deduce whether or not the SATA/USB protocol converter in the Patriot external enclosure was messing with the size apparent size of the drive, but I am currently stymied by a lack of adequate documentation for the diskinfo(8) utility, a fact which I have just filed a formal PR on:

http://www.freebsd.org/cgi/query-pr.cgi?pr=177457

What I can tell you is that I'm not trusting anything that diskinfo -v says, at least until I have a better understanding of where and how it is getting its information.

Nontheless, here are some numbers relating to the drive in question:

1,000,204,886,016 -- size as reported by smartctl
1,000,203,804,160 -- size as reported by diskinfo when the drive is connected via SATA
1,000,203,802,112 -- size as reported by diskinfo when the drive is connected via USB and is mounted inside of the Patriot Gauntlet 2 external enclosure.

The difference between the first and second numbers is == 2,113 * 512
The difference between the second and third numbers is 2,048

FYI -- This is an "AF" (4KB physical block size) drive.

It is definitely possible that somebody, or maybe even several somebodies, is/are messing with the logical drive size, e.g. related to HPA and/or DCO. That could be the FreeBSD kernel, or gpart, or the Windows kernel, or the system in-built USB controller or the SATA/USB protocol converter in some but not all external enclosures (e.g. Patriot).

One final bit of info: I verified today that the problem is entirely reproducible even in the absence of any actual partitions. I connected the drive via SATA to one system, dd'ed a bunch of /dev/zero to it, about 50 MB, and then just did a [cmd=]gpart create -s GPT ada0[/cmd]. Then I did a clean shutdown, popped the drive into the Patriot external USB enclosure, plugged that into a running FreeBSD system, and immediately saw the same old syslog message about the GPT allegedly being "corrupted".
 
You must log in or register to reply here.
Back
Top