Solved [Solved] dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB:...

Chris_H · Apr 2, 2014

Greetings,
I've enjoyed the use of dump()(8) for years w/o issue until recently. I just finished performing a dump of /var, /usr, and /tmp. But the last mount point (/) is throwing errors about 3/4 of the way through:

Code:

(ada0:ata0:0:0:0): READ_DMA. ACB: c8 00 bf cc 41 40 00 00 00 00 80 00
(ada0:ata0:0:0:0): CAM status: ATA Status Error
(ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
(ada0:ata0:0:0:0): RES: 51 40 14 cd 41 00 00 00 00 00 00
(ada0:ata0:0:0:0): Retrying command
(ada0:ata0:0:0:0): READ_DMA. ACB: c8 00 bf cc 41 40 00 00 00 00 80 00
(ada0:ata0:0:0:0): CAM status: ATA Status Error
(ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
(ada0:ata0:0:0:0): RES: 51 40 14 cd 41 00 00 00 00 00 00
(ada0:ata0:0:0:0): Retrying command
(ada0:ata0:0:0:0): READ_DMA. ACB: c8 00 bf cc 41 40 00 00 00 00 80 00
(ada0:ata0:0:0:0): CAM status: ATA Status Error
(ada0:ata0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
(ada0:ata0:0:0:0): RES: 51 40 14 cd 41 00 00 00 00 00 00
(ada0:ata0:0:0:0): Retrying command

The drive has given me no issues, and it is modern enough to provide statistics as to it's condition. I have also dropped to single-user, and fsck()ed it. No signs of trouble. But all attempts to dump() to the external drive connected via USB, fail with the above error message(s) repeated until I ^C to break the dump() operation.
The only thing I can imagine might be causing dump() to elicit this from the drive, is that it might not consider that /tmp is a mount() point, and not simply a folder/directory. Are my suspicions true? What else might cause this?
FWIW; /etc/fstab

Code:

# relevant section
/dev/ada0s1a	/		ufs	rw	1	1
/dev/ada0s1b	none		swap	sw	0	0
/dev/ada0s1d	/var		ufs	rw	2	2
/dev/ada1s1a	/tmp		ufs	rw	2	2
/dev/ada1s1b	/usr		ufs	rw	2	2

Thank you for all your time, and consideration.

--Chris

wblock@ · Apr 2, 2014

Re: dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB:...

Install sysutils/smartmontools and run smartctl -a /dev/ada0 | less -S. Look for the final column in Reallocated Sector Count and Current Pending Sectors.

Chris_H · Apr 2, 2014

Re: dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB:...

wblock@ said:
Install sysutils/smartmontools and run smartctl -a /dev/ada0 | less -S. Look for the final column in Reallocated Sector Count and Current Pending Sectors.

Thanks, as always, @wblock@.
OK following your suggested smartctl() command returned (portions you requested):

Code:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_
  5 Reallocated_Sector_Ct   0x0033   253   253   063    Pre-fail  Always       -
197 Current_Pending_Sector  0x0008   252   252   000    Old_age   Offline      -

The above table ended with:

Code:

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA
# 1  Short offline       Completed without error       00%         0         -

Selective Self-tests/Logging not supported

I guess I'll read the associated man() pages, and see if I can discover anything of value. But as far as I can tell based on my experience with the drive prior to the use of dump(), all seems good.

Thanks again, for taking the time to respond, @wblock@.

--Chris

Chris_H · Apr 2, 2014

Re: dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB:...

OK. Now I have numbers. Simply issuing smartctl -a returns the following:

Code:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_
  5 Reallocated_Sector_Ct   0x0033   253   253   063    Pre-fail  Always       -1
197 Current_Pending_Sector  0x0008   252   252   000    Old_age   Offline      -1

Actually, it wraps. So it really looks like this:

Code:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_
  5 Reallocated_Sector_Ct   0x0033   253   253   063    Pre-fail  Always       -
      1
197 Current_Pending_Sector  0x0008   252   252   000    Old_age   Offline      -
      1

OK. I redirected the output to this box, so I could read it better.
Here's the entire table, after performing smartctl -a /dev/ada0

Code:

Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  3 Spin_Up_Time            0x0027   234   233   063    Pre-fail  Always       -       4650
  4 Start_Stop_Count        0x0032   253   253   000    Old_age   Always       -       710
  5 Reallocated_Sector_Ct   0x0033   253   253   063    Pre-fail  Always       -       1
  6 Read_Channel_Margin     0x0001   253   253   100    Pre-fail  Offline      -       0
  7 Seek_Error_Rate         0x000a   253   252   000    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0027   252   242   187    Pre-fail  Always       -       44913
  9 Power_On_Minutes        0x0032   235   235   000    Old_age   Always       -       1051h+36m
 10 Spin_Retry_Count        0x002b   253   252   223    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x002b   253   252   223    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   251   251   000    Old_age   Always       -       909
192 Power-Off_Retract_Count 0x0032   253   253   000    Old_age   Always       -       205
193 Load_Cycle_Count        0x0032   253   253   000    Old_age   Always       -       3192
194 Unknown_Attribute       0x0032   253   253   000    Old_age   Always       -       0
195 Hardware_ECC_Recovered  0x000a   253   252   000    Old_age   Always       -       2812
196 Reallocated_Event_Count 0x0008   253   253   000    Old_age   Offline      -       0
197 Current_Pending_Sector  0x0008   252   252   000    Old_age   Offline      -       1
198 Offline_Uncorrectable   0x0008   253   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0008   199   199   000    Old_age   Offline      -       0
200 Multi_Zone_Error_Rate   0x000a   253   252   000    Old_age   Always       -       0
201 Soft_Read_Error_Rate    0x000a   253   240   000    Old_age   Always       -       118
202 Data_Address_Mark_Errs  0x000a   253   250   000    Old_age   Always       -       0
203 Run_Out_Cancel          0x000b   253   252   180    Pre-fail  Always       -       92
204 Soft_ECC_Correction     0x000a   253   252   000    Old_age   Always       -       0
205 Thermal_Asperity_Rate   0x000a   253   252   000    Old_age   Always       -       0
207 Spin_High_Current       0x002a   253   252   000    Old_age   Always       -       0
208 Spin_Buzz               0x002a   253   252   000    Old_age   Always       -       0
209 Offline_Seek_Performnce 0x0024   253   253   000    Old_age   Offline      -       0
 99 Unknown_Attribute       0x0004   253   253   000    Old_age   Offline      -       0
100 Unknown_Attribute       0x0004   253   253   000    Old_age   Offline      -       0
101 Unknown_Attribute       0x0004   253   253   000    Old_age   Offline      -       0

HTH, and Thanks.

--Chris

wblock@ · Apr 2, 2014

Re: dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB:...

Chris_H said:
Actually, it wraps.

Yes, the -S for less(1) allows scrolling without messing up the screen.

That drive has grown a bad block, and it's very young. Possibly the near side of the "bathtub curve". Run the short test: smartctl -tshort /dev/ada0. Monitor with smartctl -a /dev/ada0 | less -S. If that passes, run the long test: smartctl -tlong /dev/ada0.

It may pass both of those, but I'd still replace the drive.

Chris_H · Apr 2, 2014

Re: dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB:...

Well. the results are in for smartctl -tshort /dev/ada0

Code:

Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  3 Spin_Up_Time            0x0027   234   233   063    Pre-fail  Always       -       4650
  4 Start_Stop_Count        0x0032   253   253   000    Old_age   Always       -       710
  5 Reallocated_Sector_Ct   0x0033   253   253   063    Pre-fail  Always       -       1
  6 Read_Channel_Margin     0x0001   253   253   100    Pre-fail  Offline      -       0
  7 Seek_Error_Rate         0x000a   253   252   000    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0027   252   242   187    Pre-fail  Always       -       46471
  9 Power_On_Minutes        0x0032   235   235   000    Old_age   Always       -       1051h+57m
 10 Spin_Retry_Count        0x002b   253   252   223    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x002b   253   252   223    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   251   251   000    Old_age   Always       -       909
192 Power-Off_Retract_Count 0x0032   253   253   000    Old_age   Always       -       205
193 Load_Cycle_Count        0x0032   253   253   000    Old_age   Always       -       3192
194 Unknown_Attribute       0x0032   253   253   000    Old_age   Always       -       0
195 Hardware_ECC_Recovered  0x000a   253   252   000    Old_age   Always       -       2929
196 Reallocated_Event_Count 0x0008   253   253   000    Old_age   Offline      -       0
197 Current_Pending_Sector  0x0008   252   252   000    Old_age   Offline      -       1
198 Offline_Uncorrectable   0x0008   252   252   000    Old_age   Offline      -       1
199 UDMA_CRC_Error_Count    0x0008   199   199   000    Old_age   Offline      -       0
200 Multi_Zone_Error_Rate   0x000a   253   252   000    Old_age   Always       -       0
201 Soft_Read_Error_Rate    0x000a   253   240   000    Old_age   Always       -       141
202 Data_Address_Mark_Errs  0x000a   253   250   000    Old_age   Always       -       0
203 Run_Out_Cancel          0x000b   253   252   180    Pre-fail  Always       -       96
204 Soft_ECC_Correction     0x000a   253   252   000    Old_age   Always       -       0
205 Thermal_Asperity_Rate   0x000a   253   252   000    Old_age   Always       -       0
207 Spin_High_Current       0x002a   253   252   000    Old_age   Always       -       0
208 Spin_Buzz               0x002a   253   252   000    Old_age   Always       -       0
209 Offline_Seek_Performnce 0x0024   253   253   000    Old_age   Offline      -       0
 99 Unknown_Attribute       0x0004   253   253   000    Old_age   Offline      -       0
100 Unknown_Attribute       0x0004   253   253   000    Old_age   Offline      -       0
101 Unknown_Attribute       0x0004   253   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed: read failure       60%      6106         4312340
# 2  Short offline       Completed without error       00%         0         -

Selective Self-tests/Logging not supported

I'll start the long test.

Thanks a million, @wblock@

--Chris

kpa · Apr 2, 2014

Re: dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB:...

The short test is already enough information to replace the disk as soon as possible.

Chris_H · Apr 2, 2014

Re: dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB:...

Greetings,
Well, the long test results are in. Here's the results:

Code:

Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  3 Spin_Up_Time            0x0027   234   233   063    Pre-fail  Always       -       4650
  4 Start_Stop_Count        0x0032   253   253   000    Old_age   Always       -       710
  5 Reallocated_Sector_Ct   0x0033   253   253   063    Pre-fail  Always       -       1
  6 Read_Channel_Margin     0x0001   253   253   100    Pre-fail  Offline      -       0
  7 Seek_Error_Rate         0x000a   253   252   000    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0027   250   242   187    Pre-fail  Always       -       56135
  9 Power_On_Minutes        0x0032   235   235   000    Old_age   Always       -       1052h+08m
 10 Spin_Retry_Count        0x002b   253   252   223    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x002b   253   252   223    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   251   251   000    Old_age   Always       -       909
192 Power-Off_Retract_Count 0x0032   253   253   000    Old_age   Always       -       205
193 Load_Cycle_Count        0x0032   253   253   000    Old_age   Always       -       3192
194 Unknown_Attribute       0x0032   253   253   000    Old_age   Always       -       0
195 Hardware_ECC_Recovered  0x000a   253   252   000    Old_age   Always       -       1280
196 Reallocated_Event_Count 0x0008   253   253   000    Old_age   Offline      -       0
197 Current_Pending_Sector  0x0008   252   252   000    Old_age   Offline      -       1
198 Offline_Uncorrectable   0x0008   252   252   000    Old_age   Offline      -       1
199 UDMA_CRC_Error_Count    0x0008   199   199   000    Old_age   Offline      -       0
200 Multi_Zone_Error_Rate   0x000a   253   252   000    Old_age   Always       -       0
201 Soft_Read_Error_Rate    0x000a   253   240   000    Old_age   Always       -       15
202 Data_Address_Mark_Errs  0x000a   253   250   000    Old_age   Always       -       0
203 Run_Out_Cancel          0x000b   253   252   180    Pre-fail  Always       -       24
204 Soft_ECC_Correction     0x000a   253   252   000    Old_age   Always       -       0
205 Thermal_Asperity_Rate   0x000a   253   252   000    Old_age   Always       -       0
207 Spin_High_Current       0x002a   253   252   000    Old_age   Always       -       0
208 Spin_Buzz               0x002a   253   252   000    Old_age   Always       -       0
209 Offline_Seek_Performnce 0x0024   253   253   000    Old_age   Offline      -       0
 99 Unknown_Attribute       0x0004   253   253   000    Old_age   Offline      -       0
100 Unknown_Attribute       0x0004   253   253   000    Old_age   Offline      -       0
101 Unknown_Attribute       0x0004   253   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       40%      6106         4312340
# 2  Short offline       Completed: read failure       60%      6106         4312340
# 3  Short offline       Completed without error       00%         0         -

Selective Self-tests/Logging not supported

About replacing the drive;
1) I've overclocked the CPU on this board ( 2.7Ghz ==> 3.21Ghz
So in all fairness to the drive, this may be a factor.
2) Wouldn't bother me at all to pitch the drive. Like the MB, it's a spare.
My only concern at the moment, is to get the / dumped to the external USB drive. After which, I'll know to do some more testing of the drive under normal CPU frequency, and then pitch it, if it still fails.
It's my current understanding, that if the "SMART" drive discovers a "bad" block, it re-maps it, and marks it in it's internal table as such. Leaving the drive completely usable, until the internal bad log exceeds the size of it's predefined table.
Shouldn't that leave the drive dump()able at this point?

Thank you both, @kpa, and @wblock@, for all your [continued] help.

--Chris

wblock@ · Apr 2, 2014

Re: dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB:...

Overclocking is not a factor. The drive remapped one bad block on a write, and now it has another bad block that can't be read. Bad blocks can't be remapped on reads, because there is no way to reconstruct the data that was in that block. Use dd(1) with conv=noerror,sync to save ada0s1a to a file or another disk. Then run fsck(8) on that in case the data was lost in the filesystem structure rather than inside a file. Then use dump(8) to dump from the result.

And from now on, use some form of disk redundancy. Current drives are less reliable than previous ones, and some of the previous ones weren't all that solid in the first place.

Chris_H · Apr 2, 2014

Re: dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB:...

wblock@ said:
Overclocking is not a factor. The drive remapped one bad block on a write, and now it has another bad block that can't be read. Bad blocks can't be remapped on reads, because there is no way to reconstruct the data that was in that block. Use dd(1) with conv=noerror,sync to save ada0s1a to a file or another disk. Then run fsck(8) on that in case the data was lost in the filesystem structure rather than inside a file. Then use dump(8) to dump from the result.

Sounds like a plan. Thanks for the advice!

wblock@ said:
And from now on, use some form of disk redundancy. Current drives are less reliable than previous ones, and some of the previous ones weren't all that solid in the first place.

Understood. Given this one had had fairly little use, it seemed a good candidate to set aside for a spare, for just such tasks (utility drive). Oh well. Given the cost of storage these days, I think I might just as well pitch the ones I have lying around, and just purchase a handful of new drives, from brands I know to have long lifespans. Rather than spend the $$ in time testing them all.
Great experience, even tho the future of the drive (and data on it) look grim.

Thank you very much, as always, @wblock@. I really appreciate it.

--Chris

P.S. I don't suppose anyone knows how I might discover which block can't be read. It might be interesting to see if I can see if it is actually occupied.

wblock@ · Apr 2, 2014

Re: dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB:...

Just test the drives like you did with this one. Older drives are often more reliable than the new ones.

Chris_H · Apr 2, 2014

Re: dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB:...

wblock@ said:
Just test the drives like you did with this one. Older drives are often more reliable than the new ones.

True. I might have spoken a bit hastily on that. I know of several brands that seem to last f-o-o-r-r-e-v-e-r.
Fairly trivial to hook em up to a board that has FreeBSD && smarttools installed.
On a happy note; I was able to track down the offending block. It was from the www/firefox install.
Specifically; places.sqlite. Any attempt to copy, or otherwise read the space it occupies, fires off the errors I posted in the OP. I'm going to attempt to delete it now, then see if a dump() then becomes possible.
Fingers crossed.

Thanks @wblock@

--Chris

Chris_H · Apr 2, 2014

Re: dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB:...

WOOHOO!
Deleting the file places.sqlite fixed it. I then had no problems dump()ing the / to the external USB drive. Whew! Happy ending, after all. But the drive will be removed, and disposed of. Now that I know it's status.

Thanks again! I really appreciate all your help!

--Chris

ralphbsz · Apr 3, 2014

Re: [SOLVED] dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB

There are two kinds of computer users. Those that use RAID, and those that have not lost data yet.

To be honest, I sort of made that up, from several other sayings. But I hope you (and everyone who reads this) learned a lesson here: You got within a hair's width of losing all your file system. You were lucky that the only error was in one file, which is not of great importance. Next time around, you might be less lucky. Redundant disks are one part of the prevention of data loss.

Chris_H · Apr 3, 2014

Re: [SOLVED] dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB

ralphbsz said:
There are two kinds of computer users. Those that use RAID, and those that have not lost data yet.

To be honest, I sort of made that up, from several other sayings. But I hope you (and everyone who reads this) learned a lesson here: You got within a hair's width of losing all your file system. You were lucky that the only error was in one file, which is not of great importance. Next time around, you might be less lucky. Redundant disks are one part of the prevention of data loss.

Greetings, @ralphbsz, and thank you for your reply.
On the whole, I might be inclined to agree. But, in this case it was a development/evaluation box, as first mentioned in the OP.
So in the end; while I'd have been fairly disappointed for having lost what turned out to be a good set of choices, that I would have liked to save. I knew, and expected nothing but a better acquaintance with the releng_9 branch. That was my only intention, when I set out to perform the "EVAL" install. This hardware, is simply a bunch of "spares" I have lying around. I'm surprised, when some of it actually works.

It's just stuff I've removed from other systems -- my own, or contract work.
Speaking of RAID. The MB this (eval) install runs on happens to have hardware RAID on it. I haven't even (yet) tested it with FreeBSD. Testing would be mandidory, before even considering it's use on a "production" box. No? But what if the test results are less than optimal, or a complete fail? There's always Software raid, and FreeBSD is happy to accommodate there, as well. But frankly; most, these days, choose Sun/opensolaris' zfs(). It's come a l-o-o-n-g way, since it was first introduced, and not the least of which, because of all the hard work that FreeBSD development has put into it.

So in the end, if "redundancy" is your chosen solution to data integrity, there are many options to choose from, and each solution has it's strong points. The choice should be based on use cases. Which isn't the same from one, to another.
Speaking of data integrity, and keeping ones data "safe, and sound"; did you happen to notice what triggered the disk error? As it happens, it was while performing a Backup (dump()). This too, is a good solution for keeping ones data safe. No?

In this (my) particular case; the install, and configuration was less than 24hrs old, and I was performing a backup via dump(). I can't think of a more stringent policy for data integrity than that -- with the possible exception of a busy production server. In which case, backups might be performed twice, to four times daily. So I think I was probably doing The Right Thing(tm).

But on the whole. I think your intentions are correct; no one can really be too careful, if they really value their data. How one chooses to safeguard that data, is probably best determined by "use case" -- RAID being one of those options.

Thanks again, @ralphbsz, for taking the time to reply.

--Chris

wblock@ · Apr 3, 2014

Re: [SOLVED] dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB

Motherboard RAID is generally software RAID. FreeBSD supports that: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/geom-graid.html. I set that up and used it for the first time ever to write that section of the Handbook. Having done that, I prefer either gmirror(8) or ZFS. They are easier to use, more cross-platform compatible, and probably less buggy.

Chris_H · Apr 3, 2014

Re: [SOLVED] dump(8) causes (ada0:ata0:0:0:0): READ_DMA. ACB

wblock@ said:
Motherboard RAID is generally software RAID. FreeBSD supports that: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/geom-graid.html. I set that up and used it for the first time ever to write that section of the Handbook. Having done that, I prefer either gmirror(8) or ZFS. They are easier to use, more cross-platform compatible, and probably less buggy.

REALLY appreciate you taking the time to share that, @wblock@.
It confirms the impression I had already gathered from reading other' experience(s) with the different "raid" options available on/with FreeBSD. My own experiences have all been with gmirror(). I loved it the first time I used it. It just seemed to Do The Right Thing(tm). So I never looked back. But sometimes wondered if at least trying other options might not be a bad idea. At least for the experience.

Really appreciate your chiming in, on this, @wblock@. Thanks.

--Chris

P.S. Did you know FreeBSD will run on a DEAD drive? I think the following is a real testament to FreeBSD' resiliency:

Code:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  3 Spin_Up_Time            0x0027   234   234   063    Pre-fail  Always       -       3897
  4 Start_Stop_Count        0x0032   253   253   000    Old_age   Always       -       351
  5 Reallocated_Sector_Ct   0x0033   001   001   063    Pre-fail  Always   FAILING_NOW 1276
  6 Read_Channel_Margin     0x0001   253   253   100    Pre-fail  Offline      -       0
  7 Seek_Error_Rate         0x000a   253   252   000    Old_age   Always       -       0
  8 Seek_Time_Performance   0x0027   253   237   187    Pre-fail  Always       -       38778
  9 Power_On_Minutes        0x0032   170   170   000    Old_age   Always       -       366h+17m
 10 Spin_Retry_Count        0x002b   253   252   157    Pre-fail  Always       -       0
 11 Calibration_Retry_Count 0x002b   253   248   223    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   252   252   000    Old_age   Always       -       593
192 Power-Off_Retract_Count 0x0032   253   253   000    Old_age   Always       -       211
193 Load_Cycle_Count        0x0032   253   253   000    Old_age   Always       -       1867
194 Temperature_Celsius     0x0032   253   253   000    Old_age   Always       -       39
195 Hardware_ECC_Recovered  0x000a   249   246   000    Old_age   Always       -       59948
196 Reallocated_Event_Count 0x0008   001   001   000    Old_age   Offline      -       811
197 Current_Pending_Sector  0x0008   001   001   000    Old_age   Offline      -       715
198 Offline_Uncorrectable   0x0008   001   001   000    Old_age   Offline      -       6077
199 UDMA_CRC_Error_Count    0x0008   199   199   000    Old_age   Offline      -       0
200 Multi_Zone_Error_Rate   0x000a   253   252   000    Old_age   Always       -       0
201 Soft_Read_Error_Rate    0x000a   253   245   000    Old_age   Always       -       2
202 Data_Address_Mark_Errs  0x000a   001   001   000    Old_age   Always       -       61947
203 Run_Out_Cancel          0x000b   196   149   180    Pre-fail  Always   In_the_past 47456
204 Soft_ECC_Correction     0x000a   253   077   000    Old_age   Always       -       0
205 Thermal_Asperity_Rate   0x000a   253   228   000    Old_age   Always       -       0
207 Spin_High_Current       0x002a   253   252   000    Old_age   Always       -       0
208 Spin_Buzz               0x002a   253   252   000    Old_age   Always       -       0
209 Offline_Seek_Performnce 0x0024   164   156   000    Old_age   Offline      -       0
 99 Unknown_Attribute       0x0004   253   253   000    Old_age   Offline      -       0
100 Unknown_Attribute       0x0004   253   253   000    Old_age   Offline      -       0
101 Unknown_Attribute       0x0004   253   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
Warning: ATA error count 953 inconsistent with error log pointer 5

ATA Error Count: 953 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 953 occurred at disk power-on lifetime: 26966 hours (1123 days + 14 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 5e bf c5 29 e0  Error: UNC 94 sectors at LBA = 0x0029c5bf = 2737599

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 80 bf c5 29 e0 00      04:53:57.840  READ DMA
  c8 00 80 bf c5 29 e0 00      04:53:56.096  READ DMA
  c8 00 80 bf c5 29 e0 00      04:53:54.368  READ DMA
  c8 00 80 bf c5 29 e0 00      04:53:52.624  READ DMA
  c8 00 80 bf c5 29 e0 00      04:53:50.896  READ DMA

Error 952 occurred at disk power-on lifetime: 26966 hours (1123 days + 14 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 5e bf c5 29 e0  Error: UNC 94 sectors at LBA = 0x0029c5bf = 2737599

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 80 bf c5 29 e0 00      04:53:56.096  READ DMA
  c8 00 80 bf c5 29 e0 00      04:53:54.368  READ DMA
  c8 00 80 bf c5 29 e0 00      04:53:52.624  READ DMA
  c8 00 80 bf c5 29 e0 00      04:53:50.896  READ DMA
  ca 00 40 bf d5 81 e1 00      04:53:50.176  WRITE DMA

Error 951 occurred at disk power-on lifetime: 26966 hours (1123 days + 14 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 5e bf c5 29 e0  Error: UNC 94 sectors at LBA = 0x0029c5bf = 2737599

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 80 bf c5 29 e0 00      04:53:54.368  READ DMA
  c8 00 80 bf c5 29 e0 00      04:53:52.624  READ DMA
  c8 00 80 bf c5 29 e0 00      04:53:50.896  READ DMA
  ca 00 40 bf d5 81 e1 00      04:53:50.176  WRITE DMA
  ca 00 40 ff d4 81 e1 00      04:53:50.176  WRITE DMA

Error 950 occurred at disk power-on lifetime: 26966 hours (1123 days + 14 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 5e bf c5 29 e0  Error: UNC 94 sectors at LBA = 0x0029c5bf = 2737599

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 80 bf c5 29 e0 00      04:53:52.624  READ DMA
  c8 00 80 bf c5 29 e0 00      04:53:50.896  READ DMA
  ca 00 40 bf d5 81 e1 00      04:53:50.176  WRITE DMA
  ca 00 40 ff d4 81 e1 00      04:53:50.176  WRITE DMA
  ca 00 40 bf f6 88 e0 00      04:53:50.176  WRITE DMA

Error 949 occurred at disk power-on lifetime: 26966 hours (1123 days + 14 hours)
  When the command that caused the error occurred, the device was in an unknown state.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 5e bf c5 29 e0  Error: UNC 94 sectors at LBA = 0x0029c5bf = 2737599

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 80 bf c5 29 e0 00      04:53:50.896  READ DMA
  ca 00 40 bf d5 81 e1 00      04:53:50.176  WRITE DMA
  ca 00 40 ff d4 81 e1 00      04:53:50.176  WRITE DMA
  ca 00 40 bf f6 88 e0 00      04:53:50.176  WRITE DMA
  ca 00 40 ff 65 75 e0 00      04:53:50.176  WRITE DMA

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short captive       Completed without error       00%         8         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I know; a complete waste of time. Why didn't I first check the drive, before chopping up the drive, and restoring the dump() to it? I have no excuse. I just did it. However, it was sure nice to discover how well FreeBSD works on "clinically" dead hardware. So, in the end; I'm glad I did it.

In fact, it even ran X! Sure, the system was spewing ada0 related errors, as fast as the buffer would permit. But still. I had no trouble running X, and everything I had installed (once I got past the error messages)

.
I have since replaced that drive, and restored from dump(). I writing this from it now!

Just thought I'd mention it.