zfs pool "vnode_pager_getpages: I/O read error" after geom_mbr

Beeblebrox

Beeblebrox

Well, this is certainly an unpleasant surprise...
My disk uses zfs for /var, /home + /usr, the rest is on root. After reboot, the kernel crashed with this message after having mounted /

Code: 
vnode_pager_getpages: I/O read error
vm_fault: pager read error, pid 26 (zfs)
pid 26 (zfs), uid 0: exited on signal 11
segmentation fault

I had made 2 changes to my system before re-booting:
1. added geom_mbr_load="YES" to loader.conf
2. ran portupgrade -a. Portupgrade had been giving me some problems with several ports failing to build so I just installed the package version for those with
[cmd=]portupgrade -PP pkg_name[/cmd]
I seriously doubt this is the source of the problem though.

I booted into safe + single user mode and got normal output for the following commands:

disklabel -r /dev/ada0s1
zfs list OR zpool scrub tank0

BUT I get this output:

Code: 
vnode_pager_getpages: I/O read error
Device not configured

when I run any of the following: fdisk OR mount -a OR
zpool status -v tank0 OR zpool iostat -v tank0

I also tried the bootloader command line (option 8? from the boot screen) and toggle-module geom_mbr as well as the command to unload the geom_mbr module.

When I type exit at this point I get the
Code: 
vnode_pager_getpages I/O read error
flashing endlessley on the screen; so this also led to nowhere.

A strange observation is that the main slices are now listed twice after geom_mbr was loaded - ie: two listings for ada0s1, two entries for ada0s2 etc.

I'm reasonably sure that the geom_mbr module has hosed the my zfs mapping somehow, but I do not know how to fix that and I would also like to be absolutely sure that this is indeed the problem before I attempt any surgery.
 
Update:
I booted a live cd and edited loader.conf to block out geom_mbr_load="YES". The system now boots as before (whew). Obviously the toggleing command for geom_mbr somehow was not transferring to the boot process.

Now to figure out why geom_mbr places duplicate entries into geom map thus confusing zfs mapping. I need this so I can access the logical partitions inside the extended partition.
 
Today FreeBSD 8.x and FreeBSD 9.0 uses GEOM class PART for doing partitioning. GEOM classes MBR, BSD, GPT and PC98 have been retired. When you loading some of them then it tries to do the same things that GEOM_PART does and they conflict with GEOM_PART. Do not do it.
 
I'm trying to access the logical partitions inside an extended partition, but # bsdlabel does not list (so therefore does not recognise) the slices inside the dos extended partition. How do I solve that?
 
# uname -a
Code: 
FreeBSD bsdsrv.xyz 8.1-RC1 FreeBSD 8.1-RC1 #2: Mon Jun 14 18:21:27 UTC 2010     root@build8x64.pcbsd.org:/usr/obj/storage/builds/pcbsd-
build81-x64/fbsd-source/8.1/sys/PCBSD  amd64
# gpart show
Code: 
63  625142385  ada0  MBR  (298G)
         63  167782545     3  freebsd  [active]  (80G)
  167782608        252        - free -  (126K)
  167782860  361703475     1  !5  (172G)
  529486335   94365810        - free -  (45G)
  623852145    1285200     2  !131  (628M)
  625137345       5103        - free -  (2.5M)

=>        0  167782545  ada0s3  BSD  (80G)
          0    2662400       1  freebsd-ufs  (1.3G)
    2662400    4194304       2  freebsd-swap  (2.0G)
    6856704  160915456       4  freebsd-ufs  (77G)
  167772160      10385          - free -  (5.1M)
ada0s3d is a label dedicated to the zfs pool. s2 is a very small boot partition for grub at the very end of the disk. s1 is the extended dos partition with the logical labels inside.
 
1. Can you make a copy of first sector of ada0s1 partition and put them somewhere?
Code: 
# dd if=/dev/ada0s1 of=./ebr bs=512 count=1
Or show hexdump here?
Code: 
# hexdump -Cv ./ebr
2. I think it is known problem and it is already fixed in 8.1-RELEASE. Just try to update your kernel.
 
Thanks for your reply Butcher.
1. Hexdump is:
Code: 
00000000  13 17 d7 ca b4 9c 71 66  17 b3 23 59 d7 8f e9 b4  |......qf..#Y....|
00000010  8c f1 ed a9 7c e6 b3 a2  32 33 c0 cc f4 98 01 08  |....|...23......|
00000020  21 d0 db 34 99 c2 a8 6b  62 f3 8c 56 49 ea 1a 95  |!..4...kb..VI...|
00000030  3a db 79 a1 0d b3 68 fb  70 22 c1 a8 37 bb 80 ee  |:.y...h.p"..7...|
00000040  b7 49 2e e7 8e cd c9 1c  97 b5 0a b9 d0 ea 18 10  |.I..............|
00000050  be 31 1c 1e 94 54 ba 8c  f1 f4 63 7b 14 1d 93 98  |.1...T....c{....|
00000060  99 d4 69 2e 69 42 7b 46  8d ba be 07 1f 44 86 46  |..i.iB{F.....D.F|
00000070  16 6c c0 57 29 5d 14 a3  a8 74 2f 47 6b 18 df 2a  |.l.W)]...t/Gk..*|
00000080  6b 59 4b 5d 86 01 98 f3  5b 26 87 60 c4 cc 9e a9  |kYK]....[&.`....|
00000090  6d 11 45 58 e8 22 72 64  19 c4 64 09 b4 15 ec 92  |m.EX."rd..d.....|
000000a0  9c 9c c7 25 38 d9 29 4f  09 a5 b7 f9 00 15 ff 70  |...%8.)O.......p|
000000b0  06 33 98 c2 fc 6c 95 08  1c 0e d0 5f 61 e3 6c e3  |.3...l....._a.l.|
000000c0  65 68 75 1f fa 82 a6 83  cf 16 d4 00 54 97 ed b8  |ehu.........T...|
000000d0  9c 1e 26 67 44 e4 59 0e  1d 9c e0 49 d0 58 19 0c  |..&gD.Y....I.X..|
000000e0  84 c3 03 19 ec 80 e0 38  c6 f1 74 97 03 91 11 b2  |.......8..t.....|
000000f0  7b 4d 6b d2 2e 89 1e 72  82 55 f3 2c af 45 86 c6  |{Mk....r.U.,.E..|
00000100  41 88 23 4a 3f fc 95 bd  1e 57 a5 ae e3 2b f2 19  |A.#J?....W...+..|
00000110  3a 41 0b 56 f8 d2 f3 ee  3b ae 87 2c 43 7a c5 8e  |:A.V....;..,Cz..|
00000120  93 41 1b 4e 60 66 2f a3  6d 53 ac 69 4d e3 26 37  |.A.N`f/.mS.iM.&7|
00000130  ba 69 51 3d d1 fc 22 7d  de 40 54 45 99 b1 c2 f5  |.iQ=.."}.@TE....|
00000140  43 ea 9f f3 07 5b af 9c  a9 d4 be 7c e9 0a 77 00  |C....[.....|..w.|
00000150  9f bd 81 60 c4 7b 5b f1  44 84 fc d8 46 77 17 1f  |...`.{[.D...Fw..|
00000160  ca ad 44 25 aa d0 82 77  04 9c 01 09 08 85 0c e8  |..D%...w........|
00000170  40 70 83 47 c0 52 21 80  eb 08 7f f4 35 52 cf bd  |@p.G.R!.....5R..|
00000180  5c 27 60 35 91 bf 40 22  55 7b 01 1a d4 90 b5 79  |\'`5..@"U{.....y|
00000190  a9 85 25 60 50 d9 bd ba  4e 65 95 b3 2c f2 f7 96  |..%`P...Ne..,...|
000001a0  5c 0b 3e cc 1a f4 b2 b0  05 87 14 e1 0b 5e ff f3  |\.>..........^..|
000001b0  f1 41 0b 8a d9 75 99 8b  0e cf 5e 09 15 6b 00 fe  |.A...u....^..k..|
000001c0  ff ff 83 fe ff ff 3f 00  00 00 a7 14 00 05 00 fe  |......?.........|
000001d0  ff ff 05 fe ff ff e6 14  00 05 e6 14 00 05 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

2. My kernel already is 8.1 as [cmd=]uname -a[/cmd] shows?
 
It is interessting. I know how to fix this, but can you say what tool did you use to create this extended partition?
 
1. All partns were created with gparted's gui in linux
2. I don't feel comfortable enough yet to get into K compile - maybe after a couple of months. I realized my ver is RC, so I think I'm just going to go ahead and clean install the RELEASE since I am having a lot of difficulty with the portupgrade process anyway. I assume the patch is included in the release ver?
 
No. This patch is not included nor in 8-STABLE, nor in 9.0-CURRENT. And it will not be included until i do not receive successfull testing results. So, it depends on your decision. :)
 
Are you saying that the ONLY way to access extended / logical partitions under freebsd is through the patch you have written? I am willing to be adventurous but I am not an expert with BSD so if there is a safer way maybe I should try that.

Also, is this patch is for the K compile process? I have very little knowledge about what to do with the patch, so is there a howto page or such?

About the hdd structure: I have a second hdd as spare and I have partitioned the disk by using the mhdd utility. I plan to migrate my system to this second hdd. Enclosed is the hexdump; is this better, or will this cause problems also?
Code: 
00000000  0e 1f eb 1c 5e fc b4 0e  b7 00 ac cd 10 38 3c 75  |....^........8<u|
00000010  f9 b4 00 cd 16 b8 0d 0e  cd 10 b0 0a cd 10 cd 19  |................|
00000020  e8 e1 ff 0d 0a 45 78 74  65 6e 64 65 64 20 70 61  |.....Extended pa|
00000030  72 74 69 74 69 6f 6e 20  69 73 20 6e 6f 74 20 62  |rtition is not b|
00000040  6f 6f 74 61 62 6c 65 2e  0d 0a 48 69 74 20 61 20  |ootable...Hit a |
00000050  6b 65 79 20 74 6f 20 72  65 62 6f 6f 74 2e 2e 2e  |key to reboot...|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000100  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000120  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000130  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000140  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000150  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000170  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000180  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001c0  ff ff 83 00 ff ff 3f 00  00 00 c9 7f 3e 00 00 f7  |......?.....>...|
000001d0  41 fd 05 fe ff ff 08 80  3e 00 fa 7f a9 03 00 00  |A.......>.......|
000001e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

Separate but related thread, should you have any ideas:
http://forums.freebsd.org/showthread.php?p=111880#post111880
 
Beeblebrox said:
Are you saying that the ONLY way to access extended / logical partitions under freebsd is through the patch you have written? I am willing to be adventurous but I am not an expert with BSD so if there is a safer way maybe I should try that.

Also, is this patch is for the K compile process? I have very little knowledge about what to do with the patch, so is there a howto page or such?
Click to expand...

No. Usually FreeBSD correctly detects Extended partitions and works with them well. But FreeBSD's GEOM class GEOM_PART_EBR assumes that Extended Boot Record does not contain any of boot code and each EBR record should have only 2 records in partition table. But in your hexdump you can see that EBR is not what GEOM_PART_EBR wants. My patch removes these restrictions and only prints out that EBR is not correct.

FreeBSD head/ and stable/8 has new ability to recover corrupt partition tables. My goal is to check would GEOM_PART_EBR work with this patch or not. If it would work, then i will remove these restrictions and add some code to recover EBR and make GEOM_PART_EBR happy.

How to install patch. If you have installed source code and never updated it, then you should do:
Code: 
# cd /usr/src
# patch < /path/to/downloaded/g_part_ebr.c.diff
# make buildkernel installkernel
In other cases it depends on what you have in you src directory.
 
I see what you are saying (as much as my current knowledge allows) and sort of remember that when I was setting up my linux system, I may have specified a bootable extended partition.

Unfortunately for your suggestion, I REALLY do not want to compile a kernel just yet. I would much prefer to just go with a clean install. My previous (second) ebr hexdump was apparently for the wrong partition and I have corrected the output to show the status of the extended partition on the second hdd which I had previously mentioned. As far as I can tell, this extended partn has no problems?

I really thank you for your help and diagnostic and would just like your confirmation that Geom will play well with the extended partn on the second disk with the current structure before I start the clean install on this disk.

Thanks & regards.
 
It seems that it is correct. Also i was wrong about first hexdump, the problem is only in bootcode, and i think part of proposed patch is not needed.
 
You must log in or register to reply here.
Back
Top