Software RAID problem

F

FreeDomBSD

I could use some help in figuring out what steps I need to take to solve a problem:
I want to install FreeBSD9 with a softraid mirrored option, but I want to make sure that I do not loose the information of the existing two drives that were mirrored via softraid while adding two more drives to the softraid array.

1. about four years ago my friend set up a FreeBSD7 box with software raid that mirrored 2 drives.
2. OS was on the separate drive
3. There were three disks total (1 OS, 2 identical for mirrored softRAID)
4. OS drive died two years ago and the system was just sitting there doing nothing
5. at some point it got an upgrade of two more disks that I couldn't figure out how to add to the RAID
6. so there are total of 5 disks now (1 new OS drive; 2 old softraid mirrored drives; 2 new drives that need to be a part of the softraid array)

`
 
I assume your friend used gmirror to set up the mirror? If you add two more drives to a mirror, you will have four identical copies, but no extra space. You want to use ZFS on your new machine. Look at the difference between RAID1 and RAID5. So, you will want to install FreeBSD, set up a ZFS raid-z array with the new drives, then access the mirrored drives and copy the data to the new zfs array. Then, once you have copied all your data from the old array to the new, then you can wipe the old drives and add them to the new array to increase the total available storage.
 
@dave

Yeah, but how would you boot?

@FreeDomBSD

My suggestion would be this: Install FreeBSD onto the new system drive, create a zfs mirror with the two new data drives, have both the old gmirror and your new zfs mirror mounted and copy the data over from the old to the new. Then you can destroy the old mirror and then add them as another mirror-set to your zpool. You then wind up with a RAID10, and also the combined storage of both your old mirror, and the new. Next time you´re tight on storage, you either buy two more drives and add them as yet another mirror-set into the pool, or replace the drives from the old mirror with two new ones.

/Sebulon
 
I stand corrected. Apparently, you can boot from a raidz:
http://forums.freebsd.org/showthread.php?t=29900

@FreeDomBSD

If the new system drive and the other two data drives are of the same size, do what dave says. You will then have more storage, better security- because the boot system will also be protected, and you get the performance from a striped RAID5 + RAID1.

/Sebulon
 
Well, my main concern is the integrity of the data on the old mirror (let's give these drives a name, shall we? How about drive B and drive C and they make up the BC mirror).

Anyway, I have no idea how was the softraid setup. And I never tried to mount the disks separately out of the BSD box for the fear of destroying the data. My friend also doesn't remember how the softraid was setup. Is it possible to destroy the data on the BC mirror if I assume that the mirror was setup using the gmirror command, but in reality it was setup with some other command?

I have 5 drives total:

OS drive - drive "A"
BC mirror - drives "B" and "C"
New drive 1 - drive "D"
New drive 2 - drive "E"

I plan to boot from drive A.

Thanks for all you responses.
 
Sebulon said:
@dave
Yeah, but how would you boot?
Click to expand...

You are installing FreeBSD onto, and booting from your "1 new OS drive". Leave all other drives unplugged until you have installed FreeBSD on that first disk.

But even before you do that - do you know what kind of mirror the old drives are running? You better figure that out. Boot the 9 livecd and see if you can access them by starting gmirror.
 
Excellent idea! I'll post results tonight. Although I'm not sure how a live CD would know my mirror configuration that is saved as a setting on the OS drive.
 
If the drives are indeed mirrored using gmirror, then there is metadata on the drives themselves that describes the mirror. Thus, once you boot the livecd with them connected, you should be able to

Code: 
# gmirror load

and then

Code: 
gmirror status

should show you something like

Code: 
     Name    Status  Components
mirror/gm0  COMPLETE  da0
                      da1

If you don't get similar results, then your softraid may be some other type.
 
@dave

You are installing FreeBSD onto, and booting from your "1 new OS drive". Leave all other drives unplugged until you have installed FreeBSD on that first disk.
Click to expand...
That math is limping. Drive A = OS, BC = old mirror and DE = new mirror. You need minimum 3x drives to create raidz.

@FreeDomBSD

By coincidence, I´ve been writing a HOWTO today on installing 9.0-RELEASE with ZFS boot- and root that you can follow to get your new system up and running. This setup is assuming the disks B,C,D and E are of the same size, and if you follow this guide you will be booting off of a mirrored usb zpool, while having the rest on a raidz zpool made up of disks B,C,D and E that is expandable as much as you like.


DISCLAIMER

... nuff´ said, you know the drill. I will show you a path. You decide if you want to walk it.

Start by inserting both USB´s and having the old BC mirror and DE drives connected and boot up using a FreeBSD-9.0-RELEASE CD.

Code: 
Welcome to FreeBSD! Would you
like to begin an installation
of use the live CD?

< Install >

Code: 
Would you like to set a
non-default key mapping
for your keyboard

< Yes >

- Swedish ISO-8859-1

< OK >
(You´ll want to choose your own mapping here.)

Code: 
Please choose a hostname for this machine.

If you are running on a managed network, please
ask your network administrator for an appropriate
name.

- foobar

< OK >

Code: 
Choose optional system componens to install:

[ * ]  doc
[   ]  games
[   ]  lib32
[   ]  ports
[   ]  src

< OK >

Code: 
Would you like to use the guided
partitioning tool (recommended for
beginners) or to set up partitions
manually (experts)? You can also open a
shell and set up partitions entirely by
hand.

< Shell >

Use this shell to set up partitions for the new system. When finished, mount the
system at /mnt and place an fstab file for the new system at /tmp/bsdinstall_et
c/fstab. Then type 'exit'. You can also enter the partition editor at any time b
y entering 'bsdinstall partedit'.
Click to expand...
(I´m going to assume that your drives are going to be called adaX from here on.)

THESE INSTRUCTIONS HAVE BEEN UPDATED. READ LATER POST FOR IN-DETAIL INSTRUCTIONS.

Code: 
Please select a password for the system management account (root):
Changing local password for root
New Password: ************
Retype New Password: ************

Code: 
Please select a network interface to configure:

em0		Intel® PRO/1000 Legacy Network Connection 1.0.3

< OK >
(Your card might not be an em specifically)

Code: 
Would you like to
configure IPv4 for this
interface?

< Yes >

Code: 
Would you like to use
DHCP to configure this
interface?

< No >
(You may of course choose DHCP if that is available)

Code: 
Static Network Interface Configuration

IP Address	XXX.XXX.XXX.XXX
Subnet Mask	XXX.XXX.XXX.XXX
Deafult Router	XXX.XXX.XXX.XXX

Code: 
Would you like to
configure IPv6 for this
interface?

< No >

Code: 
Resolver Configuration

Search		foo.bar
IPv4 DNS #1	XXX.XXX.XXX.XXX
IPv4 DNS #2	XXX.XXX.XXX.XXX

< OK >
("Search" is if you have entered domain name you can e.g. type "ping host" and resolver auto-adds foo.bar after "host" for you)

Code: 
Is this machine´s CMOS clock set to UTC? If it is set to local time,
or you don´t know, please choose NO here!

< No >
- 8 Europe
-- 45 Sweden

Code: 
Does the abbreviation 'CET' look reasonable?

< Yes >
(Correct as appropriate)

Code: 
Choose the services you would like to be started at
boot:

[ * ]  sshd
[ * ]  moused
[ * ]  ntpd
[ * ]  powerd

< OK >
(As you see fit)

Code: 
Would you like to enable crash dumps?
If you start having problems with the
system it can help the FreeBSD
developers debug the problem. But the
crash dumps can take up a lot of disk
space in /var.

< No >
(Not possible with ZFS)

Code: 
Would you like to add
users to the installed
system now?

< Yes >
(It´s bad practice doing everything as root)

Code: 
Exit

< OK >

Done!

Some things to get you started. Personally, I´ve read it all like, a gazillion times. Things like these are good to have as a reference:

First off, from the good book
Then the wiki
And ultimately, the admin guide. Keep it under your pillow=)

/Sebulon
 
Sebulon said:
@dave
That math is limping. Drive A = OS, BC = old mirror and DE = new mirror. You need minimum 3x drives to create raidz.
Click to expand...

Actually, it is possible to create a raidz array with only two drives, no problem. I have done it. Not ideal, but it is only for the time it takes to migrate the data. Then, add the old drives, and simply export and import the pool.
 
dave said:
Actually, it is possible to create a raidz array with only two drives, no problem. I have done it. Not ideal, but it is only for the time it takes to migrate the data.
Click to expand...

Yes, it is possible. However due to way ZFS is made there are obvious reasons why you shouldn´t:
http://constantin.glez.de/blog/2010/06/closer-look-zfs-vdevs-and-performance#mirroredperformance

Regarding write performance;

When writing to mirrored vdevs, ZFS will write in parallel to all of the mirror disks at once and return from the write operation when all of the individual disks have finished writing. This means that for writes, each mirrored vdev will have the same performance as its slowest disk.
Click to expand...

When writing to RAID-Z vdevs, each filesystem block is split up into its own stripe across (potentially) all devices of the RAID-Z vdev. This means that each write I/O will have to wait until all disks in the RAID-Z vdev are finished writing.
Click to expand...

(So for write performance, two drives mirror or raidz are about equal, BUT.)

Regarding read perfomance;

When reading from mirrored vdevs, ZFS will read blocks off the mirror's individual disks in a round-robin fashion, thereby increasing both IOPS and bandwidth performance: You'll get the combined aggregate IOPS and bandwidth performance of all disks.
Click to expand...

When reading from RAID-Z vdevs ... the process is essentially reversed (no round robin shortcut like in the mirroring case)
Click to expand...

That means that with two drives, a mirror vdev gives you the read performance from two drives, while a raidz vdev gives you the read performance of just one.


dave said:
Then, add the old drives, and simply export and import the pool.
Click to expand...
This statement is very fuzzy. It gives the impression that it is possible to grow a raidz vdev, which is false. By chance, I found a mail from Mr. Jeff Bonwick, one of the original developers of ZFS, explaining it very well:
http://mail.opensolaris.org/pipermail/zfs-discuss/2006-July/003544.html
ZFS uses dynamic striping. So rather than growing an existing RAID-Z
group, you just add another one. That is, suppose you create the pool
like this:

zpool create tank raidz disk1 disk2 disk3

To add more space, you'd say:

zpool add tank raidz disk4 disk5 disk6

The down side is that you can't add just one disk -- you have to add them
in small groups. The upside is that by doing it this way, there's no
need for block remapping -- so adding space isn't a violent act.
Click to expand...

This was written in 2006 but is still true today. To achieve growing or reordering, ZFS needs to have a feature called "Block Pointer Rewrite (BPR)" and I spent some time at the goog', trying to find anything official about it to no avail, unfortunately. Just a lot of people wishing for it;)

Perhaps I misunderstood you, dave. And in that case, I would like you to explain step by step what you ment. It is very important when working with ZFS to keep redundancy or you will end up like this:
Zpool - Incorrecly added a drive to my raidz configuration


I designed my HOWTO for OP after: 1) redunancy, 2) stability, 3) storage and 4) performance, in that order. In a perfect scenario the pool would have a spare disk (RAID5 + RAID1 +1) and also a separate secondary system to make backups onto. Redundancy is not a backup.

/Sebulon
 
Sebulon said:
Yes, it is possible. However due to way ZFS is made there are obvious reasons why you shouldn´t:
http://constantin.glez.de/blog/2010/06/closer-look-zfs-vdevs-and-performance#mirroredperformance

Regarding write performance;

(So for write performance, two drives mirror or raidz are about equal, BUT.)

Regarding read perfomance;

That means that with two drives, a mirror vdev gives you the read performance from two drives, while a raidz vdev gives you the read performance of just one.
Click to expand...

I was only suggesting it as a temporary setup, in order to migrate the data from the old drives. Read what I wrote: "Not ideal, but it is only for the time it takes to migrate the data."

Sebulon said:
This statement is very fuzzy. It gives the impression that it is possible to grow a raidz vdev, which is false. By chance, I found a mail from Mr. Jeff Bonwick, one of the original developers of ZFS, explaining it very well:
http://mail.opensolaris.org/pipermail/zfs-discuss/2006-July/003544.html


This was written in 2006 but is still true today. To achieve growing or reordering, ZFS needs to have a feature called "Block Pointer Rewrite (BPR)" and I spent some time at the goog', trying to find anything official about it to no avail, unfortunately. Just a lot of people wishing for it;)

Perhaps I misunderstood you, dave. And in that case, I would like you to explain step by step what you ment. It is very important when working with ZFS to keep redundancy or you will end up like this:
Zpool - Incorrecly added a drive to my raidz configuration


I designed my HOWTO for OP after: 1) redunancy, 2) stability, 3) storage and 4) performance, in that order. In a perfect scenario the pool would have a spare disk (RAID5 + RAID1 +1) and also a separate secondary system to make backups onto. Redundancy is not a backup.

/Sebulon
Click to expand...

OK, but I have done this succesfully:

Code: 
zfs add [poolname] [new device(s)]
zfs export [poolname]
zfs import [poolname]
zfs list  <-- will show expanded capacity

I used physically identical drives to do that.

Setting up 2 zfs mirrors would be fine, too.

Your howto is great, but I think it is perhaps too complicated for the OP. He has a dedicated disk for the OS, and doesn't need to have root on ZFS. Remember, the OP is not an expert.
 
dave said:
I was only suggesting it as a temporary setup, in order to migrate the data from the old drives. Read what I wrote: "Not ideal, but it is only for the time it takes to migrate the data."
Click to expand...

You never choose two drives in a raidz over two drives in a mirror, unless you actually want bad performance. In which case, knock yourself out:)

dave said:
zfs add [poolname] [new device(s)]
zfs export [poolname]
zfs import [poolname]
zfs list <-- will show expanded capacity
Click to expand...
This worries me that you, just like our fellow from thread Zpool - Incorrecly added a drive to my raidz configuration, has added devices into a pool without any redundancy(fault tolerance). If any of the added drives dies, your whole zpool is toast. And also, that syntax is wrong. You use zpool ..., not zfs .... The correct way to expand a pool is:
Code: 
[B]zpool set autoexpand=on [poolname]
zpool add [poolname] [mirror/raidz(1,2,3)] [new devices]
zpool list  <-- will show expanded capacity[/B]
Saves you the trouble of export/importing the pool.


FreeDomBSD said:
4. OS drive died two years ago and the system was just sitting there doing nothing
Click to expand...
This is the biggest problem with having a separate disk that the system boots off of. Using my suggestion, all of the drives would be bootable, so it doesn´t matter which drives goes boom. Also yields better performance, sequentially, and the same performance random.

And I understand that it can feel more complicated, but it is a superior setup, in every way. That´s why I made detailed instructions for the OP to follow.

/Sebulon
 
@FreeDomBSD

Here is something i learned by simulating the setup on a VMWare virtual machine. I found a logical bug:

Code: 
[CMD="#"]zpool status pool1[/CMD]
  pool: pool1
 state: ONLINE
  scan: resilvered 12.1M in 0h0m with 0 errors on Fri Feb 24 06:28:43 2012
config:

	NAME           STATE     READ WRITE CKSUM
	pool1          ONLINE       0     0     0
	  raidz1-0     ONLINE       0     0     0
	    gpt/disk1  ONLINE       0     0     0
	    gpt/disk2  ONLINE       0     0     0
	    gpt/disk3  ONLINE       0     0     0

errors: No known data errors
[CMD="#"]zpool get bootfs pool1[/CMD]
NAME   PROPERTY  [B]VALUE[/B]       SOURCE
pool1  bootfs    [B]pool1/root[/B]  local
[CMD="#"]zpool add -f pool1 mirror gpt/disk{4,5}[/CMD]
cannot add to 'pool1': root pool can not have multiple vdevs or separate logs

You cheap bastardsx( But I can understand why that is. Too bad.

However:
Code: 
[CMD="#"]zpool get bootfs rpool[/CMD]
NAME   PROPERTY  [B]VALUE[/B]   SOURCE
rpool  bootfs    [B]-[/B]       default
[CMD="#"]zpool add rpool mirror gpt/disk{4,5}[/CMD]
[CMD="#"]zpool status rpool[/CMD]
  pool: rpool
 state: ONLINE
  scan: resilvered 180K in 0h0m with 0 errors on Fri Feb 24 14:11:54 2012
config:

	NAME           STATE     READ WRITE CKSUM
	rpool          ONLINE       0     0     0
	  mirror-0     ONLINE       0     0     0
	    gpt/disk6  ONLINE       0     0     0
	    gpt/disk7  ONLINE       0     0     0
	  mirror-1     ONLINE       0     0     0
	    gpt/disk4  ONLINE       0     0     0
	    gpt/disk5  ONLINE       0     0     0

errors: No known data errors

So you cannot create a bootable RAID5 + RAID1, as I envisioned it. I am going to revise the plan for you, so that it works and still uses all of your disks, just a little differently.

/Sebulon
 
Hi I'm sorry for the lengthy delay.
Let me explain why it has been so:

I couldn't find a live version of FreeBSD to boot from so I had to figure out how to "burn" the .img to the USB drive and I'll try it in few hours.

Zfs requires atleast 1gb ram and the current box was more or less "designed" to run on an antiquated hardware (p4/512mb SDRAM) and is non-upgradable (768 mb max ram available).

Here is what I'm going to do in the next 48 hours:

Check the raid status as per Dave's instructions and report back; Upgrade hardware so it can run ddr2 memory and wait for updated instruction set.

My question to Sebulon is as follows: with your instructions, will I have no use for a dedicated OS drive?

Dave and Sebulon: I am ecstatic at the help both of you are providing me. You guys are pretty awesome. I understood that Dave's solution is temporary one from the get-go and that Sebulon's is the optimized long-term for the final build. I'm a newbie at bsd and nixes (as far as command sets and such), but very technical and can read/learn fairly easy. It is hard to jump in the middle as I'm doing now, but other than that I assimilate information quite well. I appreciate the hand-holding a lot! :)
 
FreeBSD 9 CD/DVD ISOs and the USB IMGs all have LiveCD built in, just boot any one and choose Live CD at the first prompt. Even the "bootonly" ISOs. That will give you a prompt so you can try accessing the old (gmirror?) drives.
 
I can't boot into fbsd FreeBSD 9.

:/

I'll be looking for help in another thread

-

Finally booted into live. At login prompt I typed root. At # I typed gmirror. Next # I typed gmirror status. Nothing happened . All drives are detected by FreeBSD and BIOS.

--

Great news! I was able to bring back up the old OS drive and mount the file system via single user mode. It's not happy though. More details soon.

--

Looks like I have GEOM RAID. Does that sound right? It kicks me back out to single user mode after working for a bit on a file system and encountering some problems:


Code: 
THE FOLLOWING FILE SYSTEMS HAD AN UNEXPECTED INCONSISTENCY:
ufs:  /dev/ad0s1g (/home), ifs: /dev/ad0s1e (/temp), ups: /dev/ad0s1f(/usr), ufs: /dev/ ad0s1d (/var)
Unknown error; help!
ERROR: ABORTING BOOT (sending SIGTERM to patent)!
Dec 31 18:42:24 init: /bin/sh on /etc/rc terminated abnormally, going to single user mode
Enter full pathname of shell or RETURN for /bin/sh:

Code: 
#gmirror status
Name Status Components
mirror/raid DEGRADED ad6
mirror/raid1 DEGRADED ad8
Running

# fsck -y

Still the same.

--

It tries to boot me from ad0s1a and fails so I have to manually boot it into ad2s1a. After fsck -y the "can't stat" errors are still present . I don't know how to tell FreeBSD to use ad2 instead of ad0. It also thinks my RAID mirror is broken.

[ Merged a bunch of rambling posts. -- Mod. ]
 
FreeDomBSD said:
My question to Sebulon is as follows: with your instructions, will I have no use for a dedicated OS drive?
Click to expand...

Correct. You will have no use for a dedicated OS drive.

You will be using the old mirrored drives for a zfs mirror on /, /usr, /usr/local and /var, and the three new drives in a zfs raidz on /usr/home as per the instructions. You will also be able to grow the home-pool when it is filled. Feel free to improvise the layout as you see fit. Some people have also different filesystems for /usr/ports, /var/db, etc, but in my own experience, the first ones have been enough for me, for what I do.

I have now simulated my revised setup and will update my previous post shortly. I will however state one more time that the setup requires disks A,D and E to be of the same size, and you never replied as to how big they are. Please, do share that.

Before, you said that the drives were organized like this:
OS drive - drive "A"
BC mirror - drives "B" and "C"
New drive 1 - drive "D"
New drive 2 - drive "E"

They need to reorganized for the server to be able to boot properly, like this:
BC mirror - drive "B"
BC mirror - drive "C"
OS drive - drive "A"
New drive 1 - drive "D"
New drive 2 - drive "E"

Correct this before trying to install using my instructions, or the drive-names won´t match and you´ll end up trashing everything.

BTW, are you going to using this server as a workstation or NAS?

/Sebulon
 
You must log in or register to reply here.
Back
Top