Vinum/GVinum and Shrinking Created Volumes

[ckozler]

Hi Everyone,

First and foremost I would like to let you know I have just made the jump completely from Linux to FreeBSD as my desktop computer and have to say that it is much more polished than I remember (perhaps because I am more advanced at *nix now?). In either case, I was spoiled by Linux LVM and have decided to use Vinum/GVinum on my FreeBSD machine for volume management. Here is what I have:

gvinum -> l
1 drive:
D a                     State: up	/dev/ad4s1d	A: 266239/829439 MB (32%)

2 volumes:
V VOL_HOME              State: up	Plexes:       1	Size:        450 GB
V VOL_XEN               State: up	Plexes:       1	Size:        100 GB

2 plexes:
P VOL_HOME.p0         C State: up	Subdisks:     1	Size:        450 GB
P VOL_XEN.p0          C State: up	Subdisks:     1	Size:        100 GB

2 subdisks:
S VOL_HOME.p0.s0        State: up	D: a            Size:        450 GB
S VOL_XEN.p0.s0         State: up	D: a            Size:        100 GB

With the supporting configuration file (as seen when I execute the create command in gvinum

# Vinum configuration of kozler-work, saved at Fri Mar  4 15:53:53 2011
# Current configuration:
# drive a device /dev/ad4s1d
# volume VOL_HOME
# volume VOL_XEN
# plex name VOL_HOME.p0 org concat vol VOL_HOME
# plex name VOL_XEN.p0 org concat vol VOL_XEN
# sd name VOL_HOME.p0.s0 drive a len 943718400s driveoffset 265s plex VOL_HOME.p0 plexoffset 0s
# sd name VOL_XEN.p0.s0 drive a len 209715200s driveoffset 943718665s plex VOL_XEN.p0 plexoffset 0s

I have also defined /dev/VOL_HOME to mount on /home at boot (configured vinum in /boot/loader.conf).

My question is relatively straight forward-- I see that you can grow volumes that you have created but can you shrink them? That is, I have my 100GB VOL_XEN volume defined that I want to shrink to, say, 20GB and then make a new VOL_DATA for mounting on /data which was possible with LVM.

Is it possible to shrink a created volume without deleting it? Is it possible to shrink a created volume without having it in a vinum RAID5 setup with striping? The only command(s) I see is grow and not shrink or anything of the similar

I also have my disks in a hardware RAID1 so I am not going to or want to uitlize any of vinum's "RAID" technologies [at this time].

Thank you!

 

[ckozler]

Sorry for the bump but I am going to be promoting the use of and deploying FreeBSD in some infrastructure-critical areas of my company and would like to use vinum for Volume management but being deterred from it because I am unable to find any information about being able to shrink or make smaller.

I have checked a lot of documentation but have not found anything-- can anyone give a hint? Maybe a past experience? Anything?

Thanks!

 

[blox]

Hi ckozler,

I'm rather new to FreeBSD also. So I cannot give you great hints. But, as far as my investigation have come, it turns out that [g]vinum functions pretty different and is in no matter comparable to LVM.

e.g. I turned one disk of mine into a big gvinum drive, created several volumes on it with the intention to have the possibility to enlarge these with ease on the same device. As far as I understood it, that is not possible – you can only enlarge volumes (even in concatenating mode) by adding complete new slices to this volumes (that is, by creating new vinum volumes on them).

So I guess that the way to go is using gconcat to enlarge volumes. Labeling is possible via glabel…

Anyway, I haven't seen nothing of shrinking either.

 

[ckozler]

Hi ckozler,

I'm rather new to FreeBSD also. So I cannot give you great hints. But, as far as my investigation have come, it turns out that [g]vinum functions pretty different and is in no matter comparable to LVM.

e.g. I turned one disk of mine into a big gvinum drive, created several volumes on it with the intention to have the possibility to enlarge these with ease on the same device. As far as I understood it, that is not possible – you can only enlarge volumes (even in concatenating mode) by adding complete new slices to this volumes (that is, by creating new vinum volumes on them).

So I guess that the way to go is using gconcat to enlarge volumes. Labeling is possible via glabel…

Anyway, I haven't seen nothing of shrinking either.

Ok, thank you very much for the confirmation! I just wanted to be doubly sure I did not miss something.

Perhaps there is another solution more suitable or I could just begin all volumes at a very small level and increase them as I need (though I still enjoy the ability to retract space from other Volumes if another one is more pressing).

 

[Galactic_Dominator]

There is also gvirstor(). However, on FreeBSD you can up in size/devices easily, but not down. Can I ask why shrinking a fs is a requirement? How often does happen? I work with plenty of virtualization and it's nearly always bigger, not smaller. Recreating a smaller device and then migrating is a pretty simple straight forward procedure provided you have the necessary disk space.

You might also consider using ZFS and it's integrated volume management. A ZVOL is in many ways akin to an LVM LV, and a pool to an LVM VG. ZVOL's can be resized up/down, but currently a pool can only go up.

 

[ckozler]

There is also gvirstor(). However, on FreeBSD you can up in size/devices easily, but not down. Can I ask why shrinking a fs is a requirement? How often does happen? I work with plenty of virtualization and it's nearly always bigger, not smaller. Recreating a smaller device and then migrating is a pretty simple straight forward procedure provided you have the necessary disk space.

You might also consider using ZFS and it's integrated volume management. A ZVOL is in many ways akin to an LVM LV, and a pool to an LVM VG. ZVOL's can be resized up/down, but currently a pool can only go up.

I do agree that in terms of virtualization hosts that you would need to scale up in disk size more than down (via iSCSI perhaps?).

Lets have an example like this (using Linux notation with using LVM -- it creates /dev/mapper/<Logical_Volume_Group_Name>/<Logical_Volume_Name> )

- 1 system
- 2 Logical Volume Groups
- 6 Logical Volumes
- /boot on its own partition (partition one on disk 1 [sda1] with about 200MB put to it)
- Ignore swap location/layout for now

LVG_SYSTEM
 - /dev/mapper/LVG_SYSTEM/LV_USR - 10% - mounted on /usr
 - /dev/mapper/LVG_SYSTEM/LV_HOME - 50% - mounted on /home
 - /dev/mapper/LVG_SYSTEM/LV_ROOT - 20% - mounted on /
 - /dev/mapper/LVG_SYSTEM/LV_OPT - 20% - mounted on /opt


LVG_STORE
 - /dev/mapper/LVG_STORE/LV_DATA - 100% - mounted on /data

So, now, lets assume we have 6 disks in a RAID1 which gives us a usable of 3 disks at a total of 1.5TB (each disk @ 500GB). That is, 3 disks are presented to the OS (sda, sdb, sdc -- please ignore the fact I am using single device names rather than /dev RAID device names to avoid confusion) and we are going to group sda2 and sdb1 into LVG_SYSTEM with with the aforementioned partitioning space and sdc is going to be in the LVG_STORE with a single volume of LV_DATA with 100% capacity of the logical volume group. This gives me 1TB usable for my root system which is on the LVG_SYSTEM logical volume group (see mount points above) and 500GB on /data which is apart of the LVG_STORE/LV_DATA volume.

Now, assuming my users are doing a lot of activity out of no where now and in a pinch situation I need to free up some space on LV_HOME-- I take away from LV_OPT since most of this is application installation storage space (not a lot of space was originally needed but I originally over-allocated) and give it back to LV_HOME. So, now I have a layout of

LVG_SYSTEM
 - /dev/mapper/LVG_SYSTEM/LV_USR - 10%
 - /dev/mapper/LVG_SYSTEM/LV_HOME - 60%
 - /dev/mapper/LVG_SYSTEM/LV_VAR - 20%
 - /dev/mapper/LVG_SYSTEM/LV_OPT - 10%

LVG_STORE
 - /dev/mapper/LVG_STORE/LV_DATA - 100%

Now, lets assume I wan to create another LV in LVG_STORE-- I can go ahead and resize LV_DATA down to 50% and create LV_EXPORT with the other 50% given to it -- this is not possible in Vinum/GVinum

LVG_SYSTEM
 - /dev/mapper/LVG_SYSTEM/LV_USR - 10%
 - /dev/mapper/LVG_SYSTEM/LV_HOME - 60%
 - /dev/mapper/LVG_SYSTEM/LV_VAR - 20%
 - /dev/mapper/LVG_SYSTEM/LV_OPT - 10%

LVG_STORE
 - /dev/mapper/LVG_STORE/LV_DATA - 50%
 - /dev/mapper/LVG_STORE/LV_EXPORT - 50%

I hope my example was clear. While I know ZFS would be good for what I want to accomplish, I believe it is entirely way to resource intensive for any form of a production environment other than a backup/storage facility (I use this currently for our backup storage system). I was hoping to find a scalable Volume Management System in FreeBSD and while I have not tried or seen what Virtuso is yet, I would like to see Vinum have the ability at one point or another to be able to scale a created volume down in size because thus far it takes care of almost every other issue with disks I have faced.

 

[Galactic_Dominator]

I know how device mapper/LVM works so I wasn't requesting a hypothetical situation, I was asking often this is necessary. You also left out a some steps that are major inconvenience in the process. Still, FreeBSD shrinking when not using ZVOL's is worse especially for large slices I agree since you'd have to dump, delete, create, and restore.

An additional nitpick on your scenario is if you're running your storage so close to full capacity you're killing write performance(read too over time as such conditions will rapidly increase fragmentation) on most Linux FS's.

I believe it is entirely way to resource intensive for any form of a production environment other than a backup/storage facility (I use this currently for our backup storage system).

I too run ZFS and I'm left scratching my head on this statement. First, you can restrict ZFS resource consumption down to a fairly low level if necessary. Second, you give an example of a storage system but say ZFS isn't suitable because it's only for storage systems? Third, disk subsystems are generally the slowest portion of a computing system when there is IO involved. ZFS takes from areas where speed is greatest(CPU, RAM) and gives to the poorest(disk io) with things like compression, ZIL, L2ARC. In terms of performance it's the Robin Hood of file systems with much greater guaranteed integrity.

 

[ckozler]

I know how device mapper/LVM works so I wasn't requesting a hypothetical situation, I was asking often this is necessary.

Ok, I just wanted to make sure I was conveying the right info

You also left out a some steps that are major inconvenience in the process. Still, FreeBSD shrinking when not using ZVOL's is worse especially for large slices I agree since you'd have to dump, delete, create, and restore.

Could you elaborate on the bold part?

An additional nitpick on your scenario is if you're running your storage so close to full capacity you're killing write performance(read too over time as such conditions will rapidly increase fragmentation) on most Linux FS's.

While I completely agree with you, I guess not knowing the environment I work in, its a bit hard to understand. IT budget is a bit low so I have to make due with whatever storage I can and usually am running very close to capacity and have to have the ability to scale other areas of the OS before having to climb the internal chain and request another cost to the company. Also, our internal infrastructure (somewhat prod-like because number of internal users interfacing with it) also crosses use cases very often which again is why I need to be able to scale file system sizes back and forth

I too run ZFS and I'm left scratching my head on this statement. First, you can restrict ZFS resource consumption down to a fairly low level if necessary. Second, you give an example of a storage system but say ZFS isn't suitable because it's only for storage systems? Third, disk subsystems are generally the slowest portion of a computing system when there is IO involved. ZFS takes from areas where speed is greatest(CPU, RAM) and gives to the poorest(disk io) with things like compression, ZIL, L2ARC. In terms of performance it's the Robin Hood of file systems with much greater guaranteed integrity.

I will admit that my ZFS knowledge is very, very limited and I guess my post was a little biased based on what I've read-- my knowledge only goes so far as to a basic implementation of it so I do apologize for jumping the gun on my judgement/assessment of ZFS (I havent had any personal problems either!)

To your second point, as I previously said, our internal infrastructure (not our 3 main production servers) are constantly crossing use cases so at one point a server could be a storage facility for X, Y, and Z and then all of a sudden there needs to be an extra QA environment so the ability to create new LVG (like LV_EXPORT) would allow me to install a temporary QA environment for a developer, add user and set their home to something like /export/QA/user1, chroot/jail them, and then discard it when all the testing is complete-- I know it sounds silly but its something I have to do.

To your third point, again, with a slim IT budget our servers are not the greatest of quality for our internal infrastructure. Perhaps, our best internal infrastructure server is a Dell 1950 w/ single Quad Core Xeon and 4GB of RAM of which is already being used as a DNS server, PXE Server, DHCP server, a restrictive email relay, and finally a Xen dom0 for tiny services that are very restricted or serve a different purpose-- I feel it would be hard to run ZFS on a server like this because ZFS requires more from the CPU and Memory of which is already being taken up my a load of services.

I will definitely read up on ZFS then I guess since I have only done limited reading where users have been inundated with problems, not disk related, that turned out to be ZFS as the culprit-- albeit, this could have been something they did and resulted as a faulty install as some of the posts were not responded to and/or the OP just left the thread completely.

Again, I will do more reading on ZFS and perhaps end up incorporating it more as it appears to be more what I need in terms of Volume Management for FreeBSD than Vinum.

Thank you for taking the time to respond and grow my knowledge on this subject.

 

[Galactic_Dominator]

Could you elaborate on the bold part?

As I said you left some things out. You can't simply resize LV and go. In your scenario, a file system lives on that LV and must be dealt with safely. That means shrinking the FS, then LV. To shrink the FS safely, you must have a backup of FS first, maybe that's a straight dd of the FS, or some other means but a smart sys admin is going to that first because a resize is no sure thing. This can be very time consuming and can't be done live like it could with UFS/ZFS. Research the full recommended steps for doing this and you'll see what I mean.

And yes I've seen enough places that like to short-change IT budgets, generally the same type of place that like to short their IT payroll as well. Pretty backwards.

From what you describe, having the ability to # zfs clone would save you enough time that it could pay for $200 of RAM to upgrade the system. And dedup and compression is going to probably save a lot of storage space in such an environment. Just to be clear, dedup functionality isn't here yet for STABLE and will require more RAM than a standard ZFS install.

The services you have listed aren't heavy weight either(assuming this isn't some massive network). Using FreeBSD jails and ZFS could reduce a lot of that systems overhead.

 

[ckozler]

As I said you left some things out. You can't simply resize LV and go. In your scenario, a file system lives on that LV and must be dealt with safely. That means shrinking the FS, then LV. To shrink the FS safely, you must have a backup of FS first, maybe that's a straight dd of the FS, or some other means but a smart sys admin is going to that first because a resize is no sure thing. This can be very time consuming and can't be done live like it could with UFS/ZFS. Research the full recommended steps for doing this and you'll see what I mean.

Ok, I understood what you said but I guess maybe should have made it clear that I knowingly left some things out (eg: the implications of resizing an LV because I thought they were implied). Although I do have to agree that my post did imply that I did not understand the risk of resizing an LV because I was giving the tone of "resize and go" though my infrastructure backs up nightly if not hourly depending on the use-case of the server(s) so backing up before a resize if necessary is not a real issue for me. In some instances I am using DRBD so replication is already happening at the block level.

I use all of my LV's with ReiserFS which also allows for resizing so its a pretty tried and true setup for me. I have never once resized my LV's and ran into an issue (though Im not saying it can't/won't happen).

I will do that research, thank you for the note on it.

And yes I've seen enough places that like to short-change IT budgets, generally the same type of place that like to short their IT payroll as well. Pretty backwards.

Yes, it is :\

From what you describe, having the ability to # zfs clone would save you enough time that it could pay for $200 of RAM to upgrade the system. And dedup and compression is going to probably save a lot of storage space in such an environment. Just to be clear, dedup functionality isn't here yet for STABLE and will require more RAM than a standard ZFS install.

The services you have listed aren't heavy weight either(assuming this isn't some massive network). Using FreeBSD jails and ZFS could reduce a lot of that systems overhead.

# zfs clone would be good since its a RW copy of the FS. True the services arent heavy weight either but I've always stood under the mantra that a server should have its individual role which is one of the [many] reasons I am a huge advocate for Virutalization but, again, due to the short budget can not afford the appropriate technologies for a virtualized environment (I would love IBM blades!)


Thanks again, you cleared up a lot of misconceptions and answered my original question (with the addition of the ZFS refresh/knowledge!).

 

[Galactic_Dominator]

True the services arent heavy weight either but I've always stood under the mantra that a server should have its individual role which is one of the [many] reasons I am a huge advocate for Virutalization

FreeBSD jails are a form of virtualization, and not that heavyweight, poorly documented abomination of Xen. You already mentioned chroot(), jails are enhanced, more secure form of it. Even the ubuntu man page says so.

http://manpages.ubuntu.com/manpages/dapper/en/man2/chroot.2.html

Plus with utilities like ezjail(), you can config skeleton jails and quickly roll out new ones that provide the stuff you are doing now. With ZFS running underneath you get all the befits already discussed, but even on a UFS based system with md() devices is still better than your current method of operation IMO.

What happens if there's a power outage or a crash during a resize? I'm not familiar with resier, but if you're on an ext2/4 system you better get your backups ready and hope there isn't too much data loss. DRBD wouldn't count in this situation either because both sides would be corrupt. That why HA isn't a backup and a backup isn't HA.

 

[ckozler]

FreeBSD jails are a form of virtualization, and not that heavyweight, poorly documented abomination of Xen. You already mentioned chroot(), jails are enhanced, more secure form of it. Even the ubuntu man page says so.

http://manpages.ubuntu.com/manpages/dapper/en/man2/chroot.2.html

Plus with utilities like ezjail(), you can config skeleton jails and quickly roll out new ones that provide the stuff you are doing now. With ZFS running underneath you get all the befits already discussed, but even on a UFS based system with md() devices is still better than your current method of operation IMO.

I didnt design this infrastructure-- I came into this cluster. I am doing my best to reform, organize, consolidate, and repair it the best with what I've got given the pressures from every corner of the company (one man to 200 clients & 20 employees & 3 offices). We mostly use SLES11 here and I have been trying to consolidate all of our internal services onto a single server using SLES11 & small slices of Xen VM's (which hasnt let me down yet). I have just begun using FreeBSD and plan to roll it into my more infrastructure-core systems which is why I am greatly thankful for all of your information.

Most of our infrastructure consists of development and QA machines and 1 functional infrastructure server. I have migrated infrastructure services to this one so I can reburb 2 others and I guess I will be going towards ZFS, FreeBSD, and Jails at this point. It seems like it would be much lower overhead than what I'm doing with Xen VM's, Linux, LVM's, and the likes.

I was first a huge advocate of Virtual Box until I was introduced to Xen. While VBox has its benefits, I fell hard for Xen. That said, as soon as I got on FreeBSD I began investigating Xen virtualization and you are correct, there is poor (if best) documentation on Xen virtualization for FreeBSD. Perhaps for my desktop I will work with Xen (I need Windows XP/7 images running again) but infrastructure systems will receive the jails with ZFS underlying on the file system because I must agree that they are best.

What happens if there's a power outage or a crash during a resize? I'm not familiar with resier, but if you're on an ext2/4 system you better get your backups ready and hope there isn't too much data loss. DRBD wouldn't count in this situation either because both sides would be corrupt. That why HA isn't a backup and a backup isn't HA.

ReiserFS is very powerful, smart, and resilient FS but sadly discontinued ( I think the creator is in jail for murder if I recall correctly... ) which is why I was introduced to ZFS by an ex-coworker who decided to setup one system with ZFS. I liked it right away but as we both agreed and discussed, ZFS still wasnt ready for production roll-out (he liked to be doubly & triply sure that everything works to 100% before doing anything-- not a bad thing).

I have never once lost data using ResierFS with or without another facility like LVM as ReiserFS has always recovered the journal and picked back up. One time when I lost power and was using an ext3 system the data become corrupted (flat disks, no RAID) after a power failure and I inevitably lost a good chunk of data...I was able to salvage most of it back by using a different superblock and finally getting it mounted but it eventually proved to be fruitless (and pointless to try to continue with a corrupted disk).

In terms of the DRBD where I am applying replication (I did not say backup explicitly) the backup servers are on UPS's on a totally separate power lines than that of the primary. That is, two servers in HA DRBD that if the primary became corrupted, the secondary would have more than likely not the most up to date information. The DRBD boxes though are a Dev-QA/INF-QA style box and not a dedicated infrastructure box.

 

[ckozler]

I cant edit my post but I meant to say ext2 rather than ext3.

 

[Galactic_Dominator]

By poor documentation, I was referring to official Xen opensource hypervisor documentation, not docs on how to run a para-virtualized FreeBSD guest.

I can go off in many directions here from the woefully inadequate man pages, to the spiderweb of it's crappy wiki, to it's cli reference parameters, or it's hodge podge of perl scripts stacked on python scripts stacked on who knows what.

A brief example, I have some Xen VM's that require timer_mode=2 to run stable or I can crash the entire host intermittently by changing time zone in a guest. You will not find reference to that enchantment unless you dig deep into it's internals, and that's only the tiniest bit of black magic there is buried in there. An enterprise, mission critical piece of software shouldn't be full of things like that.

I think Xen 4 has improved things a lot, but for my money I'm going Vbox where I can go read much more extensive documentation if I run into trouble regardless of my host OS.

Main 3 points when running ZFS:
1. Disable sendfile on every service using a ZFS backed storage(eg FTP, web server, SAMBA). Sometimes it's not on by default.
2. AMD64 with 4GB+ of RAM or do some tuning
3. Ensure your hardware doesn't lie about cache flushes. If there's corruption due to this, it's hardly ZFS's fault. Main culprit here are SSD drives, but normal HD's, flash drives, and controllers can all cause problems.