Solved Does FreeBSD block access to hard disk drives, from within a VirtualBox machine?

jjthomas

Member

Reaction score: 5
Messages: 93

As noted in another post, I have installed FreeBSD 12.1. I have a 4 drive RAID5 partition that I want to get the data off of.
I have tried a couple of VirtualBox machines, but I cannot seem to access the existing Linux RAID5 drive. Using ext2fs I have access to my other Linux drives.

As noted in the title, does FreeBSD block access to hard disk drives, from within a VirtualBox machine?

TIA

-JJ

The vital stats (as root):
FreeBSD anthem.027esc.net 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC amd64
Code:
VBoxManage internalcommands createrawvmdk -filename /root/VirtualBox\ VMs/KnoppixDisk0.vmdk -rawdisk /dev/ada4
VBoxManage internalcommands createrawvmdk -filename /root/VirtualBox\ VMs/KnoppixDisk1.vmdk -rawdisk /dev/ada5
VBoxManage internalcommands createrawvmdk -filename /root/VirtualBox\ VMs/KnoppixDisk2.vmdk -rawdisk /dev/ada6
VBoxManage internalcommands createrawvmdk -filename /root/VirtualBox\ VMs/KnoppixDisk3.vmdk -rawdisk /dev/ada7
crw-r-----  1 root      operator   0x9e Jul  6 17:10 ada0p1
crw-r----- 1 root operator 0x5a Jul 6 08:08 ada1
crw-r----- 1 root operator 0x5c Jul 6 08:08 ada1p1
crw-r----- 1 root operator 0x5d Jul 6 08:08 ada2
crw-r----- 1 root operator 0x74 Jul 6 08:08 ada2p1
crw-r----- 1 root operator 0x75 Jul 6 08:08 ada2p2
crw-r----- 1 root operator 0x76 Jul 6 08:08 ada2p3
crw-r----- 1 root operator 0x5e Jul 6 17:01 ada3
crw-r----- 1 root operator 0x72 Jul 6 08:08 ada4
crw-r----- 1 root operator 0x73 Jul 6 08:08 ada5
crw-r----- 1 root operator 0x7b Jul 6 08:08 ada6
crw-r----- 1 root operator 0x7c Jul 6 08:08 ada7
vboxusers:*:920:jjthomas
 

mjollnir

Daemon

Reaction score: 699
Messages: 1,154

Your configuration is not 100% clear to me. You're running a Linux host with a RAID5 setup by Linux's methods and this host manages the RAID5. FreeBSD is running as guest in VBox inside that Linux host. To give the guest access to the disks of that RAID, obviously the Linux host must not keep a "hold" on the RAID, i.e. you must stop it from managing it, so that the disks are just there, but not in use on the Linux host. Then you have to tell the FreeBSD guest that these disks are a RAID5 by Linux methods - I'm not shure if FreeBSD can handle that, grep graid(8). A standard RAID is not a cluster FS allowing concurrent access!

Maybe it's easier to export the data on the Linux host and import in the guest via NFS. Or you install the VBox guest additions and use the VBox methods (shared data or s/th like that)
pkg search vbox
pkg search virtualbox
 
OP
J

jjthomas

Member

Reaction score: 5
Messages: 93

Your configuration is not 100% clear to me.
FreeBSD is the host. The RAID5 was setup by CentOS, while I was running the machine completely under Linux. I am replacing, well have replaced, CentOS with FreeBSD. FreeBSD has been installed and is working well, if I may add. I am just trying to get my data from the Linx RAID5 array.

If it's mdadm, then you're out of luck. Not supported on FreeBSD.
Ouch, the array was created with mdadm. Well crap, I am going to have to install an additional hard disk, boot Knoppix, and copy the data over that way.

Thank you for the replies.

-Julian
 

_martin

Aspiring Daemon

Reaction score: 178
Messages: 846

It doesn't matter what is on those disks, you are passing raw devices to a guest. In this scenario host doesn't care what those disks are at all. You didn't paste the error you see (either logged from host or guest).

Unfortunately I don't have a way of testing this right now. If this is a one time test solution you could change the group ownership on those disks (on host, i.e. FreeBSD) you pasted above to the VirtualBox group (by default vboxusers) and change the permissions to 660.

If you are looking for a more permanent solution you could have a look at the package information and check how USB permissions are set in devfs (same approach):
pkg info -D virtualbox-ose\*
 

mjollnir

Daemon

Reaction score: 699
Messages: 1,154

_martin, the FreeBSD guest can not read the RAID5 done by Linux mdadm. In principle RAID5 is RAID5, but where the metadata gets written to the device and such, etc.pp. all this is implementation.
You can't put the disks of an Adaptec-RAID5 on another's brand RAID-controller either (unless it's re-branded, i.e. essentially the same model - still then, beware the firmware...). Likewise, you can't throw a Linux-mdadm RAID5 to ZFS and say: "It's RAID5, READ IT!". Neither graid(8). They all have slightly differrent on-disk metadata formats.
 

_martin

Aspiring Daemon

Reaction score: 178
Messages: 846

mjollnir I see, I understood the OP differently. I thought he has FreeBSD host and Linux guest.

Edit:
jjthomas It's unclear to me what your setup actually is. But as you are already playing around with VMs you could spawn a Linux VM and attach those disks to it as you mentioned. Just make sure the permissions/ownership is set properly for VirtualBox user. That way you can work with them without further host interference.
 

mjollnir

Daemon

Reaction score: 699
Messages: 1,154

[...] playing around with VMs you could spawn a Linux VM and attach those disks to it as you mentioned. Just make sure the permissions/ownership is set properly for VirtualBox user. That way you can work with them without further host interference.
Seems there is no mdadm executable in linux*-c7*. Don't ask me why.
paul@t450s:/usr/ports % find . -type f -name pkg-plist|xargs fgrep mdadm
./sysutils/webmin/pkg-plist:lib/webmin/raid/save_mdadm.cgi
./shells/bash-completion/pkg-plist:%%DATADIR%%/completions/mdadm
./shells/zsh/pkg-plist:%%DATADIR%%/%%ZSH_VER%%/functions/Completion/Linux/_mdadm
./shells/fish/pkg-plist:%%DATADIR%%/completions/mdadm.fish
./textproc/augeas/pkg-plist:%%DATADIR%%/lenses/dist/mdadm_conf.aug
./textproc/augeas/pkg-plist:%%DATADIR%%/lenses/dist/tests/test_mdadm_conf.aug

PS: Refering to your signature (many slovakians know a little bit german):
"Stell' Dir vor es geht und keiner kriegt's hin." (Wolfgang Neuss)
 

_martin

Aspiring Daemon

Reaction score: 178
Messages: 846

mjollnir Please note I was referring to having raw disks in the VirtualBox VM as OP is already doing. But instead of having FreeBSD guest I was referring to a Linux VM of his choice. Linux VM is then accessing those disks natively, at least as far as guest is concern. I was not referring to Linux ABI on FreeBSD host.
But for that those permissions on host do have to be set properly so that VM under vbox user can actually access them.

As a PoC I just tested it (FreeBSD host, VirtualBox VM Linux guest) and it works as expected.

Sidenote: yeah, I'm vaguely familiar with the quote.
 

_martin

Aspiring Daemon

Reaction score: 178
Messages: 846

Once VM sees those disks (raw device mapping) it can do whatever with it. From VM point of view those disks are just disks. So yes, VM can fully assemble the raid and see the LVM structures as any other physical Linux would.
OP was already doing this (mapping); it's just unclear what is the VM.
 

mjollnir

Daemon

Reaction score: 699
Messages: 1,154

So then: "Es geht und keiner kriegt's hin" (It is possible, but noone succeeds to get it right) ;)
 

_martin

Aspiring Daemon

Reaction score: 178
Messages: 846

Well, as I mentioned I did PoC (prove of concept) few hours ago. It does work. :)
 
OP
J

jjthomas

Member

Reaction score: 5
Messages: 93

Hi _martin. I not getting any error messages, the VirtualBox just freezes.

I have an 8TB drive on the way and I'll just boot Knoppix and copy the files off the Linux RAID array, that way. BTW you are correct that Oracle VB does see the drives, I just can't seem to access them.

Thank you.

-JJ
 

_martin

Aspiring Daemon

Reaction score: 178
Messages: 846

Did you try to change the permissions of those disks on FreeBSD host as I mentioned ?
 
OP
J

jjthomas

Member

Reaction score: 5
Messages: 93

No, I can try that in the morning.

However, if I am running as the root user, do the permissions matter?

Code:
(1:19)ROOT@anthem:/root# ll VirtualBox\ VMs/
total 41
drwxr-xr-x 3 root wheel 6 Jul 8 22:25 Knoppix/
drwxr-xr-x 3 root wheel 6 Jul 6 13:58 Knoppix - DVD/
-rw------- 1 root wheel 539 Jul 6 14:09 KnoppixDisk0.vmdk
-rw------- 1 root wheel 539 Jul 6 14:09 KnoppixDisk1.vmdk
-rw------- 1 root wheel 539 Jul 6 14:09 KnoppixDisk2.vmdk
-rw------- 1 root wheel 539 Jul 6 14:09 KnoppixDisk3.vmdk
-rw------- 1 root wheel 539 Jul 6 14:47 KnoppixDisk4.vmdk
(1:22)ROOT@anthem:/root#
 

_martin

Aspiring Daemon

Reaction score: 178
Messages: 846

jjthomas Please do read my first post, I mentioned it there. Yes, it does matter.
Also, in your setup (and initial question) - is your VM FreeBSD or Linux ?
 
OP
J

jjthomas

Member

Reaction score: 5
Messages: 93

I have tried to install 7 different Virtual Machines. Slackware, Debian, CentOS 7 CentOS 8 as my user jjthomas, and three different VM's under my root account. None of the machines can read the RAID5 partition. Further, none of the machines will finish installation under my user account. The fact that I cannot get a Slackware VM to install is very disturbing. Soooo, I going to set this aside, my new harddisk arrived, get my data copied off, and then install a ZFS RAID 5 volume. And then I may return to installing a VM.

Deep breath, back to trying it again. I deleted all the VM's and their files and I'm starting with new VM (Virtual Box) using Centos 7 for the VM.

Oh the VM is Linux, my host is FreeBSD 12.1.

Thank you.

-JJ
 
OP
J

jjthomas

Member

Reaction score: 5
Messages: 93

jjthomas Please do read my first post, I mentioned it there. Yes, it does matter.
Also, in your setup (and initial question) - is your VM FreeBSD or Linux ?
Hi again,

FreeBSD 12.1 is the host. Fedora is the VM. Fedora 64bit version 32

I have installed Fedora in a Virtual Machine. I booted it to make sure it works, it did. I shut it down.
Then I created the RAW access as outlined in the virtualbox manual, here entering :
Code:
VBoxManage internalcommands createrawvmdk -filename ~jjthomas/VirtualBox\ VMs/Fedora32/Fedora32Disk0.vmdk -rawdisk /dev/ada4
VBoxManage internalcommands createrawvmdk -filename ~jjthomas/VirtualBox\ VMs/Fedora32/Fedora32Disk1.vmdk -rawdisk /dev/ada5
VBoxManage internalcommands createrawvmdk -filename ~jjthomas/VirtualBox\ VMs/Fedora32/Fedora32Disk2.vmdk -rawdisk /dev/ada6
VBoxManage internalcommands createrawvmdk -filename ~jjthomas/VirtualBox\ VMs/Fedora32/Fedora32Disk3.vmdk -rawdisk /dev/ada7
/dev/sda{4,5,6,7} are the four drives that make up the RAID5 array, from my old CentOS install.
Here is the directory listing:
Code:
(23:34)ROOT@anthem:/home/jjthomas/VirtualBox VMs/Fedora32# ll
total 308
-rw-rw---- 1 jjthomas vboxusers 3773 Jul 9 22:55 Fedora32.vbox
-rw-rw---- 1 jjthomas vboxusers 3773 Jul 9 22:54 Fedora32.vbox-prev
-rw-rw---- 1 jjthomas vboxusers 69634883584 Jul 9 22:50 Fedora32.vdi
-rw-rw---- 1 jjthomas vboxusers 539 Jul 9 22:49 Fedora32Disk0.vmdk
-rw-rw---- 1 jjthomas vboxusers 539 Jul 9 22:49 Fedora32Disk1.vmdk
-rw-rw---- 1 jjthomas vboxusers 539 Jul 9 22:49 Fedora32Disk2.vmdk
-rw-rw---- 1 jjthomas vboxusers 539 Jul 9 22:49 Fedora32Disk3.vmdk
drw-rw---- 2 jjthomas vboxusers 3 Jul 9 22:43 Logs/
(23:34)ROOT@anthem:/home/jjthomas/VirtualBox VMs/Fedora32#
And finally, when I try to attach the disk image I get the error:
Code:
Failed to open the disk image file /home/jjthomas/VirtualBox VMs/Fedora32/Fedora32Disk0.vmdk.

Permission problem accessing the file for the medium '/home/jjthomas/VirtualBox VMs/Fedora32/Fedora32Disk0.vmdk' (VERR_ACCESS_DENIED).

Result Code: VBOX_E_FILE_ERROR (0x80BB0004)
Component: MediumWrap
Interface: IMedium {4afe423b-43e0-e9d0-82e8-ceb307940dda}
Callee: IVirtualBox {9570b9d5-f1a1-448a-10c5-e12f5285adad}
Callee RC: VBOX_E_OBJECT_NOT_FOUND (0x80BB0001)
My drives are as follows:
ada0 and ada1 are SATA ext4 drives
ada2 is a FreeBSD ZFS
ada3 is an unused 8TB drive
and as I said above ada4, ada5, ada6, ada7 are the drives that make up my madam RAID5 array.

I did set the directory permission to 777, it did not help.

It is past my bedtime. Do you have any ideas?

-JJ
 

mjollnir

Daemon

Reaction score: 699
Messages: 1,154

You'll have to allow the group vboxusers rw access to the disks, and your user must be in that group:
pw groupmod vboxusers -m jjthomas
You can do that with a set of devd.conf(5) rules, or devfs.rules(5). You can either
  • copy /usr/local/etc/devd/cups.conf to vbox.conf and adjust
  • add entries in /etc/devfs.rules or /etc/devd.conf
 

_martin

Aspiring Daemon

Reaction score: 178
Messages: 846

My first reply on this thread is correct then -- FreeBSD host doesn't care what is on those disks, it's up to VM to assemble the raids.
You are mapping physical disks which are seen on FreeBSD host (e.g. /dev/ada3), using VirtualBox to a Linux VM.

TL;DR version: You need to set the proper permissions of the device files on FreeBSD host so that VM can properly read it.

To recap it in longer version:
1) On FreeBSD host you need to set the ownership/permissions so that VirtualBox process (a.k.a VM) can access them. By default they are not readable by virtualbox user.
As I've mentioned if you need more permanent solution look at the package info how this was done for USB devices ; it's the same issue.
If you need this only temporarily you can set these permissions as root on FreeBSD host manually.

So from this:
Code:
host(~)# ls -la /dev/da[12]
crw-r-----  1 root  operator  0x5d Jul 10 11:54 /dev/da1
crw-r-----  1 root  operator  0x60 Jul 10 11:54 /dev/da2
host(~)#
To:
Code:
host(~)# ls -la /dev/da[12]
crw-rw----  1 root  vboxusers  0x5d Jul 10 11:54 /dev/da1
crw-rw----  1 root  vboxusers  0x60 Jul 10 11:54 /dev/da2
host(~)#
2) Create virtual disks for your Linux VM. I'm assuming you know how to attach those disks to the VM, but for the whole picture I'm showing you the commands too. I named the Linux VM "linux", I've a SATA controller named "SATA".
EDIT: I'm naming them VDIs but yeah, they are VMDKs.

Code:
host(~)# VBoxManage internalcommands createrawvmdk -filename raid00.vdi -rawdisk /dev/da1
host(~)# VBoxManage internalcommands createrawvmdk -filename raid01.vdi -rawdisk /dev/da2
host(~)# VBoxManage storageattach linux --storagectl "SATA" --port 1 --type hdd --medium raid00.vdi
host(~)# VBoxManage storageattach linux --storagectl "SATA" --port 2 --type hdd --medium raid01.vdi
Now this is a "map" of the disk:
/dev/da1 - physical disk as FreeBSD sees it ; we need to allow VirtualBox process (VM, by default user/group vboxusers) to write on it
raid00.vdi - virtual disk metadata file ; used in VM to describe the disk (you don't need to modify the permissions on this)
/dev/sdb - disk as it's seen within VM (guest).

3) Start the VM and let the OS boot:
Code:
host(~)# VBoxManage startvm linux --type headless
Waiting for VM "linux" to power on...
VM "linux" has been successfully started.
host(~)#
Check the disks from VM and assemble the raid (there are more ways of doing this, but then it's a Linux topic):

Code:
root@debian:~# lsscsi
[0:0:0:0]    cd/dvd  VBOX     CD-ROM           1.0   /dev/sr0
[2:0:0:0]    disk    ATA      VBOX HARDDISK    1.0   /dev/sda
[3:0:0:0]    disk    ATA      VBOX HARDDISK    1.0   /dev/sdb
[4:0:0:0]    disk    ATA      VBOX HARDDISK    1.0   /dev/sdc
root@debian:~#

root@debian:~# mdadm -A /dev/md0 /dev/sdb /dev/sdc
mdadm: /dev/md0 has been started with 2 drives.
root@debian:~# mdadm -D /dev/md0 --prefer=mapper
/dev/md0:
           Version : 1.2
     Creation Time : Wed Jul  8 01:15:09 2020
        Raid Level : raid1
        Array Size : 2094080 (2045.00 MiB 2144.34 MB)
     Used Dev Size : 2094080 (2045.00 MiB 2144.34 MB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Fri Jul 10 10:22:12 2020
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : sus01:0
              UUID : b1db0690:09b65d22:df8fbbce:4aea1170
            Events : 13

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
root@debian:~# vgs
  VG     #PV #LV #SN Attr   VSize  VFree
  vgdata   1   1   0 wz--n- <2.00g 1020.00m
root@debian:~# vgchange -a y vgdata
  1 logical volume(s) in volume group "vgdata" now active
root@debian:~# mount /dev/vgdata/lv_data /a
root@debian:~# cat /a/welcome_msg
this is mdraid on linux !
root@debian:~#
 
OP
J

jjthomas

Member

Reaction score: 5
Messages: 93

Wow... I had not done the VBoxManage storageattach. I woke up to grab a bite to eat, I forgot to have dinner last night.

I will do your suggestions and let you know how it works out.

Thank you.

-JJ
 
OP
J

jjthomas

Member

Reaction score: 5
Messages: 93

The Final Chapter?

By following your instructions to the "T," I discovered that my fourth drive was not readable while running Linux in a Virtual Box.

I was adding all four drives, your instructions had me add just two drives, I booted and it worked. When I added the third and fourth drive, the Virtual Box would not boot. It turns out that adding the fourth drive prevented the Virtual Machine from booting, even resulting in a core dump.

So I booted my Centos 7 live disk. I could not get the RAID array to mount. That made no sense since the RAID array was created while running Centos. Since I had previously tried Fedora 32 (test) on the computer, I created a Fedora 32 (non-test) live DVD and booted it. It picked up the RAID array and let me mount the array. And so the copying was started.

In looking for a cause, I could not find anything. geom sees the drive, even the Linux virtual machines see the drive. Using the mdadm utility does not show any errors. At one point, I did get the array assembled under CentOS, but when I tried to mount it, it kicked out a bad superblock error.

So the original questions I posted, was,
"Does FreeBSD block access to hard disk drives, from within a VirtualBox machine?" The answer, it does not.

The lesson: if it doesn't work and has multiple parts, start with a minimum of parts and keep adding parts until it breaks. Generally, the last part is the problem.

I appreciate everyone who responded. Thank you.

Marking SOLVED.

-JJ
 

Jose

Well-Known Member

Reaction score: 227
Messages: 383

...Since I had previously tried Fedora 32 (test) on the computer, I created a Fedora 32 (non-test) live DVD and booted it...
I'm gonna guess the Fedora kernel was newer, and made some incompatible changes to the md metadata on the disks. Weird that it only affected one disk, though.
 
Top