Solved Running OS off USB thumb drive or ...?

Hi. I've attempted to run FreeBSD 12.2 release off a USB 3 on the motherboard of my Dell R720 server and found the experience dauntingly sluggish. I'd like to calibrate my expectations here, I don't need flying OS but I would imagine it should't take forever to unzip a bunch of TGZ files. E.g. anything attempting to install anything (so I suspect writes) like pkg install takes awfully long. It took forever to unzip that Base.tgz during the OS install I almost gave up. Somehow I don't feel this should be the norm. Was I just unlucky with this memory stick?

Now, broadly. I'm not married to the idea of running off the thumbdrive, it was simply an obvious idea. That said I don't want to put OS on one of those hot swappable drives either - it simply doesn't belong there in a server and in fact should probably be "nimble" and "ephemeral" in the sense that I should be able to replace it with another installation in a single step and have the server working. What do people employ for such cases. Lengthy detailed tutorials aren't necessary unless you feel like educating which I'll take with gratitude, but some pointers at my options and in general what "sysadmin professionals" use in their day to day would be highly valuable. I can google and Handbook my way after that.

This also doesn't need to be "physical". Atm my servers reside in a rack next to my desk, but I may need to colocate or have them elsewhere. I toyed with an idea of something like *mfsbsd* - having my entire OS in a ramdisk or smth like that. Or maybe there's some over the wire option although this would require 24/7 machine somewhere to distribute .. something (image?).

At the end of the day I can also plug a cheapo M.2 via PCIe and have OS fly. Why the heck didn't the USB drive do its god damn job. How is it so much slower than those rusty spinners?

Thank you
 
Some thoughts:
  • If you don't mount your USB thumb drive readonly, it wil wear out quicker than you want it to. I.e. you have to have tmpfs(5) /var & /tmp, and pray that the ports(7) you're using all behave well. Beware, some (many?) do write to /usr/local/portname frequently...
  • It's easy to plug a USB3 stick into an USB2 port :(
  • USB3 speed on the interface is different from the speed of the memory chips. You already discovered that... Usually cheap=slow ... expensive=faster. Of course that heavily depends on the technology used.
Thus if you have the option to use a M.2 SSD device, preferably mirrored, I wouldn't think much about any other options.
 
If something does write to /usr/local (I was actually unaware any did), then you can possibly look at using using unionfs to mount a writable overlay ontop of /usr/local which is turn is backed by a tmpfs mount.

A bit fiddly but it would get round a few issues.
 
Thus if you have the option to use a M.2 SSD device, preferably mirrored, I wouldn't think much about any other options.
I'm thinking M.2 NVMe in a PCIe slot. But. I just remembered that people had trouble booting off of those devices. Actually, I don't know if this can be done reliably. I don't recall seeing anything related in Dell R720 BIOS. FreeBSD has no problem working with those NVMe drives, but booting off of one is a different story. Anyone has experience?

So brief looking around shows booting off PCIe on Dell PowerEdge is at best clunky and amounts to either running SATA to your adapter or kicking off boot process from USB then somehow passing control to OS installed on your PCIe SSD. I'm not liking any of those options. wtf .. how is it all so convoluted. Maybe I should just figure how to power an SSD inside that Optical Drive bay and run a SATA cable there or smth like that.
 
Hi. I've attempted to run FreeBSD 12.2 release off a USB 3 on the motherboard
I just discovered those are not USB 3 but USB 2 ports ... facepalm.

Also while running OS off M.2 NVMe SSD via PCIe adaptor would be fast ... I discovered Dell servers cannot boot from those. So, if anyone can think of a reliable boot option, do please share.

Any way to kick off from USB thumbdrive but have OS installed and running on that PCIe SSD? In fact I still have FreeBSD installed on one of the drives attached to the backplane. I can't boot from it atm cause I re-flashed that LSI controller but didn't put the LSI ROM that lets you choose the boot drive. Still, this would give me a way to experiment with booting off e.g. a USB and passing control to that drive. If only someone could give me a bit of guidance.

Better still. Could I somehow boot off the network then somehow pass control to that PCIe SSD with actual OS on it? Extra browny points for way to completely replace that OS over the wire if I ever need to (or pull entire image over the network that then is written to that SSD to run). What key terms should I google or look for in the handbook?

Just booting everything somehow from the network and running entirely in RAM? That sounds a bit scary. If ever I load so much I need to start swapping. Dunno. Also I'd want an option to somehow persist my changes e.g. packages installed. Somehow I doubt there's a reliable way to do that.
 
Didn't I tell you it's easy (USB3->USB2)? ;) Happened to me a dozen times, too...
IIRC you can boot from one media, and have rootdev="disk2s1a" in loader.conf(5) to switch to another device. See also /boot/defaults/loader.conf. That's different from vfs.root.mountfrom, which only sets the root filesystem of that device. The latter is usually autodetected, but you can have it in your loader.conf(5) as a fallback or override. I have it commented off, so when I need it on the loader prompt, I view that file and can fish it.
 
Didn't I tell you it's easy (USB3->USB2)? ;) Happened to me a dozen times, too...
IIRC you can boot from one media, and have rootdev="disk2s1a" in loader.conf(5) to switch to another device. See also /boot/defaults/loader.conf. That's different from vfs.root.mountfrom, which only sets the root filesystem of that device. The latter is usually autodetected, but you can have it in your loader.conf(5) as a fallback or override. I have it commented off, so when I need it on the loader prompt, I view that file and can fish it.
Yeah, I totally s..ed up with that USB, didn't I.

Ok. That looks like something I should try. Is there a way to figure what "disk?s??" my M.2 PCIe drive would be? Or go about it differently: how do I enter the interactive interpreter and maybe query for such properties. IIUC the loader() there is a way to interact with it - its basically either a shell of some kind or a Forth interpreter, so maybe I could query devices somehow. Note, however that this is M.2 NVMe card in PCIe slot, so my hunch is that I need to have relevant kernel modules loaded to even be able to see that drive?
 
Didn't I tell you it's easy (USB3->USB2)? ;) Happened to me a dozen times, too...
IIRC you can boot from one media, and have rootdev="disk2s1a" in loader.conf(5) to switch to another device. See also /boot/defaults/loader.conf. That's different from vfs.root.mountfrom, which only sets the root filesystem of that device. The latter is usually autodetected, but you can have it in your loader.conf(5) as a fallback or override. I have it commented off, so when I need it on the loader prompt, I view that file and can fish it.

so, I made a little bit of progress having read the most helpful https://docs.freebsd.org/en/books/handbook/boot/ chapter. I can get into the loader repl and issue commands. I got stuck about where I expected.
First, as expected lsdev shows only disk0 available, which is my USB memstick. boot-conf lets me load modules and lsmod confirms that nvme, nvd, etc are loaded, os in principle once kernel loads I should be able to access those NVMe drives in PCIe slots. But how do I access them now. I've no clue what to set rootdev="???" to given that lsdev only sees one disk.

One thing that worked sort of: set vfs.root.mountfrom="/dev/da4a" actually proceeded to boot(?) from my old drive. Of course it immediately bailed cause /etc/fstab was from an old system and no such devs existed any more. I am extremely fuzzy on the whole rootdev vs vfs.root.mountfrom dichotomy and mans are mostly quiet on exactly what the difference is.

It looks like what I have done above may allow me to have my root on one or part of NVMe SSD. Is that right?

Am I even going in the right direction or maybe all I really need to do to make the system usable with just the USB stick is mount /tmp and /var to one of those fast NVMe SSDs? Wouldn't that suffice?
 
Am I even going in the right direction or maybe all I really need to do to make the system usable with just the USB stick is mount /tmp and /var to one of those fast NVMe SSDs? Wouldn't that suffice?
Ok so I scratched my head and at least for now ended up simply moving /tmp to tmpfs and /var over to NVMe SSD. Here's part of my /etc/fstab:
Code:
tmpfs           /tmp            tmpfs   rw,size=8g,mode=01777   0       0
/dev/gpt/swap   none            swap    sw      0       0
/dev/gpt/var    /var            ufs     rw,noatime,nosuid       2       2
Feels snappier. Obviously, still dog slow when I write to e.g. /home/user or /usr. But those are rare occasions since this is a server. I think I'll go with this for now and revisit if I run into problems.

As one could notice per above I created GPT labels for var and swap. And I totally did not prefix them. Idea is that I may want (or need) to replace physical drives and I want server to continue to boot. Won't get to test it on these I suppose, but for something under /usr or /home or root partition, I think it is an interesting solution. I had boot break on me so often lately simply because drives rearranged or changed and /etc/fstab was no longer valid and all you get is "unknown error". What do people think? Yay or nay? My understanding cautious among us recommend adding prefixes to GPT labels essentially making them e.g. machine or (shudder) physical drive specific. I rather like being dynamic.
 
You can also put other filesystems (e.g. /home (symlink /usr/home -> ../home/ & /usr/obj -> ../var/obj/)) on the NVD. That's especially easy when it's a ZFS pool.
EDIT Keep in mind that your USB thumb drive will quickly wear out if you write to it frequently. Thus it's desirable to put all filesystems with frequent write access to a more robust medium. You can remount the root filesystem read-write when it's needed to update some configuration or packages, and have it readonly during normal operation.

Your usage of GPT labels in fstab(5) is fine; that's exactly one of it's intended applications. Wether to prefix or not may have it's pros & cons. I'm doing it like you and it has proven useful for me. More experienced wizzards can easily convince me to change that.
 
Feels snappier. Obviously, still dog slow when I write to e.g. /home/user or /usr. But those are rare occasions since this is a server. I think I'll go with this for now and revisit if I run into problems.
Looks like something is still writing outside of those SSD drives. Even with /var on SSD and /tmp on tmpfs as per above I'm looking at python37 package struggling to extract. I thought extraction was done either to /tmp or to /var/tmp or smth similar. Was I wrong?
I dread the moment I decide to install lang/rust ... oh the horror

*EDIT* or is it that case of writing whatever's been expanded so far to /usr/whatever?
 
Increasingly looks like I'll have to bite the bullet and move everything to SSD. Since I can't boot from those PCIe NVMe drives my only option is to mountroot there. Is there a way to only create such fresh root (/) from e.g. live CD or something or do I simply dump/restore e.g. my USB thumbdrive root partition (as described in the handbook)?
 
ended up just doing the dump/restore to move root to SSD, then modified /boot/loader.conf to mountroot there. Worked the first try :). Ordered small NVMe SSD I'm going to use as a root. Also introduced indirection via GPT labels so I no longer need to touch my /etc/fstab. It mounts e.g. root via /dev/gpt/root and swap via /dev/gpt/swap etc. If ever I need to move stuff around I'll just need to make sure partitions with these labels are present.

For now I consider experiment over and case closed. Not what I expected but I've learnt a bunch.
Big thanks to Mjölnir for engaging in my silly experiment
 
This is kind of on-topic so ... I am attempting an install from a usb stick. I found some help on dd to copy disk1 DVD to a usb stick. and I succeeded first try! cool freebsd. then i tried the same usb stick on a slightly newer dell and it doesn't work. then i mounted the usb stick and looked at its contents and it all seems good, all the folders are there and boot files etc.

when i wrote the usb stick it was not mounted and when you boot from it
it cannot be mounted. so does that mean it never can get written on during the install? i am wondering if some writing to the usb stick happened that messed it up, but i tried booting from it on the same machine and it still works. maybe the target machine brands the installer usb stick in some way. I am using:

fbsd 13.0 release
the dell it works on has 4GB and a 150GB drive. the newer one it fails on has 8GB and 250BG drive.

thanks for any comments!
 
Back
Top