Solved Any way to fix seemingly random scsi (cam) drives order

This started in this thread and ended with me reflashing my PERC H710 to IT mode, so now I can see all drives on my Dell PowerEdge R720 directly and in fact they appear as I would expect da0, da1, ..., da7, followed by USB drives if those are attached. What I didn't expect was for the drives to be assigned those numbers seemingly randomly. If you ever saw the front of that server it offers 8x 3'5 bays so one would reasonably expect those drives to pop in /dev hopefully in that order "top to bottom, left to right". Except in my case drive in bay-0 turned into /dev/da4. How is this order determined? Is there any way to get reasonable ordering? Or put differently is there a way to have /dev/da0 to always point to whatever is in bay-0, ..., /dev/da7 to always point to whatever drive is in bay-7.

I should probably mention that every single one of those drives was a RAID-0 to work around the whole RAID thing unnecessary for task at hand. So, now I keep seeing errors like:
Code:
GEOM: diskid ...: the secondary GPT header is not in the last LBA
I doubt that would cause this and I bet this is something I could fix with some gpart magic. Just thought I'd mention.

Is this ask unreasonable or pointless? Those drives are hot swappable, but the system should probably not shuffle them around say if I yank one out.

*EDIT* oh no, it also just occurred to me I can no longer blink with the LED to show me which disk is in which bay. I can no longer do mfiutil locate ... doesn't look like camcontrol has anything similar - that's so sad.
 
Well, by default, drive numbers are assigned dynamically, so the numbering is not guaranteed to be deterministic.

You can hardcode the device numbers by creating entries in /boot/device.hints. For example, the following will assign da0 to target 0 on scbus 0; see the cam(4) manual page for details:
Code:
hint.da.0.at="scbus0"
hint.da.0.target="0"
Alternatively, you can just ignore the /dev/da* devices, and instead access the drives by their disk identification. This is the name printed by diskinfo -v da0 in the line labeled “disk ident”. For example, I have these:
Code:
$ diskinfo -v nda0 nda1 | grep ident
        2019283185D0    # Disk ident.
        S462NF0K816531M # Disk ident.
So, these disks can be accessed as /dev/diskid/2019283185D0 and /dev/diskid/S462NF0K816531M (you can create symlinks if you think the names are too long) instead of /dev/nda0 and /dev/nda1, respectively. The advantage of this approach is that the names won’t change, even if you put the drives into different slots, or even attach them to a different controller.

Note that the names in /dev/diskid will disappear if you mount (or otherwise use) the regular /dev/da* names.

You can also create arbitrary labels for partitions with gpart(8) and use them for mounting. For example, this is an excerpt from my /etc/fstab:
Code:
#Device                 Mountpt FStype  Options                 Dump    Pass#
#------                 ------- ------  -------                 ----    -----
/dev/gpt/SSD_SAMS_ROOT  /       ufs     rw,noatime              1       1
/dev/gpt/HGST_3T_SWAP   none    swap    sw                      0       0
/dev/gpt/SSD_SAMS_VAR   /var    ufs     rw,noatime,nosuid       2       2
/dev/gpt/SSD_SAMS_USR   /usr    ufs     rw,noatime              2       2
/dev/gpt/SSD_SAMS_HOME  /home   ufs     rw,noatime,nosuid       2       2
/dev/gpt/HGST_3T_HALDE  /halde  ufs     rw,noatime,nosuid       2       2
/dev/gpt/SSD_CRUC_DATA  /data   ufs     rw,noatime,nosuid       2       2
...
Again, the advantage is that that the system will still work if I shuffle the drives around, add a new one, whatever. The kernel’s GEOM subsystem will always disover the partition labels and mount the correct file systems, no matter how the drive is connected and what number the device got when it was probed. I can even remove my second NVMe SSD, for example, put it into an external USB enclosure and connect it – The GPT labels will still be found, and the file systems will be mounted without any changes, even though the actual device changes from /dev/nda1 to /dev/da0.

Just some food for thoughts. ;)
 
Thank you again, olli@...
This post is wiki-worthy imho!
If I had write rights there, I'd add it to the wiki right now...

Very very nice to know there is a disk-by-id feature in FreeBSD, too 👍
What I don't understand however, is why the disk-by-id entry disappears after mounting etc.
On Linux, I don't recall such a behaviour with the contents of /dev/disk/by-id ...

Is it legal to use the /dev/diskid/... in /etc/fstab?

Regarding the OP's problem, if I remember correctly, the problem associating the drive with the drive bay can be solved via finding the correct scbus address (usually shown in dmesg).
So, if you know which physical scbus/id is associated for every drive bay, it doesn't matter much which da it is.

A related question:
A thing I like very much in OpenBSD is that they assign some high da number to USB sticks/drives.
This way USB sticks do not get a drive id inbetween the fixed-mounted drives, so that drive order does not get too chaotic.
Is something similar possible on FreeBSD, for example assigning USB drives something da10+?
Maybe any hints that can do this?
 
So, these disks can be accessed as /dev/diskid/2019283185D0 and /dev/diskid/S462NF0K816531M

I did something like this but I did not pay attention to what it did.. Long story short, the disk serial number was too long and it got truncated.
Ended up with many drives with the same label because the serial numbers were sequential. The last 4 digits were truncated.

To get congruent left to right numbering you may have to juggle around drive bay to controller cables.
With disk labels it don't matter.

Be advised there are many ways to label drives. gpt labels, glabel, UFS labels.

I think it is worth looking at the FreeBSD memstick installer to see how fstab works there. It is a good labeling example.
 
You can hardcode the device numbers by creating entries in /boot/device.hints. For example, the following will assign da0 to target 0 on scbus 0; see the cam(4) manual page for details:
Code:
hint.da.0.at="scbus0"
hint.da.0.target="0"
I believe this would solve my problem at hand. I'll try and report back.

The rest are very handy tricks and I was aware of e.g. labeling and used it in the past. Very useful but not applicable in this case where essentially I have a level of indirection - it isn't the identity of the disk that matters but the identity of the bay (or enclosure) defines what drive I address. Hope it makes sense. Put differently those disks are hot swappable and are to be treated as such, but I do need a way to identify the one I need to swap out and replace. I think we can reasonably expect them to appear in order on the scbus so fingers crossed your very first proposal should work.

Thank you very much for taking the time to educate. These are all helpful notes and appears others found them useful, too. Always a win :)
 
[…] it isn't the identity of the disk that matters but the identity of the bay (or enclosure) defines what drive I address. Hope it makes sense. Put differently those disks are hot swappable and are to be treated as such, but I do need a way to identify the one I need to swap out and replace.
Is that a standard SCSI enclosure? In that case, you should be able to identify the drive bays with sesutil(8). Have you tried this? (If you don’t use the GENERIC kernel, be sure you have device ses in your kernel configuration.)
 
Is it legal to use the /dev/diskid/... in /etc/fstab?
Basically, yes. But the names are rather long and unwieldy for the purpose of handling file systems.
My recommendation is to use gpart(8) for labeling the partitions, and then use the /dev/gpt/<label> labels in your /etc/fstab.
See my example above that shows an excerpt from my /etc/fstab.

A related question:
A thing I like very much in OpenBSD is that they assign some high da number to USB sticks/drives.
This way USB sticks do not get a drive id inbetween the fixed-mounted drives, so that drive order does not get too chaotic.
Is something similar possible on FreeBSD, for example assigning USB drives something da10+?
Maybe any hints that can do this?
Yes, you can do that via /boot/device.hints, as explained above. See the cam(4) manual page for details.

Basically you can do it like this:
  • Attach a USB storage device (flash stick, external HDD / SSD, whatever) to a USB port.
  • Look for the attachment messages in dmesg(8). You should see a line like this:
      da0 at umass-sim0 bus 0 scbus12 target 0 lun 0
    In this case the USB controllers get “virtual” SCSI busses assigned, starting at scbus12. It might be a different number on your machine.
  • Add entries to /boot/device.hints containing scbus* lines starting at the one you found out above. (I think the target and LUN are always 0, so these can be omitted, but I haven’t actually tried this):
      hint.da.10.at="scbus12"
      hint.da.11.at="scbus13"
      hint.da.12.at="scbus14"
      hint.da.13.at="scbus15"
  • Reboot for the changes to take effect.
Now, the first USB storage device that you plug in will get /dev/da10 assigned, the second one will be /dev/da11, and so on. Note that these numbers are not bound to particular USB ports, i.e. it does not matter into which USB port you plug your first device.

Alternatively, again, you might want to consider using GPT labels for your USB sticks.
For example, let’s assume you have a red 8 GB USB stick that is currently connected as /dev/da0, and it has a single partition. The following command will assign the label “RED8G” to that partition:
  gpart modify -i 1 -l "RED8G" da0
Then you can use commands like newfs /dev/gpt/RED8G and mount /dev/gpt/RED8G /mnt, and so on. This always works the same, no matter what USB port you use, no matter how many USB sticks are already attached, and even if you plug it into a different PC. You might want to write the label on the stick with a Sharpie or similar.
 
Is that a standard SCSI enclosure? In that case, you should be able to identify the drive bays with sesutil(8). Have you tried this? (If you don’t use the GENERIC kernel, be sure you have device ses in your kernel configuration.)
I don't know what "standard enclosure" is tbh. They are drive bays front of the server. I just tried sesutil but it doesn't look promising. I've no idea what ses(4)(). Be trying hinting next.

Code:
$ sudo sesutil map        
ses0:
        Enclosure Name: AHCI SGPIO Enclosure 2.00
        Enclosure ID: 3061686369656d30
        Element 0, Type: Array Device Slot
                Status: Unsupported (0x00 0x00 0x00 0x00)
                Description: Drive Slots
        Element 1, Type: Array Device Slot
                Status: Not Installed (0x05 0x00 0x00 0x00)
                Description: Slot 00
        Element 2, Type: Array Device Slot
                Status: Not Installed (0x05 0x00 0x00 0x00)
                Description: Slot 01
        Element 3, Type: Array Device Slot
                Status: Not Installed (0x05 0x00 0x00 0x00)
                Description: Slot 02
        Element 4, Type: Array Device Slot
                Status: Not Installed (0x05 0x00 0x00 0x00)
                Description: Slot 03
        Element 5, Type: Array Device Slot
                Status: Not Installed (0x05 0x00 0x00 0x00)
                Description: Slot 04
        Element 6, Type: Array Device Slot
                Status: Not Installed (0x05 0x00 0x00 0x00)
                Description: Slot 05

[user@dell720 ~]$ sudo sesutil show
ses0: <AHCI SGPIO Enclosure 2.00>; ID: 3061686369656d30
Desc     Dev     Model                     Ident                Size/Status
Slot 00  -       -                         -                    Not Installed
Slot 01  -       -                         -                    Not Installed
Slot 02  -       -                         -                    Not Installed
Slot 03  -       -                         -                    Not Installed
Slot 04  -       -                         -                    Not Installed
Slot 05  -       -                         -                    Not Installed
 
Well, by default, drive numbers are assigned dynamically, so the numbering is not guaranteed to be deterministic.

You can hardcode the device numbers by creating entries in /boot/device.hints. For example, the following will assign da0 to target 0 on scbus 0; see the cam(4) manual page for details:
Code:
hint.da.0.at="scbus0"
hint.da.0.target="0"

was about to try that hinting, but now that I'm looking at camcontrol devlist -v I'm not sure what I'm supposed to do. I expected it to be shuffled. What this output tells me is that somehow bay-0 drive is actually bu0:target4? Wat? Does that mean I need to somehow physically probe every single one of those bays e.g. by ejecting them in order somehow to figure what targets they are? That's ... crazy. I don't get it. When that machine was under RAID controller that mfi identified them correctly somehow, right? Well, wait. Maybe they are in correct order now and were in incorrect order back then. ATM I tell which disk is what by its contents. So maybe FreeBSD is right. I could verify that if I had a way to blink those LEDs for each bay. For all I know it maybe that da0 now is in fact bay-0 and was incorrect back then. I really want to blink those LEDs

Code:
sudo camcontrol devlist -v
scbus0 on mps0 bus 0:
<SEAGATE ST2000NM0001 0003>        at scbus0 target 0 lun 0 (da0,pass0)
<SEAGATE ST2000NM0001 0003>        at scbus0 target 1 lun 0 (da1,pass1)
<SEAGATE ST2000NM0001 0003>        at scbus0 target 2 lun 0 (da2,pass2)
<SEAGATE ST2000NM0001 0003>        at scbus0 target 3 lun 0 (da3,pass3)
<SEAGATE ST2000NM0001 0003>        at scbus0 target 4 lun 0 (da4,pass4)
<SEAGATE ST2000NM0001 0003>        at scbus0 target 5 lun 0 (da5,pass5)
<SEAGATE ST2000NM0001 0003>        at scbus0 target 6 lun 0 (da6,pass6)
<SEAGATE ST2000NM0001 0003>        at scbus0 target 7 lun 0 (da7,pass7)
 
Does that mean I need to somehow physically probe every single one of those bays e.g. by ejecting them in order somehow to figure what targets they are?
well I just camcontrol stop da4 and it stopped ... bay-0 drive. So I guess it was correct back then and somehow incorrect now. Just kill me. I guess I can write those hints by stopping those drives in order ... just mercy shoot me now. Why ...
 
I don't know what "standard enclosure" is tbh. They are drive bays front of the server. I just tried sesutil but it doesn't look promising. I've no idea what ses(4). Be trying hinting next.
Well, by “standard” I meant something that is recognized by the ses(4) driver. That seems to be the case, according to the output that you posted.
I have to admit that I don’t know how it’s supposed to look exactly, because I don’t have a SES enclosure myself at home.

Maybe you can just try various combinations of the sesutil(8) command, and see if they change any of the LEDs on your drive bays.
For example:
Code:
sesutil locate all on
sesutil fault all on
sesutil locate da0 on
sesutil fault da0 on
sesutil locate -u /dev/ses0 1 on
sesutil fault -u /dev/ses0 1 on
If the LEDs are already on, replace “on” with “off”.
 
Does that mean I need to somehow physically probe every single one of those bays e.g. by ejecting them in order somehow to figure what targets they are?
Most LSI controllers have a feature in the Firmware/BIOS screen. You can blink the individual drive sled lights to figure out what is what.
So I would look at your PERC BIOS and see if there is an option there for flashing drive lights. It really helps to know how the controller is seeing the drives. You would think Dell have it wired Left to Right but I would use the controller tools to figure out how the drive mapping works at the controller level..

Mine is 12drivesX2 Supermicro 2U and it took a while to get it right.

Using /boot/device.hints works but it feels brutish. 24 drives will give you some work to do.
Labels are the proper way in my opinion.
 
I think at this point it is fair to say big thank you olli@
I finally solved the case of mysterious drive order but olli@ deserves most of the glory.

In case anyone else runs into similar puzzling behavior allow me to sum up both the problem and its solution.
Problem: reflashed Dell PERC H710 RAID controller into IT aka JBOD mode and having done so noticed that suddenly drives in the 8 front bays of my Dell PowerEdge R720 appear in seemingly random order in that SCSI device da4 is actually the drive in bay-0 etc. While not the end of the world it'd quickly become a nightmare if you actually want to hot-swap those drives often and not just when one of them fails.

Solution: First you can always check driver mapping (daN) to SCSIbus:target with camcontrol devlist -v. You can "remap" those drives and restore order to your universe by adding proper mapping to /boot/device.hints as well explained in cam (4)(). Here's what I ended up with:
Code:
# bay-0
hint.da.0.at="scbus0"
hint.da.0.target="4"

# bay-1
hint.da.1.at="scbus0"
hint.da.1.target="5"

# bay-2
hint.da.2.at="scbus0"
hint.da.2.target="6"

# bay-3
hint.da.3.at="scbus0"
hint.da.3.target="7"

# bay-4
hint.da.4.at="scbus0"
hint.da.4.target="0"

# bay-5
hint.da.5.at="scbus0"
hint.da.5.target="1"

# bay-6
hint.da.6.at="scbus0"
hint.da.6.target="2"

# bay-7
hint.da.7.at="scbus0"
hint.da.7.target="3"
How does one go about determining said order? Just stop them one by one and see which one starts blinking then stops spinning e.g. camcontrol stop da4.

Is this a proper solution. Well. Look carefully at the above mapping and you'll spot a pattern. Effectively, we have targets 0-3 (first 4 drives) swapped with targets 4-7 (last 4 drives). If you ever looked under the hood of your server this should give you pause. Yep. Chances are good this can be trivially solved by ... swapping the two SAS cables that run from your card to the two ports on the backplane. Sadly, I wasn't even able to confirm that would work (although I'm like 99% confident it would) because m..f... Dell supplies its own f..ed up "double" that splits into SFF-8087 but one is much shorter than the other and there's no way it can reach the other port. So, we end up with the above ugly hinting. If I were to use a decent LSI PCIe card to drive those disks I'd simply swap the cables.
 
I should've probably mentioned one unpleasantness with the above "hint" solution. It only ensures you get what you expect after boot. If you end up hot-swapping one of those drives, the "newly inserted" one will simply become daN where N is the next available integer that's not been used yet. This means you can't really rely on the stuff you put in your /etc/fstab that's specific for those drives. It isn't just that driver number! There appears to be a 1-to-1 correspondence between that N and the scsi bus target number. Sigh

Basically, so far as I can tell there's no obvious way to refer to contents of bay-0 ... bay-N (by way of some indirection). At least I've not found any.
 
Mhh... doesn't there exist a hint that causes <n> to increment no matter whether a device actually has been found there?
In other words, a hint that creates a fixed relation between da<n> and <scsiID>, and so reserves da<n> for exclusive usage on <scsiID<n>?
 
Mhh... doesn't there exist a hint that causes <n> to increment no matter whether a device actually has been found there?
In other words, a hint that creates a fixed relation between da<n> and <scsiID>, and so reserves da<n> for exclusive usage on <scsiID<n>?
cam(4)() does mention wired vs counted. But from their example I don't understand how to establish bi-directional binding there. That is: my hints establish one part of necessary binding. There has to be a way to go the other direction but for that you need a way to refer to "physical wires" if you know what I mean. I'm not even sure I understand entirely what they mean exactly when they say "wired". When I wrote my hints I thought scbus0 and targetN meant physical connections, but I no longer think that's true. There's some tacit knowledge that's not in the manual. Basically, if someone more knowledgeable could explain to me what's going on there, then great. Right now, those paragraphs remain somewhat cryptic to me.
I came to hate magic. Sadly, too many things in computing are unnecessarily magical. Often simply for lack of explanation - hard work that.

*EDIT* note how you (in passing) bring up <scsiID<n>> but you never actually define what those are and how ID and that n get established :) Problem with that is that it creates a feeling of "undrestanding" that's in my case is not quite there: scsi - sounds familiar; ID is typical and in general we all understand what it means, n - is an integer. Upon closer inspection you realize you've no clue what the other person really means.
 
Looking at the cam() manpage, I read this:
Code:
This assigns CAM bus 0 to the bus 1 instance on ahc1.  Peripheral drivers
     can be wired to a specific    bus, target, and lun as    so:

       hint.da.0.at="scbus0"
       hint.da.0.target="0"
      [B] hint.da.0.unit="0"[/B]

     This assigns da0 to target    0, unit    (lun) 0    of scbus 0.  [B]Omitting the tar-
     get or unit hints will instruct CAM to treat them as wildcards and    use
     the first respective counted instances.[/B]  These examples can be combined
     together to allow a peripheral device to be wired to any particular con-
     troller, bus, target, and/or unit instance.

     This also works with nvme(4) drives as well.

       hint.nvme.4.at="pci7:0:0"
       hint.scbus.10.at="nvme4"
       hint.nda.10.at="scbus10"
       hint.nda.10.target="1"
       hint.nda.10.unit="12"
       hint.nda.11.at="scbus10"
       hint.nda.11.target="1"
       hint.nda.11.unit="2"
If I understand the wording correctly. both the target and unit hint seem necessary to use the 'wired' instead of the 'counted' address.
As far as I have seen in past, LUN 0 is commonly used with spinning disk drives, so I think setting "unit" to 0 should not cause problems. No idea whether this is still true with NVMe...
 
Omitting the LUN isn't a big deal in this case IIUC, it'll get assigned the first unit which in this case will be 0 every time?

That said I've just run into something I didn't expect at all. Shows I really don't understand what I'm doing and have no clue what the cam manual talks about. With the above mentioned hints, I hot-swapped all 8 drives yesterday. Console reported that each newly replaced drive got a higher device number. Old disks were numbered da0-da7, my USB stick was da8. New drives that replaced the old ones got assigned da9-da16, which in itself was a bit puzzling since I assumed that with the above hinting I "wired" them. Here's the puzzling bit. After reboot. These drives were assigned da8-da15. WTF. I must at this point admit I don't understand the "wiring" or hinting or anything at all. What the hell do those hints actually do then? What is "wiring"? That behavior only makes sense if there is some internal table that kept actual disk identity somehow? I am lost.

Obviously that messed up the mountroot, so I had to figure where the USB drive was. And of course after that it dropped me into single user because ... it couldn't stat devices that obviously didn't exist any more e.g. /dev/da5p1 etc. Just mercy shoot me.
 
Omitting the LUN isn't a big deal in this case IIUC, it'll get assigned the first unit which in this case will be 0 every time?
Yes, but it is the wording in the man page that caught my attention.
Omitting the target or unit hints will instruct CAM to treat them as wildcards and use the first respective counted instances.
When I read this sentence in another way, I understand it like this:
Setting both the target and unit hints will instruct CAM to use the respective wired instances.
Maybe I understand wrong. But if I understand correctly, the things you observed, e.g. using the ever-incrementing counted instances were exactly what to expect when setting only one and not both of target or unit...
 
From mps(4)() which i believe is my adapter driver:
The driver can map devices discovered by the adapter so that target IDs
corresponding to a specific device persist across resets and reboots. In
some cases it is possible for devices to lose their mapped IDs due to un-
expected behavior from certain hardware, such as some types of enclo-
sures. To overcome this problem, a tunable is provided that will force
the driver to map devices using the Phy number associated with the de-
vice. This feature is not recommended if the topology includes multiple
enclosures/expanders. If multiple enclosures/expanders are present in
the topology, Phy numbers are repeated, causing all devices at these Phy
numbers except the first device to fail enumeration. To control this
feature for all adapters, set the

hw.mps.use_phy_num
Trying it in 3 .. 2 .. 1 .... and it fixed it. Good question is why "driver device map" logic failed for my card?
 
One (hopefully last) mystery remaining for me (not counting that failed "device mapping" failure) is the enclosure that shows up in my camcontrol devlist -v:
Code:
sudo camcontrol devlist -v
scbus0 on mps0 bus 0:
<SEAGATE ST8000NM0075 E003>        at scbus0 target 0 lun 0 (da4,pass0)
<SEAGATE ST8000NM0075 E003>        at scbus0 target 1 lun 0 (da5,pass1)
<SEAGATE ST8000NM0075 E003>        at scbus0 target 2 lun 0 (da6,pass2)
<SEAGATE ST8000NM0075 E003>        at scbus0 target 3 lun 0 (da7,pass3)
<SEAGATE ST8000NM0075 E003>        at scbus0 target 4 lun 0 (da0,pass4)
<SEAGATE ST8000NM0075 E003>        at scbus0 target 5 lun 0 (da1,pass5)
<SEAGATE ST8000NM0075 E003>        at scbus0 target 6 lun 0 (da2,pass6)
<SEAGATE ST8000NM0075 E003>        at scbus0 target 7 lun 0 (da3,pass7)
<>                                 at scbus0 target -1 lun ffffffff ()
scbus1 on ahcich0 bus 0:
<>                                 at scbus1 target -1 lun ffffffff ()
scbus2 on ahcich1 bus 0:
<>                                 at scbus2 target -1 lun ffffffff ()
scbus3 on ahcich2 bus 0:
<>                                 at scbus3 target -1 lun ffffffff ()
scbus4 on ahcich3 bus 0:
<>                                 at scbus4 target -1 lun ffffffff ()
scbus5 on ahcich4 bus 0:
<>                                 at scbus5 target -1 lun ffffffff ()
scbus6 on ahcich5 bus 0:
<>                                 at scbus6 target -1 lun ffffffff ()
scbus7 on ahciem0 bus 0:
<AHCI SGPIO Enclosure 2.00 0001>   at scbus7 target 0 lun 0 (ses0,pass8)
<>                                 at scbus7 target -1 lun ffffffff ()
So, my 8 drives in the front bay are listed first. As expected. But then, there's this AHCI SGPIO Enclosure with ses0 driver, which I kind of expected to be the enclosure for my drives which would make it possible to refer to drives via their respective bays via sesutil or whatever, but that did nothing at all for me. ses(4)() doesn't offer much in terms of info sadly. Mentions something about SES_ENABLE_PASSTHROUGH that may help for some devices, but assumes I understand how to set this option. Pretty sure I need to recompile something somewhere with this var set or smth. /usr/share/examples/ses is a bunch of Makefiles and sources. Again, what am I supposed to do?

Also, what are those scbusN devices??? Mysteries abound really
 
Back
Top