Solved missing devices in /dev

Greetings,

I recently set up a "FreeBSD 11.1 release" box. I'm now in the process of switching out a couple of disks in the zpool to bigger ones. When i started this process I noticed that my /dev is pretty much empty (including output of sudo tree /dev below). After fiddling about a bunch i've also noticed that i'm getting error messages when it is trying to mount my swap space. I'm using zroot.

I haven't really done all that much after installing the system, but sadly i can't recall everything exactly. I've set up a bunch of jails using iocage and built my own kernel with vimage support, but that's it as far as i can recall.

If i reboot into single user mode everything seems to be ok, so i figure it is some setting i've managed to mess up, but I no longer have any clue where to look, or even what to look for. Right now i can't even log into the computer using the keyboard and monitor (ssh in works fine, all jails and services work fine), and i'm not quite sure when this issue started. I suspect it is related since the second to last line displayed on the monitor before it stops is "Starting background file system checks in 60 seconds".

I have actually managed to perform the upgrades of my drives using the single user mode, and i don't really use the keyboard and monitor, so in practice i have no problem. Everything seems fine when i run zpool status as well. However, I can't help but wonder what the issue is. Any help is appreciated.


output from sudo tree /dev:
Code:
    /dev
    ├── fd
    │   ├── 0
    │   ├── 1
    │   ├── 2
    │   └── 3
    ├── log -> /var/run/log
    ├── null
    ├── pts
    │   └── 0
    ├── random
    ├── stderr -> fd/2
    ├── stdin -> fd/0
    ├── stdout -> fd/1
    ├── urandom -> random
    ├── zero
    └── zfs

    2 directories, 14 files
output from camcontrol devlist:
Code:
camcontrol: couldn't open /dev/xpt0: No such file or directory

swap-related error message displayed while booting:

Code:
swapon: /dev/ada0p2: No such file or directory

Edit:
Added formatting tags
 
Impossible to comment, you don't give us anything to work with. Can you share the output of gpart show?
Code:
$ gpart show
=>        40  5860533088  ada0  GPT  (2.7T)
          40      409600     1  efi  (200M)
      409640        2008        - free -  (1.0M)
      411648     4194304     2  freebsd-swap  (2.0G)
     4605952  5855927176     3  freebsd-zfs  (2.7T)

=>        40  5860533088  ada1  GPT  (2.7T)
          40      409600     1  efi  (200M)
      409640        2008        - free -  (1.0M)
      411648     4194304     2  freebsd-swap  (2.0G)
     4605952  5855927176     3  freebsd-zfs  (2.7T)

=>         40  15628053088  ada2  GPT  (7.3T)
           40       409600     1  efi  (200M)
       409640         2008        - free -  (1.0M)
       411648      4194304     2  freebsd-swap  (2.0G)
      4605952  15623446528     3  freebsd-zfs  (7.3T)
  15628052480          648        - free -  (324K)

=>         40  15628053088  ada3  GPT  (7.3T)
           40       409600     1  efi  (200M)
       409640         2008        - free -  (1.0M)
       411648      4194304     2  freebsd-swap  (2.0G)
      4605952  15623446528     3  freebsd-zfs  (7.3T)
  15628052480          648        - free -  (324K)

=>        40  5860533088  ada4  GPT  (2.7T)
          40      409600     1  efi  (200M)
      409640        2008        - free -  (1.0M)
      411648     4194304     2  freebsd-swap  (2.0G)
     4605952  5855926272     3  freebsd-zfs  (2.7T)
  5860532224         904        - free -  (452K)

=>        40  5860533088  ada5  GPT  (2.7T)
          40      409600     1  efi  (200M)
      409640        2008        - free -  (1.0M)
      411648     4194304     2  freebsd-swap  (2.0G)
     4605952  5855926272     3  freebsd-zfs  (2.7T)
  5860532224         904        - free -  (452K)
 
Well, re-reading the above and comparing it with your shared paste (thanks for that) I agree that something isn't right. It almost looks to me as if devd isn't started for some reason.

Try this: sysrc devd_enable, what does that tell you? If it says anything other than YES then you got a problem on your hands. If that is the case then I'm wondering if you ever edited /etc/defaults/rc.conf manually.

Assuming it didn't say YES, try this: # service devd onestart. This should enforce a start of the devd service, and after that /dev should also become more populated.

If it did say YES earlier try: # service devd status and see what that tells you. If it is running normally then I'm starting to think that you might have edited /etc/devd.conf somehow.

Hope that this can shed some light on this.
 
Well, re-reading the above and comparing it with your shared paste (thanks for that) I agree that something isn't right. It almost looks to me as if devd isn't started for some reason.

Try this: sysrc devd_enable, what does that tell you? If it says anything other than YES then you got a problem on your hands. If that is the case then I'm wondering if you ever edited /etc/defaults/rc.conf manually.

Assuming it didn't say YES, try this: # service devd onestart. This should enforce a start of the devd service, and after that /dev should also become more populated.

If it did say YES earlier try: # service devd status and see what that tells you. If it is running normally then I'm starting to think that you might have edited /etc/devd.conf somehow.

Hope that this can shed some light on this.

Thanks for the tips! Sadly they didn't yield anything useful.

sysrc devd_enable
Code:
devd_enable: YES
service devd status
Code:
devd is running as pid 1474.

I also hade a quick look into /etc/defaults/rc.conf and /etc/devd.conf and I'm sure i've never consciously edited them.
 
Not to worry, one step at a time. So now we know it's enabled, running, but..

What does: mount | grep dev yet you? It should give you something like:

Code:
peter@zefiris:/etc/defaults# mount | grep dev
devfs on /dev (devfs, local, multilabel)
And I'm guessing that you're not seeing this, meaning that devfs probably isn't mounted on /dev/, would that be correct?

Which brings me to: did you customize anything? For example, do you run a customized kernel or the default GENERIC kernel?

It's a peculiar issue because when I look at your OP again I'm noticing that basically everything from /etc/defaults/devfs.rules seems to be in there (which is the minimalistic setup). Hmm, does /etc/devd.conf actually exist?
 
Or maybe there is need for a cam rescan? You are not switching disks while powered up, are you?
Also, I had once disks disappear in my file server, which I could trace back to dodgy cables. Throw-in-the-box stuff should be evaluated if it is best thrown-in-the-bin (recycling, of course).
 
Or maybe there is need for a cam rescan? You are not switching disks while powered up, are you?
Also, I had once disks disappear in my file server, which I could trace back to dodgy cables. Throw-in-the-box stuff should be evaluated if it is best thrown-in-the-bin (recycling, of course).

No, not doing anything special with the hardware while it's running, at the most i unplugged the keyboard a few times.

Regarding dodgy cables; i suppose, if so it will be sorted soon. Was hoping to clean up the mess inside so waiting for some new sata-cables in the mail. Although since everything shows up in single user mode i kinda doubt it is hardware related.

Code:
$ sudo camcontrol rescan all
camcontrol: error opening transport layer device /dev/xpt0
camcontrol: /dev/xpt0: No such file or directory
 
Contents of /etc/rc.conf in case it is relevant, but to my newbie eyes it doesn't look like anything special.

Code:
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
clear_tmp_enable="YES"
keymap="swedish.iso.kbd"

sshd_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"

ifconfig_igb0="DHCP"

# firewall
firewall_enable="YES"
firewall_type="open"

# stuff for vnet
devfs_system_ruleset="localrules"

# NFS
rpcbind_enable="YES"
nfs_server_enable="YES"
mountd_enable="YES"
mountd_flags="-r"


# jails
iocage_enable="YES"

# bridge stuff for vnet and jails
# set up two bridge interfaces for iocage
cloned_interfaces="bridge0 bridge1"

# plumb interface igb0 into bridge0
ifconfig_bridge0="addm igb0 SYNCDHCP"
ifconfig_igb0="up"

Edit:
Hmm, going to read up on what the contents of /etc/devfs.ruleset does. Added it due to some guide for iocage recommending it, but right now i can't recall what each row does. However, from the variable names it sounds like it could be relevant.

Code:
[localrules=4]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path zfs unhide
add path 'tun*' unhide

## IOCAGE -- Add DHCP to ruleset 4
[devfsrules_jail_dhcp=5]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path zfs unhide
add path 'bpf*' unhide
 
Eureka!

Commenting out the devfs rules in rc.conf made everything ok! Going to go over it and figure out what does what, what i actually need, and what was wrong :)

Appreciate the help from everyone!

Edit:

Oh damn, got a bunch of replies that didn't even show up in my browser. Sorry for not reading them, didn't see them until now
 
Back
Top