Solved No space left on device

[Gumlokt]

When I run command pkg update on FreeBSD 11.0 I'm getting this error:

[srv]# pkg update
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01
Fetching packagesite.txz: 100%    6 MiB   2.0MB/s    00:03

/tmp: write failed, filesystem is full
pkg: archive_read_extract(extract error): No space left on device
pkg: No signature found
Unable to update repository FreeBSD
Error updating repositories!

That is, the system reports that there is no free space on device, while running command zpool list displays 344Gb of free space:

[srv]# zpool list
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank   460G   116G   344G         -    24%    25%  1.00x  ONLINE  -

What could be the error and how to fix it?

 

[acheron]

Your /tmp is too small.

 

[SirDice]

I'm guessing /tmp is a tmpfs(5) and not a regular filesystem. And, as acheron already noted, it's too small.

 

[Gumlokt]

I found out that there are 2 file systems have been mounted on /tmp:

# df -m
Filesystem        1M-blocks   Used  Avail Capacity  Mounted on
tank/ROOT/default    455682 118467 337214    26%    /
devfs                     0      0      0   100%    /dev
tank                 337214      0 337214     0%    /tank
tank/tmp             337215      0 337214     0%    /tmp
tank/usr/home        337214      0 337214     0%    /usr/home
tank/usr/ports       337394    179 337214     0%    /usr/ports
tank/usr/src         337648    433 337214     0%    /usr/src
tank/var/audit       337214      0 337214     0%    /var/audit
tank/var/crash       337214      0 337214     0%    /var/crash
tank/var/log         337216      1 337214     0%    /var/log
tank/var/mail        337214      0 337214     0%    /var/mail
tank/var/tmp         337214      0 337214     0%    /var/tmp
/dev/md0                 18      0     17     0%    /tmp

How is this possible?
And how to unmount /dev/md0 so it is no longer mounted automatically?

 

[SirDice]

Look at your /etc/fstab, it's probably being mounted from there.

 

[Gumlokt]

My fstab contains only 2 lines:
# cat /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/ada0p2 none swap sw 0 0
/dev/ada1p2 none swap sw 0 0


 

[SirDice]

Then it's something you've done by hand.

 

[Gumlokt]

# umount /dev/md0
umount: unmount of /tmp failed: Device busy


 

[SirDice]

There are applications running with files open on that filesystem.

 

[acheron]

Do you have

tmpmfs="AUTO"

in /etc/rc.conf?

 

[Gumlokt]

# cat /etc/rc.conf
# System was installed: 2017-01-06

hostname="server.my.local"
keymap="ru.koi8-r.kbd"

ifconfig_fxp0="inet 192.168.0.108 netmask 255.255.255.0"
ifconfig_sk0="inet 172.16.0.1 netmask 255.255.255.224"
defaultrouter="192.168.0.100"
gateway_enable="YES"

sshd_enable="YES"
powerd_enable="YES"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
zfs_enable="YES"


apache24_enable="YES"
squid_enable="YES"
samba_server_enable="YES"
mysql_enable="YES"


apcupsd_enable="YES"

pf_enable="YES"
pflog_enable="YES"
pflog_logfile="/var/log/pf/pf.log"

bsdstats_enable="YES"

 

[Gumlokt]

I just restarted my machine, and after that everything became fine:

# pkg update
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01
Fetching packagesite.txz: 100%    6 MiB   1.5MB/s    00:04
Processing entries: 100%
FreeBSD repository update completed. 26288 packages processed.
All repositories are up to date.

# df -m
Filesystem        1M-blocks   Used  Avail Capacity  Mounted on
tank/ROOT/default    455682 118530 337151    26%    /
devfs                     0      0      0   100%    /dev
tank                 337151      0 337151     0%    /tank
tank/tmp             337151      0 337151     0%    /tmp
tank/usr/home        337151      0 337151     0%    /usr/home
tank/usr/ports       337331    179 337151     0%    /usr/ports
tank/usr/src         337584    433 337151     0%    /usr/src
tank/var/audit       337151      0 337151     0%    /var/audit
tank/var/crash       337151      0 337151     0%    /var/crash
tank/var/log         337152      1 337151     0%    /var/log
tank/var/mail        337151      0 337151     0%    /var/mail
tank/var/tmp         337151      0 337151     0%    /var/tmp

But I still do not understand where device /dev came from???...

 

[SirDice]

But I still do not understand where device /dev came from???...

Like I said, you probably mounted it at some point by hand.

 

[Gumlokt]

No no... Manually I did not mount any devices.

 

[Gumlokt]

Thanks for the help. The issue has been resolved. Rebooting server solved the problem.

 

[phoenix]

But I still do not understand where device /dev came from???...

/dev is a pseudo-filesystem that is mounted automatically by the OS. Without this filesystem, you wouldn't be able to access any of the hardware in the system, as the device nodes for the harddrives, the keyboard, the mouse, the screens, the NICs, etc are all in there.

 

[jrronimo]

Sorry for the necro bump, but since I'm having the same problem, I wasn't sure if I should start a new thread or not.

I'm confident that I did something wrong setting up my server. In the past, rebooting it has fixed the problem. I rebooted last night and am out of space again, so now it's time to actually *learn* what I did wrong and fix it, haha.

First, the symptom:

$ sudo pkg upgrade
sudo: unable to open /var/run/sudo/ts/jrr: No space left on device

Now, some info:

$ zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zjrr  13.6T  13.2T   436G        -         -    38%    96%  1.00x  ONLINE  -

$ df -m
Filesystem        1M-blocks    Used Avail Capacity  Mounted on
zjrr/ROOT/default   6756918 6756918     0   100%    /
devfs                     0       0     0   100%    /dev
linprocfs                 0       0     0   100%    /usr/compat/linux/proc
linsysfs                  0       0     0   100%    /usr/compat/linux/sys
tmpfs                 16974       0 16974     0%    /usr/compat/linux/dev/shm
fdescfs                   0       0     0   100%    /dev/fd
zjrr/tmp                  5       5     0   100%    /tmp
zjrr/usr/home       1424240 1424240     0   100%    /usr/home
zjrr/usr/ports         7743    7743     0   100%    /usr/ports
zjrr/usr/src            960     960     0   100%    /usr/src
zjrr/var/audit            0       0     0   100%    /var/audit
zjrr/var/crash          640     640     0   100%    /var/crash
zjrr/var/log             15      15     0   100%    /var/log
zjrr/var/mail             4       4     0   100%    /var/mail
zjrr/var/tmp            116     116     0   100%    /var/tmp
zjrr                      0       0     0   100%    /zjrr

$ less /etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/da0p2              none    swap    sw              0       0
/dev/da1p2              none    swap    sw              0       0
/dev/da2p2              none    swap    sw              0       0
/dev/da3p2              none    swap    sw              0       0
/dev/da4p2              none    swap    sw              0       0
linprocfs   /compat/linux/proc  linprocfs       rw      0       0
linsysfs    /compat/linux/sys   linsysfs        rw      0       0
tmpfs                   /compat/linux/dev/shm  tmpfs    rw,mode=1777    0       0
fdesc                   /dev/fd fdescfs                 rw              0       0

I do use Linux emulation to run some game servers. Generally, I would consider myself pretty new to FreeBSD and Linux though. What have I done wrong, or what further information can I chase down that might clue me in?

Thanks in advance...

 

[ralphbsz]

You simply ran out of disk space on "file system" zjrr (I know, it's ZFS, and the real ZFS word for it is zpool, but in the output of the df command it is a set of file systems). That file system has about 13-14 TB of disk space (judging by the zpool line above), and it is now 100% full (judging by the df command output).

You need to either find more disks or more disk space on the existing disks and make that zpool larger, or you need to delete some data. The biggest source of the problem seems to be the /usr/home file system, but that's not a surprise, that's where the user's home directories are.

I would start by looking for very large files; typically, those are the source of disk space usage. Here is a starting point (do this as root): find /usr/home -type f -size +1G -print0 | xargs -0 -n 100 ls -lF. This will give you a listing of all files larger than 1gigabyte, in batches of 100. Then figure out how many of these files you really need. If the output from this is still too large, then either set the limit higher than 1G and start with larger files, or sort the output so the largest files are listed first, by appending | sort -rn -k 5,5 | more to that command.

 

[jrronimo]

You simply ran out of disk space on "file system" zjrr (I know, it's ZFS, and the real ZFS word for it is zpool, but in the output of the df command it is a set of file systems). That file system has about 13-14 TB of disk space (judging by the zpool line above), and it is now 100% full (judging by the df command output).

You need to either find more disks or more disk space on the existing disks and make that zpool larger, or you need to delete some data. The biggest source of the problem seems to be the /usr/home file system, but that's not a surprise, that's where the user's home directories are.

I would start by looking for very large files; typically, those are the source of disk space usage. Here is a starting point (do this as root): find /usr/home -type f -size +1G -print0 | xargs -0 -n 100 ls -lF. This will give you a listing of all files larger than 1gigabyte, in batches of 100. Then figure out how many of these files you really need. If the output from this is still too large, then either set the limit higher than 1G and start with larger files, or sort the output so the largest files are listed first, by appending | sort -rn -k 5,5 | more to that command.

Oh wow, that's a much easier solution than I expected. It threw me off since zpool list is showing 436G free. Getting rid of big files is easy: I use this box as my NAS and store backups of another box on it -- three backups of 1.5 TB each are easy targets.

Thank you so much!

 

[Spartrekus]

When I run command pkg update on FreeBSD 11.0 I'm getting this error:

[srv]# pkg update
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01
Fetching packagesite.txz: 100%    6 MiB   2.0MB/s    00:03

/tmp: write failed, filesystem is full
pkg: archive_read_extract(extract error): No space left on device
pkg: No signature found
Unable to update repository FreeBSD
Error updating repositories!

That is, the system reports that there is no free space on device, while running command zpool list displays 344Gb of free space:

[srv]# zpool list
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank   460G   116G   344G         -    24%    25%  1.00x  ONLINE  -

What could be the error and how to fix it?

Pkg works usually well considering space.

Likely there is something to do with the above install.

Recommendation rm -rf dir allows to delete a directory.
mc allows to check the disk, with a cool file manager.
df -h allows to look at space of disks.
du -hs directory allows to study space, occupied by a dir.

 

[pyret]

du -h /filesystem | awk '$1~/G/'

A lot shorter, simpler and faster than find.

 

[Spartrekus]

Concerning find, it is ok sure, but sometimes, it is good to use hands and C solutions.

FIND:
A simple C opendir makes it easier even, and very extremely portable. C is preferable as a bash/sh/ash... *sh script.
If one day FreeBSD is taken by *Linux* industry, C will remain.

///////////////
void listdir(const char *name, int indent, const char *searchitem )
{
    DIR *dir;
    struct dirent *entry;

    if (!(dir = opendir(name)))
        return;

    while ((entry = readdir(dir)) != NULL)
    {
        if (entry->d_type == DT_DIR)
    {
            char path[1024];

            if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
                continue;

            snprintf( path, sizeof(path), "%s/%s", name, entry->d_name);

            listdir( path, indent + 2, searchitem );
        }
    else
    {
        if ( strstr( entry->d_name , searchitem ) != 0 )
       
              printf("%s/%s\n", name , entry->d_name );
       
        }
    }
    closedir(dir);
}