Solved No space left on device

G

Gumlokt

Member

Reaction score: 4
Messages: 30

When I run command pkg update on FreeBSD 11.0 I'm getting this error:
Code: 
[srv]# pkg update
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01
Fetching packagesite.txz: 100%    6 MiB   2.0MB/s    00:03

/tmp: write failed, filesystem is full
pkg: archive_read_extract(extract error): No space left on device
pkg: No signature found
Unable to update repository FreeBSD
Error updating repositories!

That is, the system reports that there is no free space on device, while running command zpool list displays 344Gb of free space:
Code: 
[srv]# zpool list
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank   460G   116G   344G         -    24%    25%  1.00x  ONLINE  -

What could be the error and how to fix it?
 
SirDice

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,678
Messages: 39,278

I'm guessing /tmp is a tmpfs(5) and not a regular filesystem. And, as acheron already noted, it's too small.
 
OP
G

Gumlokt

Member

Reaction score: 4
Messages: 30

I found out that there are 2 file systems have been mounted on /tmp:
Code: 
# df -m
Filesystem        1M-blocks   Used  Avail Capacity  Mounted on
tank/ROOT/default    455682 118467 337214    26%    /
devfs                     0      0      0   100%    /dev
tank                 337214      0 337214     0%    /tank
tank/tmp             337215      0 337214     0%    /tmp
tank/usr/home        337214      0 337214     0%    /usr/home
tank/usr/ports       337394    179 337214     0%    /usr/ports
tank/usr/src         337648    433 337214     0%    /usr/src
tank/var/audit       337214      0 337214     0%    /var/audit
tank/var/crash       337214      0 337214     0%    /var/crash
tank/var/log         337216      1 337214     0%    /var/log
tank/var/mail        337214      0 337214     0%    /var/mail
tank/var/tmp         337214      0 337214     0%    /var/tmp
/dev/md0                 18      0     17     0%    /tmp

How is this possible?
And how to unmount /dev/md0 so it is no longer mounted automatically?
 
SirDice

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,678
Messages: 39,278

Look at your /etc/fstab, it's probably being mounted from there.
 
OP
G

Gumlokt

Member

Reaction score: 4
Messages: 30

My fstab contains only 2 lines:
# cat /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/ada0p2 none swap sw 0 0
/dev/ada1p2 none swap sw 0 0
 
SirDice

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,678
Messages: 39,278

Then it's something you've done by hand.
 
OP
G

Gumlokt

Member

Reaction score: 4
Messages: 30

# umount /dev/md0
umount: unmount of /tmp failed: Device busy
 
SirDice

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,678
Messages: 39,278

There are applications running with files open on that filesystem.
 
OP
G

Gumlokt

Member

Reaction score: 4
Messages: 30

Code: 
# cat /etc/rc.conf
# System was installed: 2017-01-06

hostname="server.my.local"
keymap="ru.koi8-r.kbd"

ifconfig_fxp0="inet 192.168.0.108 netmask 255.255.255.0"
ifconfig_sk0="inet 172.16.0.1 netmask 255.255.255.224"
defaultrouter="192.168.0.100"
gateway_enable="YES"

sshd_enable="YES"
powerd_enable="YES"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
zfs_enable="YES"


apache24_enable="YES"
squid_enable="YES"
samba_server_enable="YES"
mysql_enable="YES"


apcupsd_enable="YES"

pf_enable="YES"
pflog_enable="YES"
pflog_logfile="/var/log/pf/pf.log"

bsdstats_enable="YES"
 
OP
G

Gumlokt

Member

Reaction score: 4
Messages: 30

I just restarted my machine, and after that everything became fine:
Code: 
# pkg update
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01
Fetching packagesite.txz: 100%    6 MiB   1.5MB/s    00:04
Processing entries: 100%
FreeBSD repository update completed. 26288 packages processed.
All repositories are up to date.

Code: 
# df -m
Filesystem        1M-blocks   Used  Avail Capacity  Mounted on
tank/ROOT/default    455682 118530 337151    26%    /
devfs                     0      0      0   100%    /dev
tank                 337151      0 337151     0%    /tank
tank/tmp             337151      0 337151     0%    /tmp
tank/usr/home        337151      0 337151     0%    /usr/home
tank/usr/ports       337331    179 337151     0%    /usr/ports
tank/usr/src         337584    433 337151     0%    /usr/src
tank/var/audit       337151      0 337151     0%    /var/audit
tank/var/crash       337151      0 337151     0%    /var/crash
tank/var/log         337152      1 337151     0%    /var/log
tank/var/mail        337151      0 337151     0%    /var/mail
tank/var/tmp         337151      0 337151     0%    /var/tmp

But I still do not understand where device /dev came from???...
 
OP
G

Gumlokt

Member

Reaction score: 4
Messages: 30

Thanks for the help. The issue has been resolved. Rebooting server solved the problem.
 
phoenix

phoenix

Administrator
Staff member
Administrator
Moderator

Reaction score: 1,290
Messages: 4,099

Gumlokt said:
But I still do not understand where device /dev came from???...
Click to expand...

/dev is a pseudo-filesystem that is mounted automatically by the OS. Without this filesystem, you wouldn't be able to access any of the hardware in the system, as the device nodes for the harddrives, the keyboard, the mouse, the screens, the NICs, etc are all in there. :)
 
jrronimo

jrronimo

New Member

Reaction score: 2
Messages: 16

Sorry for the necro bump, but since I'm having the same problem, I wasn't sure if I should start a new thread or not.

I'm confident that I did something wrong setting up my server. In the past, rebooting it has fixed the problem. I rebooted last night and am out of space again, so now it's time to actually *learn* what I did wrong and fix it, haha.

First, the symptom:
Code: 
$ sudo pkg upgrade
sudo: unable to open /var/run/sudo/ts/jrr: No space left on device

Now, some info:
Code: 
$ zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zjrr  13.6T  13.2T   436G        -         -    38%    96%  1.00x  ONLINE  -

Code: 
$ df -m
Filesystem        1M-blocks    Used Avail Capacity  Mounted on
zjrr/ROOT/default   6756918 6756918     0   100%    /
devfs                     0       0     0   100%    /dev
linprocfs                 0       0     0   100%    /usr/compat/linux/proc
linsysfs                  0       0     0   100%    /usr/compat/linux/sys
tmpfs                 16974       0 16974     0%    /usr/compat/linux/dev/shm
fdescfs                   0       0     0   100%    /dev/fd
zjrr/tmp                  5       5     0   100%    /tmp
zjrr/usr/home       1424240 1424240     0   100%    /usr/home
zjrr/usr/ports         7743    7743     0   100%    /usr/ports
zjrr/usr/src            960     960     0   100%    /usr/src
zjrr/var/audit            0       0     0   100%    /var/audit
zjrr/var/crash          640     640     0   100%    /var/crash
zjrr/var/log             15      15     0   100%    /var/log
zjrr/var/mail             4       4     0   100%    /var/mail
zjrr/var/tmp            116     116     0   100%    /var/tmp
zjrr                      0       0     0   100%    /zjrr

Code: 
$ less /etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/da0p2              none    swap    sw              0       0
/dev/da1p2              none    swap    sw              0       0
/dev/da2p2              none    swap    sw              0       0
/dev/da3p2              none    swap    sw              0       0
/dev/da4p2              none    swap    sw              0       0
linprocfs   /compat/linux/proc  linprocfs       rw      0       0
linsysfs    /compat/linux/sys   linsysfs        rw      0       0
tmpfs                   /compat/linux/dev/shm  tmpfs    rw,mode=1777    0       0
fdesc                   /dev/fd fdescfs                 rw              0       0


I do use Linux emulation to run some game servers. Generally, I would consider myself pretty new to FreeBSD and Linux though. What have I done wrong, or what further information can I chase down that might clue me in?

Thanks in advance...
 
R

ralphbsz

Son of Beastie

Reaction score: 2,389
Messages: 3,277

You simply ran out of disk space on "file system" zjrr (I know, it's ZFS, and the real ZFS word for it is zpool, but in the output of the df command it is a set of file systems). That file system has about 13-14 TB of disk space (judging by the zpool line above), and it is now 100% full (judging by the df command output).

You need to either find more disks or more disk space on the existing disks and make that zpool larger, or you need to delete some data. The biggest source of the problem seems to be the /usr/home file system, but that's not a surprise, that's where the user's home directories are.

I would start by looking for very large files; typically, those are the source of disk space usage. Here is a starting point (do this as root): find /usr/home -type f -size +1G -print0 | xargs -0 -n 100 ls -lF. This will give you a listing of all files larger than 1gigabyte, in batches of 100. Then figure out how many of these files you really need. If the output from this is still too large, then either set the limit higher than 1G and start with larger files, or sort the output so the largest files are listed first, by appending | sort -rn -k 5,5 | more to that command.
 
jrronimo

jrronimo

New Member

Reaction score: 2
Messages: 16

ralphbsz said:
You simply ran out of disk space on "file system" zjrr (I know, it's ZFS, and the real ZFS word for it is zpool, but in the output of the df command it is a set of file systems). That file system has about 13-14 TB of disk space (judging by the zpool line above), and it is now 100% full (judging by the df command output).

You need to either find more disks or more disk space on the existing disks and make that zpool larger, or you need to delete some data. The biggest source of the problem seems to be the /usr/home file system, but that's not a surprise, that's where the user's home directories are.

I would start by looking for very large files; typically, those are the source of disk space usage. Here is a starting point (do this as root): find /usr/home -type f -size +1G -print0 | xargs -0 -n 100 ls -lF. This will give you a listing of all files larger than 1gigabyte, in batches of 100. Then figure out how many of these files you really need. If the output from this is still too large, then either set the limit higher than 1G and start with larger files, or sort the output so the largest files are listed first, by appending | sort -rn -k 5,5 | more to that command.
Click to expand...

Oh wow, that's a much easier solution than I expected. It threw me off since zpool list is showing 436G free. Getting rid of big files is easy: I use this box as my NAS and store backups of another box on it -- three backups of 1.5 TB each are easy targets. :)

Thank you so much!
 
Spartrekus

Spartrekus

Daemon

Reaction score: 163
Messages: 1,150

Gumlokt said:
When I run command pkg update on FreeBSD 11.0 I'm getting this error:
Code: 
[srv]# pkg update
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD load error: access repo file(/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01
Fetching packagesite.txz: 100%    6 MiB   2.0MB/s    00:03

/tmp: write failed, filesystem is full
pkg: archive_read_extract(extract error): No space left on device
pkg: No signature found
Unable to update repository FreeBSD
Error updating repositories!

That is, the system reports that there is no free space on device, while running command zpool list displays 344Gb of free space:
Code: 
[srv]# zpool list
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank   460G   116G   344G         -    24%    25%  1.00x  ONLINE  -

What could be the error and how to fix it?
Click to expand...

Pkg works usually well considering space.

Likely there is something to do with the above install.

Recommendation rm -rf dir allows to delete a directory.
mc allows to check the disk, with a cool file manager.
df -h allows to look at space of disks.
du -hs directory allows to study space, occupied by a dir.
 
Spartrekus

Spartrekus

Daemon

Reaction score: 163
Messages: 1,150

Concerning find, it is ok sure, but sometimes, it is good to use hands and C solutions.

FIND:
A simple C opendir makes it easier even, and very extremely portable. C is preferable as a bash/sh/ash... *sh script.
If one day FreeBSD is taken by *Linux* industry, C will remain. ;)

Code: 
///////////////
void listdir(const char *name, int indent, const char *searchitem )
{
    DIR *dir;
    struct dirent *entry;

    if (!(dir = opendir(name)))
        return;

    while ((entry = readdir(dir)) != NULL)
    {
        if (entry->d_type == DT_DIR)
    {
            char path[1024];

            if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0)
                continue;

            snprintf( path, sizeof(path), "%s/%s", name, entry->d_name);

            listdir( path, indent + 2, searchitem );
        }
    else
    {
        if ( strstr( entry->d_name , searchitem ) != 0 )
       
              printf("%s/%s\n", name , entry->d_name );
       
        }
    }
    closedir(dir);
}
 
M

mohamedchebbi

New Member


Messages: 4

Hi,

i got the same problem in freebsd 12 and i just addes tmpfs to /etc/fstab and then reboot whith tmpfs="AUTO" in /etc/rc.conf.
after reboot the size of tmpfs is rised from 20M to 11G.

Code: 
root@freebsd:~ # df -h
Filesystem      Size    Used   Avail Capacity  Mounted on
/dev/ada0s1a    190G    8.4G    166G     5%    /
devfs           1.0K    1.0K      0B   100%    /dev
procfs          4.0K    4.0K      0B   100%    /proc
fdescfs         1.0K    1.0K      0B   100%    /dev/fd
tmpfs            11G    476K     11G     0%    /tmp
 
J

jagg2k5

New Member


Messages: 2

Hello community,

I am newbie for freeBSD I was checking this forum and I found this is my problem now.
Here is my situation:

root@desktop:/ # pkg update
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100% 944 B 0.9kB/s 00:01
Fetching packagesite.txz: 100% 6 MiB 151.3kB/s 00:43
pkg: archive_read_extract(extract error): No space left on device
pkg: No signature found
Unable to update repository FreeBSD
Error updating repositories!
root@desktop:/ # uname -a
FreeBSD desktop 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC amd64
root@desktop:/ # df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ada1p2 104G 8.4G 88G 9% /
devfs 1.0K 1.0K 0B 100% /dev
tmpfs 20M 1.9M 18M 9% /tmp
tmpfs 1.0G 4.0K 1.0G 0% /mnt

Please help me to understand what's going on otherwise I will have to spend half day re-installing and that is not funny.
Thanks in advanced.
 
J

jagg2k5

New Member


Messages: 2

Hello community,

I am newbie for FreeBSD I was checking this forum and I found this is my problem now.
Here is my situation:
Code: 
root@desktop:/ # pkg update
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01   
Fetching packagesite.txz: 100%    6 MiB 151.3kB/s    00:43   
pkg: archive_read_extract(extract error): No space left on device
pkg: No signature found
Unable to update repository FreeBSD
Error updating repositories!
root@desktop:/ # uname -a
FreeBSD desktop 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
root@desktop:/ # df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ada1p2    104G    8.4G     88G     9%    /
devfs          1.0K    1.0K      0B   100%    /dev
tmpfs           20M    1.9M     18M     9%    /tmp
tmpfs          1.0G    4.0K    1.0G     0%    /mnt
Please help me to understand what's going on otherwise I will have to spend half day re-installing and that is not funny.
Thanks in advanced.


ups... I found the way ...! Never mind it.

I just simply add this line to my fstab
Code: 
tmpfs        /tmp        tmpfs    rw,size=469762048    0    0
and remove the tmpfs 1G from /mnt
and now here it is:
Code: 
root@desktop:/ # df -k
Filesystem  1024-blocks    Used    Avail Capacity  Mounted on
/dev/ada1p2   109478652 8887856 91832504     9%    /
devfs                 1       1        0   100%    /dev
tmpfs            458752     828   457924     0%    /tmp
root@desktop:/ #
it is working with the double I had initially.
:)
 
You must log in or register to reply here.
Top