/var: filesystem full, but plenty of space

ak147va

ak147va

Hi,

Well I have an issue that I really feel like stems from overlooking something. I am fairly new to FreeBSD, but have a fair amount of experience with Red Hat based Operating Systems.

I have set up a file server in my house to store pictures and videos. It has been in place for almost a year and working great. Just recently, I had a drive fail in my ZFS pool. I replaced the drive, rebooted the server, and another drive was giving connection issues. I think I may have knocked the SATA cable loose while switching the first drive. Anyways, I got ZFS to recognize the drives, and after a few tries, everything resilvered fine. About a week later, I was having connectivity issues with my file server. I checked the log files and found this:

Code: 
May 23 09:35:58 filer kernel: pid 680 (syslogd), uid 0 inumber 47150 on /var: filesystem full
May 23 09:35:58 filer kernel: pid 19154 (smbd), uid 0 inumber 515210 on /var: filesystem full
May 23 09:35:58 filer kernel: pid 680 (syslogd), uid 0 inumber 47150 on /var: filesystem full
May 23 09:35:58 filer kernel: pid 19154 (smbd), uid 0 inumber 515210 on /var: filesystem full
May 23 09:35:58 filer kernel: pid 680 (syslogd), uid 0 inumber 47150 on /var: filesystem full
May 23 09:35:58 filer kernel: pid 19154 (smbd), uid 0 inumber 47114 on /var: filesystem full
May 23 09:35:58 filer kernel: pid 680 (syslogd), uid 0 inumber 47150 on /var: filesystem full
May 23 09:35:58 filer kernel: pid 19154 (smbd), uid 0 inumber 47114 on /var: filesystem full


I thought it would be an easy fix, but it doesn't look like /var partition is actually full:

Code: 
filer# df -hi
Filesystem        Size Used Avail Capacity iused ifree %iused  Mounted on
/dev/ad14s1a      989M 172M  738M  19%     1.9k  139k    1%    /
devfs             1.0K 1.0K    0B 100%     0      0    100%    /dev
/dev/ad14s1e      989M 68K   910M   0%     9      141k   0%    /tmp
/dev/ad14s1f      214G 324M  197G   0%     14k    29M    0%    /usr
/dev/ad14s1d      5.7G 98M   5.2G   2%     24k    754k   3%    /var
storage           1.0T 37K   1.0T   0%     10     2.2G   0%    /storage
storage/apps      1.0T 28K   1.0T   0%     4      2.2G   0%    /storage/apps
storage/documents 1.0T 15G   1.0T   1%     14k    2.2G   0%    /storage/documents
storage/downloads 1.0T 8.0G  1.0T   1%     14     2.2G   0%    /storage/downloads
storage/videos    1.3T 293G  1.0T  22%     281    2.2G   0%    /storage/videos
storage/pictures  1.0T 13G   1.0T   1%     1.3k   2.2G   0%    /storage/pictures
storage/backups   1.5T 453G  1.0T  30%     1.7k   2.2G   0%    /storage/backups
Code: 
filer# zfs list storage
NAME USED AVAIL REFER MOUNTPOINT
storage 782G 1.03T 37.3K /storage

I've been searching for days trying to find a solution, but everything I find is people with a completely full /var partition. I'm thinking the hard drive the FreeBSD is on is going bad, but I cannot find any evidence to suggest it is.

In short, I am just stuck and need some fresh eyes. Any suggestions would be much appreciated.
 
Usually that's due to a log file or other large file that has been deleted but not closed, so the space is still occupied. Restarting the application that has the file open will close it. Offhand, I can't think of an easy way to see which application is doing that, but there probably is one. If Apache is running, it's a good candidate.
 
Thank you for the replies. Rebooting the server should close the file shouldn't it? I rebooted the server, then tried to add the fuser package, however the result was filesystem full:

Code: 
filer# pkg_add -r fuser
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-8.2-release/Latest/fuser.tbz...
/var: write failed, filesystem is full
+MTREE_DIRS: Write failed

/var: write failed, filesystem is full

I do not have Apache running, but maybe it is Samba causing this?

Code: 
filer# ps aux
USER    PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
root     11 200.0  0.0     0    16  ??  RL    9:44AM   4:26.83 [idle]
root      0  0.0  0.0     0   416  ??  DLs   9:44AM   0:00.15 [kernel]
root      1  0.0  0.0  2912   484  ??  ILs   9:44AM   0:00.01 /sbin/init --
root      2  0.0  0.0     0     8  ??  DL    9:44AM   0:00.02 [g_event]
root      3  0.0  0.0     0     8  ??  DL    9:44AM   0:00.25 [g_up]
root      4  0.0  0.0     0     8  ??  DL    9:44AM   0:00.10 [g_down]
root      5  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [fdc0]
root      6  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [sctp_iterator]
root      7  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [xpt_thrd]
root      8  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [pagedaemon]
root      9  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [vmdaemon]
root     10  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [audit]
root     12  0.0  0.0     0   144  ??  WL    9:44AM   0:06.61 [intr]
root     13  0.0  0.0     0     8  ??  DL    9:44AM   0:00.02 [yarrow]
root     14  0.0  0.0     0   160  ??  DL    9:44AM   0:00.00 [usb]
root     15  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [acpi_thermal]
root     16  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [pagezero]
root     17  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [bufdaemon]
root     18  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [vnlru]
root     19  0.0  0.0     0     8  ??  DL    9:44AM   0:36.20 [syncer]
root     20  0.0  0.0     0     8  ??  DL    9:44AM   1:10.07 [softdepflush]
root     21  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [flowcleaner]
root     40  0.0  0.0     0    36  ??  DL    9:44AM   0:00.01 [zfskern]
root    123  0.0  0.0  1544   868  ??  Is    9:44AM   0:00.00 adjkerntz -i
root    531  0.0  0.1  3292  1312  ??  Is    9:44AM   0:00.00 dhclient: vr0 [priv] (dhclient)
_dhcp   565  0.0  0.1  3292  1460  ??  Is    9:44AM   0:00.00 dhclient: vr0 (dhclient)
root    566  0.0  0.0  1888   584  ??  Is    9:44AM   0:00.00 /sbin/devd
root    698  0.0  0.1  3352  1360  ??  Is    9:44AM   0:02.23 /usr/sbin/syslogd -s
root    834  0.0  0.1  3384  1448  ??  Is    9:45AM   0:00.01 /usr/sbin/rpcbind
root    855  0.0  0.1  3352  1400  ??  Is    9:45AM   0:00.00 /usr/sbin/mountd -r /etc/exports /etc/zfs/exports
root    857  0.0  0.1  3292  1368  ??  Is    9:45AM   0:00.03 nfsd: master (nfsd)
root    858  0.0  0.1  3292  1228  ??  S     9:45AM   0:00.00 nfsd: server (nfsd)
root    900  0.0  0.2  9368  4204  ??  Ss    9:45AM   0:00.31 /usr/local/sbin/nmbd -D -s /usr/local/etc/smb.conf
root    906  0.0  0.3 14220  6608  ??  Is    9:45AM   0:00.02 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
root   1039  0.0  0.3 14220  6568  ??  I     9:45AM   0:00.00 /usr/local/sbin/smbd -D -s /usr/local/etc/smb.conf
root   1047  0.0  0.2  6712  3776  ??  Is    9:45AM   0:00.00 /usr/sbin/sshd
root   1055  0.0  0.2  6092  3532  ??  Ss    9:45AM   0:00.01 sendmail: accepting connections (sendmail)
smmsp  1059  0.0  0.2  6092  3404  ??  Is    9:45AM   0:00.00 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail)
root   1067  0.0  0.1  3380  1368  ??  Is    9:45AM   0:00.00 /usr/sbin/cron -s
root   1150  0.0  0.2  9436  4488  ??  Is    9:45AM   0:00.07 sshd: adam [priv] (sshd)
adam   1153  0.0  0.2  9436  4500  ??  S     9:45AM   0:00.02 sshd: adam@pts/0 (sshd)
root   1139  0.0  0.1  3352  1124  v0  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv0
root   1140  0.0  0.1  3352  1124  v1  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv1
root   1141  0.0  0.1  3352  1124  v2  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv2
root   1142  0.0  0.1  3352  1124  v3  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv3
root   1143  0.0  0.1  3352  1124  v4  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv4
root   1144  0.0  0.1  3352  1124  v5  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv5
root   1145  0.0  0.1  3352  1124  v6  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv6
root   1146  0.0  0.1  3352  1124  v7  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv7
adam   1154  0.0  0.1  3632  1612   0  Is    9:45AM   0:00.01 -sh (sh)
root   1155  0.0  0.1  3812  1736   0  I     9:45AM   0:00.01 su
root   1156  0.0  0.1  5656  2484   0  S     9:45AM   0:00.03 _su (csh)
root   1170  0.0  0.1  3432  1208   0  R+    9:47AM   0:00.00 ps aux

I deleted a bunch of small log files, and was able to quickly install the fuser package. Below is the result of command:

Code: 
filer# fuser -cu /var/log
/var/log:   565rjcw(_dhcp)   566w(root)   698wa(root)   834(root)   855w(root)  1055cw(root)
  1059cw(smmsp)  1067cw(root)  1156c(root)  1228c(root)
Code: 
filer# ps aux
USER    PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
root     11 200.0  0.0     0    16  ??  RL    9:44AM  18:28.46 [idle]
root      0  0.0  0.0     0   416  ??  DLs   9:44AM   0:00.15 [kernel]
root      1  0.0  0.0  2912   484  ??  ILs   9:44AM   0:00.01 /sbin/init --
root      2  0.0  0.0     0     8  ??  DL    9:44AM   0:00.05 [g_event]
root      3  0.0  0.0     0     8  ??  DL    9:44AM   0:00.29 [g_up]
root      4  0.0  0.0     0     8  ??  DL    9:44AM   0:00.14 [g_down]
root      5  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [fdc0]
root      6  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [sctp_iterator]
root      7  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [xpt_thrd]
root      8  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [pagedaemon]
root      9  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [vmdaemon]
root     10  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [audit]
root     12  0.0  0.0     0   144  ??  WL    9:44AM   0:11.91 [intr]
root     13  0.0  0.0     0     8  ??  DL    9:44AM   0:00.04 [yarrow]
root     14  0.0  0.0     0   160  ??  DL    9:44AM   0:00.01 [usb]
root     15  0.0  0.0     0     8  ??  DL    9:44AM   0:00.01 [acpi_thermal]
root     16  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [pagezero]
root     17  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [bufdaemon]
root     18  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [vnlru]
root     19  0.0  0.0     0     8  ??  DL    9:44AM   0:59.60 [syncer]
root     20  0.0  0.0     0     8  ??  DL    9:44AM   1:53.68 [softdepflush]
root     21  0.0  0.0     0     8  ??  DL    9:44AM   0:00.00 [flowcleaner]
root     40  0.0  0.0     0    36  ??  DL    9:44AM   0:00.02 [zfskern]
root    123  0.0  0.0  1544   868  ??  Is    9:44AM   0:00.00 adjkerntz -i
root    531  0.0  0.1  3292  1312  ??  Is    9:44AM   0:00.00 dhclient: vr0 [priv] (dhclient)
_dhcp   565  0.0  0.1  3292  1460  ??  Is    9:44AM   0:00.00 dhclient: vr0 (dhclient)
root    566  0.0  0.0  1888   584  ??  Is    9:44AM   0:00.00 /sbin/devd
root    698  0.0  0.1  3352  1360  ??  Ss    9:44AM   0:04.00 /usr/sbin/syslogd -s
root    834  0.0  0.1  3384  1448  ??  Ss    9:45AM   0:00.01 /usr/sbin/rpcbind
root    855  0.0  0.1  3352  1400  ??  Is    9:45AM   0:00.00 /usr/sbin/mountd -r /etc/exports /etc/zfs/exports
root    857  0.0  0.1  3292  1368  ??  Is    9:45AM   0:00.03 nfsd: master (nfsd)
root    858  0.0  0.1  3292  1228  ??  S     9:45AM   0:00.00 nfsd: server (nfsd)
root   1047  0.0  0.2  6712  3776  ??  Is    9:45AM   0:00.00 /usr/sbin/sshd
root   1055  0.0  0.2  6092  3532  ??  Ss    9:45AM   0:00.01 sendmail: accepting connections (sendmail)
smmsp  1059  0.0  0.2  6092  3404  ??  Is    9:45AM   0:00.00 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail)
root   1067  0.0  0.1  3380  1368  ??  Ss    9:45AM   0:00.00 /usr/sbin/cron -s
root   1150  0.0  0.2  9436  4488  ??  Is    9:45AM   0:00.07 sshd: adam [priv] (sshd)
adam   1153  0.0  0.2  9436  4500  ??  S     9:45AM   0:00.07 sshd: adam@pts/0 (sshd)
root   1139  0.0  0.1  3352  1124  v0  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv0
root   1140  0.0  0.1  3352  1124  v1  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv1
root   1141  0.0  0.1  3352  1124  v2  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv2
root   1142  0.0  0.1  3352  1124  v3  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv3
root   1143  0.0  0.1  3352  1124  v4  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv4
root   1144  0.0  0.1  3352  1124  v5  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv5
root   1145  0.0  0.1  3352  1124  v6  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv6
root   1146  0.0  0.1  3352  1124  v7  Is+   9:45AM   0:00.00 /usr/libexec/getty Pc ttyv7
adam   1154  0.0  0.1  3632  1612   0  Is    9:45AM   0:00.01 -sh (sh)
root   1155  0.0  0.1  3812  1736   0  I     9:45AM   0:00.01 su
root   1156  0.0  0.1  5656  2504   0  S     9:45AM   0:00.07 _su (csh)
root   1257  0.0  0.1  3432  1208   0  R+    9:55AM   0:00.00 ps aux
 
The fuser(1) utility is already in the base system, no need to install the package. I can't spot anything unusual in the process list. Boot into single user mode and run the following command, do not mount /var, this is meant to be used on unmounted filesystem:

# fsck_ffs -f /dev/ad14s1d
 
Thank you for the suggestion. I have run the disk check, fixing all errors. Now /var is showing as full:

Code: 
filer# df -h
Filesystem           Size    Used   Avail Capacity  Mounted on
/dev/ad14s1a         989M    172M    738M    19%    /
devfs                1.0K    1.0K      0B   100%    /dev
/dev/ad14s1e         989M     12K    910M     0%    /tmp
/dev/ad14s1f         214G    324M    197G     0%    /usr
/dev/ad14s1d         5.7G    5.7G   -466M   109%    /var

All the space is now being taken up by /var/lost+found/. Should I just delete all the files in here?

Code: 
filer# ls -lah
total 5907964
-r--------   1 root      operator   5.9G May  2 21:08 #000004
-r--------   1 root      operator   5.9G May 10 21:55 #000005
-rw-r--r--   1 root      wheel        0B May  2 22:00 #047152
-rw-r--r--   1 root      wheel      192K May 10 21:38 #047155
-rw-r--r--   1 root      wheel        0B May  5 10:47 #047159
-rw-r--r--   1 root      wheel        0B May  6 11:42 #047177
-rw-r--r--   1 root      wheel      5.6G May 10 21:38 #047178
-rw-r--r--   1 root      wheel        0B May  5 04:33 #047179
-rw-r--r--   1 root      wheel        0B May  6 05:29 #047180
-rw-r--r--   1 root      wheel        0B May  3 08:51 #047184
-rw-r--r--   1 root      wheel        0B May  3 15:02 #047190
-rw-r--r--   1 root      wheel        0B May  5 17:04 #047191
-rw-r--r--   1 root      wheel        0B May  4 03:35 #047193
-rw-r--r--   1 root      wheel        0B May  3 21:19 #047194
-rw-r--r--   1 root      wheel        0B May  4 09:52 #047196
-rw-r--r--   1 root      wheel        0B May  4 16:04 #047198
-rw-r--r--   1 root      wheel        0B May  4 22:21 #047200
-rw-r--r--   1 root      wheel      6.0K May  3 02:34 #047211
-rw-r--r--   1 root      wheel        0B May  5 23:18 #047212
-rw-r--r--   1 root      wheel      231B May  2 20:59 #494601
-r--------   1 operator  operator   2.0K Apr 29 09:44 #494603
-r--------   1 operator  operator   2.0K Apr 29 10:33 #494607
-r--------   1 operator  operator   2.0K Apr 29 10:00 #494608
-r--------   1 operator  operator   2.0K Apr 29 09:33 #494610
-r--------   1 operator  operator   2.0K Apr 29 10:22 #494611
-r--------   1 operator  operator   2.0K May  2 21:00 #494662
-r--------   1 operator  operator   2.0K Apr 29 09:55 #494667
-r--------   1 operator  operator   2.0K Apr 29 10:11 #494668
-rw-rw----   1 smmsp     smmsp       18B May  5 08:00 #565258
-rw-rw----   1 smmsp     smmsp      755B May  5 08:00 #565259
-rw-------   1 root      daemon     1.1K May  6 03:58 #565263
-rw-------   1 root      daemon     1.1K May  6 03:28 #565264

Thanks again for all of your help.
 
The files (and directories) in /some/mountpoint/lost+found/ are generally those that fsck or a similar utility couldn't "connect back up" when an inconsistency was found in the filesystem. They may contain valuable information, or they may just be file fragments of apparent gibberish. How you decide to deal with them has to do with what you keep on the filesystem in question, if it can be replaced, and how much time and patience you have to deal with detective work.
 
The only valuable information I have is in my ZFS pool. I have removed all the lost+found files and everything seems to be running just fine for the time being. Thanks!
 
You must log in or register to reply here.
Back
Top