Finding what is writing to my zpool (dtrace perhaps?)

J

jkcarrol

I have a RAIDZ1 zpool consisting of three 1TB WD Black drives, and I'm seeing some periodic writes to the pool that I'd like to track down. It's usually a few hundred KB/s or up to 1-2 MB/s according to zpool iostat and I've had no luck watching top -m io -s .5 to look for an I/O intensive process. As far as I know, the zpool should be pretty much idle. So I'm just curious what these periodic writes are.

The zpool in question consists of the following mounts/ds:

Code: 
data                       514G  1.28T  58.6K  /data
data/home                 7.04G  1.28T  1.21G  /usr/home
data/home_video           31.5G  1.28T  31.5G  /data/home_video
data/jails                2.65G  1.28T  2.65G  /usr/jails
data/music                48.2G  1.28T  48.1G  /data/music
data/pictures             86.2G  1.28T  85.6G  /data/pictures
data/usr_local            5.79G  1.28T  4.82G  /usr/local
data/var                   887M  1.28T   369M  /var

The only processes/file systems that aren't entirely idle are the following:

  • data/home - my user's home directory. The only active processes with anything open are irssi inside a tmux session and a couple of eggdrops. None of the files open for write on here are writing more than a few bytes every few seconds. Certainly not hundreds of KB/s or MB/s
  • data/pictures - fed via web server for a gallery. Should be only reads (uploads are disabled except for admins).
  • data/usr_local - nothing writing here other than the occasional port update. certainly nothing writing here every few seconds
  • data/var - the usual logs/etc being written here, but nothing constituting MB/s of writes. I have smokeping running update two rrd files, but again we're talking bytes or 10's of KB, not MB/s.

I went ahead and built a kernel with proper dtrace support and I was hoping someone might know of a way to use it to see what encompasses these writes. This is my first time fiddling with dtrace, so I'm not yet sure if it's even possible with dtrace in 8.2-RELEASE.

Another piece of data is that usually the number of writes is about 100 or so, so it seems to be something that's fairly consistent in whatever it is writing.

Any help (dtrace or otherwise) in tracking down these writes would be greatly appreciated.

Thanks in advance! Here is a snapshot of zpool iostat to give you an idea of what I'm seeing:

Code: 
# zpool iostat data 1
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
data         772G  1.96T      6     21  67.8K   112K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    106      0   463K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    279      0  1.81M
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    130      0   507K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    103      0   254K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    105      0   335K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    108      0   475K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0    105      0   338K
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0      0
data         772G  1.96T      0      0      0   128K
data         772G  1.96T      0    133      0   628K
 
If you're brave enough to move to 8-STABLE and upgrade to ZFS v28 you could take a couple of snapshots and do a zfs diff.

You can also take a snapshot and run the following to find files modified. (Posted by Alt in the following thread about finding differences in ZFS filesystems - http://forums.freebsd.org/showthread.php?t=10883)

[cmd=]find /mnt/store1/ -newer /mnt/store1/.zfs/snapshot/yesterday -type f[/cmd]

I don't know anything about dtrace but no-one else has answered so I thought I'd throw a few ideas out there...
 
usdmatt said:
If you're brave enough to move to 8-STABLE and upgrade to ZFS v28 you could take a couple of snapshots and do a zfs diff.
Click to expand...

Excellent! I'm using mm's zfs v28 patch on 8.2-RELEASE so I will definitely try this. I've already got some daily snapshots, but obviously would need some taken closer to each other (and when my other machines' backups aren't running). Thanks for the idea though!
 
AndyUKG said:
Hi,

I think rwsnoop should do what you want, should be included in the DTraceToolkit port:

http://www.freshports.org/sysutils/DTraceToolkit/

Not tried it though, feel free to report back to enlighten me and anyone else reading ;)

cheers Andy.
Click to expand...

Thanks for the tip! I installed the port, but the sh-bang for the rwsnoop script is using ksh (which I lack) and it doesn't want to work properly with /bin/sh nor with /usr/local/bin/zsh. I'll fiddle with it later today and try using the ksh93 port, but thanks for the info. This definitely looks promising!
 
Yeah you will need ksh, just having a little look at the handbook where it states:

support for the Korn shell should be added. This is needed as the DTrace toolkit has several utilities written in ksh. Install the shells/ksh93. It is also possible to run these tools under shells/pdksh or shells/mksh.
Click to expand...

In case you didn't know dtrace support isn't in the default GENERIC kernel, so if you didn't already you will have to recompile with the relevant dtrace options enabled. Shame it's not in by default IMO, maybe there are reason for that...?

Andy.
 
AndyUKG said:
Yeah you will need ksh, just having a little look at the handbook where it states:
Click to expand...

Whoops, didn't actually read the section in the handbook on it (honestly, didn't know there was a chapter on it, sweet!), sorry. Why not make ksh93 a runtime dependency for the port? I guess I'll contact the maintainer with a patch. Seems like it'd make sense to install it or at least make it toggle-able with a knob. I'll play with this tonight, thanks again for everyone's input! :)


In case you didn't know dtrace support isn't in the default GENERIC kernel, so if you didn't already you will have to recompile with the relevant dtrace options enabled. Shame it's not in by default IMO, maybe there are reason for that...?

Andy.
Click to expand...

Right, I have already built a proper kernel and world per the DTrace wiki, and the examples I've tried seem to work fine. Thanks though!
 
You must log in or register to reply here.
Back
Top