'usem' State in `top` (dtrace help?)

Hello!

I'm working on a problem and I don't know how to find out more about it!

I'm having an application that is hanging at a certain point on my production system and in top it is listed as usem (screenshot attached). If it is helpful, the processing that is hanging is part of a multiprocessing script. In terms of system, I'm on 12.0-RELEASE-p3.

As far as I can tell, this "event on which the process waits" (from the top man page), usem stands for "unlink semaphore", but I can't really find out anything more about the event itself and why it is happening.

Can anyone offer any advice? I've tried a few base dtrace one-liners and dwatch commands from this page, but I haven't been successful in finding any more useful information on the issue. I'd ideally like to be able to identify the reason why the process doesn't die.

Thanks!
 

Attachments

  • top-state.png
    top-state.png
    49.3 KB · Views: 201
Have you tried running truss? Also if the process is not dying you could take a core dump using gcore and then use gdb to check what was doing at that time.
 
You can peek at where process is stuck in the kernel with procstat(1): procstat -kk <pid>. Also take look at other information procstat can provide, like a list of file descriptors and virtual memory mappings.

You can peek with truss(1), as well, but it won't output anything, until syscall returns: truss -p <pid>.

What does this script do?

PS: A bit brave to run ZFS with just 1 Gig of RAM; is it necessary? Can it be replaced with UFS?
 
Thanks so much for these responses. I don't know why this problem didn't come up in my dev environment, but I was able to fix it. It was a multiprocessing script that maybe wasn't joining the child processes to the parent processes.

Bobi B: My understanding was that as long as I kept an eye on the ARC, it wouldn't be an issue. I was running with 3 GB on the dev environment and it didn't look like I was able to take advantage of the resources. The files on this server from now on will be logs and duplicates. I'm using dedup, which I know is frowned upon, but the hash table should be mostly duplicates, right? 😅
 
Thanks so much for these responses. I don't know why this problem didn't come up in my dev environment, but I was able to fix it. It was a multiprocessing script that maybe wasn't joining the child processes to the parent processes.
That would be easy to tell by looking at the output of ps dax.
Bobi B: My understanding was that as long as I kept an eye on the ARC, it wouldn't be an issue. I was running with 3 GB on the dev environment and it didn't look like I was able to take advantage of the resources. The files on this server from now on will be logs and duplicates. I'm using dedup, which I know is frowned upon, but the hash table should be mostly duplicates, right? 😅
Oh? dedup is known to be quite memory-hungry. Check out this Oracle article on ZFS Deduplication. You better put logs on a separate dataset/filesystem and implement log rotation, and see if you can avoid persisting duplicate files at all, if possible. I haven't work with a data set that would benefit from dedup, therefore my experience is more or less nil. However I believe you can't really turn off dedup once enabled.
 

SirDice

Administrator
Staff member
Administrator
Moderator
I'm using dedup, which I know is frowned upon, but the hash table should be mostly duplicates, right?
It's not frowned upon, but you need to have around 5GB of RAM per 1TB of storage in order to make this work.

As it's mostly log files, I assume these are plain text? Then you're probably better off enabling compression.
 
Thanks to you both - I never knew about the 'd' option for `ps`. That's super helpful.

I have dedup enabled only on a single dataset of files where the only thing that is changed in around 16 GB of files is in-file metadata (as opposed to extended attributes). So if I am thinking about this right, everything would be deduplicated excepted for the block where that metadata is changed, which is very small in my testing.
 
Top