High load with sh script using wait_on

O

ondra_knezour

I would like to ask how to debug high load with simple script.

Situation - I have IMAP shared folder, where can users move messages using their mail clients. When such situation occurs, this file should be cated through pipe to another script and then removed. Let say this occurs once a week. I wrote simple script using sysutils/wait_on which waits on file write in given directory and process all found files as described above. Generally, most of time this script does nothing.
Code: 
# cat /usr/local/bin/otrspipe.sh
#!/bin/sh

while :; do
    for FILE in `find /path_to_directory -type f`
        do
            cat $FILE | /path_to_script
            rm $FILE
        done
    wait_on -w /path_to_directory
done

but this script, which does almost nothing generate high load on machine with four cores

Code: 
# top -P
last pid: 23427;  load averages:  1.00,  1.00,  1.00 
188 processes: 2 running, 186 sleeping
CPU 0:  0.4% user,  0.0% nice, 48.3% system,  0.0% interrupt, 51.3% idle
CPU 1:  0.4% user,  0.0% nice, 33.2% system,  0.0% interrupt, 66.4% idle
CPU 2:  0.0% user,  0.0% nice, 23.6% system,  0.0% interrupt, 76.4% idle
CPU 3:  1.8% user,  0.0% nice,  4.8% system,  0.0% interrupt, 93.4% idle
Mem: 2552M Active, 7797M Inact, 973M Wired, 196M Cache, 1236M Buf, 370M Free
Swap: 4174M Total, 68K Used, 4174M Free

  PID USERNAME       THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 1555 root             1  76    0  1482M  1467M piperd  2  61.2H 51.17% sh

which is getting even worse when script is started from /etc/rc.local as following
Code: 
# cat /etc/rc.local
/usr/sbin/daemon /usr/local/bin/otrspipe.sh > /var/log/otrspipe.log 2>&1
### Following line isn't interesting for this case
/usr/local/bin/java -jar /usr/local/unifi/lib/ace.jar start &

Questions
  1. How to debug high load?
  2. Can anybody suggest better way to immediate processing of message which appears in IMAP folder (Dovecot 2) from time to time?
 
You're spinning through the loop as fast as possible, many times per second. Add a sleep(1) into the while loop to have it only loop every second (at most). Consider even sleeping 30 or 60 seconds.
 
I also considered this, but man page for sysutils/wait_on says:
The wait_on command waits for something to happen to the files or directories given as arguments and then exits. It is anticipated that the wait_on command will be called from a shell script, allowing, for example, a script to sleep until files have been added to a directory or data is appended to a file. No polling is required.
Click to expand...
And even more there is an example basically similar to my usage
Code: 
while :; do
   mv $SOURCE/* $DESTINATION
   wait_on -w $SOURCE
done
So my understanding is, that after initial find run will be script stopped until directory change occurs.
 
As quick overview using truss() showed up, we both was right. sysutils/wait_on does what manpage says, however while loop was iterated as quick as possible. There was missing /usr/local/bin entry in PATH enviroment of given script.

Every time I forget to use full path to file in script, I shoot myself in the foot. Get lesson and keep your feet healthy :)
 
I too am working on something with FreeBSD and need to detect when a file changes in a specific directory but since I'm a novice I am in need of some assistance or pointing in the right direction.

I'd like to make a script which would perform the following...

1) Look at a specific directory "/mnt/farm/movies"
2) Once ANY change (addition/deletion/file name change/file size change) has been performed to the files in the directory then fall through and conduct a command like "minidlna -R" or it could be an echo text message for testing purposes.
3) Frequency to call the minidlna command should not occur more often than every 10 minutes, or whatever I decide is reasonable. I'm sure I could use a sleep command after the minidlna command is issued before heading back into the loop.
4) It would be nice to monitor multiple directories if possible with the same script, again mainly to reduce the amount of times the minidlna command is issued.

And I'm not looking for anyone to make all this work for me, I'm just asking for some guidance.

An example of the script that I can see is:
Code: 
#!/usr/local/bin/sh
#
# Call: sh checkfilechange.sh /directorypath
switch1=$1
while :; do
  wait_on -w ${switch1}
# Once a change is detected, echo a change was detected
  echo "File Changed"  
# sleep 10 minutes after change is detected  
  sleep 600
done

Thanks in advance...
 
Thanks for the vote of confidence. I'm patiently waiting on my computer to finish downloading a large set of files and once it's done then I will be able to test this out. A few more hours before I can play.
 
Use /bin/sh.

Also, why sleep for ten minutes after a change? That's what wait_on is supposed to do.
 
wblock@ said:
Use /bin/sh.

Also, why sleep for ten minutes after a change? That's what wait_on is supposed to do.
Click to expand...

The project I'm working is FreeNAS and trying to get MiniDLNA to rescan the database once a file has been added or removed. The way wait_on works, it triggers on the event immediately meaning if I'm writing a 5GB video file, at the start of the write wait_on is satisfied and then I need to sleep a period of time before I rescan the database.

The /usr/local/bin/sh is required for use in the FreeNAS environment.

Thanks for your comments though.

I have been able to get the script to run just fine for wait_on but making minidlna rescan the database has it's own problems, not really a topic for this thread.
 
You must log in or register to reply here.
Back
Top