D
Deleted member 54719
Guest
A couple of years ago I wrote a small pyton 2.x batch processing system in Linux to transcode AV files. When running the deamon I could select how many concurrent jobs would run, and how often the daemon rescanned the inqueue directory for new jobs. In Linux it works fine, and within the rescan interval it keeps the running queue full of jobs. However, the same code executed in freeBSD fills the running queue and stalls new jobs until the running queue is completely empty before grabbing another "group of jobs", instead of adding one job at a time as slots become available. It's as if the backgrounding of the jobs is blocking for some reason.
The worker thread (of which there can be as many as are allowed concurrent jobs) is subclassed from
This is my background thread class...
Does anyone have any idea what I'm talking about?
The worker thread (of which there can be as many as are allowed concurrent jobs) is subclassed from
threading.Thread
and in its run()
method calls subprocess.call(job,shell=True)
. Does the freeBSD implementation of subprocesses and threads have semantic differences that are causing me problems?This is my background thread class...
Python:
class executeJob(threading.Thread):
baseDir=""
batchFile=""
debugLevel=0
params={"LOG":"", "PWD":"", "NICE":"", "CMD":"",}
# ------------------------------------------------------------------
def __init__(self, f, d, dl):
threading.Thread.__init__(self)
self.baseDir = d
self.batchFile = f
self.debugLevel = dl
# ------------------------------------------------------------------
def run(self):
script = "%s/running/%s" % (self.baseDir, self.batchFile)
self.params["LOG"] = "%s.log" % script
self.params["NICE"] = ""
self.params["PWD"] = "%s/running" % self.baseDir
# read control file
fd = open(script, "r")
for ln in fd:
l = ln.replace("\n", "") # trim newlines
if (l[0] == "#"): # comment found
for i in self.params.keys():
g = re.search("#%s=(.*)" % i, l)
if (g):
self.params[i] = g.group(1) # assign parameters
if (self.debugLevel > 2):
print "(%s) %s=%s" % (time.strftime("%x %X"), i, g.group(1))
sys.stdout.flush()
fd.close()
# get command from control file
cmd = "(cd %s/; %s sh %s > %s 2>&1)" % \
(self.params["PWD"], self.params["NICE"], script, self.params["LOG"])
if (self.debugLevel > 2):
print "(%s) CMD=%s" % (time.strftime("%x %X"), cmd)
sys.stdout.flush()
try:
subprocess.call(cmd, shell=True)
except subprocess.CalledProcessError, e:
print type(e)
print e
sys.stdout.flush()
# run the command
Does anyone have any idea what I'm talking about?
Last edited by a moderator: