Python threading/subprocess differences freeBSD vs Linux

  • Thread starter Deleted member 54719
  • Start date
D

Deleted member 54719

Guest


A couple of years ago I wrote a small pyton 2.x batch processing system in Linux to transcode AV files. When running the deamon I could select how many concurrent jobs would run, and how often the daemon rescanned the inqueue directory for new jobs. In Linux it works fine, and within the rescan interval it keeps the running queue full of jobs. However, the same code executed in freeBSD fills the running queue and stalls new jobs until the running queue is completely empty before grabbing another "group of jobs", instead of adding one job at a time as slots become available. It's as if the backgrounding of the jobs is blocking for some reason.

The worker thread (of which there can be as many as are allowed concurrent jobs) is subclassed from threading.Thread and in its run() method calls subprocess.call(job,shell=True). Does the freeBSD implementation of subprocesses and threads have semantic differences that are causing me problems?

This is my background thread class...

Python:
class executeJob(threading.Thread):
   
    baseDir=""
    batchFile=""
    debugLevel=0
    params={"LOG":"", "PWD":"", "NICE":"", "CMD":"",}
   
   
   
    # ------------------------------------------------------------------
    def __init__(self, f, d, dl):
        threading.Thread.__init__(self)
        self.baseDir = d
        self.batchFile = f
        self.debugLevel = dl
       
       
       
    # ------------------------------------------------------------------
    def run(self):
        script = "%s/running/%s" % (self.baseDir, self.batchFile)
        self.params["LOG"] = "%s.log" % script
        self.params["NICE"] = ""
        self.params["PWD"] = "%s/running" % self.baseDir
       
        # read control file
        fd = open(script, "r")
        for ln in fd:
            l = ln.replace("\n", "")                    # trim newlines
            if (l[0] == "#"):                           # comment found
                 for i in self.params.keys():
                    g = re.search("#%s=(.*)" % i, l)
                    if (g):
                        self.params[i] = g.group(1)     # assign parameters
                        if (self.debugLevel > 2):
                            print "(%s) %s=%s" % (time.strftime("%x %X"), i, g.group(1))
                            sys.stdout.flush()

        fd.close()
       
        # get command from control file
        cmd = "(cd %s/; %s sh %s > %s 2>&1)" % \
            (self.params["PWD"], self.params["NICE"], script, self.params["LOG"])
        if (self.debugLevel > 2):
            print "(%s) CMD=%s" % (time.strftime("%x %X"), cmd)
            sys.stdout.flush()

        try:
            subprocess.call(cmd, shell=True)
        except subprocess.CalledProcessError, e:
            print type(e)
            print e
            sys.stdout.flush()
            # run the command

Does anyone have any idea what I'm talking about?
 
Last edited by a moderator:
Top