Multiprocess help!

[jeffrey.pry]

Hey,

I am trying to use multiprocessing via the Process module. I run two processes in unison and for some reason python or the os kills my process after roughly 50 seconds or during long processes (~3 min). I just get the following output:

File 3 of 8
2010-10-11 13:28:09.090816: Building XML structure...
Killed
pry@linu:~/Share/scripts>

I implemented it as follows (Obviously there's more, but I can't post everything here because it's extremely long.)

from multiprocessing import Process

for i in proclist:
    p = MainProcess(i,csvMap)
    runlist.append(p)
    p.start()

for i in runlist:
    i.join()

while True:
    for i in runlist:
        if not i.isAlive():
            i.terminate()

Thanks!

[woooee]

Assuming MainProcess is a class, you probably want something along the lines of the following code. If that doesn't help, post back.

for item in proclist:
    MP = MainProcess(item,csvMap)
    p = Process(target=MP.run_something(), args=('XX',))
    p.start()
    runlist.append(p)

[jeffrey.pry]

Assuming MainProcess is a class, you probably want something along the lines of the following code. If that doesn't help, post back.

for item in proclist:
    MP = MainProcess(item,csvMap)
    p = Process(target=MP.run_something(), args=('XX',))
    p.start()
    runlist.append(p)

Hey,

Thanks for the reply. That didn't do the trick. I have it implemented a little differently:

class MainProcess(Process):

    def __init__(self,__fileListConstraint,csvMap):
        self.__fileListConstraint = __fileListConstraint
        self._csvMap = csvMap
        Process.__init__ ( self )
    def run(self):
        count = 0.0
        for line in self.__fileListConstraint:
            __main__(__isVerbose__,line,self._csvMap,options.path,options.compress)
            count +=1.0

Therefore I can call via:

c = MainProcess(['blablabla'],['blablabla'])
c.start()

Both processes start but they die after a few records are processed. If I run them without processes, it works fine. Any other suggestions?

Also, can someone explain the significance of process.join()?

If I do the following, it looks like it doesn't call the proc1:

proc0.start()
proc0.join()
proc1.start()
proc1.join()

Thanks,

Jeffrey Kevin Pry

[Gribouillis]

I remember a restriction in the threading module, that the __init__ function of a user subclass of Thread should call Thread.__init__(self) before any other statement. Could there be a similar restriction for the Process class ?

[jeffrey.pry]

I remember a restriction in the threading module, that the __init__ function of a user subclass of Thread should call Thread.__init__(self) before any other statement. Could there be a similar restriction for the Process class ?

Hey,

This isn't enforced in Process as the script runs for a short while. I'll rearrange it and see if that fixes the issue.

Thanks!

Jeffrey Kevin Pry

[jeffrey.pry]

Hey,

This isn't enforced in Process as the script runs for a short while. I'll rearrange it and see if that fixes the issue.

Thanks!

Jeffrey Kevin Pry

Too funny... Just moving the Process.__init__ ( self ) fixed the issue. Not letting the console know that this has to be first seems like a bug to me.

class MainProcess(Process):
    def __init__(self,__fileListConsPtraint,csvMap):
        Process.__init__ ( self )
        self.__fileListConstraint = __fileListConstraint
        self._csvMap = csvMap

Thanks for your help!

Jeffrey Kevin Pry

[Gribouillis]

That's great! I had issues with the Thread class one day where I forgot this rule. The class constructor must probably manipulate the instance's __dict__ in a non standard way...

[woooee]

The point is in this code, and it should be giving you problems as p is a class instance of MainProcess, not multiprocessing's Process.

for i in runlist:
    i.join()
 
while True:
    for i in runlist:
        if not i.isAlive():
            i.terminate()