I came across a problem that I can't solve and it's associated with multiprocessing and use it inside the decorator.
When I'm calling the method run_in_parallels using multiprocessing I 'm getting the error:
Can't pickle <function run_testcase at 0x00000000027789C8>: it's not found as __main__.run_testcase
The call takes place inside the decorator, then followed the above-mentioned problem. At the time of calling the same method run_in_parallels without a decorator all working properly.
What is the reason of this problem?
file: w_PythonHelper.py
desc: Function 'run_in_parallel' is used to run multiple processes simultaneously. The first method, which will end operation stops the others.
from multiprocessing import Process,Event
class ExtProcess(Process):
def __init__(self, event,*args,**kwargs):
self.event=event
Process.__init__(self,*args,**kwargs)
def run(self):
Process.run(self)
self.event.set()
class PythonHelper(object):
@staticmethod
def run_in_parallel(*functions):
event=Event()
processes=dict()
for function in functions:
fname=function[0]
try:fargs=function[1]
except:fargs=list()
try:fproc=function[2]
except:fproc=1
for i in range(fproc):
process=ExtProcess(event,target=fname,args=fargs)
process.start()
processes[process.pid]=process
event.wait()
for process in processes.values():
process.terminate()
for process in processes.values():
process.join()
file: w_Recorder.py
desc: function 'capture' is used to grab a screenshot
from PIL import ImageGrab
class Recorder(object):
def capture(self,filename=time.time(),extension="png"):
ImageGrab.grab().save("{f}.{e}".format(f=filename,e=extension))
file: w_Decorators.py
desc: Running parallel a given function along with a method 'capture' of class 'Recorder'
from w_Recorder import Recorder
from w_PythonHelper import PythonHelper
def check(function):
def wrapper(*args):
try:
recorder=Recorder()
PythonHelper.run_in_parallel([function,args],[recorder.capture])
print("success")
except Exception as e:
print("failure: {}".format(e))
return function
return wrapper
file: w_Logger.py
desc: Main program (generates error)
from w_Decorators import check
import time
class Logger(object):
@check
def run_testcase(self):
# example function (runtime: 20s)
for i in range(20):
print("number: {}".format(i))
time.sleep(1)
def run_logger(self):
self.run_testcase()
if __name__=="__main__":
logger=Logger()
logger.run_logger()
file: w_Logger.py
desc: Main program (works corectly)
from w_PythonHelper import PythonHelper
from w_Recorder import Recorder
import time
class Logger(object):
def run_testcase(self):
# example function (runtime: 20s)
for i in range(20):
print("number: {}".format(i))
time.sleep(1)
def run_logger(self):
recorder=Recorder()
PythonHelper.run_in_parallel([self.run_testcase],[recorder.capture])
if __name__=="__main__":
logger=Logger()
logger.run_logger()
What is the difference that these same methods presented in the two cases work differently?
I got only one answer on this strange question:
import traceback
def wrapper(function,*args):
try:
recorder=Recorder()
PythonHelper().run_in_parallel([function,args],[recorder.capture])
print("success")
except Exception,e:
print("failure: "+traceback.format_exc(10))
from w_Decorators import wrapper
if __name__=="__main__":
logger=Logger()
wrapper(logger.run_testcase)
I can use wrapper as a function, but can't use it as decorator. I have many functions decorated by this decorator and the simpliest way is to use multiprocessing inside decorator. But unfortunatly I can't solve this problem. Maybe someone has already solved a similar problem. I would be grateful for any hint.
I've trying to make run_testcase a top level function but still nothing.