An example of a threading background decorator (Python)

Updated vegaseat 5 Tallied Votes 3K Views Share

Run two calls to a function simultaneously with this threading @background decorator.

''' threading_run_background_deco2.py
apply threading to a function with a function decorator
to allow it to run in the background
tested with Python27 and Python33  by  vegaseat  27jun2014
'''

import time
import threading


def background(f):
    '''
    a threading decorator
    use @background above the function you want to run in the background
    '''
    def bg_f(*a, **kw):
        threading.Thread(target=f, args=a, kwargs=kw).start()
    return bg_f


@background
def counter(name, n):
    """show the count every second"""
    for k in range(1, n+1):
        print("{} counts {}".format(name, k))
        time.sleep(1)

# start the counters
# note that with the @background decorator
# Frank and Doris count simultaneously
# note from SOS: Frank may not always count first
counter("Frank", 5)
counter("Doris", 5)

'''
result with the @background decorator ...
Frank counts 1
Doris counts 1
Frank counts 2
Doris counts 2
Frank counts 3
Doris counts 3
Frank counts 4
Doris counts 4
Frank counts 5
Doris counts 5

result without the @background decorator ...
Frank counts 1
Frank counts 2
Frank counts 3
Frank counts 4
Frank counts 5
Doris counts 1
Doris counts 2
Doris counts 3
Doris counts 4
Doris counts 5
'''
~s.o.s~ 2,560 Failure as a human Team Colleague Featured Poster

Just for the clarification of the readers; the output posted using the background decorator is not always guaranteed and depends on how your OS schedules the Python threads. So another possible output can be:

Frank counts 1
Doris counts 1
Frank counts 2
Doris counts 2
Doris counts 3
Frank counts 3
Doris counts 4
Frank counts 4
Doris counts 5
Frank counts 5

(in case you haven't noticed, Doris and Frank changed places after the 2nd count)

vegaseat 1,735 DaniWeb's Hypocrite Team Colleague

Thanks SOS!
After running the code about a dozen times, I also found that Frank does not always count first in the sequence. The count however is still simultaneous.

You could also do this with a Process.

import functools
import multiprocessing
def background(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        p = multiprocessing.Process(target=func, args=args, kwargs=kwargs)
        p.start()
    return wrapper

Also, I wanted to point something out for new people that may be reading this. I think it's a good practice to use the functools.wraps decorator when making things like this.

Example:

def decoratorwithout(f):
    def wrapper():
        return f()
    return wrapper

def decoratorwith(f):
    @functools.wraps(f)
    def wrapper():
        return f()
    return wrapper

@decoratorwithout
def myfuncwithout():
    """ You can't see this with help() or other tools. """
    return None

@decoratorwith
def myfuncwith():
    """ This is correct documentation. """
    return None

Checking the docs:

help(myfuncwith)

Help on function myfuncwith in module __main__:

myfuncwith()
    This is correct documentation.

And the function without functools.wraps:

help(myfuncwithout)

Help on function wrapper in module __main__:

wrapper()

functools.wraps sets the __doc__ and other things to help avoid confusion when wrapping functions.

print(myfuncwithout.__name__)
wrapper

print(myfuncwith.__name__)
myfuncwith
HiHe 174 Junior Poster

I always use the @functools.wraps(f) decorator for decorator functions to help with debugging.

vegaseat 1,735 DaniWeb's Hypocrite Team Colleague

multiprocessing.Process() might only work for Unix type OS?

@vegaseat, from the Python doc page for multiprocessing:

It runs on both Unix and Windows.

It's stdlib stuff so I assumed it was cross-platform, but to be honest I've never tested it on Windows.

I see on the doc page that there are some differences with spawn and fork in the multiprocessing API itself. Windows has spawn and fork, but uses spawn by default. Unix only has fork pre Python 3.4, and it uses fork by default. Support for spawn was added in 3.4. I see no differences in the Process API, which by the way is compatible with the Thread API (awesome).

HiHe 174 Junior Poster

I tested multiprocessing approach on Windoze and it failed:

import time
import functools
import multiprocessing


def background(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        p = multiprocessing.Process(target=func, args=args, kwargs=kwargs)
        p.start()
    return wrapper


@background
def counter2(n):
    """show the count every second"""
    for k in range(n):
        print("counting %d" % k)
        time.sleep(1)

# start the counter
counter2(7)
time.sleep(0.9)
print('hello')   # prints hello, before counter prints 1
time.sleep(3)
print('world')   # prints world, before counter prints 4

''' should give result ...
counting 0
hello
counting 1
counting 2
counting 3
world
counting 4
counting 5
counting 6

... instead gives this error (Windows7) ...
Traceback (most recent call last):
  File "C:\Python34\Atest34\aatest34\multiprocessing_run_background_deco1.py", line 43, in <module>
    counter2(7)
  File "C:\Python34\Atest34\aatest34\multiprocessing_run_background_deco1.py", line 18, in wrapper
    p.start()
  File "C:\Python34\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Python34\lib\multiprocessing\context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Python34\lib\multiprocessing\context.py", line 313, in _Popen
    return Popen(process_obj)
  File "C:\Python34\lib\multiprocessing\popen_spawn_win32.py", line 66, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Python34\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function counter2 at 0x02E6A1E0>: it's not the same object as __main__.counter2
'''
~s.o.s~ 2,560 Failure as a human Team Colleague Featured Poster

Multiprocessing on Windows does work, but is a bit crippled when compared to multiprocessing on *nix. For e.g., the following code snippet works fine on Windows:

from __future__ import print_function
import time
import functools
import multiprocessing

def runFuncInMp(func, *args, **kwargs):
    p = multiprocessing.Process(target=func, args=args, kwargs=kwargs)
    p.start()

def counter(n):
    """show the count every second"""
    for k in range(n):
        print("counting %d" % k)
        time.sleep(1)

if __name__ == '__main__':
    # start the counter
    runFuncInMp(counter, 7)
    time.sleep(0.9)
    print('hello')   # prints hello, before counter prints 1
    time.sleep(3)
    print('world')   # prints world, before counter prints 4

Notice the removal of the decorator and adding in the if __name__ == '__main__' check.

vegaseat 1,735 DaniWeb's Hypocrite Team Colleague
from __future__ import print_function
import time
import functools
import multiprocessing

def runFuncInMp(func, *args, **kwargs):
    p = multiprocessing.Process(target=func, args=args, kwargs=kwargs)
    p.start()

def counter(n):
    """show the count every second"""
    for k in range(n):
        print("counting %d" % k)
        time.sleep(1)

if __name__ == '__main__':
    # start the counter
    runFuncInMp(counter, 7)
    time.sleep(0.9)
    print('hello')   # prints hello, before counter prints 1
    time.sleep(3)
    print('world')   # prints world, before counter prints 4

Sorry to say, but the output using Windows 8.1 is not the desired one. All you get is:
hello
world

HiHe 174 Junior Poster
~s.o.s~ 2,560 Failure as a human Team Colleague Featured Poster

Sorry to say, but the output using Windows 8.1 is not the desired one. All you get is:

Are you running this from the command line or some sort of an IDE? It works fine for me on Windows 8.1

D:\misc>python test.py
counting 0
hello
counting 1
counting 2
counting 3
world
counting 4
counting 5
counting 6
vegaseat 1,735 DaniWeb's Hypocrite Team Colleague

@~s.o.s~
You are right, running it from a DOS batch file works. Using any of several IDEs does not do well. Looks like the IDE output window in this case blocks multiprocessing from working properly.

The original threading version does not have this problem.

I don't want to go off on a tangent here, but maybe some nice person can test HiHe's multiprocessor-decorator version on a Unix box.

HiHe 174 Junior Poster

I thought the idea was to have a function decorator.

~s.o.s~ 2,560 Failure as a human Team Colleague Featured Poster

some nice person can test HiHe's multiprocessor-decorator version on a Unix box.

It works fine on a *nix box (as expected)

sos@ubuntu_14.04:~/personal$ python test.py 
counting 0
hello
counting 1
counting 2
counting 3
world
counting 4
counting 5
counting 6
HiHe 174 Junior Poster

I guess Python is made for Unix/Linux and not Windoze.

snippsat 661 Master Poster

A little look at concurrent.futures
concurrent.futures has a minimalistic API for threading and multiprocessing.
Only change one word to switch ThreadPoolExecutor(threading) and ProcessPoolExecutor(multiprocessing).
concurrent.futures is backportet to Python 2.7

A look at ProcessPoolExecutor(multiprocessing)

from __future__ import print_function
from time_code import timeit
import concurrent.futures
import time

def counter(n):
    """show the count every second"""
    for k in range(n):
        print("counting {}".format(k))
        time.sleep(1)

@timeit
def main():
    with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:
        for i in range(20):
            executor.submit(counter, i)

if __name__ == "__main__":
    main()

For Windows if __name__ == "__main__": and run from command line to see print function.
It can work from a IDE if no need to see output from print function.

So here max_workers=4 i got a time of 55 sec.
So if a spawn load over more processes,i should see a faster time.
Change to max_workers=30 and time go down to 19 sec.

The @timeit code i use for this.

#time_code.py
import time

def timeit(f):
    '''Timing a function'''
    def timed(*args):
        ts = time.time()
        result = f(*args)
        te = time.time()
        print 'func:{!r}  {!r} took: {:.2f} sec'.format\
        (f.__name__, args, te-ts)
        return result
    return timed
vegaseat 1,735 DaniWeb's Hypocrite Team Colleague

Responding to the PicklingError that HiHe observed:
The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock (GIL) by using subprocesses instead of threads. Due to this only picklable objects can be executed and returned.

Lardmeister 461 Posting Virtuoso
Lardmeister 461 Posting Virtuoso

I tested some of the multiprocessing examples in the above article. On a Windows machine they only work if you use the commandline option to run it. In other words the ugly DOS window.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.