Multi-processing or multi-threading in Python

Python Standard Library comes with a library called "multiprocessing" which can be used to do multi-threaded programming or multi-processing in Python. The multiprocessing library has objects such as Process, Queue and Pool among others, but the most straight-forward way to implement multi-processing when you have a function and you want to use the function in parallel for various data is by using the Pool object. The following example demonstrated multi-processing with Pool object:


import multiprocessing as mp

def f(x):
    return x*x

if __name__ == "__main__":
    # creates the number of worker processes 
    # as available in the machine
    pool = mp.Pool()  
    
    for data in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]:
        result = pool.apply_async(f, args=(data,))
        print result

In addition to apply_async method, the Pool object also comes with other methods such as map among others. But, the map method blocks until the result is ready whereas the apply_async method does not. So, to process data faster, you should use the apply_async method.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.