Python Standard Library comes with a library called “multiprocessing” which can be used to do multi-threaded programming or multi-processing in Python. The multiprocessing library has objects such as Process, Queue and Pool among others, but the most straight-forward way to implement multi-processing when you have a function and you want to use the function in parallel for various data is by using the Pool object. The following example demonstrated multi-processing with Pool object:
import multiprocessing as mp def f(x): return x*x if __name__ == "__main__": # creates the number of worker processes # as available in the machine pool = mp.Pool() for data in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]: result = pool.apply_async(f, args=(data,)) print result
In addition to apply_async method, the Pool object also comes with other methods such as map among others. But, the map method blocks until the result is ready whereas the apply_async method does not. So, to process data faster, you should use the apply_async method.