Multiprocessing Pool

The scripts show how to make use of the mutiprocessing ability of Python to handle a batch of processes.

All CPUs of the computer running the scripts will be called at the same time. The job handler (thrd-m.py) see all CPUs available as a pool, it fetches jobs from a list stored in it. The jobs are 10 copies of a time killing work (long01.py) which generates the number pi from a series.

1. Check the number of CPUs available.
2. Define a multiprocessing pool.
3. Fetch jobs from a list and submit them, one by one, to the pool.
4. Wait for the completion of all jobs.

Source Code

---

Here is the job handler (thrd-m.py)
#!/usr/bin/python import multiprocessing import subprocess gData = ((1,'long01.py'), (2,'long01.py'), (3,'long01.py'), (4,'long01.py'), (5,'long01.py'), (6,'long01.py'), (7,'long01.py'), (8,'long01.py'), (9,'long01.py'), (10,'long01.py')) # multiprocessing def mp_handler(pCpuCnt): p = multiprocessing.Pool(pCpuCnt) p.map(mp_worker, gData) def mp_worker(pJob): print 'Start Process: %s ...' % pJob[0] mcmd = 'python ' + pJob[1] print ' '+mcmd subprocess.call(mcmd.split(' ')) print 'Process: %s DONE' % pJob[0] # main program if __name__ == '__main__': mcpuCnt = multiprocessing.cpu_count() mp_handler(mcpuCnt) print print 'All processes are DONE' print 'Number of CPU: %s' % mcpuCnt

Here is the time killing process (long01.py)
import time, math, random random.seed() mF = random.randint(1,8) mN = 1000000*mF mSum = 0 mt1 = time.time() for ma in range(1,mN): mSum = mSum+1.0/(ma**2) mPi = math.sqrt(6*mSum) mt2 = time.time() print " Pi : %10.8f" % mPi print " Factor : %s" %mF print " Processing Time : %6.2f sec." % (mt2-mt1)

A sample result of running thrd-m.py by a Core i5 is given below # python thrd-m.py Start Process: 1 ... python long01.py Start Process: 2 ... python long01.py Start Process: 3 ... python long01.py Start Process: 4 ... python long01.py Pi : 3.14159170 Factor : 1 Processing Time : 0.12 sec. Process: 4 DONE Start Process: 5 ... python long01.py Pi : 3.14159218 Factor : 2 Processing Time : 0.24 sec. Process: 2 DONE Start Process: 6 ... python long01.py Pi : 3.14159218 Factor : 2 Processing Time : 0.23 sec. Process: 5 DONE Start Process: 7 ... python long01.py Pi : 3.14159252 Factor : 7 Processing Time : 0.82 sec. Process: 3 DONE Start Process: 8 ... python long01.py Pi : 3.14159252 Factor : 7 Processing Time : 0.84 sec. Process: 1 DONE Start Process: 9 ... python long01.py Pi : 3.14159246 Factor : 5 Processing Time : 0.58 sec. Process: 7 DONE Start Process: 10 ... python long01.py Pi : 3.14159252 Factor : 7 Processing Time : 0.82 sec. Process: 6 DONE Pi : 3.14159218 Factor : 2 Processing Time : 0.24 sec. Process: 9 DONE Pi : 3.14159234 Factor : 3 Processing Time : 0.34 sec. Process: 10 DONE Pi : 3.14159253 Factor : 8 Processing Time : 0.90 sec. Process: 8 DONE All processes are DONE Number of CPU: 4