1. Python Multiprocessing Spoon-fed
Brendan Collins Blue Raster @blueraster
Flickr - jystewart's photostream

2. Hello Multiprocessing Module!
• Python standard library module
• Allows for parallel code execution
• Easy to use
• Nice compliment to arcpy module
Flickr - jystewart's photostream

3. The Multiprocessing Mission:
• Fully leverage all CPUs
• Increase Scalability
• Improve Performance
• Alternative to Threading
Flickr - jystewart's photostream

4. Threads: Fluffy the three-headed dog
Flickr - jegcarpio's photostream

5. Threads: Fluffy the three-headed dog
• Each thread has same parent process
• Threads share memory
• Good for I/O Bound tasks (Scraping websites)
• Poor for CPU Bound Tasks (Geoprocessing)
• At the mercy of Global Interpreter Lock (GIL)
• Problems can affect other threads
Flickr - jegcarpio's photostream

6. Flickr - Blacksmith's photostream
Processes: Hounds of Resurrection

7. Processes: Hounds of Resurrection
• Each process is a separate python instance
• Memory space is NOT shared
• Easily distributed across all cores
• If one process dies, others are not affected
• Communication via IPC
Flickr - Blacksmith's photostream

9. Step 1: Basic Setup

11. Multiprocessing Pitfalls
• All worker arguments must be picklable
• Schema locks (File Geodatabase…)
• Script tools must be run “out-of-process”
• Worker startup cost

12. In Closing:
• Jason Pardy’s Multiprocessing post
– blogs.esri.com/esri/arcgis/2011/08/29/multiproce
ssing (link)
• Follow Me:
– @brendancol
– bcdcspatial.blogspot.com (link)
• Follow Blue Raster:
– @blueraster
– blueraster.com/blog (link)
Flickr - jystewart's photostream