6 Python libraries for parallel processing

Want to distribute that heavy Python workload across multiple CPUs or a compute cluster? These frameworks can make it happen.

6 Python libraries for parallel processing
graemenicholson / Getty Images
Table of Contents
Show More

Python is long on convenience and programmer-friendliness, but it isn’t the fastest programming language around. Some of its speed limitations are due to its default implementation, cPython, being single-threaded. That is, cPython doesn’t use more than one hardware thread at a time.

And while you can use the threading module built into Python to speed things up, threading only gives you concurrency, not parallelism. It’s good for running multiple tasks that aren’t CPU-dependent, but does nothing to speed up multiple tasks that each require a full CPU. 

Python does include a native way to run a Python workload across multiple CPUs. The multiprocessing module spins up multiple copies of the Python interpreter, each on a separate core, and provides primitives for splitting tasks across cores. But sometimes even multiprocessing isn’t enough.

Copyright © 2020 IDG Communications, Inc.

How to choose a low-code development platform