Python threading and subprocesses explained

Take advantage of Python’s ability to parallelize workloads using threads for I/O-bound tasks and subprocesses for CPU-bound tasks

Become An Insider

Sign up now and get FREE access to hundreds of Insider articles, guides, reviews, interviews, blogs, and other premium content. Learn more.

By default, Python’s runtime executes in a single thread, traffic-directed by its Global Interpreter Lock (GIL). Most of the time this isn’t a significant bottleneck, but it becomes one when you want to run many jobs in parallel.

Python provides two ways to work around this: threading and multiprocessing. Each allows you to take long-running jobs, break them into parallel batches, and work on them side-by-side.

Depending on the job in question, you can sometimes speed up operations tremendously. At the very least, you can treat tasks in such a way that they don’t block other work while they wait to be completed.

In this article we’ll look at one of the fastest ways to make use of threading and subprocesses in Python, the thread and process pool.

Python threads versus Python processes

To continue reading this article register now