Thread Pools and Process Pools in Python (75/100 Days of Python)

Day 75 of the “100 Days of Python” blog post series covering multithreading pools and multiprocessing pools

In today’s fast-paced world, performance is a critical aspect of software development. As applications become more complex and handle larger amounts of data, it’s important to utilize all available resources to achieve optimal performance. Multithreading and multiprocessing are two ways to achieve parallelism in Python, which can significantly improve the performance of your code. By creating thread and process pools, you can efficiently distribute tasks across multiple threads and processes, taking advantage of available resources and reducing the time required to complete tasks.

Multithreading Pools

Multithreading is a way to achieve parallelism by executing multiple threads of code simultaneously within the same process. A thread is a lightweight process that shares the same memory space as the parent process. Python has a built-in threading module that allows you to create and manage threads.

To use multithreading pools in Python, you can use the ThreadPoolExecutor class from the concurrent.futures module. This class provides a high-level interface for creating and managing a pool of worker threads that can execute tasks asynchronously:

from concurrent.futures import ThreadPoolExecutor
import requests


def download_page(url):
    response = requests.get(url)
    return response.content


urls = ['https://stackoverflow.com', 'https://google.com', 'https://facebook.com']

with ThreadPoolExecutor(max_workers=3) as executor:
    results = list(executor.map(download_page, urls))

for result in results:
    print(len(result))

In this example, we define a function download_page that downloads the content of a web page using the requests library. We also define a list of URLs that we want to download.

We then create a ThreadPoolExecutor object with a maximum of 3 worker threads, and use the map method to execute the download_page function on each URL in the list asynchronously. The map method returns an iterator that yields the results of each function call in the order that they were started. Finally, we print the length of the content of each downloaded web page:

173536
15285
70610

Multiprocessing Pools

Multiprocessing is a way to achieve parallelism by executing multiple processes simultaneously. A process is a separate instance of a running program that has its own memory space. Python has a built-in multiprocessing module that allows you to create and manage processes.

To use multiprocessing pools in Python, you can use the Pool class from the multiprocessing module. This class provides a high-level interface for creating and managing a pool of worker processes that can execute tasks asynchronously:

from multiprocessing import Pool


def square(x):
    return x * x


numbers = [1, 2, 3, 4, 5]

if __name__ == '__main__':
    with Pool(processes=4) as pool:
        results = pool.map(square, numbers)
    print(results)

In this example, we define a function square that calculates the square of a number. We also define a list of numbers that we want to square.

We then create a Pool object with 4 worker processes, and use the map method to execute the square function on each number in the list asynchronously. The map method returns a list of the results of each function call in the order that they were started.

What is the Optimal Number of Processes?

When using multiprocessing in Python, it’s important to consider the number of processes you create versus the number of CPUs available on your machine to achieve the best performance.

Creating too many processes can lead to resource contention and overhead, which can reduce performance. On the other hand, creating too few processes can leave unused resources, which is also not ideal.

To determine the optimal number of processes, you should take into account the number of available CPUs on your machine and the nature of your workload. A common rule of thumb is to create one process per CPU, or one less than the number of CPUs available to leave some room for the operating system and other processes.

For example, if your machine has 4 CPUs, you may want to create 3 or 4 processes to achieve the best performance.

However, this rule of thumb is not always applicable, especially if your workload is CPU-bound, which means that it requires a lot of CPU time to complete. In this case, creating more processes than the number of CPUs available can actually reduce performance due to increased context switching and overhead.

On the other hand, if your workload is I/O-bound, which means that it spends a lot of time waiting for I/O operations to complete (e.g. reading files, network requests), you can create more processes than the number of CPUs available to take advantage of the waiting time and achieve better performance.

What’s next?

• If you found this story valuable, please consider clapping multiple times (this really helps a lot!)
• Hands-on Practice: Free Python Course
• Full series: 100 Days of Python
• Previous topic: Multithreading VS Multiprocessing
• Next topic: Async Await in Python — Asyncio Deep Dive