Synchronizing Threads in Python With Semaphores (70/100 Days of Python)
Thread semaphores are another synchronization mechanism used in multithreaded applications. Semaphores are similar to thread locks in that they prevent race conditions by allowing only a limited number of threads to access a shared resource at a time. However, semaphores offer more flexibility than locks by allowing multiple threads to access the shared resource simultaneously, up to a specified limit.
What are Thread Semaphores?
Thread semaphores are synchronization mechanisms that allow a limited number of threads to access a shared resource simultaneously. When a thread wants to access the shared resource, it must first acquire
a semaphore. Each semaphore has a specified limit that determines how many threads can access the resource simultaneously. Once the limit is reached, any additional threads that try to acquire the semaphore will be blocked until a thread releases
the semaphore.
How do Thread Semaphores Work?
Thread semaphores work by allowing a limited number of threads to access a shared resource simultaneously. When a thread wants to access the resource, it must first acquire
a semaphore. If the semaphore is available and the limit has not been reached, the thread can acquire the semaphore and access the resource. If the semaphore is not available or the limit has been reached, the thread will be blocked until a semaphore becomes available.
Once a thread has acquired a semaphore, it can access the shared resource simultaneously with other threads that have also acquired the semaphore. When the thread is done accessing the resource, it must release
the semaphore so that other threads can acquire it and access the resource.
Thread semaphores can be used to protect any shared resource that is accessed by multiple threads, including variables, files, and network connections.
How to Use Thread Semaphores in Python
In Python, thread semaphores are implemented using the threading.Semaphore()
class. To use a semaphore in your code, you first need to create an instance of the Semaphore()
class:
import threading
semaphore = threading.Semaphore(limit)
where limit
is the maximum number of threads that can acquire the semaphore simultaneously.
Once you have created a semaphore instance, you can use it to protect a shared resource. To acquire the semaphore, you use the acquire()
method:
semaphore.acquire()
This will block the thread until a semaphore becomes available. Once a semaphore is available and the limit has not been reached, the thread can acquire the semaphore and access the shared resource simultaneously with other threads that have also acquired the semaphore. When the thread is done accessing the resource, it must release the semaphore using the release()
method:
semaphore.release()
This will release the semaphore and allow other threads to acquire it and access the shared resource.
It’s important to note that when using semaphores, you must ensure that you release the semaphore in all possible code paths. If you acquire a semaphore and then exit the function without releasing the semaphore, the semaphore will remain locked, preventing other threads from accessing the shared resource. To avoid this, it’s a good practice to use a try
-finally
block to ensure that the semaphore is released, even if an exception occurs:
semaphore.acquire()
try:
# access the shared resource
finally:
semaphore.release()
Another way of handling the semaphore is using the with
statement:
from threading import Semaphore
semaphore = Semaphore(limit) # Create a global semaphore object
# ...
with semaphore: # Access the semaphore in a thread
... # Access the shared resource
Example of Using Thread Semaphores in Python
Let’s look at an example of how to use thread semaphores in Python. Suppose we have a list of items that need to be processed, and we want to process them in parallel using multiple threads. We can use a semaphore to limit the number of threads that can access the list simultaneously:
import threading
from time import sleep
items = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# semaphore to limit the number of threads that can access the list simultaneously to 4
semaphore = threading.Semaphore(value=4)
def process_item(item):
semaphore.acquire() # acquire the semaphore
try:
sleep(3) # simulate some processing time
print(f'Processing item {item}') # process the item
finally: # Make sure we always release the semaphore
semaphore.release() # release the semaphore
# create a list of threads to process the items
threads = [
threading.Thread(target=process_item, args=(item,))
for item in items
]
[thread.start() for thread in threads] # start all threads
[thread.join() for thread in threads] # wait for all threads to finish
In this example, we create a list of items to be processed, and a semaphore with a limit of 4 threads. We define a process_item()
function that acquires the semaphore, processes the item, and releases the semaphore. We then create a list of threads to process the items, and start the threads. Finally, we wait for all threads to finish using the join()
method.
When we run this code, we will see that the items are processed in parallel, but no more than 4 items are processed simultaneously, because of the semaphore limit.
The same code can be implemented using with
statements, which makes it more clean and intuitive:
import threading
from time import sleep
items = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# semaphore to limit the number of threads that can access the list simultaneously to 4
semaphore = threading.Semaphore(value=4)
def process_item(item):
with semaphore: # acquire the semaphore
sleep(3) # simulate some processing time
print(f'Processing item {item}') # process the item
# create a list of threads to process the items
threads = [
threading.Thread(target=process_item, args=(item,))
for item in items
]
[thread.start() for thread in threads] # start all threads
[thread.join() for thread in threads] # wait for all threads to finish
These programs can have different outputs based on the order of threads accessing the items. One of the outputs might be:
Processing item 1
Processing item 2
Processing item 4
Processing item 3
Processing item 6
Processing item 5
Processing item 8
Processing item 7
Processing item 10
Processing item 9
When running the code, you’ll notice that the code processes items in groups of 4 making sure all the other threads wait for their “turn”.
What’s next?
- If you found this story valuable, please consider clapping multiple times (this really helps a lot!)
- Hands-on Practice: Free Python Course
- Full series: 100 Days of Python
- Previous topic: Synchronizing Threads in Python With Locks
- Next topic: Synchronizing Threads in Python With Barriers