Show Progress with the ThreadPool in Python

Last Updated on October 29, 2022

You can show the progress of tasks in the ThreadPool using a callback function.

In this tutorial, you will discover how to show the progress of tasks in the ThreadPool in Python.

Let’s get started.

Need To Show Progress of Tasks in the ThreadPool

The multiprocessing.pool.ThreadPool in Python provides a pool of reusable threads for executing ad hoc tasks.

A thread pool object which controls a pool of worker threads to which jobs can be submitted.
— multiprocessing — Process-based parallelism

The ThreadPool class extends the Pool class. The Pool class provides a pool of worker processes for process-based concurrency.

Although the ThreadPool class is in the multiprocessing module it offers thread-based concurrency and is best suited to IO-bound tasks, such as reading or writing from sockets or files.

A ThreadPool can be configured when it is created, which will prepare the new threads.

We can issue one-off tasks to the ThreadPool using methods such as apply() or we can apply the same function to an iterable of items using methods such as map().

Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the methods such as apply_async() and map_async().

When issuing many tasks to the ThreadPool, we may need to keep track of how much work remains to be completed.

How can we show the progress of completed tasks in the ThreadPool?

We can show the progress of tasks in the ThreadPool using the callback function.

This can be achieved by issuing tasks asynchronously to the ThreadPool, such as via the apply_async() function and specifying a callback function via the “callback” argument.

For example:

...

# issue a task to the thread pool

pool.apply_async(task, callback=progress)

Our custom callback function will then be called after each task in the ThreadPool is completed.

We can then perform some action to show the progress of completed tasks in the callback function, such as printing a dot to standard out.

For example:

# progress indicator for tasks in the thread pool

def progress(results):

print('.', end='')

Now that we know how to show the progress of tasks in the ThreadPool in a standard way, let’s look at a worked example.

Example of Showing Progress in the ThreadPool

We can explore how to show the progress of tasks issued to the ThreadPool.

In this example, we will define a task that will block for a fraction of a second. We will then issue many of these tasks to the ThreadPool. A callback will be called as each task is finished and will print a message to show the progress of tasks as they are completed.

Firstly, we can define a callback function to call each time a task is completed in the ThreadPool.

The function takes the return value of a custom task function, if present, and prints a single dot to standard out without a newline.

The progress() function below implements this.

# progress indicator for tasks in the thread pool

def progress(results):

print('.', end='')

Next, we can define a custom task function.

The task will first generate a random floating point value between 0 and 1. It will then sleep for that many seconds, which will be less than one second. This sleeping will simulate computational work that takes a variable amount of time.

The task() function below implements this.

# task executed in a worker thread

def task():

# generate a random value

value = random()

# block for a moment

sleep(value)

Next, in the main thread, we can create the ThreadPool.

We will create the ThreadPool with the default configuration and use the context manager interface.

...

# create and configure the thread pool

with ThreadPool() as pool:

# ...

You can learn more about the context manager interface in the tutorial:

How to Use the ThreadPool Context Manager

We can then issue 20 tasks to the ThreadPool, each calling our custom task() function and using the custom progress() function as a callback to show progress.

We can issue each task one by one using the apply_asyc() function which returns an AsyncResult object.

This can be achieved in a list comprehension, providing a list of AsyncResult objects, one for each task.

...

# issue many tasks asynchronously to the thread pool

results = [pool.apply_async(task, callback=progress) for _ in range(20)]

You can learn more about the apply_async() function in the tutorial:

How to Use ThreadPool apply_async() in Python

Finally, the main thread will close the ThreadPool and block, waiting for the issued task to complete.

...

# close the pool

pool.close()

# wait for all issued tasks to complete

pool.join()

You can learn more about joining the ThreadPool in the tutorial:

How to Join a ThreadPool in Python

After the ThreadPool is closed, a final message can be reported.

...

# report all done

print('\nDone!')

Tying this together, the complete example is listed below.

# SuperFastPython.com

# example of showing progress in the thread pool with separate tasks

from time import sleep

from random import random

from multiprocessing.pool import ThreadPool

# progress indicator for tasks in the thread pool

def progress(results):

print('.', end='')

# task executed in a worker thread

def task():

# generate a random value

value = random()

# block for a moment

sleep(value)

# protect the entry point

if __name__ == '__main__':

# create and configure the thread pool

with ThreadPool() as pool:

# issue many tasks asynchronously to the thread pool

results = [pool.apply_async(task, callback=progress) for _ in range(20)]

# close the pool

pool.close()

# wait for all issued tasks to complete

pool.join()

# report all done

print('\nDone!')

Running the example first creates the ThreadPool.

Then, 20 tasks are issued to the pool, one at a time.

The main thread then closes the ThreadPool and blocks waiting for all issued tasks to be completed.

Each task blocks for a fraction of a second and finishes.

As each task is finished, the callback function is called, printing a dot.

The dots accumulate on standard output, showing the progress of all issued tasks.

Once all tasks are finished, the main thread continues and reports a final message.

....................

Done!

Free Python ThreadPool Course

Download your FREE ThreadPool PDF cheat sheet and get BONUS access to my free 7-day crash course on the ThreadPool API.

Discover how to use the ThreadPool including how to configure the number of worker threads and how to execute tasks asynchronously

Learn more

Takeaways

You now know how to show the progress of tasks in the ThreadPool in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Nick Fewings on Unsplash

Need To Show Progress of Tasks in the ThreadPool

Example of Showing Progress in the ThreadPool

Further Reading

Takeaways