You can execute multiple tasks in the ThreadPool using the map() method.

In this tutorial you will discover how to use the map() method with the ThreadPool in Python.

Let’s get started.

Need a Concurrent Version of map()

The multiprocessing.pool.ThreadPool in Python provides a pool of reusable threads for executing ad hoc tasks.

A thread pool object which controls a pool of worker threads to which jobs can be submitted.

— multiprocessing — Process-based parallelism

The ThreadPool class extends the Pool class. The Pool class provides a pool of worker processes for process-based concurrency.

Although the ThreadPool class is in the multiprocessing module it offers thread-based concurrency and is best suited to IO-bound tasks, such as reading or writing from sockets or files.

A ThreadPool can be configured when it is created, which will prepare the new threads.

We can issue one-off tasks to the ThreadPool using methods such as apply() or we can apply the same function to an iterable of items using functions such as map().

Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async().

The built-in map() function allows you to apply a function to each item in an iterable.

The ThreadPool provides a map() method for executing multiple tasks in the pool.

How can we use the ThreadPool map() method?

The ThreadPool provides the map() method for executing multiple tasks.

Recall that the built-in map() function will apply a given function to each item in a given iterable.

Return an iterator that applies function to every item of iterable, yielding the results.

— Built-in Functions

It yields one result returned from the given target function called with one item from a given iterable. It is common to call map and iterate the results in a for-loop.

For example:

The ThreadPool provides a version of the map() function where the target function is called for each item in the provided iterable with separate worker threads.

A parallel equivalent of the map() built-in function […]. It blocks until the result is ready.

— multiprocessing — Process-based parallelism

For example:

Each item in the iterable is taken as a separate task in the ThreadPool.

Like the built-in map() function, the returned iterator of results will be in the order of the provided iterable. This means that tasks are issued (and perhaps executed) in the same order as the results are returned.

Unlike the built-in map() function, the ThreadPool map() method only takes one iterable as an argument. This means that the target function executed in the thread can only take a single argument.

The iterable of items that is passed is iterated in order to issue all tasks to the ThreadPool. Therefore, if the iterable is very long, it may result in many tasks waiting in memory to execute, e.g. one per worker thread.

It is possible to split up the items in the iterable evenly to worker threads.

For example, if we had a ThreadPool with 4 worker threads and an iterable with 40 items, we can split up the items into 4 chunks of 10 items, with one chunk allocated to each worker thread.

The effect is less overhead in transmitting tasks to worker threads and collecting results.

This can be achieved via the “chunksize” argument to map().

This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. The (approximate) size of these chunks can be specified by setting chunksize to a positive integer.

— multiprocessing — Process-based parallelism

For example:

Difference Between map() and map_async()

How does the map() method compare to the map_async() method for issuing tasks to the ThreadPool?

Both the map() and map_async() may be used to issue tasks that call a function to all items in an iterable via the ThreadPool.

The following summarizes the key differences between these two methods:

• The map() method blocks, whereas the map_async() method does not block.
• The map() method returns an iterable of return values from the target function, whereas the map_async() function returns an AsyncResult.
• The map() method does not support callback functions, whereas the map_async() method can execute callback functions on return values and errors.

The map() method should be used for issuing target task functions to the ThreadPool where the caller can or must block until all function calls are complete.

The map_async() method should be used for issuing target task functions to the ThreadPool where the caller cannot or must not block while the task is executing.

Now that we know how to use the map() method to execute tasks in the ThreadPool, let’s look at some worked examples.

Example of ThreadPool map()

We can explore how to use the map() method on the ThreadPool.

In this example, we can define a target task function that takes an integer as an argument, generates a random number, reports the value then returns the value that was generated. We can then call this function for each integer between 0 and 9 using the ThreadPool map() method.

This will apply the function to each integer in parallel using as many cores as are available in the system.

Firstly, we can define the target task function.

The function takes an argument, generates a random number between 0 and 1, reports the integer and generated number. It then blocks for a fraction of a second to simulate computational effort, then returns the number that was generated.

The task() function below implements this.

We can then create and configure a ThreadPool.

We will use the context manager interface to ensure the pool is shutdown automatically once we are finished with it.

We can then call the map() function on the ThreadPool to apply our task() function to each value in a range between 0 and 1.

This returns an iterator over the results returned from the task() function, in the order that function calls are completed. We will iterate over the results and report each in turn.

This can all be achieved in a for-loop.

Tying this together, the complete example is listed below.

Running the example first creates the ThreadPool with a default configuration.

It will have one worker thread for each logical CPU in your system.

The map() function is then called for the range.

This issues ten calls to the task() function, one for each integer between 0 and 9. An iterator is returned with the result for each function call, in order.

Each call to the task function generates a random number between 0 and 1, reports a message, blocks, then returns a value.

The main thread iterates over the values returned from the calls to the task() function and reports the generated values, matching those generated in each worker thread.

Importantly, all task() function calls are issued and executed before the iterator of results is returned. We cannot iterate over results as they are completed by the caller.

Next, let’s look at an example where we might call a map for a function with no return value.

Example of ThreadPool map() with No Return Value

We can explore using the map() function to call a function for each item in an iterable that does not have a return value.

This means that we are not interested in the iterable of results returned by the call to map() and instead are only interested that all issued tasks get executed.

This can be achieved by updating the previous example so that the task() function does not return a value.

The updated task() function with this change is listed below.

Then, in the main thread, we can call map() with our task() function and the range, and not iterate the results.

Importantly, the call to map() on the ThreadPool will block the main thread until all issued tasks are completed.

Once completed, the call will return and the ThreadPool will be closed by the context manager.

This is a helpful pattern to issue many tasks to the ThreadPool with a single function call.

Tying this together, the complete example is listed below.

Running the example first creates the ThreadPool with a default configuration.

The map() function is then called for the range. This issues ten calls to the task() function, one for each integer between 0 and 9. An iterator is returned with the result for each function call, but is ignored in this case.

Each call to the task function generates a random number between 0 and 1, reports a message, then blocks.

The main thread blocks until the map() function returns.

The tasks finish, map() returns, then the ThreadPool is closed.

This example again highlights that the call to map() blocks until all issued tasks are completed.

Next, let’s explore the chunksize argument to the map() function.

Example of ThreadPool map() with chunksize

The map() function will apply a function to each item in an iterable.

If the iterable has a large number of items, it may be inefficient to issue function calls to the target function for each item.

A more efficient approach would be to divide the items in the iterable into chunks and issue chunks of items to each worker thread to which the target function can be applied.

This can be achieved with the “chunksize” argument to the map() function.

We can demonstrate this with an example, first without using the “chunksize” argument, then using the “chunksize” to speed up the division of work.

Example Without Chunks

Before we demonstrate the “chunksize” argument, we can devise an example that has a reasonably large iterable.

In this example, we can update the previous example to call the task() function that returns a value as before, but to have 4 worker threads and to call the task() function 40 times, for integers 0 and 39

This number of worker threads and calls to task() were chosen so that we can test the “chunksize” argument in the next section. If you have fewer than 4 logical CPU cores in your system, change the example accordingly, e.g. 2 worker threads and 20 tasks.

Tying this together, the complete example is listed below.

Running the example first creates the ThreadPool with 4 thread workers.

The map() function is then called for the range.

This issues 40 calls to the task() function, one for each integer between 0 and 39. An iterator is returned with the result for each function call, in order.

Each call to the task function generates a random number between 0 and 1, reports a message, blocks, then returns a value.

The main thread iterates over the values returned from the calls to the task() function and reports the generated values, matching those generated in each worker thread.

On my system, the example took about 12.2 seconds to complete.

Next, let’s look at the same example using a chunksize.

Example With Chunks

We can update the previous example to use a chunksize.

This can be achieved by adding a “chunksize” argument to the call to map().

Given that we are issuing 40 calls to the task() function to the ThreadPool and that we have 4 worker threads, we can divide the work evenly into 4 groups of ten function calls.

This even division is a good starting point, but it is always a good idea to test different values of the chunksize and optimize it for the performance of your specific application with some trial and error.

This means we can set the chunksize to 10.

Tying this together, the complete example is listed below.

Running the example first creates the ThreadPool with 4 thread workers.

The map() function is then called for the range with a chunksize of 10.

This issues 4 units of work to the ThreadPool, one for each worker thread and each composed of 10 calls to the task() function.

Each call to the task function generates a random number between 0 and 1, reports a message, blocks, then returns a value.

The main thread iterates over the values returned from the calls to the task() function and reports the generated values, matching those generated in each worker thread.

On my system, the example took about 10.2 seconds to complete. This is about 1.20x faster.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Python ThreadPool Jump-Start, Jason Brownlee (my book!)
• Threading API Interview Questions
• ThreadPool PDF Cheat Sheet

I also recommend specific chapters from the following books:

• Python Cookbook, David Beazley and Brian Jones, 2013.
  • See: Chapter 12: Concurrency
• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python ThreadPool: The Complete Guide
• Python Multiprocessing Pool: The Complete Guide
• Python ThreadPoolExecutor: The Complete Guide
• Python Threading: The Complete Guide

APIs

• multiprocessing - Process-based parallelism

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.
• Thread Pool, Wikipedia.

Takeaways

You now know how to use the map() method to issue many tasks to the ThreadPool.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Kevin Schmid on Unsplash