How to Use asyncio.gather() in Python

We need ways to work with collections of tasks in a way that they can be treated as a group.

The asyncio.gather() module function provides this capability and will return an iterable of return values from the awaited asks.

In this tutorial, you will discover how to await asyncio tasks concurrently with asyncio.gather().

After completing this tutorial, you will know:

That the asyncio.gather() function will wait for a collection of tasks to complete and retrieve all return values.
How to use asyncio.gather() with collections of coroutines and collections of tasks.
How to use asyncio.gather() to create nested groups of tasks that can be awaited and cancelled.

Let’s get started.

What is Asyncio gather()

The asyncio.gather() module function allows the caller to group multiple awaitables together.

Once grouped, the awaitables can be executed concurrently, awaited, and canceled.

Run awaitable objects in the aws sequence concurrently.
— Coroutines and Tasks

It is a helpful utility function for both grouping and executing multiple coroutines or multiple tasks.

For example:

...

# run a collection of awaitables

results = await asyncio.gather(coro1(), asyncio.create_task(coro2()))

We may use the asyncio.gather() function in situations where we may create many tasks or coroutines up-front and then wish to execute them all at once and wait for them all to complete before continuing on.

This is a likely situation where the result is required from many like-tasks, e.g. same task or coroutine with different data.

The awaitables can be executed concurrently, results returned, and the main program can resume by making use of the results on which it is dependent.

The gather() function is more powerful than simply waiting for tasks to complete.

It allows a group of awaitables to be treated as a single awaitable.

This allows:

Executing and waiting for all awaitables in the group to be done via an await expression.
Getting results from all grouped awaitables to be retrieved later via the result() method.
The group of awaitables to be canceled via the cancel() method.
Checking if all awaitables in the group are done via the done() method.
Executing callback functions only when all tasks in the group are done.

And more.

The asyncio.gather() function is similar to asyncio.wait(). It will suspend until all provided tasks are done, except it will return an iterable of return values from all tasks, whereas wait(), by default, will not retrieve task results.

You can learn more about how asyncio.gather() is different from asyncio.wait() in the tutorial:

Asyncio gather() vs wait() in Python

Now that we know what the asyncio.gather() function is, let’s look at how we might use it.

In this section, we will take a closer look at how we might use the asyncio.gather() function.

gather() Takes Tasks and Coroutines

The asyncio.gather() function takes one or more awaitables as arguments.

Recall an awaitable may be a coroutine, a Future or a Task.

Therefore, we can call the gather() function with:

Multiple tasks
Multiple coroutines
Mixture of tasks and coroutines

For example:

...

# execute multiple coroutines

asyncio.gather(coro1(), coro2())

If Task objects are provided to gather(), they will already be running because Tasks are scheduled as part of being created.

The asyncio.gather() function takes awaitables as position arguments.

We cannot create a list or collection of awaitables and provide it to gather, as this will result in an error.

For example:

...

# cannot provide a list of awaitables directly

asyncio.gather([coro1(), coro2()])

A list of awaitables can be provided if it is first unpacked into separate expressions using the star operator (*), also called the asterisk operator.

This operator specifically unpacks iterables, like lists, into separate expressions. It is often referred to as the iterable unpacking operator.

An asterisk * denotes iterable unpacking. Its operand must be an iterable. The iterable is expanded into a sequence of items, which are included in the new tuple, list, or set, at the site of the unpacking.
Expression lists

For example:

...

# gather with an unpacked list of awaitables

asyncio.gather(*[coro1(), coro2()])

If coroutines are provided to gather(), they are wrapped in Task objects automatically.

gather() Returns a Future

The gather() function does not block.

Instead, it returns an asyncio.Future object that represents the group of awaitables.

For example:

...

# get a future that represents multiple awaitables

group = asyncio.gather(coro1(), coro2())

Once the Future object is created it is scheduled automatically within the event loop.

The awaitable represents the group, and all awaitables in the group will execute as soon as they are able.

This means that if the caller did nothing else, the scheduled group of awaitables will run (assuming the caller suspends).

It also means that you do not have to await the Future that is returned from gather().

For example:

...

# get a future that represents multiple awaitables

group = asyncio.gather(coro1(), coro2())

# suspend and wait a while, the group may be executing..

await asyncio.sleep(10)

Awaiting gather()’s Future

The returned Future object can be awaited which will wait for all awaitables in the group to be done.

For example:

...

# run the group of awaitables

await group

Awaiting the Future returned from gather() will return a list of return values from the awaitables.

If the awaitables do not return a value, then this list will contain the default “None” return value.

For example:

...

# run the group of awaitables and get return values

results = await group

This is more commonly performed in one line.

For example:

...

# run tasks and get results on one line

results = await asyncio.gather(coro1(), coro2())

gather() Can Nest Groups of Awaitables

The gather() function takes awaitables and itself returns an awaitable.

Therefore, we can create nested groups of awaitables.

For example:

...

# create a group of tasks

group1 = asyncio.gather(coro1(), coro2())

# create another group of tasks

group2 = asyncio.gather(group1, coro3())

# run group2 which will also run group1

await group2

gather() and Exceptions

If an awaitable fails with an exception, the exception is re-raised in the caller and may need to be handled.

For example:

...

try:

# run tasks and get results

results = await asyncio.gather(coro1(), coro2())

except Exception as e:

# ...

Similarly, if a task in the group is canceled, it will re-raise a CancelledError exception in the caller and may need to be handled.

The “return_exceptions” argument to gather() can be set to True which will catch exceptions and provide them as return values instead of re-raising them in the caller.

This applies to both exceptions raised in awaitables, as well as CancelledError exceptions if the awaitables are canceled.

For example:

...

# run tasks and retrieve exceptions as a result

results = await asyncio.gather(coro1(), coro2(), return_exceptions=True)

gather()’s Future Can Be Canceled

The Future that is returned from gather() can be used just like a normal asyncio.Future object.

We can check if it is done by calling the done() method.

For example:

...

# check if a gather group of tasks is done

if group.done():

# ...

We can also cancel the gather()‘s Future, which will cancel all tasks within the group.

For example:

...

# cancel all tasks in the group

group.cancel()

Other helpful methods we might use include adding a done callback function, getting a result, and checking if the group was canceled.

You can learn more about using a done callback on a gather’s task in the tutorial:

Asyncio gather() Add Done Callback Function

Now that we know how to use asyncio.gather(), let’s look at some worked examples.

Examples of gather() with Coroutines

In this section, we will look at how we can run coroutines concurrently using the asyncio.gather() function.

Example of gather() For One Coroutine

We can use gather() to manage a single coroutine.

A coroutine can be executed directly by creating it and await it. Nevertheless, we can use the gather() function to wrap it in a Future object and create a group of one awaitable that can be managed.

The example below defines a task coroutine that is gathered and run.

# SuperFastPython.com

# example of gather for one coroutine

import asyncio

# coroutine used for a task

async def task_coro():

# report a message

print('task executing')

# sleep for a moment

await asyncio.sleep(1)

# coroutine used for the entry point

async def main():

# report a message

print('main starting')

# create a coroutine

coro = task_coro()

# gather one coroutine

await asyncio.gather(coro)

# report a message

print('main done')

# start the asyncio program

asyncio.run(main())

Running the example creates and executes the main() coroutine as the entry point to the asyncio program.

The main() coroutine reports a message and then creates the task coroutine.

The gather() function is then called, passing the single task coroutine as an argument. This returns a Future object that is awaited.

The main() coroutine is suspended and the task coroutine is executed, reporting a message and sleeping before terminating

The main() coroutine then resumes and reports a final message.

This highlights how we might run a single coroutine using gather().

main starting

task executing

main done

Example of gather() For Many Coroutines

We can use gather() to group many coroutines together into a single task.

This is the most common use case for the gather() function.

The example below defines a task coroutine that takes an argument, reports a message with the argument, and sleeps for a moment.

The main coroutine then calls gather() with three separate calls to the task coroutine, each with different arguments. This allows all three coroutines to be executed concurrently and for the caller to wait for all three coroutines to complete before resuming.

The complete example is listed below.

# SuperFastPython.com

# example of gather for many coroutines

import asyncio

# coroutine used for a task

async def task_coro(value):

# report a message

print(f'>task {value} executing')

# sleep for a moment

await asyncio.sleep(1)

# coroutine used for the entry point

async def main():

# report a message

print('main starting')

# run the tasks

await asyncio.gather(task_coro(0),

task_coro(1),

task_coro(2))

# report a message

print('main done')

# start the asyncio program

asyncio.run(main())

Running the example executes the main() coroutine as the entry point to the program.

A message is reported and then the gather() function is called. It is passed three different coroutine instances that call the same coroutine function with different arguments.

The coroutines are run concurrently and the main coroutine() suspends and waits for the group of awaitables to be done.

Each coroutine runs, reports its message, sleeps for a moment, and terminates.

The main() coroutine resumes only after all three coroutines are completed, and then reports its final message,

This highlights the most common use case of gathering multiple coroutines and waiting for them to complete.

main starting

>task 0 executing

>task 1 executing

>task 2 executing

main done

Example of gather() For Many Coroutines in a List

It is common to create multiple coroutines beforehand and then gather them later.

This allows a program to prepare the tasks that are to be executed concurrently and then trigger their concurrent execution all at once and wait for them to complete.

We can collect many coroutines together into a list either manually or using a list comprehension.

For example:

...

# create many coroutines

coros = [task_coro(i) for i in range(10)]

We can then call gather() with all coroutines in the list.

The list of coroutines cannot be provided directly to the gather() function as this will result in an error.

Instead, the gather() function requires each awaitable to be provided as a separate positional argument.

This can be achieved by unwrapping the list into separate expressions and passing them to the gather() function. The star operator (*) will perform this operation for us.

For example:

...

# run the tasks

await asyncio.gather(*coros)

Are you new to the star or asterisk (*) operator? It is used for iterable unpacking. You can learn more in the expression API documentation.

Tying this together, the complete example of running a list of pre-prepared coroutines with gather() is listed below.

# SuperFastPython.com

# example of gather for many coroutines in a list

import asyncio

# coroutine used for a task

async def task_coro(value):

# report a message

print(f'>task {value} executing')

# sleep for a moment

await asyncio.sleep(1)

# coroutine used for the entry point

async def main():

# report a message

print('main starting')

# create many coroutines

coros = [task_coro(i) for i in range(10)]

# run the tasks

await asyncio.gather(*coros)

# report a message

print('main done')

# start the asyncio program

asyncio.run(main())

Running the example executes the main() coroutine as the entry point to the program.

The main() coroutine then creates a list of 10 coroutine objects using a list comprehension.

This list is then provided to the gather() function and unpacked into 10 separate expressions using the star operator.

The main() coroutine then awaits the Future object returned from the call to gather(), suspending and waiting for all scheduled coroutines to complete their execution.

The coroutines run as soon as they are able, reporting their unique messages and sleeping before terminating.

Only after all coroutines in the group are complete does the main() coroutine resume and report its final message.

This highlights how we might prepare a collection of coroutines and provide them as separate expressions to the gather() function.

main starting

>task 0 executing

>task 1 executing

>task 2 executing

>task 3 executing

>task 4 executing

>task 5 executing

>task 6 executing

>task 7 executing

>task 8 executing

>task 9 executing

main done

Example of gather() With Return Values

We may execute coroutines that return a value.

Using the gather() function to execute multiple coroutines that return values allows the results of all coroutines to be gathered together at one point for use, giving the gather() function its name.

Recall that the gather() function does not block, but returns immediately with a Future object.

When the Future object that is returned from gather() is awaited, it will return a list of return values from the grouped tasks.

The example below demonstrates this by updating the task coroutine to return a value and gathering and reporting the list of return values from all tasks in the main coroutine.

# SuperFastPython.com

# example of gather for many coroutines that return values

import asyncio

# coroutine used for a task

async def task_coro(value):

# report a message

print(f'>task {value} executing')

# sleep for a moment

await asyncio.sleep(1)

# return a value

return value * 10

# coroutine used for the entry point

async def main():

# report a message

print('main starting')

# create many tasks

tasks = [task_coro(i) for i in range(10)]

# run the tasks

values = await asyncio.gather(*tasks)

# report the values

print(values)

# report a message

print('main done')

# start the asyncio program

asyncio.run(main())

Running the example first creates and runs the main() coroutine as the entry point into the program.

A list of coroutines is then created, each with a different argument.

The gather function is called and the list of coroutines is then unpacked into separate expressions using the star operator.

This schedules each coroutine for execution.

The main coroutine suspends and waits for all coroutines in the group to complete.

Each coroutine runs, reporting a message, sleeping, and returning a value that is a multiple of 10 of the input argument.

All coroutines are completed and the awaited Future returned from gather() provides a list of return values.

The main() coroutine then reports the list of return values.

This highlights how we can retrieve return values from multiple coroutines that return values.

main starting

>task 0 executing

>task 1 executing

>task 2 executing

>task 3 executing

>task 4 executing

>task 5 executing

>task 6 executing

>task 7 executing

>task 8 executing

>task 9 executing

[0, 10, 20, 30, 40, 50, 60, 70, 80, 90]

main done

Example of gather() With Nested Groups

The gather() function can be used to create one awaitable that represents multiple awaitables.

The natural consequence of this is that we can nest groups of awaitables.

We can demonstrate this with a worked example.

In this example, we create two separate groups of three coroutines. We then define a third group that combines the two previously created groups of coroutines.

This creates a hierarchy with three levels of awaitables, for example:

group 3
- group 1
  - coroutine 0
  - coroutine 1
  - coroutine 2
- group 2
  - coroutine 3
  - coroutine 4
  - coroutine 5

Performing an operation on the top-level awaitable will perform an operation on all nested awaitables.

Most importantly, running the top-level will run all nested awaitables concurrently.

The complete example is listed below.

# SuperFastPython.com

# example of gather nested groups of coroutines

import asyncio

# coroutine used for a task

async def task_coro(value):

# report a message

print(f'>task {value} executing')

# sleep for a moment

await asyncio.sleep(1)

# coroutine used for the entry point

async def main():

# report a message

print('main starting')

# create group level 1

group1 = asyncio.gather(task_coro(0), task_coro(1), task_coro(2))

# create group level 2

group2 = asyncio.gather(task_coro(3), task_coro(4), task_coro(5))

# create group level 3

group3 = asyncio.gather(group1, group2)

# execute 3, which executes 1 and 2

await group3

# report a message

print('main done')

# start the asyncio program

asyncio.run(main())

Running the example first creates and runs the main() coroutine.

The main() coroutine first uses the gather() function to create two separate groups of coroutines. It then calls gather() to create a third group that combines the first two groups of coroutines.

It then awaits the third group.

This schedules all awaitables for execution and suspends the main coroutine until the hierarchy of awaitables is completed.

All awaitables are completed and the main coroutine resumes and reports a final message.

This highlights how we can easily create hierarchies of groups of awaitables using the gather() function.

main starting

>task 0 executing

>task 1 executing

>task 2 executing

>task 3 executing

>task 4 executing

>task 5 executing

main done

Example of gather() Mix of Tasks and Coroutines

Like a coroutine, an asyncio.Task is an awaitable.

This means that we can provide a mixture of Task objects as well as coroutines to the gather() function.

The coroutines provided to gather() will be wrapped in Task objects and scheduled for execution.
The Task objects will already be scheduled for execution.
The Future object that is returned from the gather() function will be scheduled for execution.

Although we can await the Future that is returned from the gather() function, this does not execute the group, the group is already scheduled.

Mixing tasks and coroutines in the call to gather makes this more apparent.

It also means that some awaitables provided to a call to gather() could already be done. Awaiting the group only waits for those awaitables that are not already done.

The example below explores the ability to mix coroutines and awaitables in a call to gather().

It also does not explicitly await the Future object returned from the gather() function but instead sleeps while the group of awaitables is executed.

The complete example is listed below.

# SuperFastPython.com

# example of gather with of tasks and coroutines

import asyncio

# coroutine used for a task

async def task_coro(value):

# report a message

print(f'>task {value} executing')

# sleep for a moment

await asyncio.sleep(1)

# coroutine used for the entry point

async def main():

# report a message

print('main starting')

# create a mix of awaitables

awaitables = [task_coro(0),

asyncio.create_task(task_coro(1)),

task_coro(2),

asyncio.create_task(task_coro(3)),

task_coro(4),]

# schedule the group

_ = asyncio.gather(*awaitables)

# wait around for a while

await asyncio.sleep(2)

# report a message

print('main done')

# start the asyncio program

asyncio.run(main())

Running the example first creates and runs the main() coroutine.

The main() coroutine then creates a list that contains a mixture of coroutine objects and Task objects.

The Task objects are scheduled for execution as soon as they are created. The coroutines are not.

The list is then provided to the gather() function and the returned Future that represents the group is ignored.

This schedules the group of awaitables for execution in the event loop.

The main() coroutine then suspends with a call to sleep.

All tasks and coroutines are given an opportunity to execute, reporting their message and sleeping.

The main() coroutine resumes after its sleep and all coroutines and tasks happen to be complete by this time.

A final message is reported and the asyncio program then terminates.

This highlights that we can create a group that contains a mixture of tasks and coroutines and that we do not have to explicitly await the Future returned from gather() in order for the group to be executed.

main starting

>task 1 executing

>task 3 executing

>task 0 executing

>task 2 executing

>task 4 executing

main done

Free Python Asyncio Course

Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.

Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.

Learn more

Examples of gather() with Failures

This section explores using gather with awaitables that fail with an exception or are canceled.

Example of gather() Where One Awaitable Fails

An awaitable in group may fail with an unhandled exception.

If this happens, the exception will be re-raised in the caller and may need to be handled.

Recall that all awaitables are scheduled after the call to gather().

This means that even though one awaitable may have failed with an exception, the remaining awaitable may continue to execute and complete normally, if given the opportunity, e.g. the program is not terminated.

The example below explores this. A total of 10 coroutines are passed to a call to gather() and are scheduled. One of the coroutines will fail with an exception that is re-raised in the caller.

# SuperFastPython.com

# example of gather where one task fails with an exception

import asyncio

# coroutine used for a task

async def task_coro(value):

# report a message

print(f'>task {value} executing')

# sleep for a moment

await asyncio.sleep(1)

# check for failure

if value == 0:

raise Exception('Something bad happened')

# coroutine used for the entry point

async def main():

# report a message

print('main starting')

# create many coroutines

coros = [task_coro(i) for i in range(10)]

# run the tasks

await asyncio.gather(*coros)

# report a message

print('main done')

# start the asyncio program

asyncio.run(main())

Running the example creates and runs the main() coroutine as the entry point into the program.

The main() coroutine then creates 10 task coroutines in a list comprehension. These are then provided to a call to gather().

All coroutines are scheduled for execution along with the Future returned from gather().

The main() coroutine then suspends and waits for the tasks to complete.

All coroutines execute, reporting a message and blocking.

One coroutine then raises an exception.

The exception does not impact any other coroutines in the group.

The main() coroutine resumes and the exception raised in the coroutine is re-raised and not handled, terminating the application.

This highlights that exceptions that occur within a group of awaitables is re-raised in the caller.

main starting

>task 0 executing

>task 1 executing

>task 2 executing

>task 3 executing

>task 4 executing

>task 5 executing

>task 6 executing

>task 7 executing

>task 8 executing

>task 9 executing

Traceback (most recent call last):

...

Exception: Something bad happened

Example of gather() With Return Exceptions

We can prevent an exception in the group of awaitables from being re-raised in the caller by setting the “return_exceptions” argument to True when calling gather().

This will trap the exception and return it as a return value for the awaitable.

The example below demonstrates this, updating the above example to set the return_exceptions argument to True.

# SuperFastPython.com

# example of gather with returned exceptions

import asyncio

# coroutine used for a task

async def task_coro(value):

# report a message

print(f'>task {value} executing')

# sleep for a moment

await asyncio.sleep(1)

# check for failure

if value == 0:

raise Exception('Something bad happened')

# coroutine used for the entry point

async def main():

# report a message

print('main starting')

# create many coroutines

coros = [task_coro(i) for i in range(10)]

# run the tasks

results = await asyncio.gather(*coros, return_exceptions=True)

# report results

print(results)

# report a message

print('main done')

# start the asyncio program

asyncio.run(main())

Running the example creates and runs the main() coroutine as the entry point into the program.

The main() coroutine then creates 10 task coroutines in a list comprehension. These are then provided to a call to gather().

All coroutines are scheduled for execution along with the Future returned from gather().

The main() coroutine then suspends and waits for the tasks to complete.

All coroutines execute, reporting a message and blocking.

One coroutine then raises an exception.

The exception does not impact any other coroutines in the group and is not re-raised in the caller.

The caller retrieves a list of return values from all awaitables. It then reports the values.

We can see that all awaitables return None, except the one that failed, where we can see the Exception object that was returned.

This highlights that exceptions can be prevented from being re-raised and can be retrieved as a return value from a failed awaitable.

main starting

>task 0 executing

>task 1 executing

>task 2 executing

>task 3 executing

>task 4 executing

>task 5 executing

>task 6 executing

>task 7 executing

>task 8 executing

>task 9 executing

[Exception('Something bad happened'), None, None, None, None, None, None, None, None, None]

main done

Example of gather() Where One Task is Canceled

Tasks can be canceled.

It is possible that we call gather() with one or more Tasks or Future objects that have been canceled.

If a task is canceled in a group, it will raise a CancelledError in the wrapped coroutine and this exception will be re-raised in the caller.

The example below demonstrates this.

Two tasks are created, and the second is provided as a reference to the first. I then cancel the other task.

Both tasks are then grouped with a call to gather and the CancelledError is re-raised in the caller.

# SuperFastPython.com

# example of gather where one task is canceled

import asyncio

# coroutine used for a task

async def task_coro(value, friend):

# report a message

print(f'>task {value} executing')

# cancel friend task

if friend:

friend.cancel()

# sleep for a moment

await asyncio.sleep(1)

# coroutine used for the entry point

async def main():

# report a message

print('main starting')

# create many tasks

task0 = asyncio.create_task(task_coro(0, None))

task1 = asyncio.create_task(task_coro(1, task0))

# run the tasks

await asyncio.gather(task0, task1)

# report a message

print('main done')

# start the asyncio program

asyncio.run(main())

Running the example creates and runs the main() coroutine as the entry point into the program.

The main() coroutine then creates one task with no paired task. This task is scheduled immediately.

A second task is created and provides the first task as a reference. This task too is scheduled immediately.

Both tasks are then provided to the gather() function and the main coroutine awaits both tasks.

The first task is given an opportunity to execute, report a message, and sleep.

The second task is given an opportunity to execute, it reports a message, then cancels the first task, then sleeps.

The first task resumes and raises a CancelledError exception.

The main() task resumes and re-raises the CancelledError that terminates the asyncio program.

This highlights that tasks in a call to gather() can be canceled which causes the CancelledError to be re-raised in the calling coroutine.

main starting

>task 0 executing

>task 1 executing

Traceback (most recent call last):

...

asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

...

asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

...

asyncio.exceptions.CancelledError

Example of gather() Where One Task is Canceled With Return Exceptions

It is possible to not re-raise a CancelledError for a canceled task in a call to gather() by setting the “return_exceptions” argument to True.

This will cause the CancelledError to be returned as a return value, just as though the task failed with any arbitrary exception.

The example below demonstrates this by updating the previous example to set the “return_exceptions” argument to True, then retrieve and report the return values from all awaitables.

# SuperFastPython.com

# example of gather where one task is canceled with returned exceptions

import asyncio

# coroutine used for a task

async def task_coro(value, friend):

# report a message

print(f'>task {value} executing')

# cancel friend task

if friend:

friend.cancel()

# sleep for a moment

await asyncio.sleep(1)

# coroutine used for the entry point

async def main():

# report a message

print('main starting')

# create many tasks

task0 = asyncio.create_task(task_coro(0, None))

task1 = asyncio.create_task(task_coro(1, task0))

# run the tasks

results = await asyncio.gather(task0, task1, return_exceptions=True)

# report results

print(results)

# report a message

print('main done')

# start the asyncio program

asyncio.run(main())

Running the example creates and runs the main() coroutine as the entry point into the program.

The main() coroutine then creates one task with no paired task. This task is scheduled immediately.

A second task is created and provides the first task as a reference. This task too is scheduled immediately.

Both tasks are then provided to the gather() function with return_exceptions set to True and the main coroutine awaits both tasks.

The first task is given an opportunity to execute, report a message, and sleep.

The second task is given an opportunity to execute, it reports a message, then cancels the first task, then sleeps.

The first task resumes and raises a CancelledError exception.

The main() task resumes and the return values from both tasks are retrieved and reported. We can see the first task returned a CancelledError and the second returned the default value of None.

This highlights that a task in the group can be canceled, but does not have to raise a CancelledError in the calling coroutine.

main starting

>task 0 executing

>task 1 executing

[CancelledError(''), None]

main done

Example of Canceling All Tasks in gather()

As discussed, a call to gather() returns a Future object.

This Future object can be canceled by calling the cancel() method.

If canceled, a CancelledError exception is raised in the Future itself which in turn cancels all tasks in the group, if they are not done.

We can demonstrate this with a worked example.

In this example, we define a global variable that will reference the Future object returned from the call to gather(). Initially, it is set to None. It is then assigned within the main() coroutine.

Note, this is different from canceling all tasks in gather() if one task fails. For an example of this, see the tutorial:

Asyncio gather() Cancel All Tasks if One Task Fails

The complete example is listed below.

# SuperFastPython.com

# example of canceling a gather with many coroutines

import asyncio

# coroutine used for a task

async def task_coro(value):

# report a message

print(f'>task {value} executing')

# check if this is the special task

if value == 0:

global group

group.cancel()

# sleep for a moment

await asyncio.sleep(1)

# coroutine used for the entry point

async def main():

# report a message

print('main starting')

# create many coroutines

coros = [task_coro(i) for i in range(10)]

# create a group of tasks

global group

group = asyncio.gather(*coros)

# await the task

await group

# report a message

print('main done')

# define a global variable

group = None

# start the asyncio program

asyncio.run(main())

Running the example first defines a global variable named “group” then creates the main() coroutine and uses it as the entry point to the program.

The main() coroutine runs, first reporting a message, then creating a list of coroutines. They are provided with a call to gather and the Future object that is returned is assigned to the “group” global variable.

All coroutines are scheduled as tasks at this point.

The main() coroutine then awaits the group and suspends, giving the coroutines an opportunity to execute.

The first coroutine executes, reporting a message. It then accesses the “group” global variable which is a Future object and cancels it.

This has the effect of raising a CancelledError exception in the Future object returned from the gather() function, and in turn, canceling all coroutines in the group.

The tasks terminate with the exception.

The main() coroutine returns and the CancelledError exception is raised, terminating the asyncio program.

This highlights that all awaitables in a group can be canceled.

main starting

>task 0 executing

asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

...

asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

...

asyncio.exceptions.CancelledError

Common Errors With asyncio.gather()

This section considers some of the more common errors when using the asyncio.gather() function.

Example of gather() Error With a List

Perhaps the most common error when using the gather() function occurs when providing a list of awaitables to the function.

For example:

...

# create many coroutines

coros = [task_coro(i) for i in range(10)]

# run the tasks

await asyncio.gather(coros)

This is a problem because the gather() function does not take a list of awaitables, it takes multiple awaitables as separate positional arguments.

The example below demonstrates this common error.

# SuperFastPython.com

# example of gather with a list that results in an error

import asyncio

# coroutine used for a task

async def task_coro(value):

# report a message

print(f'>task {value} executing')

# sleep for a moment

await asyncio.sleep(1)

# coroutine used for the entry point

async def main():

# report a message

print('main starting')

# create many coroutines

coros = [task_coro(i) for i in range(10)]

# run the tasks

await asyncio.gather(coros)

# report a message

print('main done')

# start the asyncio program

asyncio.run(main())

Running the example creates and runs the main() coroutine as the entry point into the program.

It reports a message and then creates a list of coroutines.

The coroutines are then provided with a call to the gather() function.

This fails with a TypeError.

The reason is that the gather() function expects separate awaitable arguments (e.g. a variable number of awaitable arguments), not a list of arguments.

main starting

Traceback (most recent call last):

...

TypeError: unhashable type: 'list'

sys:1: RuntimeWarning: coroutine 'task_coro' was never awaited

Example of gather() Error Without Await

Another common error is to call gather() but not give the tasks an opportunity to execute.

Recall that when we call gather(), all awaitables will be scheduled for execution.

We do not have to await the Future that is returned from the gather() function.

Nevertheless, we do need to give the awaitables an opportunity to execute.

This can be achieved by suspending the calling coroutine, such as by awaiting a call to sleep() or some other task.

If this is not done, the coroutines are scheduled and never executed and the program will exit.

The example below demonstrates this problem.

# SuperFastPython.com

# example of gather without await results in an error

import asyncio

# coroutine used for a task

async def task_coro(value):

# report a message

print(f'>task {value} executing')

# sleep for a moment

await asyncio.sleep(1)

# coroutine used for the entry point

async def main():

# report a message

print('main starting')

# create many coroutines

coros = [task_coro(i) for i in range(10)]

# run the tasks

asyncio.gather(*coros)

# report a message

print('main done')

# start the asyncio program

asyncio.run(main())

Running the example creates and runs the main() coroutine as the entry point into the program.

It reports a message and then creates a list of coroutines.

The coroutines are then provided with a call to the gather() function.

The main() coroutine then reports a message and exits, terminating the program.

The scheduled tasks are never given an opportunity to execute.

The garbage collector attempts to cancel the scheduled tasks and is reported before the program finishes terminating.

main starting

main done

>task 0 executing

>task 1 executing

>task 2 executing

>task 3 executing

>task 4 executing

>task 5 executing

>task 6 executing

>task 7 executing

>task 8 executing

>task 9 executing

_GatheringFuture exception was never retrieved

future: <_GatheringFuture finished exception=CancelledError()>

Traceback (most recent call last):

...

asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

asyncio.exceptions.CancelledError

Loving The Tutorials?

Why not take the next step? Get the book.

Learn more

Takeaways

You now know how to run asyncio tasks concurrent with gather() in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Kahl Orr on Unsplash

What is Asyncio gather()

gather() Takes Tasks and Coroutines

gather() Returns a Future

Awaiting gather()’s Future

gather() Can Nest Groups of Awaitables

gather() and Exceptions

gather()’s Future Can Be Canceled

Examples of gather() with Coroutines

Example of gather() For One Coroutine

Example of gather() For Many Coroutines

Example of gather() For Many Coroutines in a List

Example of gather() With Return Values

Example of gather() With Nested Groups

Example of gather() Mix of Tasks and Coroutines

Examples of gather() with Failures

Example of gather() Where One Awaitable Fails

Example of gather() With Return Exceptions

Example of gather() Where One Task is Canceled

Example of gather() Where One Task is Canceled With Return Exceptions

Example of Canceling All Tasks in gather()

Common Errors With asyncio.gather()

Example of gather() Error With a List

Example of gather() Error Without Await

Further Reading

Takeaways