You can create a daemon process in Python via the “daemon” argument to the multiprocessing.Process constructor or via the “daemon” property on a Process instance
In this tutorial you will discover how to create, configure and use daemon processes in Python.
Let’s get started.
Need for Daemon Processes
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create new child processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
You can learn more about multiprocessing in the tutorial:
In concurrent programming, we may need to execute sporadic, periodic or long-running tasks in the background.
A special type of process is used for background tasks, called a daemon process.
What is a daemon process and how can we use it in Python?
What is a Daemon Process
A daemon thread or a daemon process runs in the background.
Daemon is pronounced “dee-mon“, like the alternate spelling “demon“.
The idea is that background processes are like “daemons” or spirits (from the ancient Greek) that do tasks for you in the background. You might also refer to daemon processes as daemonic.
A process may be configured to be a daemon or not, and most processes in concurrent programming, including the main parent process, are non-daemon processes (not background processes) by default.
The property of being a daemon process or not may be supported by the underlying operating system that actually creates and manages the execution of threads and processes.
Daemon processes are helpful for executing tasks in the background to support the non-daemon processes in an application.
Uses of daemon processes might include:
- Background logging to file or database.
- Background data retrieval, updates, refresh.
- Background data storage to disk or database.
Background tasks can be varied in type and specific to your application.
Some properties of these tasks might include:
- Sporadic: Tasks that run only when specific conditions arise (e.g. ad hoc logging).
- Periodic: Tasks that run after a consistent interval (e.g. data save/load every minute).
- Long-Running: Tasks that run for the duration of the program (e.g. monitoring a resource).
Typically daemon processes execute non-critical tasks that although may be useful to the application are not critical if they fail (perhaps silently) or are canceled mid-operation.
By definition, a daemon process will not have control over when it is terminated.
The main parent process will terminate once all non-daemon processes finish, even if there are daemon processes still running. Therefore, the code it executes must be robust to arbitrary termination, such as the flushing and closing of external resources like streams and files that may not be closed correctly.
When a process exits, it attempts to terminate all of its daemonic child processes.
— multiprocessing — Process-based parallelism
Note, a daemon process is an extension of a daemon thread to processes. You can learn more about daemon threads in the tutorial:
Now that we know what daemon processes are, let’s compare them to non-daemon processes.
Daemon vs Non-Daemon Processes
We can better understand daemon processes by comparing them to non-daemon processes.
The difference between daemon processes and non-daemon processes is that the main parent process will exit if only daemon processes are running, whereas it cannot exit if at least one non-daemon process is running.
- Daemon: A main parent process will exit if only daemon processes are running (or if no processes are running).
- Non-Daemon: A main parent process will not exit if at least one non-daemon process is running.
This property makes daemon processes suited to executing background tasks that support the application, but are not critical if they are terminated at an ad hoc time by the parent process.
Now that we understand when to use daemon processes, let’s look at how we might create them in Python.
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
How to Create Daemon Processes
A Python multiprocessing.Process instance can be configured to be a daemon process.
There are 2 ways to create a new daemon process, they are:
- Set the daemon argument to True of the multiprocessing.Process class constructor.
- Set the daemon property to True on a multiprocessing.Process instance.
Set Daemon Argument
We can configure a new process to be a daemon process by specifying the “daemon” argument to True in the constructor of the multiprocessing.Process class.
For example:
... # create a new daemon process process = Process(daemon=True, ...) |
Set Daemon Property
Alternatively, we can configure a new process to be a daemon process after it has been constructed via the “daemon” property.
For example:
... # create a new daemon process process = Process(...) # configure the process to be a daemon process process.daemon = True |
The “daemon” property can only be set on a new process if the new process is not running. That is, before calling the start() method, also signified by the is_alive() method on the process returning False.
If a running process is configured to be daemon, then perhaps an is raised (as is the case with re-configuring threads to be daemons).
Inhering Daemon Property
A new multiprocessing.Process instance will inherit the value of the “daemon” property from the process in which they are created.
If provided, the keyword-only daemon argument sets the process daemon flag to True or False. If None (the default), this flag will be inherited from the creating process.
— multiprocessing — Process-based parallelism
The main parent process is the default process created in a Python program. By default this process is not a daemon process.
Nevertheless, a daemon process is not able to create new child processes. Only non-daemon processes are able to create child processes.
Note that a daemonic process is not allowed to create child processes. Otherwise a daemonic process would leave its children orphaned if it gets terminated when its parent process exits.
— multiprocessing — Process-based parallelism
Now that we know how to create and configure a daemon process, let’s look at some worked examples.
Example of Checking if the Current Process is Daemon
We can explore how to check if the current process is a daemon process or not.
First, we must get a multiprocessing.Process instance for the current process.
This can be achieved via the multiprocessing.current_process() function.
... # get the current process process = current_process() |
Next, we can report whether the current process is a daemon process via the “daemon” property on the multiprocessing.Process instance.
... # report if daemon process print(f'Daemon process: {process.daemon}') |
Tying this together, the complete example is listed below.
# SuperFastPython.com # example of checking if the current process is a daemon from multiprocessing import current_process # get the current process process = current_process() # report if daemon process print(f'Daemon process: {process.daemon}') |
Running the example first gets the multiprocessing.Process instance for the current process.
The daemon property of the current process retrieved and reported, showing that in this case the current process is not a daemon process.
This is to be expected because the code is running in the main parent process (the default process of a Python program), which is not a daemon process.
Next, let’s look at how we can check if a new process is a daemon process or not.
Example of Checking if a Process is a Daemon
We can explore how we might check if a new process is a daemon process or not.
First, we can define a function named task() that will be executed by a new process.
# function to be executed in a new process def task(): # ... |
Within the function, we will first get the multiprocessing.Process instance for the process executing the function.
... # get the current process process = current_process() |
Next, we can report whether the process is a daemon process or not.
... # report if daemon process print(f'Daemon process: {process.daemon}') |
Next, in the main process we can create a new multiprocessing.Process instance and configure it to execute our task() function in a new process via the “target” argument, specifying the name of our function.
... # create a new process with a default value for "daemon" process = Process(target=task) |
Finally, we can start the new process.
... # start the new process process.start() |
Tying this together, the complete example of checking if a new process is a daemon process is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# SuperFastPython.com # example of checking if a new process is a daemon process from multiprocessing import current_process from multiprocessing import Process # function to be executed in a new process def task(): # get the current process process = current_process() # report if daemon process print(f'Daemon process: {process.daemon}') # entry point if __name__ == '__main__': # create a new process with a default value for "daemon" process = Process(target=task) # start the new process process.start() |
Running the example first creates a new process with the default value of the “daemon” property. This will be inherited from the parent process, which is the main parent process in this case and not a daemon process.
Next, the new process executes and retrieves the multiprocessing.Process instance for the process that is executing the task() function and reports whether it is a daemon or not.
In this case, we can see that the new process is not a daemon process, as we expected, because it inherited the default daemon value of False from the main process.
Next, let’s look at how we might create a new daemon process.
Example of Creating a Daemon Process
We can explore how to create a new daemon process.
This can be achieved by creating a new multiprocessing.Process instance and setting the “daemon” argument to True.
We can update the previous example to create a daemon process to execute our task() function.
... # create a new daemon process process = Process(target=task, daemon=True) |
The task() function will then report whether it is being executed by a daemon process or not.
Because the new process is a daemon process, we will block the main parent process for a moment so that the process does not terminate too soon.
... # wait for the new process to finish process.join() |
Tying this together, the complete example of creating a new daemon process is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# SuperFastPython.com # example of creating a new daemon process from time import sleep from multiprocessing import current_process from multiprocessing import Process # function to be executed in a new process def task(): # get the current process process = current_process() # report if daemon process print(f'Daemon process: {process.daemon}') # entry point if __name__ == '__main__': # create a new daemon process process = Process(target=task, daemon=True) # start the new process process.start() # wait for the new process to finish process.join() |
Running the example creates a new daemon process configured to execute our custom task() function.
Our task() function is then executed by the daemon process, which acquires an instance of the multiprocessing.Process and reports whether it is a daemon process or not.
In this case, it shows that indeed, the new process executing our function is a daemon process.
Next, let’s look at how we might configure a new process to be a daemon process after it has been created.
Example of Changing a Process to be a Daemon
We can explore how to configure a process to be a daemon process after it has been created.
This can be achieved by setting the “daemon” property to True.
We can update the previous example to create a new multiprocessing.Process that uses the default value of “daemon” inherited from the main parent process (e.g. not a daemon process).
... # create a new process process = Process(target=task) |
Once created, we can configure the process to be a daemon process.
... # configure the process to be a daemon process process.daemon = True |
We can then start the process as per normal which will execute our task() function.
... # start the new process process.start() |
Because the new process is a daemon process, we will block the main parent process for a moment so that the program does not terminate too soon.
... # block for a moment to let the daemon process run process.join() |
Tying this together, the complete example of configuring a new process to be a daemon process is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of configuring a process to be a daemon process from time import sleep from multiprocessing import current_process from multiprocessing import Process # function to be executed in a new process def task(): # get the current process process = current_process() # report if daemon process print(f'Daemon process: {process.daemon}') # entry point if __name__ == '__main__': # create a new process process = Process(target=task) # configure the process to be a daemon process process.daemon = True # start the new process process.start() # block for a moment to let the daemon process run process.join() |
Running the example first creates a new process configured to execute our custom target function, but the process is not a daemon process.
We then configure the new multiprocessing.Process instance to be a daemon process.
Finally, we start the new process which executes our task() function and reports that the process executing the function is a daemon process as we specified.
Next, let’s look at how we cannot create a daemon process from a daemon process.
Example of Creating a Daemon Process from a Daemon Process
We can explore how to create a daemon process from an existing daemon process.
Recall that a daemon process is unable to create new child processes, therefore it is not possible to create a daemon process from a daemon child process.
Let’s explore this limitation with a worked example.
We can update the previous example so that the task() function is executed by a daemon process, and in turn the task() function creates a new process with the default value of “daemon“, which will be inherited from the daemon process.
We expect this to fail with an error.
First, we can define a function for the second child process to execute that will get the multiprocessing.Process instance for the process running the function and report whether it is a daemon or not.
# function to be executed by second daemon process def task2(): # get the current process process = current_process() # report if daemon process print(f'Daemon process 2: {process.daemon}') |
We can then update the first task() function to create a new process and start it.
The new process will be configured to execute the task2() function we just defined above, and to inherit the default value of daemon from the current process, which we know is a daemon process.
... # create a new process new_process = Process(target=task2) # start the new process new_process.start() |
The updated version of the task() function is listed below.
# function to be executed in a new process def task(): # get the current process process = current_process() # report if daemon process print(f'Daemon process 1: {process.daemon}') # create a new process new_process = Process(target=task2) # start the new process new_process.start() |
Finally the main parent process will create a process instance and configure it to be a daemon process and to execute our task() function.
... # create a new daemon process process = Process(target=task, daemon=True) # start the new process process.start() |
We can then start the new daemon process which will report a message and in turn start a second daemon process that will report a message.
Because we have two new daemon processes and the main parent process has nothing left to do, we will block the main parent process for a moment and allow the daemon processes to report their messages before exiting the program.
... # block for a moment to let the daemon process run process.join() |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
# SuperFastPython.com # example of creating a daemon process from a daemon process from time import sleep from multiprocessing import current_process from multiprocessing import Process # function to be executed by second daemon process def task2(): # get the current process process = current_process() # report if daemon process print(f'Daemon process 2: {process.daemon}') # function to be executed in a new process def task(): # get the current process process = current_process() # report if daemon process print(f'Daemon process 1: {process.daemon}') # create a new process new_process = Process(target=task2) # start the new process new_process.start() # entry point if __name__ == '__main__': # create a new daemon process process = Process(target=task, daemon=True) # start the new process process.start() # block for a moment to let the daemon process run process.join() |
Running the example first creates a daemon process that executes our task() function.
The task() function reports that it is running in a daemon process and then attempts to start a second process that executes our task2() function.
Because this new process is created in a daemon process, it fails with an error, specifically an AssertionError indicating that daemon processes are not permitted to create child processes of their own.
Process Process-1: Traceback (most recent call last): ... AssertionError: daemonic processes are not allowed to have children Daemon process 1: True |
Next, let’s look at what happens if we try to configure the currently running process to be a daemon process.
Example of Changing The Current Process to Daemon While Running
We can explore what happens if we attempt to change the current process to be a daemon process after it has started.
Recall, if a process is already running then we cannot change the daemon property of the process.
The process’s daemon flag, a Boolean value. This must be set before start() is called.
— multiprocessing — Process-based parallelism
We can demonstrate this with an example.
First, we can get the multiprocessing.Process instance for the current process, which in this case will be the main parent process.
... # get the current process process = current_process() |
We can then report whether the current process is a daemon process or not.
... # report if daemon process print(f'Daemon process: {process.daemon}') |
Finally, we can attempt to change the current process to be a daemon process. That is, we will try and change the main parent process to be a daemon process, then report if the change took effect.
... # try and change the current process to be a daemon process.daemon = True # report if daemon process print(f'Daemon process: {process.daemon}') |
Tying this together, the complete example is listed below.
# SuperFastPython.com # example of changing the current process to be a daemon process from multiprocessing import current_process # get the current process process = current_process() # report if daemon process print(f'Daemon process: {process.daemon}') # try and change the current process to be a daemon process.daemon = True # report if daemon process print(f'Daemon process: {process.daemon}') |
Running the example first gets the multiprocessing.Process instance for the current process and reports that it is not a daemon process. This is to be expected because the current process is the main parent process.
Next, we attempt to change the configuration of the current process (the main process) to be a daemon process.
Thus succeeds, which conflicts with the documentation which indicates that it cannot be changed.
Daemon process: False Daemon process: True |
Note, this was tested with Python v3.9.10.
What result do you get?
Next, let’s look at an example of how daemon processes are terminated abruptly.
Example of Daemon Process Terminated Abruptly
We can explore how daemon child processes are stopped abruptly by the parent process.
You may recall that a parent process will exit once all non-daemon child processes have terminated.
The main parent process is a non-daemon process, so if it is the only non-daemon process running then the Python program will exit.
This is still the case even if we have one or more daemon processes currently running.
We can demonstrate this with an example.
First, we can update our custom task() function executed in a daemon process to perform a long-running task that will not be able to be completed in time.
In this case, it will count from 0 to 999 and print each integer with 1/10th of a second gap between each print statement.
... # loop for a while for i in range(1000): print(i, flush=True) # block for a moment sleep(0.1) |
The updated version of the task() function with this change is listed below.
# function to be executed in a new process def task(): # get the current process process = current_process() # report if daemon process print(f'Daemon process: {process.daemon}') # loop for a while for i in range(1000): print(i, flush=True) # block for a moment sleep(0.1) |
Next, we can create a new daemon process and configure it to execute our task() function.
... # create a new daemon process process = Process(target=task, daemon=True) # start the new process process.start() |
Finally, we can block the main parent process for a moment to allow the daemon process to make some progress on its task.
... # block for a moment to let the daemon process run sleep(3) # prepare the user print('Main process exiting...') |
The idea is that the daemon process will not be able to complete its task before the main parent process terminates. This will in turn cause the Python program to exit and abruptly terminate the daemon process in the middle of its task.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of daemon processes being terminated abruptly from time import sleep from multiprocessing import current_process from multiprocessing import Process # function to be executed in a new process def task(): # get the current process process = current_process() # report if daemon process print(f'Daemon process: {process.daemon}') # loop for a while for i in range(1000): print(i, flush=True) # block for a moment sleep(0.1) # entry point if __name__ == '__main__': # create a new daemon process process = Process(target=task, daemon=True) # start the new process process.start() # block for a moment to let the daemon process run sleep(3) # prepare the user print('Main process exiting...') |
Running the example creates a new daemon process that executes our task() function.
The task() function reports that it is running in a daemon process then starts its long-running task.
The main process blocks for a few seconds, then reports that it is done and that the Python program will exit.
The Python program then exits, abruptly terminating the daemon process in the middle of its task. In this case, we can see that it only managed to count to 28 out of 999.
If the process executing the task() function was configured to not be a daemon process, then the task() function would complete before the Python process exited. You can experiment with this as an exercise.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
Daemon process: True 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Main process exiting... |
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to use daemon processes in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Simon Hesthaven on Unsplash