You can extend the multiprocessing.Process class and override the run() function to run code in a new Process.
In this tutorial you will discover how to extend the multiprocessing.Process class.
Let’s get started.
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. Each Python process has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create additional processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
You can learn more about multiprocessing in the tutorial:
One way of running code in a new process is to extend the multiprocessing.Process class. This provides flexibility and allows data to be stored as instance variables and accessed by the new process.
It also allows functions associated with the new process to be grouped with the new object.
How can we extend the multiprocessing.Process class?
How to Extend the Process Class
The multiprocessing.Process class can be extended to run code in another process.
This can be achieved by first extending the class, just like any other Python class.
For example:
# custom process class class CustomProcess(multiprocessing.Process): # ... |
Then the run() function of the multiprocessing.Process class must be overridden to contain the code that you wish to execute in another process.
For example:
# override the run function def run(self): # ... |
And that’s it.
Given that it is a custom class, you can define a constructor for the class and use it to pass in data that may be needed in the run() function, stored such as instance variables (attributes).
You can also define additional functions in the class to split up the work you may need to complete in another process.
Finally, attributes can also be used to store the results of any calculation or IO performed in another process that may need to be retrieved afterward.
Next, let’s look at a worked example of extending the multiprocessing.Process class.
Example of Extending the Process Class
First, we can define a class that extends the multiprocessing.Process class.
We will name the class something arbitrary such as “CustomProcess“.
# custom process class class CustomProcess(Process): # ... |
We can then override the run() instance method and define the code that we wish to execute in another process.
In this case, we will block for a moment and then print a message.
# override the run function def run(self): # block for a moment sleep(1) # display a message print('This is coming from another process') |
Next, we can create an instance of our CustomProcess class and call the start() function to begin executing our run() function in another process. Internally, the start() function will call the run() function.
The code will then run in a new process as soon as the operating system can schedule it.
... # create the process process = CustomProcess() # start the process process.start() |
Finally, we wait for the new process to finish executing.
... # wait for the process to finish print('Waiting for the process to finish') process.join() |
Tying this together, the complete example of executing code in another process by extending the multiprocessing.Process class is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# SuperFastPython.com # example of extending the Process class from time import sleep from multiprocessing import Process # custom process class class CustomProcess(Process): # override the run function def run(self): # block for a moment sleep(1) # display a message print('This is coming from another process') # entry point if __name__ == '__main__': # create the process process = CustomProcess() # start the process process.start() # wait for the process to finish print('Waiting for the process to finish') process.join() |
Running the example first creates an instance of the process, then executes the content of the run() function.
Meanwhile, the main thread waits for the new process to finish its execution, before exiting.
Waiting for the process to finish This is coming from another process |
Free Python Multiprocessing Course
Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.
Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.
Example of Passing Arguments When Extending the Process Class
We may need data or variables within the run() function when extending the multiprocessing.Process class.
This data cannot be passed to the run() function via the start() function when starting the process.
Instead, we can pass any data needed by the run() function to the constructor of the extended multiprocessing.Process class and store them as instance variables (properties or attributes) in the class. We can then access these attributes from within the run() function.
Let’s make this concrete with a worked example.
We will update the example of extending the multiprocessing.Process class in the previous section to use an argument that specifies how long the process will block. We need to get this value from the main thread into the run() function of the extended multiprocessing.Process class.
Firstly, we can override the constructor of the multiprocessing.Process class and define a new constructor to take any data we require. In this case, the “value” time in seconds that we wish to block.
# override the constructor def __init__(self, value): # ... |
Within the constructor, we must first call the parent constructor on the multiprocessing.Process class.
... # execute the base constructor Process.__init__(self) |
Next, we can store the “value” argument as an attribute on the object instance.
... # store the value self.value = value |
We can then update the code in the run() function to use the “value” instance attribute.
... # block for a moment sleep(self.value) # display a message print(f'This is coming from another process: {self.value}') |
Tying this together, the updated CustomProcess class that overrides the constructor to get data into the run() function is listed below.
# custom process class class CustomProcess(Process): # override the constructor def __init__(self, value): # execute the base constructor Process.__init__(self) # store the value self.value = value # override the run function def run(self): # block for a moment sleep(self.value) # display a message print(f'This is coming from another process: {self.value}') |
Finally, we can pass a value to the constructor when creating an instance of our CustomProcess class.
... # create the process process = CustomProcess(0.5) |
The complete example of getting data variables to the run() function when extending the multiprocessing.Process class is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of extending the Process class and passing arguments from time import sleep from multiprocessing import Process # custom process class class CustomProcess(Process): # override the constructor def __init__(self, value): # execute the base constructor Process.__init__(self) # store the value self.value = value # override the run function def run(self): # block for a moment sleep(self.value) # display a message print(f'This is coming from another process: {self.value}') # entry point if __name__ == '__main__': # create the process process = CustomProcess(0.5) # start the process process.start() # wait for the process to finish print('Waiting for the process to finish') process.join() |
Running the example first creates an instance of our extended multiprocessing.Process class and passing data into the constructor. The override constructor first calls the parent constructor, then stores the new data as an instance variable.
Finally the new process is started by calling start() which in turn internally calls the run() function which makes use of the instance variable data.
Waiting for the process to finish This is coming from another process: 0.5 |
Further Reading
This section provides additional resources that you may find helpful.
Python Multiprocessing Books
- Python Multiprocessing Jump-Start, Jason Brownlee (my book!)
- Multiprocessing API Interview Questions
- Multiprocessing API Cheat Sheet
I would also recommend specific chapters in the books:
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- High Performance Python, Ian Ozsvald and Micha Gorelick, 2020.
- See: Chapter 9: The multiprocessing Module
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Multiprocessing: The Complete Guide
- Python Multiprocessing Pool: The Complete Guide
- Python ProcessPoolExecutor: The Complete Guide
APIs
References
Takeaways
You now know how to extend the multiprocessing.Process class in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Peter Fogden on Unsplash