Дочерний процесс fork
Начал разбираться с функцией os.fork() в Python. В процессе гугления понял, что это стандартная функция в Linux для создания дочерних процессов. Как её использовать в терминале у себя на Ubuntu не понял. Внятного примера применения в Python тоже не нашел. Обычно это pid = os.fork() , а дальше какая-то магия. Что при этом происходит?
Отслеживать
19.1k 6 6 золотых знаков 30 30 серебряных знаков 44 44 бронзовых знака
задан 29 июл 2016 в 13:18
7,688 17 17 золотых знаков 70 70 серебряных знаков 131 131 бронзовый знак
29 июл 2016 в 13:28
29 июл 2016 в 14:16
1 ответ 1
Сортировка: Сброс на вариант по умолчанию
Создаётся полная КОПИЯ процесса, выполнившего вызов fork(). С точки зрения ОС они различаются только тем, что первый процесс был запущен Вами (допустим — из консоли), а второй — вашей программой, вызвавшей fork. Соответственно, у них всё абсолютно одинаковое. За исключением ppid — id родительского процесса. Первый будет иметь ppid==pid Вашего shell, а второй — pid этого процесса.
Соответственно, для того, что бы эти два ОДИНАКОВЫХ процесса могли работать далее по разному, ОС возвращает через fork() pid созданного процесса.
Для родителя это будет реальный pid процесса, который он создал. Для дочернего — ноль, так как он ничего не создавал.
Отсюда следует, что сразу после вызова fork(), нужно ОБЯЗАТЕЛЬНО использовать конструкцию вида:
if (pid > 0 ) < // Делаем что-то в родителе >else if (pid == 0) < // Делаем что-то в дочернем >else < // Обрабатываем ошибку >Python:How os.fork() works?
I am learning multiprocessing in python. I tried multiprocessing and after I read the source code of multiprocessing module, I found it use os.fork() , so I wrote some code to test os.fork() , but I am stuck. My code is as following:
#!/usr/bin/env python # -*- coding: utf-8 -*- import os import time for i in range(2): print '**********%d***********' % i pid = os.fork() print "Pid %d" % pid I think that each print will be executed two times but they execute three times. I can’t understand how this works? I read this Need to know how fork works?
From what this article says it also will be executed twice, so I am so stuck.
383 2 2 silver badges 7 7 bronze badges
asked Nov 6, 2015 at 6:32
419 3 3 gold badges 6 6 silver badges 11 11 bronze badges
3 Answers 3
First of all, remove that print ‘******. ‘ line. It just confuses everyone. Instead, let’s try this code.
import os import time for i in range(2): print("I'm about to be a dad!") time.sleep(5) pid = os.fork() if pid == 0: print("I'm <>, a newborn that knows to write to the terminal!".format(os.getpid())) else: print("I'm the dad of <>, and he knows to use the terminal!".format(pid)) os.waitpid(pid, 0) Okay, first of all, what is «fork»? Fork is a feature of modern and standard-compliant operating systems (except of M$ Windows: that joke of an OS is all but modern and standard-compliant) that allows a process (a.k.a: «program», and that includes the Python interpreter!) to literally make an exact duplicate of itself, effectively creating a new process (another instance of the «program»). Once that magic is done, both processes are independent. Changing anything in one of them does not affect the other one.
The process responsible for spelling out this dark and ancient incantation is known as the parent process. The soulless result of this immoral abomination towards life itself is known as the child process.
As shall be obvious to all, including those for which it isn’t, you can become a member of that select group of programmers who have sold their soul by means of os.fork() . This function performs a fork operation, and thus results in a second process being created out of thin air.
Now, what does this function return, or more importantly, how does it even return? If you want not to become insane, please don’t go and read the Linux kernel’s /kernel/fork.c file! Once the kernel does what we know it has to do, but we don’t want to accept it, os.fork() returns in the two processes! Yes, even the call stack is copied on!
So, if they are exact copies, how does one differentiate between parent and child? Simple. If the result of os.fork() is zero, then you’re working in the child. Otherwise, you’re working in the parent, and the return value is the PID (Process IDentifier) of the child. Anyway, the child can get its own PID from os.getpid() , no?
Now, taking this into account, and the fact that doing fork() inside a loop is the recipe for mess, this is what happens. Let’s call the original process the «master» process.
- Master: i = 0 , forks into child-#1-of-master
- Child-#1-of-master: i = 1 forks into child-#1-of-child-#1-of-master
- Child-#1-of-child-#1-of-master: for loop over, exits
- Child-#1-of-master: for loop over, exits
- Child-#2-of-master: i = 1 forks into child-#1-of-child-#2-of-master
- Child-#1-of-child-#2-of-master: for loop over, exits
- Child-#2-of-master: for loop over, exits
As you can see, there are a total of 6 parent/child prints coming from 4 unique processes, resulting in 6 lines of output, something like.
I’m the dad of 12120, and he knows to use the terminal!
I’m 12120, a newborn that knows to write to the terminal!
I’m the dad of 12121, and he knows to use the terminal!
I’m 12121, a newborn that knows to write to the terminal!
I’m the dad of 12122, and he knows to use the terminal!
I’m 12122, a newborn that knows to write to the terminal!
But that’s just arbitrary, it could have output this out instead.
I’m 12120, a newborn that knows to write to the terminal!
I’m the dad of 12120, and he knows to use the terminal!
I’m 12121, a newborn that knows to write to the terminal!
I’m the dad of 12121, and he knows to use the terminal!
I’m 12122, a newborn that knows to write to the terminal!
I’m the dad of 12122, and he knows to use the terminal!
Or anything other than that. The OS (and your motherboard’s funky clocks) is solely responsible for the order in which processes get timeslices, so go blame on Torvalds (and expect no self-steem when back) if you dislike how the kernel manages to organize your processes ;).
I hope this has led some light on you!
914 13 13 silver badges 19 19 bronze badges
answered Nov 6, 2015 at 7:07
8,358 2 2 gold badges 19 19 silver badges 42 42 bronze badges
Very nice explanation.
Dec 31, 2016 at 15:35This is quite possibly the most entertaining and insightful answer I’ve ever read on this site! Thumbs up for the sense knocks! 😀
Apr 3, 2018 at 17:44
I believe there are a total of 4 processes, not 6 which is very misleading. Otherwise. I love the entertaining writeup and gave upvote.
Jan 11, 2019 at 1:28
An excellent explanation!
Sep 25, 2021 at 15:38@RazzleShazl you are actually right, there will be only 4 processes/threads rather than 6. I loved the explanation though!
Mar 13 at 20:22
To answer the question directly, os.fork() works by calling the underlying OS function fork() .
But you are surely interested in what this does. Well this creates another process which will resume at exactly the same place as this one. So within the first loop run, you get a fork after which you have two processes, the «original one» (which gets a pid value of the PID of the child process) and the forked one (which gets a pid value of 0 ).
They both print their pid value and go on with the 2nd loop run, which they both print. Then they both fork, leaving you with 4 processes which all print their respective pid values. Two of them should be 0 , the other two should be the PIDs of the child they just created.
Changing the code to
#!/usr/bin/env python # -*- coding: utf-8 -*- import os import time for i in range(2): print '**********%d***********' % i pid = os.fork() if pid == 0: # We are in the child process. print "%d (child) just was created by %d." % (os.getpid(), os.getppid()) else: # We are in the parent process. print "%d (parent) just created %d." % (os.getpid(), pid)you will see better what happens: each process will tell you its own PID and what happened on the fork.
answered Nov 6, 2015 at 6:55
89.4k 13 13 gold badges 149 149 silver badges 219 219 bronze badges
Just for clarity: the bit inside if pid == 0: is the child process, the else is the parent.
Jan 5, 2019 at 9:51@RalphBolton Thank you. I just added this clarification to the code, along with the determination of the parent PID.
Jan 5, 2019 at 15:40
Just to add — in the child, you can get the PID of the parent by looking it up (with os.getpid() ) before you fork 😉
Jan 12, 2019 at 16:56
@RalphBolton Before the fork, it’s not yet in the child, strictly speaking. But if I call os.getpid() in the parent or os.getppid() in the child has the same result.
Jan 12, 2019 at 20:11
As both answers cover almost everything about how os.fork() works. I wanted to add some more specific information regarding the case.
Fork system call is used for creating a new process which is called the child process, which runs concurrently with the process that makes the fork() call (parent process). After a new child process created both processes will execute (Execution state; heap, stack, and processor registers) the next instructions following fork() system call(system call that creates a new process identical to the calling one, makes a copy of the text, data, stack, and heap). fork() does not take parameters and returns an integer value. After successfully call to fork(), the child process is basically an exact duplicate of the parent process. fork is Unix system call which is used to create a new process. System call way to request services from the kernel. fork returns a value, if the value greater than 0, that means successful fork but you may get below 0 as well, this might be that process out of memory or other errors, it cannot be created and it will raise an error. If it is equal to zero it is a child greater than 0, which means parent process. So the process runs concurrently, we archive here parallelization, but everything after fork call will be executed.
Calling fork duplicates process that is referred to parent. fork means two different identical copies of address base one for child one for the parent process. Both proses are exact copy of each other except PID and a few others. Since both processes have identical but separate address spaces, those variables initialized before the fork() call have the same values in both address spaces. Since every process has its own address space, any modifications will be independent of the others. In other words, if the parent changes the value of its variable, the modification will only affect the variable in the parent process address space. Other address spaces created by fork() calls will not be affected even though they have identical variable names.
Upon fork() all resources owned by the parent are duplicated and the copy is given to the child, In Linux, fork() is implemented through the use of Copy-on-Write pages. Copy on Write or simply CoW is a resource management technique. More about CoW
You can use fork, in case if you want to create parallel program with multiple processes.
Fork vs Spawn in Python Multiprocessing
I recently got stuck trying to plot multiple figures in parallel with Matplotlib. It took five hours to find a two-line fix to make it work. Afterwards I spent even more hours learning about multiprocessing in order to understand what had gone wrong and how the fix worked. This post is an attempt to capture what I learned.
The problem
The British Geological Survey’s Ash Model Plotting tool makes maps of simulated of volcanic ash clouds. Properties are calculated on a grid of locations for many different times and elevations, resulting in hundreds of maps. Creating the maps is CPU-bound and each map is independent, so plotting with multiple processes is a logical way to speed it up.
Python’s multiprocessing pool makes this easy. Using pool.map(plot_function, args) sets up multiple processes to call plot_function on the different args in parallel.
It didn’t take long to configure a pool for a simple script. Unfortunately, however, calling the plot function within the test suite caused pytest to hang/freeze. Quitting with reported that the code was stuck at waiter.aquire() . Thus began a long search through Stack Overflow, bug reports and blog posts for a way to make it run.
The fix
I eventually found the answer in a blog post called Why your multiprocessing Pool is stuck (it’s full of sharks!). It suggested modifying the code to “spawn” new processes in the multiprocessing pool, instead of using the default “fork” method. This is as simple as changing:
import multiprocessing with multiprocessing.Pool() as pool: pool.map(plot_function, args)import multiprocessing with multiprocessing.get_context('spawn').Pool() as pool: pool.map(plot_function, args)I made the change, pytest ran to completion and all the tests turned green. I was very happy. I was also curious about how fork and spawn work.
What happens when you start a new process?
Forking and spawning are two different start methods for new processes. Fork is the default on Linux (it isn’t available on Windows), while Windows and MacOS use spawn by default.
When a process is forked the child process inherits all the same variables in the same state as they were in the parent. Each child process then continues independently from the forking point. The pool divides the args between the children and they work though them sequentially.
On the other hand, when a process is spawned , it begins by starting a new Python interpreter. The current module is reimported and new versions of all the variables are created. The plot_function is then called on each of the the args allocated to that child process. As with forking, the child processes are independent of each other and the parent.
Neither method copies running threads into the child processes.
Similarities and differences between the two start methods are:
Action fork spawn Create new PID for processes yes yes Module-level variables and functions present yes yes Each child process calls plot_function on multiple pool args yes yes Child processes independently track variable state yes yes Import module at start of each child process no yes Variables have same id as in parent process yes no Child process gets variables defined in name == main block yes no Parent process variables are updated from child process state no no Threads from parent process run in child processes no no Threads from parent process modify child variables no no See the appendix for a script that illustrates these differences.
Why my code was hanging
The problem with my test suite was caused by threading, either within Matplotlib or Pytest. Threads are not transferred to child processes (see Why your multiprocessing Pool is stuck for more details). Resources that have been locked by threads in the parent process remain locked when you fork the process. However, the thread that holds the lock (and would eventually release the resource) is not transferred. Anything else that needs the resource is stuck waiting and the process hangs at waiter.acquire() . Using spawn creates of fresh instances of each resource so none are in a locked state.
Other multiprocessing tricks
Multiprocessing processes are independent and state is not shared between.
Sometimes, however, it is necessary to update a dictionary with information from each process. In this case, state can be shared between processes using a Manager() object.
Furthermore, things such as logging configuration that are normally defined in the __name__ == ‘__main__’ block of a script are not passed to the spawned processes. Configuring loggers or other global varaibles in the child prcesses can be done by defining an initializer function that is called at the beginning of each process.
See the Ash Model Plotting code for examples of each: ash_model_plotting/plotting.py
Learn more
Hopefully these notes have given you (or future me) some insight into multiprocessing and a possible fix for processes that freeze. There is a bug report on Python.org that suggests making “spawn” the default start method (multiprocessing’s default start method of fork()-without-exec() is broken). It may be worth checking back there to see if things change in future.
Below is a script to demonstrate some differences between fork and spawn . There is also a copy of the output that it produces. Experimenting with it may help deepen your understanding of multiprocessing.
Happy parallel computing!
Appendix
Variables and processes with fork and spawn
The script below uses a multiprocessing.Pool() with both fork and spawn start methods to repeatedly call a function that prints information about current processes and variables. Comparing the two start methods gives insight into how the child processes are created in each context.
Notice especially what happens to the LOCK in each case. In the fork version, the lock is released after the child processes have begun so their version of it remains locked. In the spawn version, the thread that acquires the lock is never started as it is not called when the module is imported.
# multi_demo.py import datetime as dt import logging from multiprocessing import get_context import os import sys import time from threading import Lock, Thread, enumerate if sys.platform in ('win32', 'msys', 'cygwin'): print("Script works only on *NIX type operating systems.") sys.exit(1) print(f"Importing 'multi_demo.py' at dt.datetime.now()>") logger = logging.getLogger("multi_demo") # Define some module-level variables CONSTANT = 3.14 MUTABLE = "mutated": False> LOCK = Lock() def run_multi(context): """ Run multiprocessing job with given context type """ with get_context(context).Pool(2, initializer=init) as pool: pool.map(run_task, (1, 2, 3, 4)) def init(): """This function is called when new processes start.""" print(f'Initializing process os.getpid()>') # Uncomment the following to see pool process log messages with spawn # logging.basicConfig(level=logging.INFO) def hold_lock(lock, hold_time=1): """Hold a lock item for "hold_time" seconds""" lock.acquire() logger.info("*** Lock acquired in thread process ***") time.sleep(hold_time) lock.release() logger.info("*** Lock released in thread process ***") def run_task(index): """Print 'index' and state of different variables.""" time.sleep(4) logger.info("Hello from run_task(%s) with root logger id %s", index, id(logging.getLogger())) print(f"Index: index>") print(f"process ID: os.getpid()>") public_globals = [g for g in globals().keys() if not g.startswith('__')] print(f"Global vars: ', '.join(public_globals)>") print(f"CONSTANT: CONSTANT> (with id id(CONSTANT)>)") MUTABLE[index] = os.getpid() print(f"MUTABLE: MUTABLE>") print(f"Number of running threads: len(enumerate())>") print(f"LOCK is locked? LOCK.locked()>") # Uncomment the following to make "fork" process hang at waiter.acquire() # LOCK.acquire() print() if __name__ == '__main__': # Configure root logger with handler to print messages from multi_demo # logger to std_err logging.basicConfig(level=logging.INFO) logger.info("Original process ID: %s", os.getpid()) logger.info("root logger id: %s", id(logging.getLogger())) # modify mutable global var MUTABLE['mutated'] = True logger.info("MUTABLE before tasks: %s", MUTABLE) # Start a thread to hold the lock. This will unlock after the pool has # started but while the process is still sleeping. lock_holder_thread = Thread(target=hold_lock, args=(LOCK, 1)) lock_holder_thread.start() logger.info("Number of running threads: %s", len(enumerate())) # Run pool processes with different contexts for context in ('fork', 'spawn'): print('\n\n') logger.info("Running as '%s' pool at %s", context, dt.datetime.now()) logger.info("_" * 20 + " pool process begin " + "_" * 20) run_multi(context) logger.info("_" * 20 + " pool process end " + "_" * 20) # Log final MUTABLE value print('\n') logger.info("MUTABLE after tasks: %s", MUTABLE)Script output
Importing 'multi_demo.py' at 2020-11-16 17:19:32.825109 INFO:multi_demo:Original process ID: 439383 INFO:multi_demo:root logger id: 140472182712832 INFO:multi_demo:MUTABLE before tasks: INFO:multi_demo:*** Lock acquired in thread process *** INFO:multi_demo:Number of running threads: 2 INFO:multi_demo:Running as 'fork' pool at 2020-11-16 17:19:32.826051 INFO:multi_demo:____________________ pool process begin ____________________ Initializing process 439385 Initializing process 439386 INFO:multi_demo:*** Lock released in thread process *** INFO:multi_demo:Hello from run_task(1) with root logger id 140472182712832 Index: 1 process ID: 439385 Global vars: dt, logging, get_context, os, time, Lock, Thread, enumerate, logger, CONSTANT, MUTABLE, LOCK, run_multi, init, hold_lock, run_task, lock_holder_thread, context CONSTANT: 3.14 (with id 140472183846128) MUTABLE: Number of running threads: 1 LOCK is locked? True INFO:multi_demo:Hello from run_task(2) with root logger id 140472182712832 Index: 2 process ID: 439386 Global vars: dt, logging, get_context, os, time, Lock, Thread, enumerate, logger, CONSTANT, MUTABLE, LOCK, run_multi, init, hold_lock, run_task, lock_holder_thread, context CONSTANT: 3.14 (with id 140472183846128) MUTABLE: Number of running threads: 1 LOCK is locked? True INFO:multi_demo:Hello from run_task(3) with root logger id 140472182712832 Index: 3 process ID: 439385 INFO:multi_demo:Hello from run_task(4) with root logger id 140472182712832 Index: 4 process ID: 439386 Global vars: dt, logging, get_context, os, time, Lock, Thread, enumerate, logger, CONSTANT, MUTABLE, LOCK, run_multi, init, hold_lock, run_task, lock_holder_thread, context CONSTANT: 3.14 (with id 140472183846128) MUTABLE: Number of running threads: 1 LOCK is locked? True Global vars: dt, logging, get_context, os, time, Lock, Thread, enumerate, logger, CONSTANT, MUTABLE, LOCK, run_multi, init, hold_lock, run_task, lock_holder_thread, context CONSTANT: 3.14 (with id 140472183846128) MUTABLE: Number of running threads: 1 LOCK is locked? True INFO:multi_demo:____________________ pool process end ____________________ INFO:multi_demo:Running as 'spawn' pool at 2020-11-16 17:19:40.908982 INFO:multi_demo:____________________ pool process begin ____________________ Importing 'multi_demo.py' at 2020-11-16 17:19:41.097592 Importing 'multi_demo.py' at 2020-11-16 17:19:41.098022 Initializing process 439393 Initializing process 439394 Index: 1 process ID: 439393 Global vars: dt, logging, get_context, os, time, Lock, Thread, enumerate, logger, CONSTANT, MUTABLE, LOCK, run_multi, init, hold_lock, run_task CONSTANT: 3.14 (with id 140312126736656) MUTABLE: Number of running threads: 1 LOCK is locked? False Index: 2 process ID: 439394 Global vars: dt, logging, get_context, os, time, Lock, Thread, enumerate, logger, CONSTANT, MUTABLE, LOCK, run_multi, init, hold_lock, run_task CONSTANT: 3.14 (with id 140298227288336) MUTABLE: Number of running threads: 1 LOCK is locked? False Index: 3 process ID: 439393 Global vars: dt, logging, get_context, os, time, Lock, Thread, enumerate, logger, CONSTANT, MUTABLE, LOCK, run_multi, init, hold_lock, run_task CONSTANT: 3.14 (with id 140312126736656) MUTABLE: Number of running threads: 1 LOCK is locked? False Index: 4 process ID: 439394 Global vars: dt, logging, get_context, os, time, Lock, Thread, enumerate, logger, CONSTANT, MUTABLE, LOCK, run_multi, init, hold_lock, run_task CONSTANT: 3.14 (with id 140298227288336) MUTABLE: Number of running threads: 1 LOCK is locked? False INFO:multi_demo:____________________ pool process end ____________________ INFO:multi_demo:MUTABLE after tasks:
Categories: science
Updated: November 13, 2020
Функция fork() модуля os в Python
Функция fork() модуля os создает клон текущего процесса как дочерний процесс. Возвращает 0 в дочернем процессе и идентификатор дочернего процесса в родительском элементе. Если возникает ошибка, то возникает исключение OSError .
Вызывает событие аудита os.fork без аргументов.
Изменено в версии Python 3.8: вызов os.fork() в субинтерпретаторе больше не поддерживается (возникает ошибка RuntimeError ).
Изменено в версии Python 3.12: если Python обнаруживает, что процесс имеет несколько потоков, то os.fork() теперь выдает предупреждение об устаревании DeprecationWarning .
Чтобы лучше информировать разработчиков о проблеме проектирования, которую платформа POSIX специально отмечает как не поддерживаемую, команда разработчиков Python решила отображать это как предупреждение. Даже в коде, который, кажется, что работает, на платформах POSIX небезопасно смешивать многопоточность с os.fork() . Сама среда выполнения CPython всегда выполняла вызовы API, которые небезопасны для использования в дочернем процессе, когда потоки существовали в родительском (например, malloc и free ).
Пользователи macOS или пользователи реализаций libc или malloc , отличных от тех, которые обычно встречаются в glibc на сегодняшний день, относятся к числу тех, кто уже с большей вероятностью столкнется с взаимоблокировками при запуске такого кода.
Доступность: POSIX, но недоступно на Emscripten и WASI.
Примеры создания дочерних процессов
После разветвления, два процесса выполняют один и тот же код. Чтобы программа узнала, в каком она находится, она должна проверить возвращаемое значение os.fork() . Если значение равно 0 , текущий процесс является дочерним. Если это не 0 , программа выполняется в родительском процессе, а возвращаемое значение является идентификатором процесса дочернего процесса.
# test_fork.py import os pid = os.fork() if pid: print('Child process id:', pid) else: print('I am the child') # $ python3 test_fork.py # Child process id: 20131 # I am the child
Родитель может отправлять сигналы дочернему процессу, используя функцию os.kill() используя модуль signal . В этом примере используется короткая пауза, чтобы дать дочернему процессу время для настройки обработчика сигнала. Реальное приложение, не будет нуждаться в вызове time.sleep() . В дочернем процессе установим обработчик сигнала и сделаем небольшую паузу, чтобы предоставить родительскому процессу достаточно времени для отправки сигнала.
# test_kill.py import os import signal import time def signal_usr1(signum, frame): "Обратный вызов вызывается при получении сигнала" pid = os.getpid() print(f'Получен USR1 в процессе pid>') print('Forking. ') child_pid = os.fork() if child_pid: print('PARENT: Пауза перед отправкой сигнала . ') time.sleep(1) print(f'PARENT: передача сигналов child_pid>') os.kill(child_pid, signal.SIGUSR1) else: print('CHILD: Настройка обработчика сигнала') signal.signal(signal.SIGUSR1, signal_usr1) print('CHILD: Пауза в ожидании сигнала') time.sleep(5) # $ python3 test_kill.py # Forking. # PARENT: Пауза перед отправкой сигнала . # CHILD: Настройка обработчика сигнала # CHILD: Пауза в ожидании сигнала # PARENT: передача сигналов 21168 # Получен USR1 в процессе 21168
Простой способ обработать отдельное поведение в дочернем процессе — проверить возвращаемое значение os.fork() и branch . Более сложное поведение может потребовать большего разделения кода, чем простая ветвь. В других случаях может существовать программа, которую необходимо обернуть. Для обеих этих ситуаций можно использовать серию функций os.exec*() для запуска другой программы.
# test_exec.py import os child_pid = os.fork() if child_pid: os.waitpid(child_pid, 0) else: os.execlp('pwd', 'pwd', '-P') # $ python3 test_exec.py # /home/docs-python
Когда программа запускается функцией os.exec() , код из этой программы заменяет код из существующего процесса.
Примеры ожидания дочерних процессов
Много ресурсоемких программ, используется несколько процессов, чтобы обойти ограничения многопоточности в Python и глобальную блокировку интерпретатора GIL. При запуске нескольких процессов для выполнения отдельных задач мастеру нужно будет дождаться завершения одного или нескольких из них, прежде чем запускать новые, чтобы избежать перегрузки сервера. Существует несколько различных способов сделать это с помощью функции os.wait() и связанных с ней функций.
Когда не имеет значения, какой дочерний процесс может завершиться первым, используйте os.wait() . Он возвращается, как только завершается любой дочерний процесс.
# test_wait.py import os import sys import time for i in range(2): print(f'PARENT os.getpid()>: Forking i>') worker_pid = os.fork() if not worker_pid: print(f'WORKER i>: Starting') time.sleep(2 + i) print(f'WORKER i>: Finishing') sys.exit(i) for i in range(2): print(f'PARENT: Waiting for i>') done = os.wait() print(f'PARENT: Child done: done>') # $ python3 test_wait.py # PARENT 24512: Forking 0 # PARENT 24512: Forking 1 # WORKER 0: Starting # PARENT: Waiting for 0 # WORKER 1: Starting # WORKER 0: Finishing # PARENT: Child done: (24513, 0) # PARENT: Waiting for 1 # WORKER 1: Finishing # PARENT: Child done: (24514, 256)
Функция os.wait() возвращает идентификатор процесса и код завершения, упакованный в 16-битовое значение. Младший байт представляет номер сигнала, прекратившего выполнение процесса, а старший — код состояния, возвращенный процессом по его завершении.
Чтобы дождаться определенного процесса, используйте функцию os.waitpid() .
Функция os.waitpid() , которой передан идентификатор целевого процесса, будет блокироваться до тех пор, пока процесс не завершится.
# test_waitpid.py import os import sys import time workers = [] for i in range(2): print(f'PARENT os.getpid()>: Forking i>') worker_pid = os.fork() if not worker_pid: print(f'WORKER i>: Starting') time.sleep(2 + i) print(f'WORKER i>: Finishing') sys.exit(i) workers.append(worker_pid) for pid in workers: print(f'PARENT: Waiting for pid>') done = os.waitpid(pid, 0) print(f'PARENT: Child done: done>') # $ python3 test_waitpid.py # PARENT 25144: Forking 0 # PARENT 25144: Forking 1 # WORKER 0: Starting # PARENT: Waiting for 25145 # WORKER 1: Starting # WORKER 0: Finishing # PARENT: Child done: (25145, 0) # PARENT: Waiting for 25146 # WORKER 1: Finishing # PARENT: Child done: (25146, 256)
- ОБЗОРНАЯ СТРАНИЦА РАЗДЕЛА
- Управление переменной средой окружения системы
- Представление пути в файловой системе
- Извлечение/установка uid, gid и pid процесса
- Наследование файловых дескрипторов
- Создание дескриптора файла, чтение, запись и его закрытие
- Функция listdir() модуля os
- Функция walk() модуля os
- Функция scandir() модуля os
- Объект DirEntry() модуля os
- Функция stat() модуля os
- Объект stat_result, результаты выполнения os.stat()
- Функция lstat() модуля os
- Функция access() модуля os
- Функция chdir() модуля os
- Функция chmod() модуля os
- Функция chown() модуля os
- Функция chroot() модуля os
- Функция getcwd() модуля os
- Функция link() модуля os
- Функция mkdir() модуля os
- Функция makedirs() модуля os
- Функция symlink() модуля os
- Функция readlink() модуля os
- Функция remove() модуля os, удалить файл
- Функция removedirs() модуля os
- Функция rename() модуля os
- Функция renames() модуля os
- Функция replace() модуля os
- Функция rmdir() модуля os
- Функция strerror() модуля os
- Функция supports_dir_fd модуля os
- Функция supports_effective_ids модуля os
- Функция supports_fd модуля os
- Функция supports_follow_symlinks модуля os
- Функция truncate() модуля os
- Функция utime() модуля os
- Манипулирование списком контроля доступа ACL в Linux
- Функция abort() модуля os
- Функция exec*() модуля os
- Функция popen() модуля os
- Функция system() модуля os
- Функция _exit() модуля os
- Функция fork() модуля os
- Функция kill() модуля os
- Функция spawn() модуля os
- Функция umask() модуля os
- Функция uname() модуля os
- Функция wait() модуля os
- Функция waitpid() модуля os
- Определение состояния процесса
- Константы для поддержки операций с путями
- Генератор случайных байтов на основе модуля os
- Функция startfile() модуля os
- Функция times() модуля os
- Функции getloadavg() и cpu_count() модуля os
- Функция waitstatus_to_exitcode() модуля os
- Функция listdrives() модуля os, имена дисков Windows
- Функция listmounts() модуля os, точки монтирования Windows
- Функция listvolumes() модуля os, список томов Windows