Python Internals Serie : Subprocess.Popen07 Nov 2020
The purpose of this serie is to review some parts of the CPython’s code.
Well there are multiple reasons :
- Because we can. The project is opensource and the source code is freely available here : https://github.com/python/cpython
- I strongly believe that we, as developpers, can learn a lot by studying good and clean code. And I think we can safely assume that CPython’s code, which is used practicaly everywhere, meet this crietria.
- I also think that studying python internals can make us better python programmers, for example ever wondered why changing sys.stdout seems to have no effect on subprocesses ? Well at the end of this article You will know why.
I will try to regulary realease articles on random functions/modules/data strucutres/algorithms.
Starting today I don’t really have a plan or a timeline for future articles, I’ll just start with subprocess.Popen and go from there.
I’ll try to link PEPs, articles and disscussions whenever I can.
DISCLAIMER : I am not a Python Core Developper, therefore a lot of claims that I’ll make here may be wrong, especially if they’re not backed by any evidence.
With that being said, let’s read some code !
Wait … What the heck is CPython ?
CPython is the official python implementation. You need to understand that there is a difference between python specification and a python implementation.
A simple example can be the sorting function :
The python specification specify that a valid implementation should offer a
sorted function which can take a list and sort it.
But as you may recall from your painfull algorithms course, there are multiple sorting algorithms, so which one are being used ? Well, as a lot of things, it depends on the implementation.
In fact you can create your own implementation and use bubble sort if you want, not a very practical choice but you can !
CPython is widely used nowadays, and a lot of its code is wrote using C, again this is an implementation detail not a specification one.
There are other python implementations like PyPy (which use JIT compilation), or Jython which is ran on a JVM machine.
Getting the code
the first step is to get the cpython code. For that we gonna just clone the official repo on github :
Subprocess.Popen : The class
Please note that we won’t really analyze ALL the code of Popen, there are way too many options and use cases to fit in this article.
Instead we juste gonna try to understand what happens when you execute this line of code :
We need to find a starting point, we can simply do :
we can safely assume that the module we’re searching for is
By searching for
Popen in the file we find a class :
The first instruction is :
__del__ is basicaly a destructor method in python, this method is automatically called when the garbage collector try to free the object.
in CPython the garbage collector use what we call reference counting, this means that if you write something like this :
In the second line we no longer have a reference to the list
[1, 2, 3], which means that the garbage collector may delete it (or not) any time.
my guess, is that if you run a command like this :
since you’re not referencing the Popen object, it is a garbage candidate, and the gc may try to delete it before it run. Which is not the excepected behaviour since you except the command to run even if you don’t care about what it’ll return.
What surprises me tough is the fact that it’s a class attribute, there is surely a reason behind it, but i’m not sure what.
And the explanation is as follow :
If subprocess.Popen is called with a keyword argument whose name is undefined or simply too many arguments, an instance of the Popen class is created but its __init__ method call fails due to the invalid argument list. (Immediately) afterwards, the new instance is destroyed and its __del__ method is called, which checks the _child_created field that is defined by __init__; but __init__ never got the chance to execute, so the field is not defined, and __del__ raises an AttributeError, which is written out to stderr.
- Oleg Oshmyan (chortos) 
the first instruction in the init method is :
from the comment it looks like the purpose of the function is to avoid zombie processes. one thing that i like about this function is how the autor of the except explictly explain why he passes when he catches a ValueError and why it’s okay. I see a lot of developpers using the pattern
try: except (some exception or just Exception) pass instead of treating the error proply. This can be very risky as it may hide a real error that you wish you saw before wasting 2 days trying to debug a random issue.
After that we can see some basic parameters validation : like is bufsize an integer ?
This is also a very good habit to have, especialy in a dynamicaly typed language like Python.
your code may still work with the wrong types and give you a wrong result, as an example :
As you can see altough it works, example 2 is clearly not what the user would expect.
You can also start to see that some code parts depends on the target plateforme (windows / POSIX), for the rest of this article, I’ll focus on the POSIX part.
next we get the file decriptors related to the optional arguments stdin/stdout/stderr.
A file descriptor is an abstraction managed by the process, each file descriptor points to an entry in a system wide table called Open file table.
In python you can do :
This will return the file descriptor.
Based on the comments and common sense it looks like, we will not always need 6 file descriptors, but only if we choose to communicate with the child process (which is not our case for our example).
In our example it looks like :
c2pwrite will be the fd of the output of our script (1 if stdout) and
p2cread the fd of the input of our script (0 if stdin).
_get_handles function :
looks like the default values (if no alternative stdout/in/err) is provided) : -1.
the function seems to confirm our hypothesis :
it does bug me how similar the stdin / stdout / stderr code is. I know if I had to write this code I will desperatly try to somehow refactore it in one block, I’m not sure tough if it will be a good idea, refactoring is not always a good thing.
I found the comment funny, for refecence this the xkcd comic that it refers to :
next we call the function :
_execute_child (POSIX version in our case) :
Again some basic parameter validation.
this will send a signal and subsribed functions may do things accordenly (no idea what tough).
_USE_POSIX_SPAWN is true, python will try to use it to create the child process.
posix_spawn will not copy the data of the parent process, which can make the script start faster for larger processes.
that if you are on a linux plateform and the version of glibc is greater than 2.24 you ll use
the 2.24 version of glibc has been released around 2016 so it’s fair to assume that if you have a recent linux distribution you ll also use
If you want to check
If you’re interessted you can read the disccusion (on the python bug tracker) which lead to trying to use
posix_span : https://bugs.python.org/issue35537
looks like our child process will inherit all our defined env variables
one interessting thing :
we iterate over the fds (in/out/err) of our child process, which are -1 by default as you can see if they’re different than -1, based on https://linux.die.net/man/2/dup2, will close fd2 (standard input/output), and copy the value of fd (our new fd) place of it.
which mean in our case (-1) nothing happen ! so even if sys.stdout is different than 1, this won’t affect children process, and this explain why contextredirect will not affect children output/error
os.py (same folder), we do a
/f posix_spawn : nada. There is no function in this module with this name, my guess is that the function added to the exported functions somehow, and that it may be defined in the c code.
by doing a
grep -R 'posix_spawn', we seem to have a strong candidate here :
I guess that our function is
Note : the
PyObject is a C wrapper aroud python objects, so each variable you use in python, will be presented by a PyObject in C.
we first see some initializations.
and then some … gotos ? gonna admit that is kinda weird, since, well :
After some googling it appear that it’s not a bad practice to use for this kind of situation :
source : https://stackoverflow.com/a/245761
gotos are apprently also appropriate for jumping out of nested loops (fun fact : this is the only accepted form of goto in java, which is called label)
after multiple conditions we go here :
grep -R os_posix_spawnp_impl, Modules/posixmodule.c looks like a strong candidate.
if you’re intriged by this line :
clinic is a boilerpate code generator for CPython, we may inspect it in futur articles.
As above, some checks and gotos.
then we call
Which looks like our final step, since
posix_spawn is a native function in c : https://man7.org/linux/man-pages/man3/posix_spawn.3.html
And that’s it, as you can see, this is definetly not an exaustive analysis of subprocess code. We left a lot of code/conditions. Maybe we’ll revisit them in later articles.
Hope you learned a thing or two in python, cpython or even general developpement best practices.
If you have any suggestion/request/comment feel free to contact me ! (more details here : about page.