Design and History FAQ¶
Why does Python use indentation for grouping of statements?¶
Guido van Rossum believes that using indentation for grouping is extremely elegant and contributes a lot to the clarity of the average Python program. Most people learn to love this feature after a while.
Since there are no begin/end brackets there cannot be a disagreement between grouping perceived by the parser and the human reader. Occasionally C programmers will encounter a fragment of code like this:
if (x <= y)
x++;
y--;
z++;
Only the x++
statement is executed if the condition is true, but the
indentation leads many to believe otherwise. Even experienced C programmers will
sometimes stare at it a long time wondering as to why y
is being decremented even
for x > y
.
Because there are no begin/end brackets, Python is much less prone to coding-style conflicts. In C there are many different ways to place the braces. After becoming used to reading and writing code using a particular style, it is normal to feel somewhat uneasy when reading (or being required to write) in a different one.
Many coding styles place begin/end brackets on a line by themselves. This makes programs considerably longer and wastes valuable screen space, making it harder to get a good overview of a program. Ideally, a function should fit on one screen (say, 20–30 lines). 20 lines of Python can do a lot more work than 20 lines of C. This is not solely due to the lack of begin/end brackets – the lack of declarations and the high-level data types are also responsible – but the indentation-based syntax certainly helps.
Why am I getting strange results with simple arithmetic operations?¶
See the next question.
Why are floating-point calculations so inaccurate?¶
Users are often surprised by results like this:
>>> 1.2 - 1.0
0.19999999999999996
and think it is a bug in Python. It’s not. This has little to do with Python, and much more to do with how the underlying platform handles floating-point numbers.
The float
type in CPython uses a C double
for storage. A
float
object’s value is stored in binary floating-point with a fixed
precision (typically 53 bits) and Python uses C operations, which in turn rely
on the hardware implementation in the processor, to perform floating-point
operations. This means that as far as floating-point operations are concerned,
Python behaves like many popular languages including C and Java.
Many numbers that can be written easily in decimal notation cannot be expressed exactly in binary floating point. For example, after:
>>> x = 1.2
the value stored for x
is a (very good) approximation to the decimal value
1.2
, but is not exactly equal to it. On a typical machine, the actual
stored value is:
1.0011001100110011001100110011001100110011001100110011 (binary)
which is exactly:
1.1999999999999999555910790149937383830547332763671875 (decimal)
The typical precision of 53 bits provides Python floats with 15–16 decimal digits of accuracy.
For a fuller explanation, please see the floating-point arithmetic chapter in the Python tutorial.
Why are Python strings immutable?¶
There are several advantages.
One is performance: knowing that a string is immutable means we can allocate space for it at creation time, and the storage requirements are fixed and unchanging. This is also one of the reasons for the distinction between tuples and lists.
Another advantage is that strings in Python are considered as “elemental” as numbers. No amount of activity will change the value 8 to anything else, and in Python, no amount of activity will change the string “eight” to anything else.
Why must ‘self’ be used explicitly in method definitions and calls?¶
The idea was borrowed from Modula-3. It turns out to be very useful, for a variety of reasons.
First, it’s more obvious that you are using a method or instance attribute
instead of a local variable. Reading self.x
or self.meth()
makes it
absolutely clear that an instance variable or method is used even if you don’t
know the class definition by heart. In C++, you can sort of tell by the lack of
a local variable declaration (assuming globals are rare or easily recognizable)
– but in Python, there are no local variable declarations, so you’d have to
look up the class definition to be sure. Some C++ and Java coding standards
call for instance attributes to have an m_
prefix, so this explicitness is
still useful in those languages, too.
Second, it means that no special syntax is necessary if you want to explicitly
reference or call the method from a particular class. In C++, if you want to
use a method from a base class which is overridden in a derived class, you have
to use the ::
operator – in Python you can write
baseclass.methodname(self, <argument list>)
. This is particularly useful
for __init__()
methods, and in general in cases where a derived class
method wants to extend the base class method of the same name and thus has to
call the base class method somehow.
Finally, for instance variables it solves a syntactic problem with assignment:
since local variables in Python are (by definition!) those variables to which a
value is assigned in a function body (and that aren’t explicitly declared
global), there has to be some way to tell the interpreter that an assignment was
meant to assign to an instance variable instead of to a local variable, and it
should preferably be syntactic (for efficiency reasons). C++ does this through
declarations, but Python doesn’t have declarations and it would be a pity having
to introduce them just for this purpose. Using the explicit self.var
solves
this nicely. Similarly, for using instance variables, having to write
self.var
means that references to unqualified names inside a method don’t
have to search the instance’s directories. To put it another way, local
variables and instance variables live in two different namespaces, and you need
to tell Python which namespace to use.
Why can’t I use an assignment in an expression?¶
Starting in Python 3.8, you can!
Assignment expressions using the walrus operator :=
assign a variable in an
expression:
while chunk := fp.read(200):
print(chunk)
See PEP 572 for more information.
Why does Python use methods for some functionality (e.g. list.index()) but functions for other (e.g. len(list))?¶
As Guido said:
(a) For some operations, prefix notation just reads better than postfix – prefix (and infix!) operations have a long tradition in mathematics which likes notations where the visuals help the mathematician thinking about a problem. Compare the easy with which we rewrite a formula like x*(a+b) into x*a + x*b to the clumsiness of doing the same thing using a raw OO notation.
(b) When I read code that says len(x) I know that it is asking for the length of something. This tells me two things: the result is an integer, and the argument is some kind of container. To the contrary, when I read x.len(), I have to already know that x is some kind of container implementing an interface or inheriting from a class that has a standard len(). Witness the confusion we occasionally have when a class that is not implementing a mapping has a get() or keys() method, or something that isn’t a file has a write() method.
—https://mail.python.org/pipermail/python-3000/2006-November/004643.html
Why is join() a string method instead of a list or tuple method?¶
Strings became much more like other standard types starting in Python 1.6, when methods were added which give the same functionality that has always been available using the functions of the string module. Most of these new methods have been widely accepted, but the one which appears to make some programmers feel uncomfortable is:
", ".join(['1', '2', '4', '8', '16'])
which gives the result:
"1, 2, 4, 8, 16"
There are two common arguments against this usage.
The first runs along the lines of: “It looks really ugly using a method of a string literal (string constant)”, to which the answer is that it might, but a string literal is just a fixed value. If the methods are to be allowed on names bound to strings there is no logical reason to make them unavailable on literals.
The second objection is typically cast as: “I am really telling a sequence to
join its members together with a string constant”. Sadly, you aren’t. For some
reason there seems to be much less difficulty with having split()
as
a string method, since in that case it is easy to see that
"1, 2, 4, 8, 16".split(", ")
is an instruction to a string literal to return the substrings delimited by the given separator (or, by default, arbitrary runs of white space).
join()
is a string method because in using it you are telling the
separator string to iterate over a sequence of strings and insert itself between
adjacent elements. This method can be used with any argument which obeys the
rules for sequence objects, including any new classes you might define yourself.
Similar methods exist for bytes and bytearray objects.
How fast are exceptions?¶
A try
/except
block is extremely efficient if no exceptions
are raised. Actually
catching an exception is expensive. In versions of Python prior to 2.0 it was
common to use this idiom:
try:
value = mydict[key]
except KeyError:
mydict[key] = getvalue(key)
value = mydict[key]
This only made sense when you expected the dict to have the key almost all the time. If that wasn’t the case, you coded it like this:
if key in mydict:
value = mydict[key]
else:
value = mydict[key] = getvalue(key)
For this specific case, you could also use value = dict.setdefault(key,
getvalue(key))
, but only if the getvalue()
call is cheap enough because it
is evaluated in all cases.
Why isn’t there a switch or case statement in Python?¶
In general, structured switch statements execute one block of code
when an expression has a particular value or set of values.
Since Python 3.10 one can easily match literal values, or constants
within a namespace, with a match ... case
statement.
An older alternative is a sequence of if... elif... elif... else
.
For cases where you need to choose from a very large number of possibilities, you can create a dictionary mapping case values to functions to call. For example:
functions = {'a': function_1,
'b': function_2,
'c': self.method_1}
func = functions[value]
func()
For calling methods on objects, you can simplify yet further by using the
getattr()
built-in to retrieve methods with a particular name:
class MyVisitor:
def visit_a(self):
...
def dispatch(self, value):
method_name = 'visit_' + str(value)
method = getattr(self, method_name)
method()
It’s suggested that you use a prefix for the method names, such as visit_
in
this example. Without such a prefix, if values are coming from an untrusted
source, an attacker would be able to call any method on your object.
Imitating switch with fallthrough, as with C’s switch-case-default, is possible, much harder, and less needed.
Can’t you emulate threads in the interpreter instead of relying on an OS-specific thread implementation?¶
Answer 1: Unfortunately, the interpreter pushes at least one C stack frame for each Python stack frame. Also, extensions can call back into Python at almost random moments. Therefore, a complete threads implementation requires thread support for C.
Answer 2: Fortunately, there is Stackless Python, which has a completely redesigned interpreter loop that avoids the C stack.
Why can’t lambda expressions contain statements?¶
Python lambda expressions cannot contain statements because Python’s syntactic framework can’t handle statements nested inside expressions. However, in Python, this is not a serious problem. Unlike lambda forms in other languages, where they add functionality, Python lambdas are only a shorthand notation if you’re too lazy to define a function.
Functions are already first class objects in Python, and can be declared in a local scope. Therefore the only advantage of using a lambda instead of a locally defined function is that you don’t need to invent a name for the function – but that’s just a local variable to which the function object (which is exactly the same type of object that a lambda expression yields) is assigned!
Can Python be compiled to machine code, C or some other language?¶
Cython compiles a modified version of Python with optional annotations into C extensions. Nuitka is an up-and-coming compiler of Python into C++ code, aiming to support the full Python language.
How does Python manage memory?¶
The details of Python memory management depend on the implementation. The
standard implementation of Python, CPython, uses reference counting to
detect inaccessible objects, and another mechanism to collect reference cycles,
periodically executing a cycle detection algorithm which looks for inaccessible
cycles and deletes the objects involved. The gc
module provides functions
to perform a garbage collection, obtain debugging statistics, and tune the
collector’s parameters.
Other implementations (such as Jython or PyPy), however, can rely on a different mechanism such as a full-blown garbage collector. This difference can cause some subtle porting problems if your Python code depends on the behavior of the reference counting implementation.
In some Python implementations, the following code (which is fine in CPython) will probably run out of file descriptors:
for file in very_long_list_of_files:
f = open(file)
c = f.read(1)
Indeed, using CPython’s reference counting and destructor scheme, each new
assignment to f
closes the previous file. With a traditional GC, however,
those file objects will only get collected (and closed) at varying and possibly
long intervals.
If you want to write code that will work with any Python implementation,
you should explicitly close the file or use the with
statement;
this will work regardless of memory management scheme:
for file in very_long_list_of_files:
with open(file) as f:
c = f.read(1)
Why doesn’t CPython use a more traditional garbage collection scheme?¶
For one thing, this is not a C standard feature and hence it’s not portable. (Yes, we know about the Boehm GC library. It has bits of assembler code for most common platforms, not for all of them, and although it is mostly transparent, it isn’t completely transparent; patches are required to get Python to work with it.)
Traditional GC also becomes a problem when Python is embedded into other
applications. While in a standalone Python it’s fine to replace the standard
malloc()
and free()
with versions provided by the GC library, an application
embedding Python may want to have its own substitute for malloc()
and free()
,
and may not want Python’s. Right now, CPython works with anything that
implements malloc()
and free()
properly.
Why isn’t all memory freed when CPython exits?¶
Objects referenced from the global namespaces of Python modules are not always deallocated when Python exits. This may happen if there are circular references. There are also certain bits of memory that are allocated by the C library that are impossible to free (e.g. a tool like Purify will complain about these). Python is, however, aggressive about cleaning up memory on exit and does try to destroy every single object.
If you want to force Python to delete certain things on deallocation use the
atexit
module to run a function that will force those deletions.
Why are there separate tuple and list data types?¶
Lists and tuples, while similar in many respects, are generally used in
fundamentally different ways. Tuples can be thought of as being similar to
Pascal records
or C structs
; they’re small collections of related data which may
be of different types which are operated on as a group. For example, a
Cartesian coordinate is appropriately represented as a tuple of two or three
numbers.
Lists, on the other hand, are more like arrays in other languages. They tend to
hold a varying number of objects all of which have the same type and which are
operated on one-by-one. For example, os.listdir('.')
returns a list of
strings representing the files in the current directory. Functions which
operate on this output would generally not break if you added another file or
two to the directory.
Tuples are immutable, meaning that once a tuple has been created, you can’t replace any of its elements with a new value. Lists are mutable, meaning that you can always change a list’s elements. Only immutable elements can be used as dictionary keys, and hence only tuples and not lists can be used as keys.
How are lists implemented in CPython?¶
CPython’s lists are really variable-length arrays, not Lisp-style linked lists. The implementation uses a contiguous array of references to other objects, and keeps a pointer to this array and the array’s length in a list head structure.
This makes indexing a list a[i]
an operation whose cost is independent of
the size of the list or the value of the index.
When items are appended or inserted, the array of references is resized. Some cleverness is applied to improve the performance of appending items repeatedly; when the array must be grown, some extra space is allocated so the next few times don’t require an actual resize.
How are dictionaries implemented in CPython?¶
CPython’s dictionaries are implemented as resizable hash tables. Compared to B-trees, this gives better performance for lookup (the most common operation by far) under most circumstances, and the implementation is simpler.
Dictionaries work by computing a hash code for each key stored in the dictionary
using the hash()
built-in function. The hash code varies widely depending
on the key and a per-process seed; for example, 'Python'
could hash to
-539294296
while 'python'
, a string that differs by a single bit, could hash
to 1142331976
. The hash code is then used to calculate a location in an
internal array where the value will be stored. Assuming that you’re storing
keys that all have different hash values, this means that dictionaries take
constant time – O(1), in Big-O notation – to retrieve a key.
Why must dictionary keys be immutable?¶
The hash table implementation of dictionaries uses a hash value calculated from the key value to find the key. If the key were a mutable object, its value could change, and thus its hash could also change. But since whoever changes the key object can’t tell that it was being used as a dictionary key, it can’t move the entry around in the dictionary. Then, when you try to look up the same object in the dictionary it won’t be found because its hash value is different. If you tried to look up the old value it wouldn’t be found either, because the value of the object found in that hash bin would be different.
If you want a dictionary indexed with a list, simply convert the list to a tuple
first; the function tuple(L)
creates a tuple with the same entries as the
list L
. Tuples are immutable and can therefore be used as dictionary keys.
Some unacceptable solutions that have been proposed:
Hash lists by their address (object ID). This doesn’t work because if you construct a new list with the same value it won’t be found; e.g.:
mydict = {[1, 2]: '12'} print(mydict[[1, 2]])
would raise a
KeyError
exception because the id of the[1, 2]
used in the second line differs from that in the first line. In other words, dictionary keys should be compared using==
, not usingis
.Make a copy when using a list as a key. This doesn’t work because the list, being a mutable object, could contain a reference to itself, and then the copying code would run into an infinite loop.
Allow lists as keys but tell the user not to modify them. This would allow a class of hard-to-track bugs in programs when you forgot or modified a list by accident. It also invalidates an important invariant of dictionaries: every value in
d.keys()
is usable as a key of the dictionary.Mark lists as read-only once they are used as a dictionary key. The problem is that it’s not just the top-level object that could change its value; you could use a tuple containing a list as a key. Entering anything as a key into a dictionary would require marking all objects reachable from there as read-only – and again, self-referential objects could cause an infinite loop.
There is a trick to get around this if you need to, but use it at your own risk:
You can wrap a mutable structure inside a class instance which has both a
__eq__()
and a __hash__()
method.
You must then make sure that the
hash value for all such wrapper objects that reside in a dictionary (or other
hash based structure), remain fixed while the object is in the dictionary (or
other structure).
class ListWrapper:
def __init__(self, the_list):
self.the_list = the_list
def __eq__(self, other):
return self.the_list == other.the_list
def __hash__(self):
l = self.the_list
result = 98767 - len(l)*555
for i, el in enumerate(l):
try:
result = result + (hash(el) % 9999999) * 1001 + i
except Exception:
result = (result % 7777777) + i * 333
return result
Note that the hash computation is complicated by the possibility that some members of the list may be unhashable and also by the possibility of arithmetic overflow.
Furthermore it must always be the case that if o1 == o2
(ie o1.__eq__(o2)
is True
) then hash(o1) == hash(o2)
(ie, o1.__hash__() == o2.__hash__()
),
regardless of whether the object is in a dictionary or not. If you fail to meet
these restrictions dictionaries and other hash based structures will misbehave.
In the case of ListWrapper
, whenever the wrapper object is in a dictionary the
wrapped list must not change to avoid anomalies. Don’t do this unless you are
prepared to think hard about the requirements and the consequences of not
meeting them correctly. Consider yourself warned.
Why doesn’t list.sort() return the sorted list?¶
In situations where performance matters, making a copy of the list just to sort
it would be wasteful. Therefore, list.sort()
sorts the list in place. In
order to remind you of that fact, it does not return the sorted list. This way,
you won’t be fooled into accidentally overwriting a list when you need a sorted
copy but also need to keep the unsorted version around.
If you want to return a new list, use the built-in sorted()
function
instead. This function creates a new list from a provided iterable, sorts
it and returns it. For example, here’s how to iterate over the keys of a
dictionary in sorted order:
for key in sorted(mydict):
... # do whatever with mydict[key]...
How do you specify and enforce an interface spec in Python?¶
An interface specification for a module as provided by languages such as C++ and Java describes the prototypes for the methods and functions of the module. Many feel that compile-time enforcement of interface specifications helps in the construction of large programs.
Python 2.6 adds an abc
module that lets you define Abstract Base Classes
(ABCs). You can then use isinstance()
and issubclass()
to check
whether an instance or a class implements a particular ABC. The
collections.abc
module defines a set of useful ABCs such as
Iterable
, Container
, and
MutableMapping
.
For Python, many of the advantages of interface specifications can be obtained by an appropriate test discipline for components.
A good test suite for a module can both provide a regression test and serve as a
module interface specification and a set of examples. Many Python modules can
be run as a script to provide a simple “self test.” Even modules which use
complex external interfaces can often be tested in isolation using trivial
“stub” emulations of the external interface. The doctest
and
unittest
modules or third-party test frameworks can be used to construct
exhaustive test suites that exercise every line of code in a module.
An appropriate testing discipline can help build large complex applications in
Python as well as having interface specifications would. In fact, it can be
better because an interface specification cannot test certain properties of a
program. For example, the list.append()
method is expected to add new elements
to the end of some internal list; an interface specification cannot test that
your list.append()
implementation will actually do this correctly, but it’s
trivial to check this property in a test suite.
Writing test suites is very helpful, and you might want to design your code to make it easily tested. One increasingly popular technique, test-driven development, calls for writing parts of the test suite first, before you write any of the actual code. Of course Python allows you to be sloppy and not write test cases at all.
Why is there no goto?¶
In the 1970s people realized that unrestricted goto could lead
to messy “spaghetti” code that was hard to understand and revise.
In a high-level language, it is also unneeded as long as there
are ways to branch (in Python, with if
statements and or
,
and
, and if
/else
expressions) and loop (with while
and for
statements, possibly containing continue
and break
).
One can also use exceptions to provide a “structured goto”
that works even across
function calls. Many feel that exceptions can conveniently emulate all
reasonable uses of the go
or goto
constructs of C, Fortran, and other
languages. For example:
class label(Exception): pass # declare a label
try:
...
if condition: raise label() # goto label
...
except label: # where to goto
pass
...
This doesn’t allow you to jump into the middle of a loop, but that’s usually
considered an abuse of goto
anyway. Use sparingly.
Why can’t raw strings (r-strings) end with a backslash?¶
More precisely, they can’t end with an odd number of backslashes: the unpaired backslash at the end escapes the closing quote character, leaving an unterminated string.
Raw strings were designed to ease creating input for processors (chiefly regular expression engines) that want to do their own backslash escape processing. Such processors consider an unmatched trailing backslash to be an error anyway, so raw strings disallow that. In return, they allow you to pass on the string quote character by escaping it with a backslash. These rules work well when r-strings are used for their intended purpose.
If you’re trying to build Windows pathnames, note that all Windows system calls accept forward slashes too:
f = open("/mydir/file.txt") # works fine!
If you’re trying to build a pathname for a DOS command, try e.g. one of
dir = r"\this\is\my\dos\dir" "\\"
dir = r"\this\is\my\dos\dir\ "[:-1]
dir = "\\this\\is\\my\\dos\\dir\\"
Why doesn’t Python have a “with” statement for attribute assignments?¶
Python has a with
statement that wraps the execution of a block, calling code
on the entrance and exit from the block. Some languages have a construct that
looks like this:
with obj:
a = 1 # equivalent to obj.a = 1
total = total + 1 # obj.total = obj.total + 1
In Python, such a construct would be ambiguous.
Other languages, such as Object Pascal, Delphi, and C++, use static types, so it’s possible to know, in an unambiguous way, what member is being assigned to. This is the main point of static typing – the compiler always knows the scope of every variable at compile time.
Python uses dynamic types. It is impossible to know in advance which attribute will be referenced at runtime. Member attributes may be added or removed from objects on the fly. This makes it impossible to know, from a simple reading, what attribute is being referenced: a local one, a global one, or a member attribute?
For instance, take the following incomplete snippet:
def foo(a):
with a:
print(x)
The snippet assumes that a
must have a member attribute called x
. However,
there is nothing in Python that tells the interpreter this. What should happen
if a
is, let us say, an integer? If there is a global variable named x
,
will it be used inside the with
block? As you see, the dynamic nature of Python
makes such choices much harder.
The primary benefit of with
and similar language features (reduction of code
volume) can, however, easily be achieved in Python by assignment. Instead of:
function(args).mydict[index][index].a = 21
function(args).mydict[index][index].b = 42
function(args).mydict[index][index].c = 63
write this:
ref = function(args).mydict[index][index]
ref.a = 21
ref.b = 42
ref.c = 63
This also has the side-effect of increasing execution speed because name bindings are resolved at run-time in Python, and the second version only needs to perform the resolution once.
Similar proposals that would introduce syntax to further reduce code volume, such as using a ‘leading dot’, have been rejected in favour of explicitness (see https://mail.python.org/pipermail/python-ideas/2016-May/040070.html).
Why don’t generators support the with statement?¶
For technical reasons, a generator used directly as a context manager
would not work correctly. When, as is most common, a generator is used as
an iterator run to completion, no closing is needed. When it is, wrap
it as contextlib.closing(generator)
in the with
statement.
Why are colons required for the if/while/def/class statements?¶
The colon is required primarily to enhance readability (one of the results of the experimental ABC language). Consider this:
if a == b
print(a)
versus
if a == b:
print(a)
Notice how the second one is slightly easier to read. Notice further how a colon sets off the example in this FAQ answer; it’s a standard usage in English.
Another minor reason is that the colon makes it easier for editors with syntax highlighting; they can look for colons to decide when indentation needs to be increased instead of having to do a more elaborate parsing of the program text.
Why does Python allow commas at the end of lists and tuples?¶
Python lets you add a trailing comma at the end of lists, tuples, and dictionaries:
[1, 2, 3,]
('a', 'b', 'c',)
d = {
"A": [1, 5],
"B": [6, 7], # last trailing comma is optional but good style
}
There are several reasons to allow this.
When you have a literal value for a list, tuple, or dictionary spread across multiple lines, it’s easier to add more elements because you don’t have to remember to add a comma to the previous line. The lines can also be reordered without creating a syntax error.
Accidentally omitting the comma can lead to errors that are hard to diagnose. For example:
x = [
"fee",
"fie"
"foo",
"fum"
]
This list looks like it has four elements, but it actually contains three: “fee”, “fiefoo” and “fum”. Always adding the comma avoids this source of error.
Allowing the trailing comma may also make programmatic code generation easier.


14. Interactive Input Editing and History Substitution¶
Some versions of the Python interpreter support editing of the current input line and history substitution, similar to facilities found in the Korn shell and the GNU Bash shell. This is implemented using the GNU Readline library, which supports various styles of editing. This library has its own documentation which we won’t duplicate here.
14.1. Tab Completion and History Editing¶
Completion of variable and module names is
automatically enabled at interpreter startup so
that the Tab key invokes the completion function; it looks at
Python statement names, the current local variables, and the available
module names. For dotted expressions such as string.a
, it will evaluate
the expression up to the final '.'
and then suggest completions from
the attributes of the resulting object. Note that this may execute
application-defined code if an object with a __getattr__()
method
is part of the expression. The default configuration also saves your
history into a file named .python_history
in your user directory.
The history will be available again during the next interactive interpreter
session.
14.2. Alternatives to the Interactive Interpreter¶
This facility is an enormous step forward compared to earlier versions of the
interpreter; however, some wishes are left: It would be nice if the proper
indentation were suggested on continuation lines (the parser knows if an
INDENT
token is required next). The completion mechanism might
use the interpreter’s symbol table. A command to check (or even suggest)
matching parentheses, quotes, etc., would also be useful.
One alternative enhanced interactive interpreter that has been around for quite some time is IPython, which features tab completion, object exploration and advanced history management. It can also be thoroughly customized and embedded into other applications. Another similar enhanced interactive environment is bpython.


Python Setup and Usage

Python Setup and Usage¶
This part of the documentation is devoted to general information on the setup of the Python environment on different platforms, the invocation of the interpreter and things that make working with Python easier.
- 1. Command line and environment
- 2. Using Python on Unix platforms
- 3. Configure Python
- 4. Using Python on Windows
- 4.1. Python install manager
- 4.2. The embeddable package
- 4.3. The nuget.org packages
- 4.4. Alternative bundles
- 4.5. Supported Windows versions
- 4.6. Removing the MAX_PATH limitation
- 4.7. UTF-8 mode
- 4.8. Finding modules
- 4.9. Additional modules
- 4.10. Compiling Python on Windows
- 4.11. The full installer (deprecated)
- 4.12. Python launcher for Windows (deprecated)
- 5. Using Python on macOS
- 6. Using Python on Android
- 7. Using Python on iOS
- 8. Editors and IDEs


Python on Windows FAQ¶
How do I run a Python program under Windows?¶
This is not necessarily a straightforward question. If you are already familiar with running programs from the Windows command line then everything will seem obvious; otherwise, you might need a little more guidance.
Unless you use some sort of integrated development environment, you will end up
typing Windows commands into what is referred to as a
“Command prompt window”. Usually you can create such a window from your
search bar by searching for cmd
. You should be able to recognize
when you have started such a window because you will see a Windows “command
prompt”, which usually looks like this:
C:\>
The letter may be different, and there might be other things after it, so you might just as easily see something like:
D:\YourName\Projects\Python>
depending on how your computer has been set up and what else you have recently done with it. Once you have started such a window, you are well on the way to running Python programs.
You need to realize that your Python scripts have to be processed by another program called the Python interpreter. The interpreter reads your script, compiles it into bytecodes, and then executes the bytecodes to run your program. So, how do you arrange for the interpreter to handle your Python?
First, you need to make sure that your command window recognises the word
“py” as an instruction to start the interpreter. If you have opened a
command window, you should try entering the command py
and hitting
return:
C:\Users\YourName> py
You should then see something like:
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
You have started the interpreter in “interactive mode”. That means you can enter Python statements or expressions interactively and have them executed or evaluated while you wait. This is one of Python’s strongest features. Check it by entering a few expressions of your choice and seeing the results:
>>> print("Hello")
Hello
>>> "Hello" * 3
'HelloHelloHello'
Many people use the interactive mode as a convenient yet highly programmable
calculator. When you want to end your interactive Python session,
call the exit()
function or hold the Ctrl key down
while you enter a Z, then hit the “Enter” key to get
back to your Windows command prompt.
You may also find that you have a Start-menu entry such as >>>
prompt in a new window. If so, the window will disappear
after you call the exit()
function or enter the Ctrl-Z
character; Windows is running a single “python”
command in the window, and closes it when you terminate the interpreter.
Now that we know the py
command is recognized, you can give your
Python script to it. You’ll have to give either an absolute or a
relative path to the Python script. Let’s say your Python script is
located in your desktop and is named hello.py
, and your command
prompt is nicely opened in your home directory so you’re seeing something
similar to:
C:\Users\YourName>
So now you’ll ask the py
command to give your script to Python by
typing py
followed by your script path:
C:\Users\YourName> py Desktop\hello.py
hello
How do I make Python scripts executable?¶
On Windows, the standard Python installer already associates the .py
extension with a file type (Python.File) and gives that file type an open
command that runs the interpreter (D:\Program Files\Python\python.exe "%1"
%*
). This is enough to make scripts executable from the command prompt as
‘foo.py’. If you’d rather be able to execute the script by simple typing ‘foo’
with no extension you need to add .py to the PATHEXT environment variable.
Why does Python sometimes take so long to start?¶
Usually Python starts very quickly on Windows, but occasionally there are bug reports that Python suddenly begins to take a long time to start up. This is made even more puzzling because Python will work fine on other Windows systems which appear to be configured identically.
The problem may be caused by a misconfiguration of virus checking software on the problem machine. Some virus scanners have been known to introduce startup overhead of two orders of magnitude when the scanner is configured to monitor all reads from the filesystem. Try checking the configuration of virus scanning software on your systems to ensure that they are indeed configured identically. McAfee, when configured to scan all file system read activity, is a particular offender.
How do I make an executable from a Python script?¶
See How can I create a stand-alone binary from a Python script? for a list of tools that can be used to make executables.
Is a *.pyd
file the same as a DLL?¶
Yes, .pyd files are dll’s, but there are a few differences. If you have a DLL
named foo.pyd
, then it must have a function PyInit_foo()
. You can then
write Python “import foo”, and Python will search for foo.pyd (as well as
foo.py, foo.pyc) and if it finds it, will attempt to call PyInit_foo()
to
initialize it. You do not link your .exe with foo.lib, as that would cause
Windows to require the DLL to be present.
Note that the search path for foo.pyd is PYTHONPATH, not the same as the path
that Windows uses to search for foo.dll. Also, foo.pyd need not be present to
run your program, whereas if you linked your program with a dll, the dll is
required. Of course, foo.pyd is required if you want to say import foo
. In
a DLL, linkage is declared in the source code with __declspec(dllexport)
.
In a .pyd, linkage is defined in a list of available functions.
How can I embed Python into a Windows application?¶
Embedding the Python interpreter in a Windows app can be summarized as follows:
Do not build Python into your .exe file directly. On Windows, Python must be a DLL to handle importing modules that are themselves DLL’s. (This is the first key undocumented fact.) Instead, link to
pythonNN.dll
; it is typically installed inC:\Windows\System
. NN is the Python version, a number such as “33” for Python 3.3.You can link to Python in two different ways. Load-time linking means linking against
pythonNN.lib
, while run-time linking means linking againstpythonNN.dll
. (General note:pythonNN.lib
is the so-called “import lib” corresponding topythonNN.dll
. It merely defines symbols for the linker.)Run-time linking greatly simplifies link options; everything happens at run time. Your code must load
pythonNN.dll
using the WindowsLoadLibraryEx()
routine. The code must also use access routines and data inpythonNN.dll
(that is, Python’s C API’s) using pointers obtained by the WindowsGetProcAddress()
routine. Macros can make using these pointers transparent to any C code that calls routines in Python’s C API.If you use SWIG, it is easy to create a Python “extension module” that will make the app’s data and methods available to Python. SWIG will handle just about all the grungy details for you. The result is C code that you link into your .exe file (!) You do not have to create a DLL file, and this also simplifies linking.
SWIG will create an init function (a C function) whose name depends on the name of the extension module. For example, if the name of the module is leo, the init function will be called initleo(). If you use SWIG shadow classes, as you should, the init function will be called initleoc(). This initializes a mostly hidden helper class used by the shadow class.
The reason you can link the C code in step 2 into your .exe file is that calling the initialization function is equivalent to importing the module into Python! (This is the second key undocumented fact.)
In short, you can use the following code to initialize the Python interpreter with your extension module.
#include <Python.h> ... Py_Initialize(); // Initialize Python. initmyAppc(); // Initialize (import) the helper class. PyRun_SimpleString("import myApp"); // Import the shadow class.
There are two problems with Python’s C API which will become apparent if you use a compiler other than MSVC, the compiler used to build pythonNN.dll.
Problem 1: The so-called “Very High Level” functions that take
FILE *
arguments will not work in a multi-compiler environment because each compiler’s notion of astruct FILE
will be different. From an implementation standpoint these are very low level functions.Problem 2: SWIG generates the following code when generating wrappers to void functions:
Py_INCREF(Py_None); _resultobj = Py_None; return _resultobj;
Alas, Py_None is a macro that expands to a reference to a complex data structure called _Py_NoneStruct inside pythonNN.dll. Again, this code will fail in a mult-compiler environment. Replace such code by:
return Py_BuildValue("");
It may be possible to use SWIG’s
%typemap
command to make the change automatically, though I have not been able to get this to work (I’m a complete SWIG newbie).Using a Python shell script to put up a Python interpreter window from inside your Windows app is not a good idea; the resulting window will be independent of your app’s windowing system. Rather, you (or the wxPythonWindow class) should create a “native” interpreter window. It is easy to connect that window to the Python interpreter. You can redirect Python’s i/o to _any_ object that supports read and write, so all you need is a Python object (defined in your extension module) that contains read() and write() methods.
How do I keep editors from inserting tabs into my Python source?¶
The FAQ does not recommend using tabs, and the Python style guide, PEP 8, recommends 4 spaces for distributed Python code; this is also the Emacs python-mode default.
Under any editor, mixing tabs and spaces is a bad idea. MSVC is no different in this respect, and is easily configured to use spaces: Take
, and for file type “Default” set “Tab size” and “Indent size” to 4, and select the “Insert spaces” radio button.Python raises IndentationError
or TabError
if mixed tabs
and spaces are causing problems in leading whitespace.
You may also run the tabnanny
module to check a directory tree
in batch mode.
How do I check for a keypress without blocking?¶
Use the msvcrt
module. This is a standard Windows-specific extension module.
It defines a function kbhit()
which checks whether a keyboard hit is
present, and getch()
which gets one character without echoing it.
How do I solve the missing api-ms-win-crt-runtime-l1-1-0.dll error?¶
This can occur on Python 3.5 and later when using Windows 8.1 or earlier without all updates having been installed. First ensure your operating system is supported and is up to date, and if that does not resolve the issue, visit the Microsoft support page for guidance on manually installing the C Runtime update.


Supporting Cyclic Garbage Collection

Supporting Cyclic Garbage Collection¶
Python’s support for detecting and collecting garbage which involves circular references requires support from object types which are “containers” for other objects which may also be containers. Types which do not store references to other objects, or which only store references to atomic types (such as numbers or strings), do not need to provide any explicit support for garbage collection.
To create a container type, the tp_flags
field of the type object must
include the Py_TPFLAGS_HAVE_GC
and provide an implementation of the
tp_traverse
handler. If instances of the type are mutable, a
tp_clear
implementation must also be provided.
Py_TPFLAGS_HAVE_GC
Objects with a type with this flag set must conform with the rules documented here. For convenience these objects will be referred to as container objects.
Constructors for container types must conform to two rules:
The memory for the object must be allocated using
PyObject_GC_New
orPyObject_GC_NewVar
.Once all the fields which may contain references to other containers are initialized, it must call
PyObject_GC_Track()
.
Similarly, the deallocator for the object must conform to a similar pair of rules:
Before fields which refer to other containers are invalidated,
PyObject_GC_UnTrack()
must be called.The object’s memory must be deallocated using
PyObject_GC_Del()
.Warning
If a type adds the Py_TPFLAGS_HAVE_GC, then it must implement at least a
tp_traverse
handler or explicitly use one from its subclass or subclasses.When calling
PyType_Ready()
or some of the APIs that indirectly call it likePyType_FromSpecWithBases()
orPyType_FromSpec()
the interpreter will automatically populate thetp_flags
,tp_traverse
andtp_clear
fields if the type inherits from a class that implements the garbage collector protocol and the child class does not include thePy_TPFLAGS_HAVE_GC
flag.
-
PyObject_GC_New(TYPE, typeobj)¶
Analogous to
PyObject_New
but for container objects with thePy_TPFLAGS_HAVE_GC
flag set.Do not call this directly to allocate memory for an object; call the type’s
tp_alloc
slot instead.When populating a type’s
tp_alloc
slot,PyType_GenericAlloc()
is preferred over a custom function that simply calls this macro.Memory allocated by this macro must be freed with
PyObject_GC_Del()
(usually called via the object’stp_free
slot).
-
PyObject_GC_NewVar(TYPE, typeobj, size)¶
Analogous to
PyObject_NewVar
but for container objects with thePy_TPFLAGS_HAVE_GC
flag set.Do not call this directly to allocate memory for an object; call the type’s
tp_alloc
slot instead.When populating a type’s
tp_alloc
slot,PyType_GenericAlloc()
is preferred over a custom function that simply calls this macro.Memory allocated by this macro must be freed with
PyObject_GC_Del()
(usually called via the object’stp_free
slot).
-
PyObject *PyUnstable_Object_GC_NewWithExtraData(PyTypeObject *type, size_t extra_size)¶
- This is Unstable API. It may change without warning in minor releases.
Analogous to
PyObject_GC_New
but allocates extra_size bytes at the end of the object (at offsettp_basicsize
). The allocated memory is initialized to zeros, except for thePython object header
.The extra data will be deallocated with the object, but otherwise it is not managed by Python.
Memory allocated by this function must be freed with
PyObject_GC_Del()
(usually called via the object’stp_free
slot).Warning
The function is marked as unstable because the final mechanism for reserving extra data after an instance is not yet decided. For allocating a variable number of fields, prefer using
PyVarObject
andtp_itemsize
instead.Added in version 3.12.
-
PyObject_GC_Resize(TYPE, op, newsize)¶
Resize an object allocated by
PyObject_NewVar
. Returns the resized object of typeTYPE*
(refers to any C type) orNULL
on failure.op must be of type PyVarObject* and must not be tracked by the collector yet. newsize must be of type
Py_ssize_t
.
-
void PyObject_GC_Track(PyObject *op)¶
- Part of the Stable ABI.
Adds the object op to the set of container objects tracked by the collector. The collector can run at unexpected times so objects must be valid while being tracked. This should be called once all the fields followed by the
tp_traverse
handler become valid, usually near the end of the constructor.
-
int PyObject_IS_GC(PyObject *obj)¶
Returns non-zero if the object implements the garbage collector protocol, otherwise returns 0.
The object cannot be tracked by the garbage collector if this function returns 0.
-
int PyObject_GC_IsTracked(PyObject *op)¶
- Part of the Stable ABI since version 3.9.
Returns 1 if the object type of op implements the GC protocol and op is being currently tracked by the garbage collector and 0 otherwise.
This is analogous to the Python function
gc.is_tracked()
.Added in version 3.9.
-
int PyObject_GC_IsFinalized(PyObject *op)¶
- Part of the Stable ABI since version 3.9.
Returns 1 if the object type of op implements the GC protocol and op has been already finalized by the garbage collector and 0 otherwise.
This is analogous to the Python function
gc.is_finalized()
.Added in version 3.9.
-
void PyObject_GC_Del(void *op)¶
- Part of the Stable ABI.
Releases memory allocated to an object using
PyObject_GC_New
orPyObject_GC_NewVar
.Do not call this directly to free an object’s memory; call the type’s
tp_free
slot instead.Do not use this for memory allocated by
PyObject_New
,PyObject_NewVar
, or related allocation functions; usePyObject_Free()
instead.See also
PyObject_Free()
is the non-GC equivalent of this function.
-
void PyObject_GC_UnTrack(void *op)¶
- Part of the Stable ABI.
Remove the object op from the set of container objects tracked by the collector. Note that
PyObject_GC_Track()
can be called again on this object to add it back to the set of tracked objects. The deallocator (tp_dealloc
handler) should call this for the object before any of the fields used by thetp_traverse
handler become invalid.
Changed in version 3.8: The _PyObject_GC_TRACK()
and _PyObject_GC_UNTRACK()
macros
have been removed from the public C API.
The tp_traverse
handler accepts a function parameter of this type:
-
typedef int (*visitproc)(PyObject *object, void *arg)¶
- Part of the Stable ABI.
Type of the visitor function passed to the
tp_traverse
handler. The function should be called with an object to traverse as object and the third parameter to thetp_traverse
handler as arg. The Python core uses several visitor functions to implement cyclic garbage detection; it’s not expected that users will need to write their own visitor functions.
The tp_traverse
handler must have the following type:
-
typedef int (*traverseproc)(PyObject *self, visitproc visit, void *arg)¶
- Part of the Stable ABI.
Traversal function for a container object. Implementations must call the visit function for each object directly contained by self, with the parameters to visit being the contained object and the arg value passed to the handler. The visit function must not be called with a
NULL
object argument. If visit returns a non-zero value that value should be returned immediately.The traversal function must not have any side effects. Implementations may not modify the reference counts of any Python objects nor create or destroy any Python objects.
To simplify writing tp_traverse
handlers, a Py_VISIT()
macro is
provided. In order to use this macro, the tp_traverse
implementation
must name its arguments exactly visit and arg:
-
Py_VISIT(o)¶
If the PyObject* o is not
NULL
, call the visit callback, with arguments o and arg. If visit returns a non-zero value, then return it. Using this macro,tp_traverse
handlers look like:static int my_traverse(Noddy *self, visitproc visit, void *arg) { Py_VISIT(self->foo); Py_VISIT(self->bar); return 0; }
The tp_clear
handler must be of the inquiry
type, or NULL
if the object is immutable.
-
typedef int (*inquiry)(PyObject *self)¶
- Part of the Stable ABI.
Drop references that may have created reference cycles. Immutable objects do not have to define this method since they can never directly create reference cycles. Note that the object must still be valid after calling this method (don’t just call
Py_DECREF()
on a reference). The collector will call this method if it detects that this object is involved in a reference cycle.
Controlling the Garbage Collector State¶
The C-API provides the following functions for controlling garbage collection runs.
-
Py_ssize_t PyGC_Collect(void)¶
- Part of the Stable ABI.
Perform a full garbage collection, if the garbage collector is enabled. (Note that
gc.collect()
runs it unconditionally.)Returns the number of collected + unreachable objects which cannot be collected. If the garbage collector is disabled or already collecting, returns
0
immediately. Errors during garbage collection are passed tosys.unraisablehook
. This function does not raise exceptions.
-
int PyGC_Enable(void)¶
- Part of the Stable ABI since version 3.10.
Enable the garbage collector: similar to
gc.enable()
. Returns the previous state, 0 for disabled and 1 for enabled.Added in version 3.10.
-
int PyGC_Disable(void)¶
- Part of the Stable ABI since version 3.10.
Disable the garbage collector: similar to
gc.disable()
. Returns the previous state, 0 for disabled and 1 for enabled.Added in version 3.10.
-
int PyGC_IsEnabled(void)¶
- Part of the Stable ABI since version 3.10.
Query the state of the garbage collector: similar to
gc.isenabled()
. Returns the current state, 0 for disabled and 1 for enabled.Added in version 3.10.
Querying Garbage Collector State¶
The C-API provides the following interface for querying information about the garbage collector.
-
void PyUnstable_GC_VisitObjects(gcvisitobjects_t callback, void *arg)¶
- This is Unstable API. It may change without warning in minor releases.
Run supplied callback on all live GC-capable objects. arg is passed through to all invocations of callback.
Warning
If new objects are (de)allocated by the callback it is undefined if they will be visited.
Garbage collection is disabled during operation. Explicitly running a collection in the callback may lead to undefined behaviour e.g. visiting the same objects multiple times or not at all.
Added in version 3.12.
-
typedef int (*gcvisitobjects_t)(PyObject *object, void *arg)¶
Type of the visitor function to be passed to
PyUnstable_GC_VisitObjects()
. arg is the same as the arg passed toPyUnstable_GC_VisitObjects
. Return1
to continue iteration, return0
to stop iteration. Other return values are reserved for now so behavior on returning anything else is undefined.Added in version 3.12.


Instrumenting CPython with DTrace and SystemTap¶
- author:
David Malcolm
- author:
Łukasz Langa
DTrace and SystemTap are monitoring tools, each providing a way to inspect what the processes on a computer system are doing. They both use domain-specific languages allowing a user to write scripts which:
filter which processes are to be observed
gather data from the processes of interest
generate reports on the data
As of Python 3.6, CPython can be built with embedded “markers”, also known as “probes”, that can be observed by a DTrace or SystemTap script, making it easier to monitor what the CPython processes on a system are doing.
CPython implementation detail: DTrace markers are implementation details of the CPython interpreter. No guarantees are made about probe compatibility between versions of CPython. DTrace scripts can stop working or work incorrectly without warning when changing CPython versions.
Enabling the static markers¶
macOS comes with built-in support for DTrace. On Linux, in order to build CPython with the embedded markers for SystemTap, the SystemTap development tools must be installed.
On a Linux machine, this can be done via:
$ yum install systemtap-sdt-devel
or:
$ sudo apt-get install systemtap-sdt-dev
CPython must then be configured with the --with-dtrace option
:
checking for --with-dtrace... yes
On macOS, you can list available DTrace probes by running a Python process in the background and listing all probes made available by the Python provider:
$ python3.6 -q &
$ sudo dtrace -l -P python$! # or: dtrace -l -m python3.6
ID PROVIDER MODULE FUNCTION NAME
29564 python18035 python3.6 _PyEval_EvalFrameDefault function-entry
29565 python18035 python3.6 dtrace_function_entry function-entry
29566 python18035 python3.6 _PyEval_EvalFrameDefault function-return
29567 python18035 python3.6 dtrace_function_return function-return
29568 python18035 python3.6 collect gc-done
29569 python18035 python3.6 collect gc-start
29570 python18035 python3.6 _PyEval_EvalFrameDefault line
29571 python18035 python3.6 maybe_dtrace_line line
On Linux, you can verify if the SystemTap static markers are present in the built binary by seeing if it contains a “.note.stapsdt” section.
$ readelf -S ./python | grep .note.stapsdt
[30] .note.stapsdt NOTE 0000000000000000 00308d78
If you’ve built Python as a shared library
(with the --enable-shared
configure option), you
need to look instead within the shared library. For example:
$ readelf -S libpython3.3dm.so.1.0 | grep .note.stapsdt
[29] .note.stapsdt NOTE 0000000000000000 00365b68
Sufficiently modern readelf can print the metadata:
$ readelf -n ./python
Displaying notes found at file offset 0x00000254 with length 0x00000020:
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 2.6.32
Displaying notes found at file offset 0x00000274 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: df924a2b08a7e89f6e11251d4602022977af2670
Displaying notes found at file offset 0x002d6c30 with length 0x00000144:
Owner Data size Description
stapsdt 0x00000031 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: gc__start
Location: 0x00000000004371c3, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf6
Arguments: -4@%ebx
stapsdt 0x00000030 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: gc__done
Location: 0x00000000004374e1, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf8
Arguments: -8@%rax
stapsdt 0x00000045 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: function__entry
Location: 0x000000000053db6c, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6be8
Arguments: 8@%rbp 8@%r12 -4@%eax
stapsdt 0x00000046 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: function__return
Location: 0x000000000053dba8, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bea
Arguments: 8@%rbp 8@%r12 -4@%eax
The above metadata contains information for SystemTap describing how it can patch strategically placed machine code instructions to enable the tracing hooks used by a SystemTap script.
Static DTrace probes¶
The following example DTrace script can be used to show the call/return hierarchy of a Python script, only tracing within the invocation of a function called “start”. In other words, import-time function invocations are not going to be listed:
self int indent;
python$target:::function-entry
/copyinstr(arg1) == "start"/
{
self->trace = 1;
}
python$target:::function-entry
/self->trace/
{
printf("%d\t%*s:", timestamp, 15, probename);
printf("%*s", self->indent, "");
printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2);
self->indent++;
}
python$target:::function-return
/self->trace/
{
self->indent--;
printf("%d\t%*s:", timestamp, 15, probename);
printf("%*s", self->indent, "");
printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2);
}
python$target:::function-return
/copyinstr(arg1) == "start"/
{
self->trace = 0;
}
It can be invoked like this:
$ sudo dtrace -q -s call_stack.d -c "python3.6 script.py"
The output looks like this:
156641360502280 function-entry:call_stack.py:start:23
156641360518804 function-entry: call_stack.py:function_1:1
156641360532797 function-entry: call_stack.py:function_3:9
156641360546807 function-return: call_stack.py:function_3:10
156641360563367 function-return: call_stack.py:function_1:2
156641360578365 function-entry: call_stack.py:function_2:5
156641360591757 function-entry: call_stack.py:function_1:1
156641360605556 function-entry: call_stack.py:function_3:9
156641360617482 function-return: call_stack.py:function_3:10
156641360629814 function-return: call_stack.py:function_1:2
156641360642285 function-return: call_stack.py:function_2:6
156641360656770 function-entry: call_stack.py:function_3:9
156641360669707 function-return: call_stack.py:function_3:10
156641360687853 function-entry: call_stack.py:function_4:13
156641360700719 function-return: call_stack.py:function_4:14
156641360719640 function-entry: call_stack.py:function_5:18
156641360732567 function-return: call_stack.py:function_5:21
156641360747370 function-return:call_stack.py:start:28
Static SystemTap markers¶
The low-level way to use the SystemTap integration is to use the static markers directly. This requires you to explicitly state the binary file containing them.
For example, this SystemTap script can be used to show the call/return hierarchy of a Python script:
probe process("python").mark("function__entry") {
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
printf("%s => %s in %s:%d\\n",
thread_indent(1), funcname, filename, lineno);
}
probe process("python").mark("function__return") {
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
printf("%s <= %s in %s:%d\\n",
thread_indent(-1), funcname, filename, lineno);
}
It can be invoked like this:
$ stap \
show-call-hierarchy.stp \
-c "./python test.py"
The output looks like this:
11408 python(8274): => __contains__ in Lib/_abcoll.py:362
11414 python(8274): => __getitem__ in Lib/os.py:425
11418 python(8274): => encode in Lib/os.py:490
11424 python(8274): <= encode in Lib/os.py:493
11428 python(8274): <= __getitem__ in Lib/os.py:426
11433 python(8274): <= __contains__ in Lib/_abcoll.py:366
where the columns are:
time in microseconds since start of script
name of executable
PID of process
and the remainder indicates the call/return hierarchy as the script executes.
For a --enable-shared
build of CPython, the markers are contained within the
libpython shared library, and the probe’s dotted path needs to reflect this. For
example, this line from the above example:
probe process("python").mark("function__entry") {
should instead read:
probe process("python").library("libpython3.6dm.so.1.0").mark("function__entry") {
(assuming a debug build of CPython 3.6)
Available static markers¶
- function__entry(str filename, str funcname, int lineno)
This marker indicates that execution of a Python function has begun. It is only triggered for pure-Python (bytecode) functions.
The filename, function name, and line number are provided back to the tracing script as positional arguments, which must be accessed using
$arg1
,$arg2
,$arg3
:$arg1
:(const char *)
filename, accessible usinguser_string($arg1)
$arg2
:(const char *)
function name, accessible usinguser_string($arg2)
$arg3
:int
line number
- function__return(str filename, str funcname, int lineno)
This marker is the converse of
function__entry()
, and indicates that execution of a Python function has ended (either viareturn
, or via an exception). It is only triggered for pure-Python (bytecode) functions.The arguments are the same as for
function__entry()
- line(str filename, str funcname, int lineno)
This marker indicates a Python line is about to be executed. It is the equivalent of line-by-line tracing with a Python profiler. It is not triggered within C functions.
The arguments are the same as for
function__entry()
.
- gc__start(int generation)
Fires when the Python interpreter starts a garbage collection cycle.
arg0
is the generation to scan, likegc.collect()
.
- gc__done(long collected)
Fires when the Python interpreter finishes a garbage collection cycle.
arg0
is the number of collected objects.
- import__find__load__start(str modulename)
Fires before
importlib
attempts to find and load the module.arg0
is the module name.Added in version 3.7.
- import__find__load__done(str modulename, int found)
Fires after
importlib
’s find_and_load function is called.arg0
is the module name,arg1
indicates if module was successfully loaded.Added in version 3.7.
- audit(str event, void *tuple)
Fires when
sys.audit()
orPySys_Audit()
is called.arg0
is the event name as C string,arg1
is aPyObject
pointer to a tuple object.Added in version 3.8.
SystemTap Tapsets¶
The higher-level way to use the SystemTap integration is to use a “tapset”: SystemTap’s equivalent of a library, which hides some of the lower-level details of the static markers.
Here is a tapset file, based on a non-shared build of CPython:
/*
Provide a higher-level wrapping around the function__entry and
function__return markers:
\*/
probe python.function.entry = process("python").mark("function__entry")
{
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
frameptr = $arg4
}
probe python.function.return = process("python").mark("function__return")
{
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
frameptr = $arg4
}
If this file is installed in SystemTap’s tapset directory (e.g.
/usr/share/systemtap/tapset
), then these additional probepoints become
available:
- python.function.entry(str filename, str funcname, int lineno, frameptr)
This probe point indicates that execution of a Python function has begun. It is only triggered for pure-Python (bytecode) functions.
- python.function.return(str filename, str funcname, int lineno, frameptr)
This probe point is the converse of
python.function.return
, and indicates that execution of a Python function has ended (either viareturn
, or via an exception). It is only triggered for pure-Python (bytecode) functions.
Examples¶
This SystemTap script uses the tapset above to more cleanly implement the example given above of tracing the Python function-call hierarchy, without needing to directly name the static markers:
probe python.function.entry
{
printf("%s => %s in %s:%d\n",
thread_indent(1), funcname, filename, lineno);
}
probe python.function.return
{
printf("%s <= %s in %s:%d\n",
thread_indent(-1), funcname, filename, lineno);
}
The following script uses the tapset above to provide a top-like view of all running CPython code, showing the top 20 most frequently entered bytecode frames, each second, across the whole system:
global fn_calls;
probe python.function.entry
{
fn_calls[pid(), filename, funcname, lineno] += 1;
}
probe timer.ms(1000) {
printf("\033[2J\033[1;1H") /* clear screen \*/
printf("%6s %80s %6s %30s %6s\n",
"PID", "FILENAME", "LINE", "FUNCTION", "CALLS")
foreach ([pid, filename, funcname, lineno] in fn_calls- limit 20) {
printf("%6d %80s %6d %30s %6d\n",
pid, filename, lineno, funcname,
fn_calls[pid, filename, funcname, lineno]);
}
delete fn_calls;
}


Platform Support

Platform Support¶
The asyncio
module is designed to be portable,
but some platforms have subtle differences and limitations
due to the platforms’ underlying architecture and capabilities.
All Platforms¶
loop.add_reader()
andloop.add_writer()
cannot be used to monitor file I/O.
Windows¶
Source code: Lib/asyncio/proactor_events.py, Lib/asyncio/windows_events.py, Lib/asyncio/windows_utils.py
Changed in version 3.8: On Windows, ProactorEventLoop
is now the default event loop.
All event loops on Windows do not support the following methods:
loop.create_unix_connection()
andloop.create_unix_server()
are not supported. Thesocket.AF_UNIX
socket family is specific to Unix.loop.add_signal_handler()
andloop.remove_signal_handler()
are not supported.
SelectorEventLoop
has the following limitations:
SelectSelector
is used to wait on socket events: it supports sockets and is limited to 512 sockets.loop.add_reader()
andloop.add_writer()
only accept socket handles (e.g. pipe file descriptors are not supported).Pipes are not supported, so the
loop.connect_read_pipe()
andloop.connect_write_pipe()
methods are not implemented.Subprocesses are not supported, i.e.
loop.subprocess_exec()
andloop.subprocess_shell()
methods are not implemented.
ProactorEventLoop
has the following limitations:
The
loop.add_reader()
andloop.add_writer()
methods are not supported.
The resolution of the monotonic clock on Windows is usually around 15.6 milliseconds. The best resolution is 0.5 milliseconds. The resolution depends on the hardware (availability of HPET) and on the Windows configuration.
Subprocess Support on Windows¶
On Windows, the default event loop ProactorEventLoop
supports
subprocesses, whereas SelectorEventLoop
does not.
macOS¶
Modern macOS versions are fully supported.
macOS <= 10.8
On macOS 10.6, 10.7 and 10.8, the default event loop
uses selectors.KqueueSelector
, which does not support
character devices on these versions. The SelectorEventLoop
can be manually configured to use SelectSelector
or PollSelector
to support character devices on
these older versions of macOS. Example:
import asyncio
import selectors
selector = selectors.SelectSelector()
loop = asyncio.SelectorEventLoop(selector)
asyncio.set_event_loop(loop)


Execution model

4. Execution model¶
4.1. Structure of a program¶
A Python program is constructed from code blocks.
A block is a piece of Python program text that is executed as a unit.
The following are blocks: a module, a function body, and a class definition.
Each command typed interactively is a block. A script file (a file given as
standard input to the interpreter or specified as a command line argument to the
interpreter) is a code block. A script command (a command specified on the
interpreter command line with the -c
option) is a code block.
A module run as a top level script (as module __main__
) from the command
line using a -m
argument is also a code block. The string
argument passed to the built-in functions eval()
and exec()
is a
code block.
A code block is executed in an execution frame. A frame contains some administrative information (used for debugging) and determines where and how execution continues after the code block’s execution has completed.
4.2. Naming and binding¶
4.2.1. Binding of names¶
Names refer to objects. Names are introduced by name binding operations.
The following constructs bind names:
formal parameters to functions,
class definitions,
function definitions,
assignment expressions,
targets that are identifiers if occurring in an assignment:
import
statements.type
statements.
The import
statement of the form from ... import *
binds all
names defined in the imported module, except those beginning with an underscore.
This form may only be used at the module level.
A target occurring in a del
statement is also considered bound for
this purpose (though the actual semantics are to unbind the name).
Each assignment or import statement occurs within a block defined by a class or function definition or at the module level (the top-level code block).
If a name is bound in a block, it is a local variable of that block, unless
declared as nonlocal
or global
. If a name is bound at
the module level, it is a global variable. (The variables of the module code
block are local and global.) If a variable is used in a code block but not
defined there, it is a free variable.
Each occurrence of a name in the program text refers to the binding of that name established by the following name resolution rules.
4.2.2. Resolution of names¶
A scope defines the visibility of a name within a block. If a local variable is defined in a block, its scope includes that block. If the definition occurs in a function block, the scope extends to any blocks contained within the defining one, unless a contained block introduces a different binding for the name.
When a name is used in a code block, it is resolved using the nearest enclosing scope. The set of all such scopes visible to a code block is called the block’s environment.
When a name is not found at all, a NameError
exception is raised.
If the current scope is a function scope, and the name refers to a local
variable that has not yet been bound to a value at the point where the name is
used, an UnboundLocalError
exception is raised.
UnboundLocalError
is a subclass of NameError
.
If a name binding operation occurs anywhere within a code block, all uses of the name within the block are treated as references to the current block. This can lead to errors when a name is used within a block before it is bound. This rule is subtle. Python lacks declarations and allows name binding operations to occur anywhere within a code block. The local variables of a code block can be determined by scanning the entire text of the block for name binding operations. See the FAQ entry on UnboundLocalError for examples.
If the global
statement occurs within a block, all uses of the names
specified in the statement refer to the bindings of those names in the top-level
namespace. Names are resolved in the top-level namespace by searching the
global namespace, i.e. the namespace of the module containing the code block,
and the builtins namespace, the namespace of the module builtins
. The
global namespace is searched first. If the names are not found there, the
builtins namespace is searched next. If the names are also not found in the
builtins namespace, new variables are created in the global namespace.
The global statement must precede all uses of the listed names.
The global
statement has the same scope as a name binding operation
in the same block. If the nearest enclosing scope for a free variable contains
a global statement, the free variable is treated as a global.
The nonlocal
statement causes corresponding names to refer
to previously bound variables in the nearest enclosing function scope.
SyntaxError
is raised at compile time if the given name does not
exist in any enclosing function scope. Type parameters
cannot be rebound with the nonlocal
statement.
The namespace for a module is automatically created the first time a module is
imported. The main module for a script is always called __main__
.
Class definition blocks and arguments to exec()
and eval()
are
special in the context of name resolution.
A class definition is an executable statement that may use and define names.
These references follow the normal rules for name resolution with an exception
that unbound local variables are looked up in the global namespace.
The namespace of the class definition becomes the attribute dictionary of
the class. The scope of names defined in a class block is limited to the
class block; it does not extend to the code blocks of methods. This includes
comprehensions and generator expressions, but it does not include
annotation scopes,
which have access to their enclosing class scopes.
This means that the following will fail:
class A:
a = 42
b = list(a + i for i in range(10))
However, the following will succeed:
class A:
type Alias = Nested
class Nested: pass
print(A.Alias.__value__) # <type 'A.Nested'>
4.2.3. Annotation scopes¶
Annotations, type parameter lists
and type
statements
introduce annotation scopes, which behave mostly like function scopes,
but with some exceptions discussed below.
Annotation scopes are used in the following contexts:
Type parameter lists for generic type aliases.
Type parameter lists for generic functions. A generic function’s annotations are executed within the annotation scope, but its defaults and decorators are not.
Type parameter lists for generic classes. A generic class’s base classes and keyword arguments are executed within the annotation scope, but its decorators are not.
The bounds, constraints, and default values for type parameters (lazily evaluated).
The value of type aliases (lazily evaluated).
Annotation scopes differ from function scopes in the following ways:
Annotation scopes have access to their enclosing class namespace. If an annotation scope is immediately within a class scope, or within another annotation scope that is immediately within a class scope, the code in the annotation scope can use names defined in the class scope as if it were executed directly within the class body. This contrasts with regular functions defined within classes, which cannot access names defined in the class scope.
Expressions in annotation scopes cannot contain
yield
,yield from
,await
, or:=
expressions. (These expressions are allowed in other scopes contained within the annotation scope.)Names defined in annotation scopes cannot be rebound with
nonlocal
statements in inner scopes. This includes only type parameters, as no other syntactic elements that can appear within annotation scopes can introduce new names.While annotation scopes have an internal name, that name is not reflected in the qualified name of objects defined within the scope. Instead, the
__qualname__
of such objects is as if the object were defined in the enclosing scope.
Added in version 3.12: Annotation scopes were introduced in Python 3.12 as part of PEP 695.
Changed in version 3.13: Annotation scopes are also used for type parameter defaults, as introduced by PEP 696.
4.2.4. Lazy evaluation¶
Most annotation scopes are lazily evaluated. This includes annotations,
the values of type aliases created through the type
statement, and
the bounds, constraints, and default values of type
variables created through the type parameter syntax.
This means that they are not evaluated when the type alias or type variable is
created, or when the object carrying annotations is created. Instead, they
are only evaluated when necessary, for example when the __value__
attribute on a type alias is accessed.
Example:
>>> type Alias = 1/0
>>> Alias.__value__
Traceback (most recent call last):
...
ZeroDivisionError: division by zero
>>> def func[T: 1/0](): pass
>>> T = func.__type_params__[0]
>>> T.__bound__
Traceback (most recent call last):
...
ZeroDivisionError: division by zero
Here the exception is raised only when the __value__
attribute
of the type alias or the __bound__
attribute of the type variable
is accessed.
This behavior is primarily useful for references to types that have not yet been defined when the type alias or type variable is created. For example, lazy evaluation enables creation of mutually recursive type aliases:
from typing import Literal
type SimpleExpr = int | Parenthesized
type Parenthesized = tuple[Literal["("], Expr, Literal[")"]]
type Expr = SimpleExpr | tuple[SimpleExpr, Literal["+", "-"], Expr]
Lazily evaluated values are evaluated in annotation scope, which means that names that appear inside the lazily evaluated value are looked up as if they were used in the immediately enclosing scope.
Added in version 3.12.
4.2.5. Builtins and restricted execution¶
CPython implementation detail: Users should not touch __builtins__
; it is strictly an implementation
detail. Users wanting to override values in the builtins namespace should
import
the builtins
module and modify its
attributes appropriately.
The builtins namespace associated with the execution of a code block
is actually found by looking up the name __builtins__
in its
global namespace; this should be a dictionary or a module (in the
latter case the module’s dictionary is used). By default, when in the
__main__
module, __builtins__
is the built-in module
builtins
; when in any other module, __builtins__
is an
alias for the dictionary of the builtins
module itself.
4.2.6. Interaction with dynamic features¶
Name resolution of free variables occurs at runtime, not at compile time. This means that the following code will print 42:
i = 10
def f():
print(i)
i = 42
f()
The eval()
and exec()
functions do not have access to the full
environment for resolving names. Names may be resolved in the local and global
namespaces of the caller. Free variables are not resolved in the nearest
enclosing namespace, but in the global namespace. [1] The exec()
and
eval()
functions have optional arguments to override the global and local
namespace. If only one namespace is specified, it is used for both.
4.3. Exceptions¶
Exceptions are a means of breaking out of the normal flow of control of a code block in order to handle errors or other exceptional conditions. An exception is raised at the point where the error is detected; it may be handled by the surrounding code block or by any code block that directly or indirectly invoked the code block where the error occurred.
The Python interpreter raises an exception when it detects a run-time error
(such as division by zero). A Python program can also explicitly raise an
exception with the raise
statement. Exception handlers are specified
with the try
… except
statement. The finally
clause of such a statement can be used to specify cleanup code which does not
handle the exception, but is executed whether an exception occurred or not in
the preceding code.
Python uses the “termination” model of error handling: an exception handler can find out what happened and continue execution at an outer level, but it cannot repair the cause of the error and retry the failing operation (except by re-entering the offending piece of code from the top).
When an exception is not handled at all, the interpreter terminates execution of
the program, or returns to its interactive main loop. In either case, it prints
a stack traceback, except when the exception is SystemExit
.
Exceptions are identified by class instances. The except
clause is
selected depending on the class of the instance: it must reference the class of
the instance or a non-virtual base class thereof.
The instance can be received by the handler and can carry additional information
about the exceptional condition.
Note
Exception messages are not part of the Python API. Their contents may change from one version of Python to the next without warning and should not be relied on by code which will run under multiple versions of the interpreter.
See also the description of the try
statement in section The try statement
and raise
statement in section The raise statement.
4.4. Runtime Components¶
4.4.1. General Computing Model¶
Python’s execution model does not operate in a vacuum. It runs on a host machine and through that host’s runtime environment, including its operating system (OS), if there is one. When a program runs, the conceptual layers of how it runs on the host look something like this:
host machineprocess (global resources)thread (runs machine code)
Each process represents a program running on the host. Think of each process itself as the data part of its program. Think of the process’ threads as the execution part of the program. This distinction will be important to understand the conceptual Python runtime.
The process, as the data part, is the execution context in which the program runs. It mostly consists of the set of resources assigned to the program by the host, including memory, signals, file handles, sockets, and environment variables.
Processes are isolated and independent from one another. (The same is true for hosts.) The host manages the process’ access to its assigned resources, in addition to coordinating between processes.
Each thread represents the actual execution of the program’s machine code, running relative to the resources assigned to the program’s process. It’s strictly up to the host how and when that execution takes place.
From the point of view of Python, a program always starts with exactly one thread. However, the program may grow to run in multiple simultaneous threads. Not all hosts support multiple threads per process, but most do. Unlike processes, threads in a process are not isolated and independent from one another. Specifically, all threads in a process share all of the process’ resources.
The fundamental point of threads is that each one does run independently, at the same time as the others. That may be only conceptually at the same time (“concurrently”) or physically (“in parallel”). Either way, the threads effectively run at a non-synchronized rate.
Note
That non-synchronized rate means none of the process’ memory is guaranteed to stay consistent for the code running in any given thread. Thus multi-threaded programs must take care to coordinate access to intentionally shared resources. Likewise, they must take care to be absolutely diligent about not accessing any other resources in multiple threads; otherwise two threads running at the same time might accidentally interfere with each other’s use of some shared data. All this is true for both Python programs and the Python runtime.
The cost of this broad, unstructured requirement is the tradeoff for the kind of raw concurrency that threads provide. The alternative to the required discipline generally means dealing with non-deterministic bugs and data corruption.
4.4.2. Python Runtime Model¶
The same conceptual layers apply to each Python program, with some extra data layers specific to Python:
host machineprocess (global resources)Python global runtime (state)Python interpreter (state)thread (runs Python bytecode and “C-API”)Python thread state
At the conceptual level: when a Python program starts, it looks exactly like that diagram, with one of each. The runtime may grow to include multiple interpreters, and each interpreter may grow to include multiple thread states.
Note
A Python implementation won’t necessarily implement the runtime
layers distinctly or even concretely. The only exception is places
where distinct layers are directly specified or exposed to users,
like through the threading
module.
Note
The initial interpreter is typically called the “main” interpreter. Some Python implementations, like CPython, assign special roles to the main interpreter.
Likewise, the host thread where the runtime was initialized is known as the “main” thread. It may be different from the process’ initial thread, though they are often the same. In some cases “main thread” may be even more specific and refer to the initial thread state. A Python runtime might assign specific responsibilities to the main thread, such as handling signals.
As a whole, the Python runtime consists of the global runtime state, interpreters, and thread states. The runtime ensures all that state stays consistent over its lifetime, particularly when used with multiple host threads.
The global runtime, at the conceptual level, is just a set of interpreters. While those interpreters are otherwise isolated and independent from one another, they may share some data or other resources. The runtime is responsible for managing these global resources safely. The actual nature and management of these resources is implementation-specific. Ultimately, the external utility of the global runtime is limited to managing interpreters.
In contrast, an “interpreter” is conceptually what we would normally think of as the (full-featured) “Python runtime”. When machine code executing in a host thread interacts with the Python runtime, it calls into Python in the context of a specific interpreter.
Note
The term “interpreter” here is not the same as the “bytecode interpreter”, which is what regularly runs in threads, executing compiled Python code.
In an ideal world, “Python runtime” would refer to what we currently call “interpreter”. However, it’s been called “interpreter” at least since introduced in 1997 (CPython:a027efa5b).
Each interpreter completely encapsulates all of the non-process-global,
non-thread-specific state needed for the Python runtime to work.
Notably, the interpreter’s state persists between uses. It includes
fundamental data like sys.modules
. The runtime ensures
multiple threads using the same interpreter will safely
share it between them.
A Python implementation may support using multiple interpreters at the
same time in the same process. They are independent and isolated from
one another. For example, each interpreter has its own
sys.modules
.
For thread-specific runtime state, each interpreter has a set of thread states, which it manages, in the same way the global runtime contains a set of interpreters. It can have thread states for as many host threads as it needs. It may even have multiple thread states for the same host thread, though that isn’t as common.
Each thread state, conceptually, has all the thread-specific runtime data an interpreter needs to operate in one host thread. The thread state includes the current raised exception and the thread’s Python call stack. It may include other thread-specific resources.
Note
The term “Python thread” can sometimes refer to a thread state, but
normally it means a thread created using the threading
module.
Each thread state, over its lifetime, is always tied to exactly one interpreter and exactly one host thread. It will only ever be used in that thread and with that interpreter.
Multiple thread states may be tied to the same host thread, whether for different interpreters or even the same interpreter. However, for any given host thread, only one of the thread states tied to it can be used by the thread at a time.
Thread states are isolated and independent from one another and don’t share any data, except for possibly sharing an interpreter and objects or other resources belonging to that interpreter.
Once a program is running, new Python threads can be created using the
threading
module (on platforms and Python implementations that
support threads). Additional processes can be created using the
os
, subprocess
, and multiprocessing
modules.
Interpreters can be created and used with the
interpreters
module. Coroutines (async) can
be run using asyncio
in each interpreter, typically only
in a single thread (often the main thread).
Footnotes


: Miscellaneous utilities

email.utils
: Miscellaneous utilities¶
Source code: Lib/email/utils.py
There are a couple of useful utilities provided in the email.utils
module:
- email.utils.localtime(dt=None)¶
Return local time as an aware datetime object. If called without arguments, return current time. Otherwise dt argument should be a
datetime
instance, and it is converted to the local time zone according to the system time zone database. If dt is naive (that is,dt.tzinfo
isNone
), it is assumed to be in local time.Added in version 3.3.
Deprecated since version 3.12, removed in version 3.14: The isdst parameter.
- email.utils.make_msgid(idstring=None, domain=None)¶
Returns a string suitable for an RFC 2822-compliant Message-ID header. Optional idstring if given, is a string used to strengthen the uniqueness of the message id. Optional domain if given provides the portion of the msgid after the ‘@’. The default is the local hostname. It is not normally necessary to override this default, but may be useful certain cases, such as a constructing distributed system that uses a consistent domain name across multiple hosts.
Changed in version 3.2: Added the domain keyword.
The remaining functions are part of the legacy (Compat32
) email API. There
is no need to directly use these with the new API, since the parsing and
formatting they provide is done automatically by the header parsing machinery
of the new API.
- email.utils.quote(str)¶
Return a new string with backslashes in str replaced by two backslashes, and double quotes replaced by backslash-double quote.
- email.utils.unquote(str)¶
Return a new string which is an unquoted version of str. If str ends and begins with double quotes, they are stripped off. Likewise if str ends and begins with angle brackets, they are stripped off.
- email.utils.parseaddr(address, *, strict=True)¶
Parse address – which should be the value of some address-containing field such as To or Cc – into its constituent realname and email address parts. Returns a tuple of that information, unless the parse fails, in which case a 2-tuple of
('', '')
is returned.If strict is true, use a strict parser which rejects malformed inputs.
Changed in version 3.13: Add strict optional parameter and reject malformed inputs by default.
- email.utils.formataddr(pair, charset='utf-8')¶
The inverse of
parseaddr()
, this takes a 2-tuple of the form(realname, email_address)
and returns the string value suitable for a To or Cc header. If the first element of pair is false, then the second element is returned unmodified.Optional charset is the character set that will be used in the RFC 2047 encoding of the
realname
if therealname
contains non-ASCII characters. Can be an instance ofstr
or aCharset
. Defaults toutf-8
.Changed in version 3.3: Added the charset option.
- email.utils.getaddresses(fieldvalues, *, strict=True)¶
This method returns a list of 2-tuples of the form returned by
parseaddr()
. fieldvalues is a sequence of header field values as might be returned byMessage.get_all
.If strict is true, use a strict parser which rejects malformed inputs.
Here’s a simple example that gets all the recipients of a message:
from email.utils import getaddresses tos = msg.get_all('to', []) ccs = msg.get_all('cc', []) resent_tos = msg.get_all('resent-to', []) resent_ccs = msg.get_all('resent-cc', []) all_recipients = getaddresses(tos + ccs + resent_tos + resent_ccs)
Changed in version 3.13: Add strict optional parameter and reject malformed inputs by default.
- email.utils.parsedate(date)¶
Attempts to parse a date according to the rules in RFC 2822. however, some mailers don’t follow that format as specified, so
parsedate()
tries to guess correctly in such cases. date is a string containing an RFC 2822 date, such as"Mon, 20 Nov 1995 19:12:08 -0500"
. If it succeeds in parsing the date,parsedate()
returns a 9-tuple that can be passed directly totime.mktime()
; otherwiseNone
will be returned. Note that indexes 6, 7, and 8 of the result tuple are not usable.
- email.utils.parsedate_tz(date)¶
Performs the same function as
parsedate()
, but returns eitherNone
or a 10-tuple; the first 9 elements make up a tuple that can be passed directly totime.mktime()
, and the tenth is the offset of the date’s timezone from UTC (which is the official term for Greenwich Mean Time) [1]. If the input string has no timezone, the last element of the tuple returned is0
, which represents UTC. Note that indexes 6, 7, and 8 of the result tuple are not usable.
- email.utils.parsedate_to_datetime(date)¶
The inverse of
format_datetime()
. Performs the same function asparsedate()
, but on success returns adatetime
; otherwiseValueError
is raised if date contains an invalid value such as an hour greater than 23 or a timezone offset not between -24 and 24 hours. If the input date has a timezone of-0000
, thedatetime
will be a naivedatetime
, and if the date is conforming to the RFCs it will represent a time in UTC but with no indication of the actual source timezone of the message the date comes from. If the input date has any other valid timezone offset, thedatetime
will be an awaredatetime
with the corresponding atimezone
tzinfo
.Added in version 3.3.
- email.utils.mktime_tz(tuple)¶
Turn a 10-tuple as returned by
parsedate_tz()
into a UTC timestamp (seconds since the Epoch). If the timezone item in the tuple isNone
, assume local time.
- email.utils.formatdate(timeval=None, localtime=False, usegmt=False)¶
Returns a date string as per RFC 2822, e.g.:
Fri, 09 Nov 2001 01:08:47 -0000
Optional timeval if given is a floating-point time value as accepted by
time.gmtime()
andtime.localtime()
, otherwise the current time is used.Optional localtime is a flag that when
True
, interprets timeval, and returns a date relative to the local timezone instead of UTC, properly taking daylight savings time into account. The default isFalse
meaning UTC is used.Optional usegmt is a flag that when
True
, outputs a date string with the timezone as an ascii stringGMT
, rather than a numeric-0000
. This is needed for some protocols (such as HTTP). This only applies when localtime isFalse
. The default isFalse
.
- email.utils.format_datetime(dt, usegmt=False)¶
Like
formatdate
, but the input is adatetime
instance. If it is a naive datetime, it is assumed to be “UTC with no information about the source timezone”, and the conventional-0000
is used for the timezone. If it is an awaredatetime
, then the numeric timezone offset is used. If it is an aware timezone with offset zero, then usegmt may be set toTrue
, in which case the stringGMT
is used instead of the numeric timezone offset. This provides a way to generate standards conformant HTTP date headers.Added in version 3.3.
- email.utils.encode_rfc2231(s, charset=None, language=None)¶
Encode the string s according to RFC 2231. Optional charset and language, if given is the character set name and language name to use. If neither is given, s is returned as-is. If charset is given but language is not, the string is encoded using the empty string for language.
- email.utils.collapse_rfc2231_value(value, errors='replace', fallback_charset='us-ascii')¶
When a header parameter is encoded in RFC 2231 format,
Message.get_param
may return a 3-tuple containing the character set, language, and value.collapse_rfc2231_value()
turns this into a unicode string. Optional errors is passed to the errors argument ofstr
’sencode()
method; it defaults to'replace'
. Optional fallback_charset specifies the character set to use if the one in the RFC 2231 header is not known by Python; it defaults to'us-ascii'
.For convenience, if the value passed to
collapse_rfc2231_value()
is not a tuple, it should be a string and it is returned unquoted.
- email.utils.decode_params(params)¶
Decode parameters list according to RFC 2231. params is a sequence of 2-tuples containing elements of the form
(content-type, string-value)
.
Footnotes