Simple statements

7. Simple statements¶
A simple statement is comprised within a single logical line. Several simple statements may occur on a single line separated by semicolons. The syntax for simple statements is:
simple_stmt:expression_stmt
|assert_stmt
|assignment_stmt
|augmented_assignment_stmt
|annotated_assignment_stmt
|pass_stmt
|del_stmt
|return_stmt
|yield_stmt
|raise_stmt
|break_stmt
|continue_stmt
|import_stmt
|future_stmt
|global_stmt
|nonlocal_stmt
|type_stmt
7.1. Expression statements¶
Expression statements are used (mostly interactively) to compute and write a
value, or (usually) to call a procedure (a function that returns no meaningful
result; in Python, procedures return the value None
). Other uses of
expression statements are allowed and occasionally useful. The syntax for an
expression statement is:
expression_stmt: starred_expression
An expression statement evaluates the expression list (which may be a single expression).
In interactive mode, if the value is not None
, it is converted to a string
using the built-in repr()
function and the resulting string is written to
standard output on a line by itself (except if the result is None
, so that
procedure calls do not cause any output.)
7.2. Assignment statements¶
Assignment statements are used to (re)bind names to values and to modify attributes or items of mutable objects:
assignment_stmt: (target_list
"=")+ (starred_expression
|yield_expression
) target_list:target
(","target
)* [","] target:identifier
| "(" [target_list
] ")" | "[" [target_list
] "]" |attributeref
|subscription
| "*"target
(See section Primaries for the syntax definitions for attributeref and subscription.)
An assignment statement evaluates the expression list (remember that this can be a single expression or a comma-separated list, the latter yielding a tuple) and assigns the single resulting object to each of the target lists, from left to right.
Assignment is defined recursively depending on the form of the target (list). When a target is part of a mutable object (an attribute reference or subscription), the mutable object must ultimately perform the assignment and decide about its validity, and may raise an exception if the assignment is unacceptable. The rules observed by various types and the exceptions raised are given with the definition of the object types (see section The standard type hierarchy).
Assignment of an object to a target list, optionally enclosed in parentheses or square brackets, is recursively defined as follows.
If the target list is a single target with no trailing comma, optionally in parentheses, the object is assigned to that target.
Else:
If the target list contains one target prefixed with an asterisk, called a “starred” target: The object must be an iterable with at least as many items as there are targets in the target list, minus one. The first items of the iterable are assigned, from left to right, to the targets before the starred target. The final items of the iterable are assigned to the targets after the starred target. A list of the remaining items in the iterable is then assigned to the starred target (the list can be empty).
Else: The object must be an iterable with the same number of items as there are targets in the target list, and the items are assigned, from left to right, to the corresponding targets.
Assignment of an object to a single target is recursively defined as follows.
If the target is an identifier (name):
If the name does not occur in a
global
ornonlocal
statement in the current code block: the name is bound to the object in the current local namespace.Otherwise: the name is bound to the object in the global namespace or the outer namespace determined by
nonlocal
, respectively.
The name is rebound if it was already bound. This may cause the reference count for the object previously bound to the name to reach zero, causing the object to be deallocated and its destructor (if it has one) to be called.
If the target is an attribute reference: The primary expression in the reference is evaluated. It should yield an object with assignable attributes; if this is not the case,
TypeError
is raised. That object is then asked to assign the assigned object to the given attribute; if it cannot perform the assignment, it raises an exception (usually but not necessarilyAttributeError
).Note: If the object is a class instance and the attribute reference occurs on both sides of the assignment operator, the right-hand side expression,
a.x
can access either an instance attribute or (if no instance attribute exists) a class attribute. The left-hand side targeta.x
is always set as an instance attribute, creating it if necessary. Thus, the two occurrences ofa.x
do not necessarily refer to the same attribute: if the right-hand side expression refers to a class attribute, the left-hand side creates a new instance attribute as the target of the assignment:class Cls: x = 3 # class variable inst = Cls() inst.x = inst.x + 1 # writes inst.x as 4 leaving Cls.x as 3
This description does not necessarily apply to descriptor attributes, such as properties created with
property()
.If the target is a subscription: The primary expression in the reference is evaluated. Next, the subscript expression is evaluated. Then, the primary’s
__setitem__()
method is called with two arguments: the subscript and the assigned object.Typically,
__setitem__()
is defined on mutable sequence objects (such as lists) and mapping objects (such as dictionaries), and behaves as follows.If the primary is a mutable sequence object (such as a list), the subscript must yield an integer. If it is negative, the sequence’s length is added to it. The resulting value must be a nonnegative integer less than the sequence’s length, and the sequence is asked to assign the assigned object to its item with that index. If the index is out of range,
IndexError
is raised (assignment to a subscripted sequence cannot add new items to a list).If the primary is a mapping object (such as a dictionary), the subscript must have a type compatible with the mapping’s key type, and the mapping is then asked to create a key/value pair which maps the subscript to the assigned object. This can either replace an existing key/value pair with the same key value, or insert a new key/value pair (if no key with the same value existed).
If the target is a slicing: The primary expression should evaluate to a mutable sequence object (such as a list). The assigned object should be iterable. The slicing’s lower and upper bounds should be integers; if they are
None
(or not present), the defaults are zero and the sequence’s length. If either bound is negative, the sequence’s length is added to it. The resulting bounds are clipped to lie between zero and the sequence’s length, inclusive. Finally, the sequence object is asked to replace the slice with the items of the assigned sequence. The length of the slice may be different from the length of the assigned sequence, thus changing the length of the target sequence, if the target sequence allows it.
Although the definition of assignment implies that overlaps between the
left-hand side and the right-hand side are ‘simultaneous’ (for example a, b =
b, a
swaps two variables), overlaps within the collection of assigned-to
variables occur left-to-right, sometimes resulting in confusion. For instance,
the following program prints [0, 2]
:
x = [0, 1]
i = 0
i, x[i] = 1, 2 # i is updated, then x[i] is updated
print(x)
See also
- PEP 3132 - Extended Iterable Unpacking
The specification for the
*target
feature.
7.2.1. Augmented assignment statements¶
Augmented assignment is the combination, in a single statement, of a binary operation and an assignment statement:
augmented_assignment_stmt:augtarget
augop
(expression_list
|yield_expression
) augtarget:identifier
|attributeref
|subscription
augop: "+=" | "-=" | "*=" | "@=" | "/=" | "//=" | "%=" | "**=" | ">>=" | "<<=" | "&=" | "^=" | "|="
(See section Primaries for the syntax definitions of the last three symbols.)
An augmented assignment evaluates the target (which, unlike normal assignment statements, cannot be an unpacking) and the expression list, performs the binary operation specific to the type of assignment on the two operands, and assigns the result to the original target. The target is only evaluated once.
An augmented assignment statement like x += 1
can be rewritten as x = x +
1
to achieve a similar, but not exactly equal effect. In the augmented
version, x
is only evaluated once. Also, when possible, the actual operation
is performed in-place, meaning that rather than creating a new object and
assigning that to the target, the old object is modified instead.
Unlike normal assignments, augmented assignments evaluate the left-hand side
before evaluating the right-hand side. For example, a[i] += f(x)
first
looks-up a[i]
, then it evaluates f(x)
and performs the addition, and
lastly, it writes the result back to a[i]
.
With the exception of assigning to tuples and multiple targets in a single statement, the assignment done by augmented assignment statements is handled the same way as normal assignments. Similarly, with the exception of the possible in-place behavior, the binary operation performed by augmented assignment is the same as the normal binary operations.
For targets which are attribute references, the same caveat about class and instance attributes applies as for regular assignments.
7.2.2. Annotated assignment statements¶
Annotation assignment is the combination, in a single statement, of a variable or attribute annotation and an optional assignment statement:
annotated_assignment_stmt:augtarget
":"expression
["=" (starred_expression
|yield_expression
)]
The difference from normal Assignment statements is that only a single target is allowed.
The assignment target is considered “simple” if it consists of a single
name that is not enclosed in parentheses.
For simple assignment targets, if in class or module scope,
the annotations are gathered in a lazily evaluated
annotation scope. The annotations can be
evaluated using the __annotations__
attribute of a
class or module, or using the facilities in the annotationlib
module.
If the assignment target is not simple (an attribute, subscript node, or parenthesized name), the annotation is never evaluated.
If a name is annotated in a function scope, then this name is local for that scope. Annotations are never evaluated and stored in function scopes.
If the right hand side is present, an annotated
assignment performs the actual assignment as if there was no annotation
present. If the right hand side is not present for an expression
target, then the interpreter evaluates the target except for the last
__setitem__()
or __setattr__()
call.
See also
- PEP 526 - Syntax for Variable Annotations
The proposal that added syntax for annotating the types of variables (including class variables and instance variables), instead of expressing them through comments.
- PEP 484 - Type hints
The proposal that added the
typing
module to provide a standard syntax for type annotations that can be used in static analysis tools and IDEs.
Changed in version 3.8: Now annotated assignments allow the same expressions in the right hand side as regular assignments. Previously, some expressions (like un-parenthesized tuple expressions) caused a syntax error.
Changed in version 3.14: Annotations are now lazily evaluated in a separate annotation scope. If the assignment target is not simple, annotations are never evaluated.
7.3. The assert
statement¶
Assert statements are a convenient way to insert debugging assertions into a program:
assert_stmt: "assert"expression
[","expression
]
The simple form, assert expression
, is equivalent to
if __debug__:
if not expression: raise AssertionError
The extended form, assert expression1, expression2
, is equivalent to
if __debug__:
if not expression1: raise AssertionError(expression2)
These equivalences assume that __debug__
and AssertionError
refer to
the built-in variables with those names. In the current implementation, the
built-in variable __debug__
is True
under normal circumstances,
False
when optimization is requested (command line option -O
). The current
code generator emits no code for an assert
statement when optimization is
requested at compile time. Note that it is unnecessary to include the source
code for the expression that failed in the error message; it will be displayed
as part of the stack trace.
Assignments to __debug__
are illegal. The value for the built-in variable
is determined when the interpreter starts.
7.4. The pass
statement¶
pass_stmt: "pass"
pass
is a null operation — when it is executed, nothing happens.
It is useful as a placeholder when a statement is required syntactically, but no
code needs to be executed, for example:
def f(arg): pass # a function that does nothing (yet)
class C: pass # a class with no methods (yet)
7.5. The del
statement¶
del_stmt: "del" target_list
Deletion is recursively defined very similar to the way assignment is defined. Rather than spelling it out in full details, here are some hints.
Deletion of a target list recursively deletes each target, from left to right.
Deletion of a name removes the binding of that name from the local or global
namespace, depending on whether the name occurs in a global
statement
in the same code block. Trying to delete an unbound name raises a
NameError
exception.
Deletion of attribute references and subscriptions is passed to the primary object involved; deletion of a slicing is in general equivalent to assignment of an empty slice of the right type (but even this is determined by the sliced object).
Changed in version 3.2: Previously it was illegal to delete a name from the local namespace if it occurs as a free variable in a nested block.
7.6. The return
statement¶
return_stmt: "return" [expression_list
]
return
may only occur syntactically nested in a function definition,
not within a nested class definition.
If an expression list is present, it is evaluated, else None
is substituted.
return
leaves the current function call with the expression list (or
None
) as return value.
When return
passes control out of a try
statement with a
finally
clause, that finally
clause is executed before
really leaving the function.
In a generator function, the return
statement indicates that the
generator is done and will cause StopIteration
to be raised. The returned
value (if any) is used as an argument to construct StopIteration
and
becomes the StopIteration.value
attribute.
In an asynchronous generator function, an empty return
statement
indicates that the asynchronous generator is done and will cause
StopAsyncIteration
to be raised. A non-empty return
statement is a syntax error in an asynchronous generator function.
7.7. The yield
statement¶
yield_stmt: yield_expression
A yield
statement is semantically equivalent to a yield
expression. The yield
statement can be used to omit the
parentheses that would otherwise be required in the equivalent yield expression
statement. For example, the yield statements
yield <expr>
yield from <expr>
are equivalent to the yield expression statements
(yield <expr>)
(yield from <expr>)
Yield expressions and statements are only used when defining a generator
function, and are only used in the body of the generator function. Using yield
in a function definition is sufficient to cause that definition to create a
generator function instead of a normal function.
For full details of yield
semantics, refer to the
Yield expressions section.
7.8. The raise
statement¶
raise_stmt: "raise" [expression
["from"expression
]]
If no expressions are present, raise
re-raises the
exception that is currently being handled, which is also known as the active exception.
If there isn’t currently an active exception, a RuntimeError
exception is raised
indicating that this is an error.
Otherwise, raise
evaluates the first expression as the exception
object. It must be either a subclass or an instance of BaseException
.
If it is a class, the exception instance will be obtained when needed by
instantiating the class with no arguments.
The type of the exception is the exception instance’s class, the value is the instance itself.
A traceback object is normally created automatically when an exception is raised
and attached to it as the __traceback__
attribute.
You can create an exception and set your own traceback in one step using the
with_traceback()
exception method (which returns the
same exception instance, with its traceback set to its argument), like so:
raise Exception("foo occurred").with_traceback(tracebackobj)
The from
clause is used for exception chaining: if given, the second
expression must be another exception class or instance. If the second
expression is an exception instance, it will be attached to the raised
exception as the __cause__
attribute (which is writable). If the
expression is an exception class, the class will be instantiated and the
resulting exception instance will be attached to the raised exception as the
__cause__
attribute. If the raised exception is not handled, both
exceptions will be printed:
>>> try:
... print(1 / 0)
... except Exception as exc:
... raise RuntimeError("Something bad happened") from exc
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
print(1 / 0)
~~^~~
ZeroDivisionError: division by zero
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
raise RuntimeError("Something bad happened") from exc
RuntimeError: Something bad happened
A similar mechanism works implicitly if a new exception is raised when
an exception is already being handled. An exception may be handled
when an except
or finally
clause, or a
with
statement, is used. The previous exception is then
attached as the new exception’s __context__
attribute:
>>> try:
... print(1 / 0)
... except:
... raise RuntimeError("Something bad happened")
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
print(1 / 0)
~~^~~
ZeroDivisionError: division by zero
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
raise RuntimeError("Something bad happened")
RuntimeError: Something bad happened
Exception chaining can be explicitly suppressed by specifying None
in
the from
clause:
>>> try:
... print(1 / 0)
... except:
... raise RuntimeError("Something bad happened") from None
...
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
RuntimeError: Something bad happened
Additional information on exceptions can be found in section Exceptions, and information about handling exceptions is in section The try statement.
Changed in version 3.3: None
is now permitted as Y
in raise X from Y
.
Added the __suppress_context__
attribute to suppress
automatic display of the exception context.
Changed in version 3.11: If the traceback of the active exception is modified in an except
clause, a subsequent raise
statement re-raises the exception with the
modified traceback. Previously, the exception was re-raised with the
traceback it had when it was caught.
7.9. The break
statement¶
break_stmt: "break"
break
may only occur syntactically nested in a for
or
while
loop, but not nested in a function or class definition within
that loop.
It terminates the nearest enclosing loop, skipping the optional else
clause if the loop has one.
If a for
loop is terminated by break
, the loop control
target keeps its current value.
When break
passes control out of a try
statement with a
finally
clause, that finally
clause is executed before
really leaving the loop.
7.10. The continue
statement¶
continue_stmt: "continue"
continue
may only occur syntactically nested in a for
or
while
loop, but not nested in a function or class definition within
that loop. It continues with the next cycle of the nearest enclosing loop.
When continue
passes control out of a try
statement with a
finally
clause, that finally
clause is executed before
really starting the next loop cycle.
7.11. The import
statement¶
import_stmt: "import"module
["as"identifier
] (","module
["as"identifier
])* | "from"relative_module
"import"identifier
["as"identifier
] (","identifier
["as"identifier
])* | "from"relative_module
"import" "("identifier
["as"identifier
] (","identifier
["as"identifier
])* [","] ")" | "from"relative_module
"import" "*" module: (identifier
".")*identifier
relative_module: "."*module
| "."+
The basic import statement (no from
clause) is executed in two
steps:
find a module, loading and initializing it if necessary
define a name or names in the local namespace for the scope where the
import
statement occurs.
When the statement contains multiple clauses (separated by commas) the two steps are carried out separately for each clause, just as though the clauses had been separated out into individual import statements.
The details of the first step, finding and loading modules, are described in greater detail in the section on the import system, which also describes the various types of packages and modules that can be imported, as well as all the hooks that can be used to customize the import system. Note that failures in this step may indicate either that the module could not be located, or that an error occurred while initializing the module, which includes execution of the module’s code.
If the requested module is retrieved successfully, it will be made available in the local namespace in one of three ways:
If the module name is followed by
as
, then the name followingas
is bound directly to the imported module.If no other name is specified, and the module being imported is a top level module, the module’s name is bound in the local namespace as a reference to the imported module
If the module being imported is not a top level module, then the name of the top level package that contains the module is bound in the local namespace as a reference to the top level package. The imported module must be accessed using its full qualified name rather than directly
The from
form uses a slightly more complex process:
find the module specified in the
from
clause, loading and initializing it if necessary;for each of the identifiers specified in the
import
clauses:check if the imported module has an attribute by that name
if not, attempt to import a submodule with that name and then check the imported module again for that attribute
if the attribute is not found,
ImportError
is raised.otherwise, a reference to that value is stored in the local namespace, using the name in the
as
clause if it is present, otherwise using the attribute name
Examples:
import foo # foo imported and bound locally
import foo.bar.baz # foo, foo.bar, and foo.bar.baz imported, foo bound locally
import foo.bar.baz as fbb # foo, foo.bar, and foo.bar.baz imported, foo.bar.baz bound as fbb
from foo.bar import baz # foo, foo.bar, and foo.bar.baz imported, foo.bar.baz bound as baz
from foo import attr # foo imported and foo.attr bound as attr
If the list of identifiers is replaced by a star ('*'
), all public
names defined in the module are bound in the local namespace for the scope
where the import
statement occurs.
The public names defined by a module are determined by checking the module’s
namespace for a variable named __all__
; if defined, it must be a sequence
of strings which are names defined or imported by that module.
Names containing non-ASCII characters must be in the normalization form
NFKC; see Non-ASCII characters in names for details. The names
given in __all__
are all considered public and are required to exist. If
__all__
is not defined, the set of public names includes all names found
in the module’s namespace which do not begin with an underscore character
('_'
). __all__
should contain the entire public API. It is intended
to avoid accidentally exporting items that are not part of the API (such as
library modules which were imported and used within the module).
The wild card form of import — from module import *
— is only allowed at
the module level. Attempting to use it in class or function definitions will
raise a SyntaxError
.
When specifying what module to import you do not have to specify the absolute
name of the module. When a module or package is contained within another
package it is possible to make a relative import within the same top package
without having to mention the package name. By using leading dots in the
specified module or package after from
you can specify how high to
traverse up the current package hierarchy without specifying exact names. One
leading dot means the current package where the module making the import
exists. Two dots means up one package level. Three dots is up two levels, etc.
So if you execute from . import mod
from a module in the pkg
package
then you will end up importing pkg.mod
. If you execute from ..subpkg2
import mod
from within pkg.subpkg1
you will import pkg.subpkg2.mod
.
The specification for relative imports is contained in
the Package Relative Imports section.
importlib.import_module()
is provided to support applications that
determine dynamically the modules to be loaded.
Raises an auditing event import
with arguments module
, filename
, sys.path
, sys.meta_path
, sys.path_hooks
.
7.11.1. Future statements¶
A future statement is a directive to the compiler that a particular module should be compiled using syntax or semantics that will be available in a specified future release of Python where the feature becomes standard.
The future statement is intended to ease migration to future versions of Python that introduce incompatible changes to the language. It allows use of the new features on a per-module basis before the release in which the feature becomes standard.
future_stmt: "from" "__future__" "import"feature
["as"identifier
] (","feature
["as"identifier
])* | "from" "__future__" "import" "("feature
["as"identifier
] (","feature
["as"identifier
])* [","] ")" feature:identifier
A future statement must appear near the top of the module. The only lines that can appear before a future statement are:
the module docstring (if any),
comments,
blank lines, and
other future statements.
The only feature that requires using the future statement is
annotations
(see PEP 563).
All historical features enabled by the future statement are still recognized
by Python 3. The list includes absolute_import
, division
,
generators
, generator_stop
, unicode_literals
,
print_function
, nested_scopes
and with_statement
. They are
all redundant because they are always enabled, and only kept for
backwards compatibility.
A future statement is recognized and treated specially at compile time: Changes to the semantics of core constructs are often implemented by generating different code. It may even be the case that a new feature introduces new incompatible syntax (such as a new reserved word), in which case the compiler may need to parse the module differently. Such decisions cannot be pushed off until runtime.
For any given release, the compiler knows which feature names have been defined, and raises a compile-time error if a future statement contains a feature not known to it.
The direct runtime semantics are the same as for any import statement: there is
a standard module __future__
, described later, and it will be imported in
the usual way at the time the future statement is executed.
The interesting runtime semantics depend on the specific feature enabled by the future statement.
Note that there is nothing special about the statement:
import __future__ [as name]
That is not a future statement; it’s an ordinary import statement with no special semantics or syntax restrictions.
Code compiled by calls to the built-in functions exec()
and compile()
that occur in a module M
containing a future statement will, by default,
use the new syntax or semantics associated with the future statement. This can
be controlled by optional arguments to compile()
— see the documentation
of that function for details.
A future statement typed at an interactive interpreter prompt will take effect
for the rest of the interpreter session. If an interpreter is started with the
-i
option, is passed a script name to execute, and the script includes
a future statement, it will be in effect in the interactive session started
after the script is executed.
See also
- PEP 236 - Back to the __future__
The original proposal for the __future__ mechanism.
7.12. The global
statement¶
global_stmt: "global"identifier
(","identifier
)*
The global
statement causes the listed identifiers to be interpreted
as globals. It would be impossible to assign to a global variable without
global
, although free variables may refer to globals without being
declared global.
The global
statement applies to the entire current scope
(module, function body or class definition).
A SyntaxError
is raised if a variable is used or
assigned to prior to its global declaration in the scope.
At the module level, all variables are global, so a global
statement has no effect.
However, variables must still not be used or
assigned to prior to their global
declaration.
This requirement is relaxed in the interactive prompt (REPL).
Programmer’s note: global
is a directive to the parser. It
applies only to code parsed at the same time as the global
statement.
In particular, a global
statement contained in a string or code
object supplied to the built-in exec()
function does not affect the code
block containing the function call, and code contained in such a string is
unaffected by global
statements in the code containing the function
call. The same applies to the eval()
and compile()
functions.
7.13. The nonlocal
statement¶
nonlocal_stmt: "nonlocal"identifier
(","identifier
)*
When the definition of a function or class is nested (enclosed) within
the definitions of other functions, its nonlocal scopes are the local
scopes of the enclosing functions. The nonlocal
statement
causes the listed identifiers to refer to names previously bound in
nonlocal scopes. It allows encapsulated code to rebind such nonlocal
identifiers. If a name is bound in more than one nonlocal scope, the
nearest binding is used. If a name is not bound in any nonlocal scope,
or if there is no nonlocal scope, a SyntaxError
is raised.
The nonlocal
statement applies to the entire scope of a function or
class body. A SyntaxError
is raised if a variable is used or
assigned to prior to its nonlocal declaration in the scope.
Programmer’s note: nonlocal
is a directive to the parser
and applies only to code parsed along with it. See the note for the
global
statement.
7.14. The type
statement¶
type_stmt: 'type'identifier
[type_params
] "="expression
The type
statement declares a type alias, which is an instance
of typing.TypeAliasType
.
For example, the following statement creates a type alias:
type Point = tuple[float, float]
This code is roughly equivalent to:
annotation-def VALUE_OF_Point():
return tuple[float, float]
Point = typing.TypeAliasType("Point", VALUE_OF_Point())
annotation-def
indicates an annotation scope, which behaves
mostly like a function, but with several small differences.
The value of the
type alias is evaluated in the annotation scope. It is not evaluated when the
type alias is created, but only when the value is accessed through the type alias’s
__value__
attribute (see Lazy evaluation).
This allows the type alias to refer to names that are not yet defined.
Type aliases may be made generic by adding a type parameter list after the name. See Generic type aliases for more.
type
is a soft keyword.
Added in version 3.12.
See also
- PEP 695 - Type Parameter Syntax
Introduced the
type
statement and syntax for generic classes and functions.


HOWTO Fetch Internet Resources Using The urllib Package¶
- Author:
Introduction¶
urllib.request is a Python module for fetching URLs (Uniform Resource Locators). It offers a very simple interface, in the form of the urlopen function. This is capable of fetching URLs using a variety of different protocols. It also offers a slightly more complex interface for handling common situations - like basic authentication, cookies, proxies and so on. These are provided by objects called handlers and openers.
urllib.request supports fetching URLs for many “URL schemes” (identified by the string
before the ":"
in URL - for example "ftp"
is the URL scheme of
"ftp://python.org/"
) using their associated network protocols (e.g. FTP, HTTP).
This tutorial focuses on the most common case, HTTP.
For straightforward situations urlopen is very easy to use. But as soon as you
encounter errors or non-trivial cases when opening HTTP URLs, you will need some
understanding of the HyperText Transfer Protocol. The most comprehensive and
authoritative reference to HTTP is RFC 2616. This is a technical document and
not intended to be easy to read. This HOWTO aims to illustrate using urllib,
with enough detail about HTTP to help you through. It is not intended to replace
the urllib.request
docs, but is supplementary to them.
Fetching URLs¶
The simplest way to use urllib.request is as follows:
import urllib.request
with urllib.request.urlopen('http://python.org/') as response:
html = response.read()
If you wish to retrieve a resource via URL and store it in a temporary
location, you can do so via the shutil.copyfileobj()
and
tempfile.NamedTemporaryFile()
functions:
import shutil
import tempfile
import urllib.request
with urllib.request.urlopen('http://python.org/') as response:
with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
shutil.copyfileobj(response, tmp_file)
with open(tmp_file.name) as html:
pass
Many uses of urllib will be that simple (note that instead of an ‘http:’ URL we could have used a URL starting with ‘ftp:’, ‘file:’, etc.). However, it’s the purpose of this tutorial to explain the more complicated cases, concentrating on HTTP.
HTTP is based on requests and responses - the client makes requests and servers
send responses. urllib.request mirrors this with a Request
object which represents
the HTTP request you are making. In its simplest form you create a Request
object that specifies the URL you want to fetch. Calling urlopen
with this
Request object returns a response object for the URL requested. This response is
a file-like object, which means you can for example call .read()
on the
response:
import urllib.request
req = urllib.request.Request('http://python.org/')
with urllib.request.urlopen(req) as response:
the_page = response.read()
Note that urllib.request makes use of the same Request interface to handle all URL schemes. For example, you can make an FTP request like so:
req = urllib.request.Request('ftp://example.com/')
In the case of HTTP, there are two extra things that Request objects allow you to do: First, you can pass data to be sent to the server. Second, you can pass extra information (“metadata”) about the data or about the request itself, to the server - this information is sent as HTTP “headers”. Let’s look at each of these in turn.
Data¶
Sometimes you want to send data to a URL (often the URL will refer to a CGI
(Common Gateway Interface) script or other web application). With HTTP,
this is often done using what’s known as a POST request. This is often what
your browser does when you submit a HTML form that you filled in on the web. Not
all POSTs have to come from forms: you can use a POST to transmit arbitrary data
to your own application. In the common case of HTML forms, the data needs to be
encoded in a standard way, and then passed to the Request object as the data
argument. The encoding is done using a function from the urllib.parse
library.
import urllib.parse
import urllib.request
url = 'http://www.someserver.com/cgi-bin/register.cgi'
values = {'name' : 'Michael Foord',
'location' : 'Northampton',
'language' : 'Python' }
data = urllib.parse.urlencode(values)
data = data.encode('ascii') # data should be bytes
req = urllib.request.Request(url, data)
with urllib.request.urlopen(req) as response:
the_page = response.read()
Note that other encodings are sometimes required (e.g. for file upload from HTML forms - see HTML Specification, Form Submission for more details).
If you do not pass the data
argument, urllib uses a GET request. One
way in which GET and POST requests differ is that POST requests often have
“side-effects”: they change the state of the system in some way (for example by
placing an order with the website for a hundredweight of tinned spam to be
delivered to your door). Though the HTTP standard makes it clear that POSTs are
intended to always cause side-effects, and GET requests never to cause
side-effects, nothing prevents a GET request from having side-effects, nor a
POST requests from having no side-effects. Data can also be passed in an HTTP
GET request by encoding it in the URL itself.
This is done as follows:
>>> import urllib.request
>>> import urllib.parse
>>> data = {}
>>> data['name'] = 'Somebody Here'
>>> data['location'] = 'Northampton'
>>> data['language'] = 'Python'
>>> url_values = urllib.parse.urlencode(data)
>>> print(url_values) # The order may differ from below.
name=Somebody+Here&language=Python&location=Northampton
>>> url = 'http://www.example.com/example.cgi'
>>> full_url = url + '?' + url_values
>>> data = urllib.request.urlopen(full_url)
Notice that the full URL is created by adding a ?
to the URL, followed by
the encoded values.
Headers¶
We’ll discuss here one particular HTTP header, to illustrate how to add headers to your HTTP request.
Some websites [1] dislike being browsed by programs, or send different versions
to different browsers [2]. By default urllib identifies itself as
Python-urllib/x.y
(where x
and y
are the major and minor version
numbers of the Python release,
e.g. Python-urllib/2.5
), which may confuse the site, or just plain
not work. The way a browser identifies itself is through the
User-Agent
header [3]. When you create a Request object you can
pass a dictionary of headers in. The following example makes the same
request as above, but identifies itself as a version of Internet
Explorer [4].
import urllib.parse
import urllib.request
url = 'http://www.someserver.com/cgi-bin/register.cgi'
user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)'
values = {'name': 'Michael Foord',
'location': 'Northampton',
'language': 'Python' }
headers = {'User-Agent': user_agent}
data = urllib.parse.urlencode(values)
data = data.encode('ascii')
req = urllib.request.Request(url, data, headers)
with urllib.request.urlopen(req) as response:
the_page = response.read()
The response also has two useful methods. See the section on info and geturl which comes after we have a look at what happens when things go wrong.
Handling Exceptions¶
urlopen raises URLError
when it cannot handle a response (though as
usual with Python APIs, built-in exceptions such as ValueError
,
TypeError
etc. may also be raised).
HTTPError
is the subclass of URLError
raised in the specific case of
HTTP URLs.
The exception classes are exported from the urllib.error
module.
URLError¶
Often, URLError is raised because there is no network connection (no route to the specified server), or the specified server doesn’t exist. In this case, the exception raised will have a ‘reason’ attribute, which is a tuple containing an error code and a text error message.
e.g.
>>> req = urllib.request.Request('http://www.pretend_server.org')
>>> try: urllib.request.urlopen(req)
... except urllib.error.URLError as e:
... print(e.reason)
...
(4, 'getaddrinfo failed')
HTTPError¶
Every HTTP response from the server contains a numeric “status code”. Sometimes
the status code indicates that the server is unable to fulfil the request. The
default handlers will handle some of these responses for you (for example, if
the response is a “redirection” that requests the client fetch the document from
a different URL, urllib will handle that for you). For those it can’t handle,
urlopen will raise an HTTPError
. Typical errors include ‘404’ (page not
found), ‘403’ (request forbidden), and ‘401’ (authentication required).
See section 10 of RFC 2616 for a reference on all the HTTP error codes.
The HTTPError
instance raised will have an integer ‘code’ attribute, which
corresponds to the error sent by the server.
Error Codes¶
Because the default handlers handle redirects (codes in the 300 range), and codes in the 100–299 range indicate success, you will usually only see error codes in the 400–599 range.
http.server.BaseHTTPRequestHandler.responses
is a useful dictionary of
response codes that shows all the response codes used by RFC 2616.
An excerpt from the dictionary is shown below
responses = {
...
<HTTPStatus.OK: 200>: ('OK', 'Request fulfilled, document follows'),
...
<HTTPStatus.FORBIDDEN: 403>: ('Forbidden',
'Request forbidden -- authorization will '
'not help'),
<HTTPStatus.NOT_FOUND: 404>: ('Not Found',
'Nothing matches the given URI'),
...
<HTTPStatus.IM_A_TEAPOT: 418>: ("I'm a Teapot",
'Server refuses to brew coffee because '
'it is a teapot'),
...
<HTTPStatus.SERVICE_UNAVAILABLE: 503>: ('Service Unavailable',
'The server cannot process the '
'request due to a high load'),
...
}
When an error is raised the server responds by returning an HTTP error code
and an error page. You can use the HTTPError
instance as a response on the
page returned. This means that as well as the code attribute, it also has read,
geturl, and info, methods as returned by the urllib.response
module:
>>> req = urllib.request.Request('http://www.python.org/fish.html')
>>> try:
... urllib.request.urlopen(req)
... except urllib.error.HTTPError as e:
... print(e.code)
... print(e.read())
...
404
b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n\n\n<html
...
<title>Page Not Found</title>\n
...
Wrapping it Up¶
So if you want to be prepared for HTTPError
or URLError
there are two
basic approaches. I prefer the second approach.
Number 1¶
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError
req = Request(someurl)
try:
response = urlopen(req)
except HTTPError as e:
print('The server couldn\'t fulfill the request.')
print('Error code: ', e.code)
except URLError as e:
print('We failed to reach a server.')
print('Reason: ', e.reason)
else:
# everything is fine
Note
The except HTTPError
must come first, otherwise except URLError
will also catch an HTTPError
.
Number 2¶
from urllib.request import Request, urlopen
from urllib.error import URLError
req = Request(someurl)
try:
response = urlopen(req)
except URLError as e:
if hasattr(e, 'reason'):
print('We failed to reach a server.')
print('Reason: ', e.reason)
elif hasattr(e, 'code'):
print('The server couldn\'t fulfill the request.')
print('Error code: ', e.code)
else:
# everything is fine
info and geturl¶
The response returned by urlopen (or the HTTPError
instance) has two
useful methods info()
and geturl()
and is defined in the module
urllib.response
.
geturl - this returns the real URL of the page fetched. This is useful because
urlopen
(or the opener object used) may have followed a redirect. The URL of the page fetched may not be the same as the URL requested.info - this returns a dictionary-like object that describes the page fetched, particularly the headers sent by the server. It is currently an
http.client.HTTPMessage
instance.
Typical headers include ‘Content-length’, ‘Content-type’, and so on. See the Quick Reference to HTTP Headers for a useful listing of HTTP headers with brief explanations of their meaning and use.
Openers and Handlers¶
When you fetch a URL you use an opener (an instance of the perhaps
confusingly named urllib.request.OpenerDirector
). Normally we have been using
the default opener - via urlopen
- but you can create custom
openers. Openers use handlers. All the “heavy lifting” is done by the
handlers. Each handler knows how to open URLs for a particular URL scheme (http,
ftp, etc.), or how to handle an aspect of URL opening, for example HTTP
redirections or HTTP cookies.
You will want to create openers if you want to fetch URLs with specific handlers installed, for example to get an opener that handles cookies, or to get an opener that does not handle redirections.
To create an opener, instantiate an OpenerDirector
, and then call
.add_handler(some_handler_instance)
repeatedly.
Alternatively, you can use build_opener
, which is a convenience function for
creating opener objects with a single function call. build_opener
adds
several handlers by default, but provides a quick way to add more and/or
override the default handlers.
Other sorts of handlers you might want to can handle proxies, authentication, and other common but slightly specialised situations.
install_opener
can be used to make an opener
object the (global) default
opener. This means that calls to urlopen
will use the opener you have
installed.
Opener objects have an open
method, which can be called directly to fetch
urls in the same way as the urlopen
function: there’s no need to call
install_opener
, except as a convenience.
Basic Authentication¶
To illustrate creating and installing a handler we will use the
HTTPBasicAuthHandler
. For a more detailed discussion of this subject –
including an explanation of how Basic Authentication works - see the Basic
Authentication Tutorial.
When authentication is required, the server sends a header (as well as the 401
error code) requesting authentication. This specifies the authentication scheme
and a ‘realm’. The header looks like: WWW-Authenticate: SCHEME
realm="REALM"
.
e.g.
WWW-Authenticate: Basic realm="cPanel Users"
The client should then retry the request with the appropriate name and password
for the realm included as a header in the request. This is ‘basic
authentication’. In order to simplify this process we can create an instance of
HTTPBasicAuthHandler
and an opener to use this handler.
The HTTPBasicAuthHandler
uses an object called a password manager to handle
the mapping of URLs and realms to passwords and usernames. If you know what the
realm is (from the authentication header sent by the server), then you can use a
HTTPPasswordMgr
. Frequently one doesn’t care what the realm is. In that
case, it is convenient to use HTTPPasswordMgrWithDefaultRealm
. This allows
you to specify a default username and password for a URL. This will be supplied
in the absence of you providing an alternative combination for a specific
realm. We indicate this by providing None
as the realm argument to the
add_password
method.
The top-level URL is the first URL that requires authentication. URLs “deeper” than the URL you pass to .add_password() will also match.
# create a password manager
password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
# Add the username and password.
# If we knew the realm, we could use it instead of None.
top_level_url = "http://example.com/foo/"
password_mgr.add_password(None, top_level_url, username, password)
handler = urllib.request.HTTPBasicAuthHandler(password_mgr)
# create "opener" (OpenerDirector instance)
opener = urllib.request.build_opener(handler)
# use the opener to fetch a URL
opener.open(a_url)
# Install the opener.
# Now all calls to urllib.request.urlopen use our opener.
urllib.request.install_opener(opener)
Note
In the above example we only supplied our HTTPBasicAuthHandler
to
build_opener
. By default openers have the handlers for normal situations
– ProxyHandler
(if a proxy setting such as an http_proxy
environment variable is set), UnknownHandler
, HTTPHandler
,
HTTPDefaultErrorHandler
, HTTPRedirectHandler
, FTPHandler
,
FileHandler
, DataHandler
, HTTPErrorProcessor
.
top_level_url
is in fact either a full URL (including the ‘http:’ scheme
component and the hostname and optionally the port number)
e.g. "http://example.com/"
or an “authority” (i.e. the hostname,
optionally including the port number) e.g. "example.com"
or "example.com:8080"
(the latter example includes a port number). The authority, if present, must
NOT contain the “userinfo” component - for example "joe:password@example.com"
is
not correct.
Proxies¶
urllib will auto-detect your proxy settings and use those. This is through
the ProxyHandler
, which is part of the normal handler chain when a proxy
setting is detected. Normally that’s a good thing, but there are occasions
when it may not be helpful [5]. One way to do this is to setup our own
ProxyHandler
, with no proxies defined. This is done using similar steps to
setting up a Basic Authentication handler:
>>> proxy_support = urllib.request.ProxyHandler({})
>>> opener = urllib.request.build_opener(proxy_support)
>>> urllib.request.install_opener(opener)
Note
Currently urllib.request
does not support fetching of https
locations
through a proxy. However, this can be enabled by extending urllib.request as
shown in the recipe [6].
Note
HTTP_PROXY
will be ignored if a variable REQUEST_METHOD
is set; see
the documentation on getproxies()
.
Sockets and Layers¶
The Python support for fetching resources from the web is layered. urllib uses
the http.client
library, which in turn uses the socket library.
As of Python 2.3 you can specify how long a socket should wait for a response before timing out. This can be useful in applications which have to fetch web pages. By default the socket module has no timeout and can hang. Currently, the socket timeout is not exposed at the http.client or urllib.request levels. However, you can set the default timeout globally for all sockets using
import socket
import urllib.request
# timeout in seconds
timeout = 10
socket.setdefaulttimeout(timeout)
# this call to urllib.request.urlopen now uses the default timeout
# we have set in the socket module
req = urllib.request.Request('http://www.voidspace.org.uk')
response = urllib.request.urlopen(req)
Footnotes¶
This document was reviewed and revised by John Lee.


Data Structures

5. Data Structures¶
This chapter describes some things you’ve learned about already in more detail, and adds some new things as well.
5.1. More on Lists¶
The list data type has some more methods. Here are all of the methods of list objects:
- list.append(x)
Add an item to the end of the list. Similar to
a[len(a):] = [x]
.
- list.extend(iterable)
Extend the list by appending all the items from the iterable. Similar to
a[len(a):] = iterable
.
- list.insert(i, x)
Insert an item at a given position. The first argument is the index of the element before which to insert, so
a.insert(0, x)
inserts at the front of the list, anda.insert(len(a), x)
is equivalent toa.append(x)
.
- list.remove(x)
Remove the first item from the list whose value is equal to x. It raises a
ValueError
if there is no such item.
- list.pop([i])
Remove the item at the given position in the list, and return it. If no index is specified,
a.pop()
removes and returns the last item in the list. It raises anIndexError
if the list is empty or the index is outside the list range.
- list.clear()
Remove all items from the list. Similar to
del a[:]
.
- list.index(x[, start[, end]])
Return zero-based index of the first occurrence of x in the list. Raises a
ValueError
if there is no such item.The optional arguments start and end are interpreted as in the slice notation and are used to limit the search to a particular subsequence of the list. The returned index is computed relative to the beginning of the full sequence rather than the start argument.
- list.count(x)
Return the number of times x appears in the list.
- list.sort(*, key=None, reverse=False)
Sort the items of the list in place (the arguments can be used for sort customization, see
sorted()
for their explanation).
- list.reverse()
Reverse the elements of the list in place.
- list.copy()
Return a shallow copy of the list. Similar to
a[:]
.
An example that uses most of the list methods:
>>> fruits = ['orange', 'apple', 'pear', 'banana', 'kiwi', 'apple', 'banana']
>>> fruits.count('apple')
2
>>> fruits.count('tangerine')
0
>>> fruits.index('banana')
3
>>> fruits.index('banana', 4) # Find next banana starting at position 4
6
>>> fruits.reverse()
>>> fruits
['banana', 'apple', 'kiwi', 'banana', 'pear', 'apple', 'orange']
>>> fruits.append('grape')
>>> fruits
['banana', 'apple', 'kiwi', 'banana', 'pear', 'apple', 'orange', 'grape']
>>> fruits.sort()
>>> fruits
['apple', 'apple', 'banana', 'banana', 'grape', 'kiwi', 'orange', 'pear']
>>> fruits.pop()
'pear'
You might have noticed that methods like insert
, remove
or sort
that
only modify the list have no return value printed – they return the default
None
. [1] This is a design principle for all mutable data structures in
Python.
Another thing you might notice is that not all data can be sorted or
compared. For instance, [None, 'hello', 10]
doesn’t sort because
integers can’t be compared to strings and None
can’t be compared to
other types. Also, there are some types that don’t have a defined
ordering relation. For example, 3+4j < 5+7j
isn’t a valid
comparison.
5.1.1. Using Lists as Stacks¶
The list methods make it very easy to use a list as a stack, where the last
element added is the first element retrieved (“last-in, first-out”). To add an
item to the top of the stack, use append()
. To retrieve an item from the
top of the stack, use pop()
without an explicit index. For example:
>>> stack = [3, 4, 5]
>>> stack.append(6)
>>> stack.append(7)
>>> stack
[3, 4, 5, 6, 7]
>>> stack.pop()
7
>>> stack
[3, 4, 5, 6]
>>> stack.pop()
6
>>> stack.pop()
5
>>> stack
[3, 4]
5.1.2. Using Lists as Queues¶
It is also possible to use a list as a queue, where the first element added is the first element retrieved (“first-in, first-out”); however, lists are not efficient for this purpose. While appends and pops from the end of list are fast, doing inserts or pops from the beginning of a list is slow (because all of the other elements have to be shifted by one).
To implement a queue, use collections.deque
which was designed to
have fast appends and pops from both ends. For example:
>>> from collections import deque
>>> queue = deque(["Eric", "John", "Michael"])
>>> queue.append("Terry") # Terry arrives
>>> queue.append("Graham") # Graham arrives
>>> queue.popleft() # The first to arrive now leaves
'Eric'
>>> queue.popleft() # The second to arrive now leaves
'John'
>>> queue # Remaining queue in order of arrival
deque(['Michael', 'Terry', 'Graham'])
5.1.3. List Comprehensions¶
List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each member of another sequence or iterable, or to create a subsequence of those elements that satisfy a certain condition.
For example, assume we want to create a list of squares, like:
>>> squares = []
>>> for x in range(10):
... squares.append(x**2)
...
>>> squares
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
Note that this creates (or overwrites) a variable named x
that still exists
after the loop completes. We can calculate the list of squares without any
side effects using:
squares = list(map(lambda x: x**2, range(10)))
or, equivalently:
squares = [x**2 for x in range(10)]
which is more concise and readable.
A list comprehension consists of brackets containing an expression followed
by a for
clause, then zero or more for
or if
clauses. The result will be a new list resulting from evaluating the expression
in the context of the for
and if
clauses which follow it.
For example, this listcomp combines the elements of two lists if they are not
equal:
>>> [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]
[(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]
and it’s equivalent to:
>>> combs = []
>>> for x in [1,2,3]:
... for y in [3,1,4]:
... if x != y:
... combs.append((x, y))
...
>>> combs
[(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]
Note how the order of the for
and if
statements is the
same in both these snippets.
If the expression is a tuple (e.g. the (x, y)
in the previous example),
it must be parenthesized.
>>> vec = [-4, -2, 0, 2, 4]
>>> # create a new list with the values doubled
>>> [x*2 for x in vec]
[-8, -4, 0, 4, 8]
>>> # filter the list to exclude negative numbers
>>> [x for x in vec if x >= 0]
[0, 2, 4]
>>> # apply a function to all the elements
>>> [abs(x) for x in vec]
[4, 2, 0, 2, 4]
>>> # call a method on each element
>>> freshfruit = [' banana', ' loganberry ', 'passion fruit ']
>>> [weapon.strip() for weapon in freshfruit]
['banana', 'loganberry', 'passion fruit']
>>> # create a list of 2-tuples like (number, square)
>>> [(x, x**2) for x in range(6)]
[(0, 0), (1, 1), (2, 4), (3, 9), (4, 16), (5, 25)]
>>> # the tuple must be parenthesized, otherwise an error is raised
>>> [x, x**2 for x in range(6)]
File "<stdin>", line 1
[x, x**2 for x in range(6)]
^^^^^^^
SyntaxError: did you forget parentheses around the comprehension target?
>>> # flatten a list using a listcomp with two 'for'
>>> vec = [[1,2,3], [4,5,6], [7,8,9]]
>>> [num for elem in vec for num in elem]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
List comprehensions can contain complex expressions and nested functions:
>>> from math import pi
>>> [str(round(pi, i)) for i in range(1, 6)]
['3.1', '3.14', '3.142', '3.1416', '3.14159']
5.1.4. Nested List Comprehensions¶
The initial expression in a list comprehension can be any arbitrary expression, including another list comprehension.
Consider the following example of a 3x4 matrix implemented as a list of 3 lists of length 4:
>>> matrix = [
... [1, 2, 3, 4],
... [5, 6, 7, 8],
... [9, 10, 11, 12],
... ]
The following list comprehension will transpose rows and columns:
>>> [[row[i] for row in matrix] for i in range(4)]
[[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]]
As we saw in the previous section, the inner list comprehension is evaluated in
the context of the for
that follows it, so this example is
equivalent to:
>>> transposed = []
>>> for i in range(4):
... transposed.append([row[i] for row in matrix])
...
>>> transposed
[[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]]
which, in turn, is the same as:
>>> transposed = []
>>> for i in range(4):
... # the following 3 lines implement the nested listcomp
... transposed_row = []
... for row in matrix:
... transposed_row.append(row[i])
... transposed.append(transposed_row)
...
>>> transposed
[[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]]
In the real world, you should prefer built-in functions to complex flow statements.
The zip()
function would do a great job for this use case:
>>> list(zip(*matrix))
[(1, 5, 9), (2, 6, 10), (3, 7, 11), (4, 8, 12)]
See Unpacking Argument Lists for details on the asterisk in this line.
5.2. The del
statement¶
There is a way to remove an item from a list given its index instead of its
value: the del
statement. This differs from the pop()
method
which returns a value. The del
statement can also be used to remove
slices from a list or clear the entire list (which we did earlier by assignment
of an empty list to the slice). For example:
>>> a = [-1, 1, 66.25, 333, 333, 1234.5]
>>> del a[0]
>>> a
[1, 66.25, 333, 333, 1234.5]
>>> del a[2:4]
>>> a
[1, 66.25, 1234.5]
>>> del a[:]
>>> a
[]
del
can also be used to delete entire variables:
>>> del a
Referencing the name a
hereafter is an error (at least until another value
is assigned to it). We’ll find other uses for del
later.
5.3. Tuples and Sequences¶
We saw that lists and strings have many common properties, such as indexing and slicing operations. They are two examples of sequence data types (see Sequence Types — list, tuple, range). Since Python is an evolving language, other sequence data types may be added. There is also another standard sequence data type: the tuple.
A tuple consists of a number of values separated by commas, for instance:
>>> t = 12345, 54321, 'hello!'
>>> t[0]
12345
>>> t
(12345, 54321, 'hello!')
>>> # Tuples may be nested:
>>> u = t, (1, 2, 3, 4, 5)
>>> u
((12345, 54321, 'hello!'), (1, 2, 3, 4, 5))
>>> # Tuples are immutable:
>>> t[0] = 88888
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> # but they can contain mutable objects:
>>> v = ([1, 2, 3], [3, 2, 1])
>>> v
([1, 2, 3], [3, 2, 1])
As you see, on output tuples are always enclosed in parentheses, so that nested tuples are interpreted correctly; they may be input with or without surrounding parentheses, although often parentheses are necessary anyway (if the tuple is part of a larger expression). It is not possible to assign to the individual items of a tuple, however it is possible to create tuples which contain mutable objects, such as lists.
Though tuples may seem similar to lists, they are often used in different
situations and for different purposes.
Tuples are immutable, and usually contain a heterogeneous sequence of
elements that are accessed via unpacking (see later in this section) or indexing
(or even by attribute in the case of namedtuples
).
Lists are mutable, and their elements are usually homogeneous and are
accessed by iterating over the list.
A special problem is the construction of tuples containing 0 or 1 items: the syntax has some extra quirks to accommodate these. Empty tuples are constructed by an empty pair of parentheses; a tuple with one item is constructed by following a value with a comma (it is not sufficient to enclose a single value in parentheses). Ugly, but effective. For example:
>>> empty = ()
>>> singleton = 'hello', # <-- note trailing comma
>>> len(empty)
0
>>> len(singleton)
1
>>> singleton
('hello',)
The statement t = 12345, 54321, 'hello!'
is an example of tuple packing:
the values 12345
, 54321
and 'hello!'
are packed together in a tuple.
The reverse operation is also possible:
>>> x, y, z = t
This is called, appropriately enough, sequence unpacking and works for any sequence on the right-hand side. Sequence unpacking requires that there are as many variables on the left side of the equals sign as there are elements in the sequence. Note that multiple assignment is really just a combination of tuple packing and sequence unpacking.
5.4. Sets¶
Python also includes a data type for sets. A set is an unordered collection with no duplicate elements. Basic uses include membership testing and eliminating duplicate entries. Set objects also support mathematical operations like union, intersection, difference, and symmetric difference.
Curly braces or the set()
function can be used to create sets. Note: to
create an empty set you have to use set()
, not {}
; the latter creates an
empty dictionary, a data structure that we discuss in the next section.
Here is a brief demonstration:
>>> basket = {'apple', 'orange', 'apple', 'pear', 'orange', 'banana'}
>>> print(basket) # show that duplicates have been removed
{'orange', 'banana', 'pear', 'apple'}
>>> 'orange' in basket # fast membership testing
True
>>> 'crabgrass' in basket
False
>>> # Demonstrate set operations on unique letters from two words
>>>
>>> a = set('abracadabra')
>>> b = set('alacazam')
>>> a # unique letters in a
{'a', 'r', 'b', 'c', 'd'}
>>> a - b # letters in a but not in b
{'r', 'd', 'b'}
>>> a | b # letters in a or b or both
{'a', 'c', 'r', 'd', 'b', 'm', 'z', 'l'}
>>> a & b # letters in both a and b
{'a', 'c'}
>>> a ^ b # letters in a or b but not both
{'r', 'd', 'b', 'm', 'z', 'l'}
Similarly to list comprehensions, set comprehensions are also supported:
>>> a = {x for x in 'abracadabra' if x not in 'abc'}
>>> a
{'r', 'd'}
5.5. Dictionaries¶
Another useful data type built into Python is the dictionary (see
Mapping Types — dict). Dictionaries are sometimes found in other languages as
“associative memories” or “associative arrays”. Unlike sequences, which are
indexed by a range of numbers, dictionaries are indexed by keys, which can be
any immutable type; strings and numbers can always be keys. Tuples can be used
as keys if they contain only strings, numbers, or tuples; if a tuple contains
any mutable object either directly or indirectly, it cannot be used as a key.
You can’t use lists as keys, since lists can be modified in place using index
assignments, slice assignments, or methods like append()
and
extend()
.
It is best to think of a dictionary as a set of key: value pairs,
with the requirement that the keys are unique (within one dictionary). A pair of
braces creates an empty dictionary: {}
. Placing a comma-separated list of
key:value pairs within the braces adds initial key:value pairs to the
dictionary; this is also the way dictionaries are written on output.
The main operations on a dictionary are storing a value with some key and
extracting the value given the key. It is also possible to delete a key:value
pair with del
. If you store using a key that is already in use, the old
value associated with that key is forgotten.
Extracting a value for a non-existent key by subscripting (d[key]
) raises a
KeyError
. To avoid getting this error when trying to access a possibly
non-existent key, use the get()
method instead, which returns
None
(or a specified default value) if the key is not in the dictionary.
Performing list(d)
on a dictionary returns a list of all the keys
used in the dictionary, in insertion order (if you want it sorted, just use
sorted(d)
instead). To check whether a single key is in the
dictionary, use the in
keyword.
Here is a small example using a dictionary:
>>> tel = {'jack': 4098, 'sape': 4139}
>>> tel['guido'] = 4127
>>> tel
{'jack': 4098, 'sape': 4139, 'guido': 4127}
>>> tel['jack']
4098
>>> tel['irv']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'irv'
>>> print(tel.get('irv'))
None
>>> del tel['sape']
>>> tel['irv'] = 4127
>>> tel
{'jack': 4098, 'guido': 4127, 'irv': 4127}
>>> list(tel)
['jack', 'guido', 'irv']
>>> sorted(tel)
['guido', 'irv', 'jack']
>>> 'guido' in tel
True
>>> 'jack' not in tel
False
The dict()
constructor builds dictionaries directly from sequences of
key-value pairs:
>>> dict([('sape', 4139), ('guido', 4127), ('jack', 4098)])
{'sape': 4139, 'guido': 4127, 'jack': 4098}
In addition, dict comprehensions can be used to create dictionaries from arbitrary key and value expressions:
>>> {x: x**2 for x in (2, 4, 6)}
{2: 4, 4: 16, 6: 36}
When the keys are simple strings, it is sometimes easier to specify pairs using keyword arguments:
>>> dict(sape=4139, guido=4127, jack=4098)
{'sape': 4139, 'guido': 4127, 'jack': 4098}
5.6. Looping Techniques¶
When looping through dictionaries, the key and corresponding value can be
retrieved at the same time using the items()
method.
>>> knights = {'gallahad': 'the pure', 'robin': 'the brave'}
>>> for k, v in knights.items():
... print(k, v)
...
gallahad the pure
robin the brave
When looping through a sequence, the position index and corresponding value can
be retrieved at the same time using the enumerate()
function.
>>> for i, v in enumerate(['tic', 'tac', 'toe']):
... print(i, v)
...
0 tic
1 tac
2 toe
To loop over two or more sequences at the same time, the entries can be paired
with the zip()
function.
>>> questions = ['name', 'quest', 'favorite color']
>>> answers = ['lancelot', 'the holy grail', 'blue']
>>> for q, a in zip(questions, answers):
... print('What is your {0}? It is {1}.'.format(q, a))
...
What is your name? It is lancelot.
What is your quest? It is the holy grail.
What is your favorite color? It is blue.
To loop over a sequence in reverse, first specify the sequence in a forward
direction and then call the reversed()
function.
>>> for i in reversed(range(1, 10, 2)):
... print(i)
...
9
7
5
3
1
To loop over a sequence in sorted order, use the sorted()
function which
returns a new sorted list while leaving the source unaltered.
>>> basket = ['apple', 'orange', 'apple', 'pear', 'orange', 'banana']
>>> for i in sorted(basket):
... print(i)
...
apple
apple
banana
orange
orange
pear
Using set()
on a sequence eliminates duplicate elements. The use of
sorted()
in combination with set()
over a sequence is an idiomatic
way to loop over unique elements of the sequence in sorted order.
>>> basket = ['apple', 'orange', 'apple', 'pear', 'orange', 'banana']
>>> for f in sorted(set(basket)):
... print(f)
...
apple
banana
orange
pear
It is sometimes tempting to change a list while you are looping over it; however, it is often simpler and safer to create a new list instead.
>>> import math
>>> raw_data = [56.2, float('NaN'), 51.7, 55.3, 52.5, float('NaN'), 47.8]
>>> filtered_data = []
>>> for value in raw_data:
... if not math.isnan(value):
... filtered_data.append(value)
...
>>> filtered_data
[56.2, 51.7, 55.3, 52.5, 47.8]
5.7. More on Conditions¶
The conditions used in while
and if
statements can contain any
operators, not just comparisons.
The comparison operators in
and not in
are membership tests that
determine whether a value is in (or not in) a container. The operators is
and is not
compare whether two objects are really the same object. All
comparison operators have the same priority, which is lower than that of all
numerical operators.
Comparisons can be chained. For example, a < b == c
tests whether a
is
less than b
and moreover b
equals c
.
Comparisons may be combined using the Boolean operators and
and or
, and
the outcome of a comparison (or of any other Boolean expression) may be negated
with not
. These have lower priorities than comparison operators; between
them, not
has the highest priority and or
the lowest, so that A and
not B or C
is equivalent to (A and (not B)) or C
. As always, parentheses
can be used to express the desired composition.
The Boolean operators and
and or
are so-called short-circuit
operators: their arguments are evaluated from left to right, and evaluation
stops as soon as the outcome is determined. For example, if A
and C
are
true but B
is false, A and B and C
does not evaluate the expression
C
. When used as a general value and not as a Boolean, the return value of a
short-circuit operator is the last evaluated argument.
It is possible to assign the result of a comparison or other Boolean expression to a variable. For example,
>>> string1, string2, string3 = '', 'Trondheim', 'Hammer Dance'
>>> non_null = string1 or string2 or string3
>>> non_null
'Trondheim'
Note that in Python, unlike C, assignment inside expressions must be done
explicitly with the
walrus operator :=
.
This avoids a common class of problems encountered in C programs: typing =
in an expression when ==
was intended.
5.8. Comparing Sequences and Other Types¶
Sequence objects typically may be compared to other objects with the same sequence type. The comparison uses lexicographical ordering: first the first two items are compared, and if they differ this determines the outcome of the comparison; if they are equal, the next two items are compared, and so on, until either sequence is exhausted. If two items to be compared are themselves sequences of the same type, the lexicographical comparison is carried out recursively. If all items of two sequences compare equal, the sequences are considered equal. If one sequence is an initial sub-sequence of the other, the shorter sequence is the smaller (lesser) one. Lexicographical ordering for strings uses the Unicode code point number to order individual characters. Some examples of comparisons between sequences of the same type:
(1, 2, 3) < (1, 2, 4)
[1, 2, 3] < [1, 2, 4]
'ABC' < 'C' < 'Pascal' < 'Python'
(1, 2, 3, 4) < (1, 2, 4)
(1, 2) < (1, 2, -1)
(1, 2, 3) == (1.0, 2.0, 3.0)
(1, 2, ('aa', 'ab')) < (1, 2, ('abc', 'a'), 4)
Note that comparing objects of different types with <
or >
is legal
provided that the objects have appropriate comparison methods. For example,
mixed numeric types are compared according to their numeric value, so 0 equals
0.0, etc. Otherwise, rather than providing an arbitrary ordering, the
interpreter will raise a TypeError
exception.
Footnotes

File
, line
:
File
, line
, in
:
File
, line
, in
:


Instance Method Objects

Instance Method Objects¶
An instance method is a wrapper for a PyCFunction
and the new way
to bind a PyCFunction
to a class object. It replaces the former call
PyMethod_New(func, NULL, class)
.
-
PyTypeObject PyInstanceMethod_Type¶
This instance of
PyTypeObject
represents the Python instance method type. It is not exposed to Python programs.
-
int PyInstanceMethod_Check(PyObject *o)¶
Return true if o is an instance method object (has type
PyInstanceMethod_Type
). The parameter must not beNULL
. This function always succeeds.
-
PyObject *PyInstanceMethod_New(PyObject *func)¶
- Return value: New reference.
Return a new instance method object, with func being any callable object. func is the function that will be called when the instance method is called.
-
PyObject *PyInstanceMethod_Function(PyObject *im)¶
- Return value: Borrowed reference.
Return the function object associated with the instance method im.
-
PyObject *PyInstanceMethod_GET_FUNCTION(PyObject *im)¶
- Return value: Borrowed reference.
Macro version of
PyInstanceMethod_Function()
which avoids error checking.
Method Objects¶
Methods are bound function objects. Methods are always bound to an instance of a user-defined class. Unbound methods (methods bound to a class object) are no longer available.
-
PyTypeObject PyMethod_Type¶
This instance of
PyTypeObject
represents the Python method type. This is exposed to Python programs astypes.MethodType
.
-
int PyMethod_Check(PyObject *o)¶
Return true if o is a method object (has type
PyMethod_Type
). The parameter must not beNULL
. This function always succeeds.
-
PyObject *PyMethod_New(PyObject *func, PyObject *self)¶
- Return value: New reference.
Return a new method object, with func being any callable object and self the instance the method should be bound. func is the function that will be called when the method is called. self must not be
NULL
.
-
PyObject *PyMethod_Function(PyObject *meth)¶
- Return value: Borrowed reference.
Return the function object associated with the method meth.
-
PyObject *PyMethod_GET_FUNCTION(PyObject *meth)¶
- Return value: Borrowed reference.
Macro version of
PyMethod_Function()
which avoids error checking.
-
PyObject *PyMethod_Self(PyObject *meth)¶
- Return value: Borrowed reference.
Return the instance associated with the method meth.
-
PyObject *PyMethod_GET_SELF(PyObject *meth)¶
- Return value: Borrowed reference.
Macro version of
PyMethod_Self()
which avoids error checking.


A Conceptual Overview of

A Conceptual Overview of asyncio
¶
This HOWTO article seeks to help you build a sturdy mental
model of how asyncio
fundamentally works, helping you understand the
how and why behind the recommended patterns.
You might be curious about some key asyncio
concepts.
By the end of this article, you’ll be able to comfortably answer these questions:
What’s happening behind the scenes when an object is awaited?
How does
asyncio
differentiate between a task which doesn’t need CPU time (such as a network request or file read) as opposed to a task that does (such as computing n-factorial)?How to write an asynchronous variant of an operation, such as an async sleep or database request.
See also
The guide that inspired this HOWTO article, by Alexander Nordin.
This in-depth YouTube tutorial series on
asyncio
created by Python core team member, Łukasz Langa.500 Lines or Less: A Web Crawler With asyncio Coroutines by A. Jesse Jiryu Davis and Guido van Rossum.
A conceptual overview part 1: the high-level¶
In part 1, we’ll cover the main, high-level building blocks of asyncio
:
the event loop, coroutine functions, coroutine objects, tasks, and await
.
Event Loop¶
Everything in asyncio
happens relative to the event loop.
It’s the star of the show.
It’s like an orchestra conductor.
It’s behind the scenes managing resources.
Some power is explicitly granted to it, but a lot of its ability to get things
done comes from the respect and cooperation of its worker bees.
In more technical terms, the event loop contains a collection of jobs to be run.
Some jobs are added directly by you, and some indirectly by asyncio
.
The event loop takes a job from its backlog of work and invokes it (or “gives
it control”), similar to calling a function, and then that job runs.
Once it pauses or completes, it returns control to the event loop.
The event loop will then select another job from its pool and invoke it.
You can roughly think of the collection of jobs as a queue: jobs are added and
then processed one at a time, generally (but not always) in order.
This process repeats indefinitely, with the event loop cycling endlessly
onwards.
If there are no more jobs pending execution, the event loop is smart enough to
rest and avoid needlessly wasting CPU cycles, and will come back when there’s
more work to be done.
Effective execution relies on jobs sharing well and cooperating; a greedy job could hog control and leave the other jobs to starve, rendering the overall event loop approach rather useless.
import asyncio
# This creates an event loop and indefinitely cycles through
# its collection of jobs.
event_loop = asyncio.new_event_loop()
event_loop.run_forever()
Asynchronous functions and coroutines¶
This is a basic, boring Python function:
def hello_printer():
print(
"Hi, I am a lowly, simple printer, though I have all I "
"need in life -- \nfresh paper and my dearly beloved octopus "
"partner in crime."
)
Calling a regular function invokes its logic or body:
>>> hello_printer()
Hi, I am a lowly, simple printer, though I have all I need in life --
fresh paper and my dearly beloved octopus partner in crime.
The async def, as opposed to just a plain def
, makes
this an asynchronous function (or “coroutine function”).
Calling it creates and returns a coroutine object.
async def loudmouth_penguin(magic_number: int):
print(
"I am a super special talking penguin. Far cooler than that printer. "
f"By the way, my lucky number is: {magic_number}."
)
Calling the async function, loudmouth_penguin
, does not execute the print statement;
instead, it creates a coroutine object:
>>> loudmouth_penguin(magic_number=3)
<coroutine object loudmouth_penguin at 0x104ed2740>
The terms “coroutine function” and “coroutine object” are often conflated
as coroutine.
That can be confusing!
In this article, coroutine specifically refers to a coroutine object, or more
precisely, an instance of types.CoroutineType
(native coroutine).
Note that coroutines can also exist as instances of
collections.abc.Coroutine
– a distinction that matters for type
checking.
A coroutine represents the function’s body or logic. A coroutine has to be explicitly started; again, merely creating the coroutine does not start it. Notably, the coroutine can be paused and resumed at various points within the function’s body. That pausing and resuming ability is what allows for asynchronous behavior!
Coroutines and coroutine functions were built by leveraging the functionality
of generators and
generator functions.
Recall, a generator function is a function that yield
s, like this
one:
def get_random_number():
# This would be a bad random number generator!
print("Hi")
yield 1
print("Hello")
yield 7
print("Howdy")
yield 4
...
Similar to a coroutine function, calling a generator function does not run it. Instead, it creates a generator object:
>>> get_random_number()
<generator object get_random_number at 0x1048671c0>
You can proceed to the next yield
of a generator by using the
built-in function next()
.
In other words, the generator runs, then pauses.
For example:
>>> generator = get_random_number()
>>> next(generator)
Hi
1
>>> next(generator)
Hello
7
Tasks¶
Roughly speaking, tasks are coroutines (not coroutine
functions) tied to an event loop.
A task also maintains a list of callback functions whose importance will become
clear in a moment when we discuss await
.
The recommended way to create tasks is via asyncio.create_task()
.
Creating a task automatically schedules it for execution (by adding a callback to run it in the event loop’s to-do list, that is, collection of jobs).
asyncio
automatically associates tasks with the event loop for you.
This automatic association was purposely designed into asyncio
for
the sake of simplicity.
Without it, you’d have to keep track of the event loop object and pass it to
any coroutine function that wants to create tasks, adding redundant clutter
to your code.
coroutine = loudmouth_penguin(magic_number=5)
# This creates a Task object and schedules its execution via the event loop.
task = asyncio.create_task(coroutine)
Earlier, we manually created the event loop and set it to run forever.
In practice, it’s recommended to use (and common to see) asyncio.run()
,
which takes care of managing the event loop and ensuring the provided
coroutine finishes before advancing.
For example, many async programs follow this setup:
import asyncio
async def main():
# Perform all sorts of wacky, wild asynchronous things...
...
if __name__ == "__main__":
asyncio.run(main())
# The program will not reach the following print statement until the
# coroutine main() finishes.
print("coroutine main() is done!")
It’s important to be aware that the task itself is not added to the event loop, only a callback to the task is. This matters if the task object you created is garbage collected before it’s called by the event loop. For example, consider this program:
1async def hello():
2 print("hello!")
3
4async def main():
5 asyncio.create_task(hello())
6 # Other asynchronous instructions which run for a while
7 # and cede control to the event loop...
8 ...
9
10asyncio.run(main())
Because there’s no reference to the task object created on line 5, it might
be garbage collected before the event loop invokes it.
Later instructions in the coroutine main()
hand control back to the event
loop so it can invoke other jobs.
When the event loop eventually tries to run the task, it might fail and
discover the task object does not exist!
This can also happen even if a coroutine keeps a reference to a task but
completes before that task finishes.
When the coroutine exits, local variables go out of scope and may be subject
to garbage collection.
In practice, asyncio
and Python’s garbage collector work pretty hard to
ensure this sort of thing doesn’t happen.
But that’s no reason to be reckless!
await¶
await
is a Python keyword that’s commonly used in one of two
different ways:
await task
await coroutine
In a crucial way, the behavior of await
depends on the type of object
being awaited.
Awaiting a task will cede control from the current task or coroutine to the event loop. In the process of relinquishing control, a few important things happen. We’ll use the following code example to illustrate:
async def plant_a_tree():
dig_the_hole_task = asyncio.create_task(dig_the_hole())
await dig_the_hole_task
# Other instructions associated with planting a tree.
...
In this example, imagine the event loop has passed control to the start of the
coroutine plant_a_tree()
.
As seen above, the coroutine creates a task and then awaits it.
The await dig_the_hole_task
instruction adds a callback (which will resume
plant_a_tree()
) to the dig_the_hole_task
object’s list of callbacks.
And then, the instruction cedes control to the event loop.
Some time later, the event loop will pass control to dig_the_hole_task
and the task will finish whatever it needs to do.
Once the task finishes, it will add its various callbacks to the event loop,
in this case, a call to resume plant_a_tree()
.
Generally speaking, when the awaited task finishes (dig_the_hole_task
),
the original task or coroutine (plant_a_tree()
) is added back to the event
loop’s to-do list to be resumed.
This is a basic, yet reliable mental model. In practice, the control handoffs are slightly more complex, but not by much. In part 2, we’ll walk through the details that make this possible.
Unlike tasks, awaiting a coroutine does not hand control back to the event
loop!
Wrapping a coroutine in a task first, then awaiting that would cede
control.
The behavior of await coroutine
is effectively the same as invoking a
regular, synchronous Python function.
Consider this program:
import asyncio
async def coro_a():
print("I am coro_a(). Hi!")
async def coro_b():
print("I am coro_b(). I sure hope no one hogs the event loop...")
async def main():
task_b = asyncio.create_task(coro_b())
num_repeats = 3
for _ in range(num_repeats):
await coro_a()
await task_b
asyncio.run(main())
The first statement in the coroutine main()
creates task_b
and schedules
it for execution via the event loop.
Then, coro_a()
is repeatedly awaited. Control never cedes to the
event loop, which is why we see the output of all three coro_a()
invocations before coro_b()
’s output:
I am coro_a(). Hi!
I am coro_a(). Hi!
I am coro_a(). Hi!
I am coro_b(). I sure hope no one hogs the event loop...
If we change await coro_a()
to await asyncio.create_task(coro_a())
, the
behavior changes.
The coroutine main()
cedes control to the event loop with that statement.
The event loop then proceeds through its backlog of work, calling task_b
and then the task which wraps coro_a()
before resuming the coroutine
main()
.
I am coro_b(). I sure hope no one hogs the event loop...
I am coro_a(). Hi!
I am coro_a(). Hi!
I am coro_a(). Hi!
This behavior of await coroutine
can trip a lot of people up!
That example highlights how using only await coroutine
could
unintentionally hog control from other tasks and effectively stall the event
loop.
asyncio.run()
can help you detect such occurrences via the
debug=True
flag, which enables
debug mode.
Among other things, it will log any coroutines that monopolize execution for
100ms or longer.
The design intentionally trades off some conceptual clarity around usage of
await
for improved performance.
Each time a task is awaited, control needs to be passed all the way up the
call stack to the event loop.
That might sound minor, but in a large program with many await
statements and a deep
call stack, that overhead can add up to a meaningful performance drag.
A conceptual overview part 2: the nuts and bolts¶
Part 2 goes into detail on the mechanisms asyncio
uses to manage
control flow.
This is where the magic happens.
You’ll come away from this section knowing what await
does behind the scenes
and how to make your own asynchronous operators.
The inner workings of coroutines¶
asyncio
leverages four components to pass around control.
coroutine.send(arg)
is the method used to start or
resume a coroutine.
If the coroutine was paused and is now being resumed, the argument arg
will be sent in as the return value of the yield
statement which originally
paused it.
If the coroutine is being used for the first time (as opposed to being resumed),
arg
must be None
.
1class Rock:
2 def __await__(self):
3 value_sent_in = yield 7
4 print(f"Rock.__await__ resuming with value: {value_sent_in}.")
5 return value_sent_in
6
7async def main():
8 print("Beginning coroutine main().")
9 rock = Rock()
10 print("Awaiting rock...")
11 value_from_rock = await rock
12 print(f"Coroutine received value: {value_from_rock} from rock.")
13 return 23
14
15coroutine = main()
16intermediate_result = coroutine.send(None)
17print(f"Coroutine paused and returned intermediate value: {intermediate_result}.")
18
19print(f"Resuming coroutine and sending in value: 42.")
20try:
21 coroutine.send(42)
22except StopIteration as e:
23 returned_value = e.value
24print(f"Coroutine main() finished and provided value: {returned_value}.")
yield, as usual, pauses execution and returns control
to the caller.
In the example above, the yield
, on line 3, is called by
... = await rock
on line 11.
More broadly speaking, await
calls the __await__()
method of
the given object.
await
also does one more very special thing: it propagates (or “passes
along”) any yield
s it receives up the call chain.
In this case, that’s back to ... = coroutine.send(None)
on line 16.
The coroutine is resumed via the coroutine.send(42)
call on line 21.
The coroutine picks back up from where it yield
ed (or paused) on line 3
and executes the remaining statements in its body.
When a coroutine finishes, it raises a StopIteration
exception with the
return value attached in the value
attribute.
That snippet produces this output:
Beginning coroutine main().
Awaiting rock...
Coroutine paused and returned intermediate value: 7.
Resuming coroutine and sending in value: 42.
Rock.__await__ resuming with value: 42.
Coroutine received value: 42 from rock.
Coroutine main() finished and provided value: 23.
It’s worth pausing for a moment here and making sure you followed the various ways that control flow and values were passed. A lot of important ideas were covered and it’s worth ensuring your understanding is firm.
The only way to yield (or effectively cede control) from a coroutine is to
await
an object that yield
s in its __await__
method.
That might sound odd to you. You might be thinking:
1. What about a
yield
directly within the coroutine function? The coroutine function becomes an async generator function, a different beast entirely.2. What about a yield from within the coroutine function to a (plain) generator? That causes the error:
SyntaxError: yield from not allowed in a coroutine.
This was intentionally designed for the sake of simplicity – mandating only one way of using coroutines. Initiallyyield
was barred as well, but was re-accepted to allow for async generators. Despite that,yield from
andawait
effectively do the same thing.
Futures¶
A future is an object meant to represent a computation’s status and result. The term is a nod to the idea of something still to come or not yet happened, and the object is a way to keep an eye on that something.
A future has a few important attributes. One is its state, which can be either “pending”, “cancelled”, or “done”. Another is its result, which is set when the state transitions to done. Unlike a coroutine, a future does not represent the actual computation to be done; instead, it represents the status and result of that computation, kind of like a status light (red, yellow, or green) or indicator.
asyncio.Task
subclasses asyncio.Future
in order to gain
these various capabilities.
The prior section said tasks store a list of callbacks, which wasn’t entirely
accurate.
It’s actually the Future
class that implements this logic, which Task
inherits.
Futures may also be used directly (not via tasks). Tasks mark themselves as done when their coroutine is complete. Futures are much more versatile and will be marked as done when you say so. In this way, they’re the flexible interface for you to make your own conditions for waiting and resuming.
A homemade asyncio.sleep¶
We’ll go through an example of how you could leverage a future to create your
own variant of asynchronous sleep (async_sleep
) which mimics
asyncio.sleep()
.
This snippet registers a few tasks with the event loop and then awaits the task
created by asyncio.create_task
, which wraps the async_sleep(3)
coroutine.
We want that task to finish only after three seconds have elapsed, but without
preventing other tasks from running.
async def other_work():
print("I like work. Work work.")
async def main():
# Add a few other tasks to the event loop, so there's something
# to do while asynchronously sleeping.
work_tasks = [
asyncio.create_task(other_work()),
asyncio.create_task(other_work()),
asyncio.create_task(other_work())
]
print(
"Beginning asynchronous sleep at time: "
f"{datetime.datetime.now().strftime("%H:%M:%S")}."
)
await asyncio.create_task(async_sleep(3))
print(
"Done asynchronous sleep at time: "
f"{datetime.datetime.now().strftime("%H:%M:%S")}."
)
# asyncio.gather effectively awaits each task in the collection.
await asyncio.gather(*work_tasks)
Below, we use a future to enable custom control over when that task will be
marked as done.
If future.set_result()
(the method
responsible for marking that future as done) is never called, then this task
will never finish.
We’ve also enlisted the help of another task, which we’ll see in a moment, that
will monitor how much time has elapsed and, accordingly, call
future.set_result()
.
async def async_sleep(seconds: float):
future = asyncio.Future()
time_to_wake = time.time() + seconds
# Add the watcher-task to the event loop.
watcher_task = asyncio.create_task(_sleep_watcher(future, time_to_wake))
# Block until the future is marked as done.
await future
Below, we use a rather bare YieldToEventLoop()
object to yield
from its __await__
method, ceding control to the event loop.
This is effectively the same as calling asyncio.sleep(0)
, but this approach
offers more clarity, not to mention it’s somewhat cheating to use
asyncio.sleep
when showcasing how to implement it!
As usual, the event loop cycles through its tasks, giving them control
and receiving control back when they pause or finish.
The watcher_task
, which runs the coroutine _sleep_watcher(...)
, will
be invoked once per full cycle of the event loop.
On each resumption, it’ll check the time and if not enough has elapsed, then
it’ll pause once again and hand control back to the event loop.
Once enough time has elapsed, _sleep_watcher(...)
marks the future as done and completes by exiting its
infinite while
loop.
Given this helper task is only invoked once per cycle of the event loop,
you’d be correct to note that this asynchronous sleep will sleep at least
three seconds, rather than exactly three seconds.
Note this is also true of asyncio.sleep
.
class YieldToEventLoop:
def __await__(self):
yield
async def _sleep_watcher(future, time_to_wake):
while True:
if time.time() >= time_to_wake:
# This marks the future as done.
future.set_result(None)
break
else:
await YieldToEventLoop()
Here is the full program’s output:
$ python custom-async-sleep.py
Beginning asynchronous sleep at time: 14:52:22.
I like work. Work work.
I like work. Work work.
I like work. Work work.
Done asynchronous sleep at time: 14:52:25.
You might feel this implementation of asynchronous sleep was unnecessarily convoluted. And, well, it was. The example was meant to showcase the versatility of futures with a simple example that could be mimicked for more complex needs. For reference, you could implement it without futures, like so:
async def simpler_async_sleep(seconds):
time_to_wake = time.time() + seconds
while True:
if time.time() >= time_to_wake:
return
else:
await YieldToEventLoop()
But that’s all for now. Hopefully you’re ready to more confidently dive into
some async programming or check out advanced topics in the
rest of the documentation
.


asynchat
— Asynchronous socket command/response handler¶
Deprecated since version 3.6, removed in version 3.12.
This module is no longer part of the Python standard library. It was removed in Python 3.12 after being deprecated in Python 3.6. The removal was decided in PEP 594.
Applications should use the asyncio
module instead.
The last version of Python that provided the asynchat
module was
Python 3.11.


Descriptor Objects

Descriptor Objects¶
“Descriptors” are objects that describe some attribute of an object. They are found in the dictionary of type objects.
-
PyTypeObject PyProperty_Type¶
- Part of the Stable ABI.
The type object for the built-in descriptor types.
-
PyObject *PyDescr_NewGetSet(PyTypeObject *type, struct PyGetSetDef *getset)¶
- Return value: New reference. Part of the Stable ABI.
-
PyObject *PyDescr_NewMember(PyTypeObject *type, struct PyMemberDef *meth)¶
- Return value: New reference. Part of the Stable ABI.
-
PyTypeObject PyMemberDescr_Type¶
- Part of the Stable ABI.
The type object for member descriptor objects created from
PyMemberDef
structures. These descriptors expose fields of a C struct as attributes on a type, and correspond totypes.MemberDescriptorType
objects in Python.
-
PyTypeObject PyGetSetDescr_Type¶
- Part of the Stable ABI.
The type object for get/set descriptor objects created from
PyGetSetDef
structures. These descriptors implement attributes whose value is computed by C getter and setter functions, and are used for many built-in type attributes.
-
PyObject *PyDescr_NewMethod(PyTypeObject *type, struct PyMethodDef *meth)¶
- Return value: New reference. Part of the Stable ABI.
-
PyTypeObject PyMethodDescr_Type¶
- Part of the Stable ABI.
The type object for method descriptor objects created from
PyMethodDef
structures. These descriptors expose C functions as methods on a type, and correspond totypes.MemberDescriptorType
objects in Python.
-
PyObject *PyDescr_NewWrapper(PyTypeObject *type, struct wrapperbase *wrapper, void *wrapped)¶
- Return value: New reference.
-
PyTypeObject PyWrapperDescr_Type¶
- Part of the Stable ABI.
The type object for wrapper descriptor objects created by
PyDescr_NewWrapper()
andPyWrapper_New()
. Wrapper descriptors are used internally to expose special methods implemented via wrapper structures, and appear in Python astypes.WrapperDescriptorType
objects.
-
PyObject *PyDescr_NewClassMethod(PyTypeObject *type, PyMethodDef *method)¶
- Return value: New reference. Part of the Stable ABI.
-
int PyDescr_IsData(PyObject *descr)¶
Return non-zero if the descriptor object descr describes a data attribute, or
0
if it describes a method. descr must be a descriptor object; there is no error checking.
-
PyObject *PyWrapper_New(PyObject*, PyObject*)¶
- Return value: New reference. Part of the Stable ABI.
Built-in descriptors¶
-
PyTypeObject PySuper_Type¶
- Part of the Stable ABI.
The type object for super objects. This is the same object as
super
in the Python layer.
-
PyTypeObject PyClassMethod_Type¶
The type of class method objects. This is the same object as
classmethod
in the Python layer.
-
PyTypeObject PyClassMethodDescr_Type¶
- Part of the Stable ABI.
The type object for C-level class method descriptor objects. This is the type of the descriptors created for
classmethod()
defined in C extension types, and is the same object asclassmethod
in Python.
-
PyObject *PyClassMethod_New(PyObject *callable)¶
Create a new
classmethod
object wrapping callable. callable must be a callable object and must not beNULL
.On success, this function returns a strong reference to a new class method descriptor. On failure, this function returns
NULL
with an exception set.
-
PyTypeObject PyStaticMethod_Type¶
The type of static method objects. This is the same object as
staticmethod
in the Python layer.
-
PyObject *PyStaticMethod_New(PyObject *callable)¶
Create a new
staticmethod
object wrapping callable. callable must be a callable object and must not beNULL
.On success, this function returns a strong reference to a new static method descriptor. On failure, this function returns
NULL
with an exception set.