Futures

Futures¶
Source code: Lib/asyncio/futures.py, Lib/asyncio/base_futures.py
Future objects are used to bridge low-level callback-based code with high-level async/await code.
Future Functions¶
- asyncio.isfuture(obj)¶
Return
True
if obj is either of:an instance of
asyncio.Future
,an instance of
asyncio.Task
,a Future-like object with a
_asyncio_future_blocking
attribute.
Added in version 3.5.
- asyncio.ensure_future(obj, *, loop=None)¶
Return:
obj argument as is, if obj is a
Future
, aTask
, or a Future-like object (isfuture()
is used for the test.)a
Task
object wrapping obj, if obj is a coroutine (iscoroutine()
is used for the test); in this case the coroutine will be scheduled byensure_future()
.a
Task
object that would await on obj, if obj is an awaitable (inspect.isawaitable()
is used for the test.)
If obj is neither of the above a
TypeError
is raised.Important
Save a reference to the result of this function, to avoid a task disappearing mid-execution.
See also the
create_task()
function which is the preferred way for creating new tasks or useasyncio.TaskGroup
which keeps reference to the task internally.Changed in version 3.5.1: The function accepts any awaitable object.
Deprecated since version 3.10: Deprecation warning is emitted if obj is not a Future-like object and loop is not specified and there is no running event loop.
- asyncio.wrap_future(future, *, loop=None)¶
Wrap a
concurrent.futures.Future
object in aasyncio.Future
object.Deprecated since version 3.10: Deprecation warning is emitted if future is not a Future-like object and loop is not specified and there is no running event loop.
Future Object¶
- class asyncio.Future(*, loop=None)¶
A Future represents an eventual result of an asynchronous operation. Not thread-safe.
Future is an awaitable object. Coroutines can await on Future objects until they either have a result or an exception set, or until they are cancelled. A Future can be awaited multiple times and the result is same.
Typically Futures are used to enable low-level callback-based code (e.g. in protocols implemented using asyncio transports) to interoperate with high-level async/await code.
The rule of thumb is to never expose Future objects in user-facing APIs, and the recommended way to create a Future object is to call
loop.create_future()
. This way alternative event loop implementations can inject their own optimized implementations of a Future object.Changed in version 3.7: Added support for the
contextvars
module.Deprecated since version 3.10: Deprecation warning is emitted if loop is not specified and there is no running event loop.
- result()¶
Return the result of the Future.
If the Future is done and has a result set by the
set_result()
method, the result value is returned.If the Future is done and has an exception set by the
set_exception()
method, this method raises the exception.If the Future has been cancelled, this method raises a
CancelledError
exception.If the Future’s result isn’t yet available, this method raises an
InvalidStateError
exception.
- set_result(result)¶
Mark the Future as done and set its result.
Raises an
InvalidStateError
error if the Future is already done.
- set_exception(exception)¶
Mark the Future as done and set an exception.
Raises an
InvalidStateError
error if the Future is already done.
- done()¶
Return
True
if the Future is done.A Future is done if it was cancelled or if it has a result or an exception set with
set_result()
orset_exception()
calls.
- cancelled()¶
Return
True
if the Future was cancelled.The method is usually used to check if a Future is not cancelled before setting a result or an exception for it:
if not fut.cancelled(): fut.set_result(42)
- add_done_callback(callback, *, context=None)¶
Add a callback to be run when the Future is done.
The callback is called with the Future object as its only argument.
If the Future is already done when this method is called, the callback is scheduled with
loop.call_soon()
.An optional keyword-only context argument allows specifying a custom
contextvars.Context
for the callback to run in. The current context is used when no context is provided.functools.partial()
can be used to pass parameters to the callback, e.g.:# Call 'print("Future:", fut)' when "fut" is done. fut.add_done_callback( functools.partial(print, "Future:"))
Changed in version 3.7: The context keyword-only parameter was added. See PEP 567 for more details.
- remove_done_callback(callback)¶
Remove callback from the callbacks list.
Returns the number of callbacks removed, which is typically 1, unless a callback was added more than once.
- cancel(msg=None)¶
Cancel the Future and schedule callbacks.
If the Future is already done or cancelled, return
False
. Otherwise, change the Future’s state to cancelled, schedule the callbacks, and returnTrue
.Changed in version 3.9: Added the msg parameter.
- exception()¶
Return the exception that was set on this Future.
The exception (or
None
if no exception was set) is returned only if the Future is done.If the Future has been cancelled, this method raises a
CancelledError
exception.If the Future isn’t done yet, this method raises an
InvalidStateError
exception.
- get_loop()¶
Return the event loop the Future object is bound to.
Added in version 3.7.
This example creates a Future object, creates and schedules an asynchronous Task to set result for the Future, and waits until the Future has a result:
async def set_after(fut, delay, value):
# Sleep for *delay* seconds.
await asyncio.sleep(delay)
# Set *value* as a result of *fut* Future.
fut.set_result(value)
async def main():
# Get the current event loop.
loop = asyncio.get_running_loop()
# Create a new Future object.
fut = loop.create_future()
# Run "set_after()" coroutine in a parallel Task.
# We are using the low-level "loop.create_task()" API here because
# we already have a reference to the event loop at hand.
# Otherwise we could have just used "asyncio.create_task()".
loop.create_task(
set_after(fut, 1, '... world'))
print('hello ...')
# Wait until *fut* has a result (1 second) and print it.
print(await fut)
asyncio.run(main())
Important
The Future object was designed to mimic
concurrent.futures.Future
. Key differences include:
unlike asyncio Futures,
concurrent.futures.Future
instances cannot be awaited.asyncio.Future.result()
andasyncio.Future.exception()
do not accept the timeout argument.asyncio.Future.result()
andasyncio.Future.exception()
raise anInvalidStateError
exception when the Future is not done.Callbacks registered with
asyncio.Future.add_done_callback()
are not called immediately. They are scheduled withloop.call_soon()
instead.asyncio Future is not compatible with the
concurrent.futures.wait()
andconcurrent.futures.as_completed()
functions.asyncio.Future.cancel()
accepts an optionalmsg
argument, butconcurrent.futures.Future.cancel()
does not.


Object Implementation Support

Object Implementation Support¶
This chapter describes the functions, types, and macros used when defining new object types.
- Allocating Objects on the Heap
- Object Life Cycle
- Common Object Structures
- Type Object Structures
- Supporting Cyclic Garbage Collection


Configure Python

3. Configure Python¶
3.1. Build Requirements¶
To build CPython, you will need:
A C11 compiler. Optional C11 features are not required.
On Windows, Microsoft Visual Studio 2017 or later is required.
Support for IEEE 754 floating-point numbers and floating-point Not-a-Number (NaN).
Support for threads.
Changed in version 3.5: On Windows, Visual Studio 2015 or later is now required.
Changed in version 3.6: Selected C99 features, like <stdint.h>
and static inline
functions,
are now required.
Changed in version 3.7: Thread support is now required.
Changed in version 3.11: C11 compiler, IEEE 754 and NaN support are now required. On Windows, Visual Studio 2017 or later is required.
See also PEP 7 “Style Guide for C Code” and PEP 11 “CPython platform support”.
3.1.1. Requirements for optional modules¶
Some optional modules of the standard library require third-party libraries installed for development (for example, header files must be available).
Missing requirements are reported in the configure
output.
Modules that are missing due to missing dependencies are listed near the end
of the make
output,
sometimes using an internal name, for example, _ctypes
for ctypes
module.
If you distribute a CPython interpreter without optional modules, it’s best practice to advise users, who generally expect that standard library modules are available.
Dependencies to build optional modules are:
Dependency |
Minimum version |
Python module |
|---|---|---|
3.3.0 recommended |
||
2.5.0 |
||
|
||
3.0.18 recommended
(1.1.1 minimum)
|
||
3.15.2 |
||
8.5.12 |
||
1.2.2.1 |
||
1.4.5 |
Note that the table does not include all optional modules; in particular,
platform-specific modules like winreg
are not listed here.
See also
The devguide includes a full list of dependencies required to build all modules and instructions on how to install them on common platforms.
--with-system-expat
allows building with an external libexpat library.
Changed in version 3.1: Tcl/Tk version 8.3.1 is now required for tkinter
.
Changed in version 3.5: Tcl/Tk version 8.4 is now required for tkinter
.
Changed in version 3.10: OpenSSL 1.1.1 is now required for hashlib
and ssl
.
SQLite 3.7.15 is now required for sqlite3
.
Changed in version 3.11: Tcl/Tk version 8.5.12 is now required for tkinter
.
Changed in version 3.13: SQLite 3.15.2 is now required for sqlite3
.
3.2. Generated files¶
To reduce build dependencies, Python source code contains multiple generated files. Commands to regenerate all generated files:
make regen-all
make regen-stdlib-module-names
make regen-limited-abi
make regen-configure
The Makefile.pre.in
file documents generated files, their inputs, and tools used
to regenerate them. Search for regen-*
make targets.
3.2.1. configure script¶
The make regen-configure
command regenerates the aclocal.m4
file and
the configure
script using the Tools/build/regen-configure.sh
shell
script which uses an Ubuntu container to get the same tools versions and have a
reproducible output.
The container is optional, the following command can be run locally:
autoreconf -ivf -Werror
The generated files can change depending on the exact versions of the
tools used.
The container that CPython uses has
Autoconf 2.72,
aclocal
from Automake 1.16.5,
and pkg-config 1.8.1.
Changed in version 3.13: Autoconf 2.71 and aclocal 1.16.5 and are now used to regenerate
configure
.
Changed in version 3.14: Autoconf 2.72 is now used to regenerate configure
.
3.3. Configure Options¶
List all configure
script options using:
./configure --help
See also the Misc/SpecialBuilds.txt
in the Python source distribution.
3.3.1. General Options¶
- --enable-loadable-sqlite-extensions¶
Support loadable extensions in the
_sqlite
extension module (default is no) of thesqlite3
module.See the
sqlite3.Connection.enable_load_extension()
method of thesqlite3
module.Added in version 3.6.
- --enable-big-digits=[15|30]¶
Define the size in bits of Python
int
digits: 15 or 30 bits.By default, the digit size is 30.
Define the
PYLONG_BITS_IN_DIGIT
to15
or30
.
- --with-suffix=SUFFIX¶
Set the Python executable suffix to SUFFIX.
The default suffix is
.exe
on Windows and macOS (python.exe
executable),.js
on Emscripten node,.html
on Emscripten browser,.wasm
on WASI, and an empty string on other platforms (python
executable).Changed in version 3.11: The default suffix on WASM platform is one of
.js
,.html
or.wasm
.
- --with-tzpath=<list of absolute paths separated by pathsep>¶
Select the default time zone search path for
zoneinfo.TZPATH
. See the Compile-time configuration of thezoneinfo
module.Default:
/usr/share/zoneinfo:/usr/lib/zoneinfo:/usr/share/lib/zoneinfo:/etc/zoneinfo
.See
os.pathsep
path separator.Added in version 3.9.
- --without-decimal-contextvar¶
Build the
_decimal
extension module using a thread-local context rather than a coroutine-local context (default), see thedecimal
module.See
decimal.HAVE_CONTEXTVAR
and thecontextvars
module.Added in version 3.9.
- --with-dbmliborder=<list of backend names>¶
Override order to check db backends for the
dbm
moduleA valid value is a colon (
:
) separated string with the backend names:ndbm
;gdbm
;bdb
.
- --without-c-locale-coercion¶
Disable C locale coercion to a UTF-8 based locale (enabled by default).
Don’t define the
PY_COERCE_C_LOCALE
macro.See
PYTHONCOERCECLOCALE
and the PEP 538.
- --with-platlibdir=DIRNAME¶
Python library directory name (default is
lib
).Fedora and SuSE use
lib64
on 64-bit platforms.See
sys.platlibdir
.Added in version 3.9.
- --with-wheel-pkg-dir=PATH¶
Directory of wheel packages used by the
ensurepip
module (none by default).Some Linux distribution packaging policies recommend against bundling dependencies. For example, Fedora installs wheel packages in the
/usr/share/python-wheels/
directory and don’t install theensurepip._bundled
package.Added in version 3.10.
- --with-pkg-config=[check|yes|no]¶
Whether configure should use pkg-config to detect build dependencies.
check
(default): pkg-config is optionalyes
: pkg-config is mandatoryno
: configure does not use pkg-config even when present
Added in version 3.11.
- --enable-pystats¶
Turn on internal Python performance statistics gathering.
By default, statistics gathering is off. Use
python3 -X pystats
command or setPYTHONSTATS=1
environment variable to turn on statistics gathering at Python startup.At Python exit, dump statistics if statistics gathering was on and not cleared.
Effects:
Add
-X pystats
command line option.Add
PYTHONSTATS
environment variable.Define the
Py_STATS
macro.Add functions to the
sys
module:sys._stats_on()
: Turns on statistics gathering.sys._stats_off()
: Turns off statistics gathering.sys._stats_clear()
: Clears the statistics.sys._stats_dump()
: Dump statistics to file, and clears the statistics.
The statistics will be dumped to a arbitrary (probably unique) file in
/tmp/py_stats/
(Unix) orC:\temp\py_stats\
(Windows). If that directory does not exist, results will be printed on stderr.Use
Tools/scripts/summarize_stats.py
to read the stats.Statistics:
Opcode:
Specialization: success, failure, hit, deferred, miss, deopt, failures;
Execution count;
Pair count.
Call:
Inlined Python calls;
PyEval calls;
Frames pushed;
Frame object created;
Eval calls: vector, generator, legacy, function VECTORCALL, build class, slot, function “ex”, API, method.
Object:
incref and decref;
interpreter incref and decref;
allocations: all, 512 bytes, 4 kiB, big;
free;
to/from free lists;
dictionary materialized/dematerialized;
type cache;
optimization attempts;
optimization traces created/executed;
uops executed.
Garbage collector:
Garbage collections;
Objects visited;
Objects collected.
Added in version 3.11.
- --disable-gil¶
Enables support for running Python without the global interpreter lock (GIL): free-threaded build.
Defines the
Py_GIL_DISABLED
macro and adds"t"
tosys.abiflags
.See Free-threaded CPython for more detail.
Added in version 3.13.
- --enable-experimental-jit=[no|yes|yes-off|interpreter]¶
Indicate how to integrate the experimental just-in-time compiler.
no
: Don’t build the JIT.yes
: Enable the JIT. To disable it at runtime, set the environment variablePYTHON_JIT=0
.yes-off
: Build the JIT, but disable it by default. To enable it at runtime, set the environment variablePYTHON_JIT=1
.interpreter
: Enable the “JIT interpreter” (only useful for those debugging the JIT itself). To disable it at runtime, set the environment variablePYTHON_JIT=0
.
--enable-experimental-jit=no
is the default behavior if the option is not provided, and--enable-experimental-jit
is shorthand for--enable-experimental-jit=yes
. SeeTools/jit/README.md
for more information, including how to install the necessary build-time dependencies.Note
When building CPython with JIT enabled, ensure that your system has Python 3.11 or later installed.
Added in version 3.13.
- PKG_CONFIG¶
Path to
pkg-config
utility.
- PKG_CONFIG_LIBDIR¶
- PKG_CONFIG_PATH¶
pkg-config
options.
3.3.2. C compiler options¶
- CC¶
C compiler command.
- CFLAGS¶
C compiler flags.
- CPP¶
C preprocessor command.
- CPPFLAGS¶
C preprocessor flags, e.g.
-Iinclude_dir
.
3.3.3. Linker options¶
- LDFLAGS¶
Linker flags, e.g.
-Llibrary_directory
.
- LIBS¶
Libraries to pass to the linker, e.g.
-llibrary
.
- MACHDEP¶
Name for machine-dependent library files.
3.3.4. Options for third-party dependencies¶
Added in version 3.11.
- BZIP2_CFLAGS¶
- BZIP2_LIBS¶
C compiler and linker flags to link Python to
libbz2
, used bybz2
module, overridingpkg-config
.
- CURSES_CFLAGS¶
- CURSES_LIBS¶
C compiler and linker flags for
libncurses
orlibncursesw
, used bycurses
module, overridingpkg-config
.
- GDBM_CFLAGS¶
- GDBM_LIBS¶
C compiler and linker flags for
gdbm
.
- LIBEDIT_CFLAGS¶
- LIBEDIT_LIBS¶
C compiler and linker flags for
libedit
, used byreadline
module, overridingpkg-config
.
- LIBFFI_CFLAGS¶
- LIBMPDEC_CFLAGS¶
- LIBMPDEC_LIBS¶
C compiler and linker flags for
libmpdec
, used bydecimal
module, overridingpkg-config
.Note
These environment variables have no effect unless
--with-system-libmpdec
is specified.
- LIBLZMA_CFLAGS¶
- LIBREADLINE_CFLAGS¶
- LIBREADLINE_LIBS¶
C compiler and linker flags for
libreadline
, used byreadline
module, overridingpkg-config
.
- LIBSQLITE3_CFLAGS¶
- LIBSQLITE3_LIBS¶
C compiler and linker flags for
libsqlite3
, used bysqlite3
module, overridingpkg-config
.
- LIBUUID_CFLAGS¶
- LIBZSTD_CFLAGS¶
- LIBZSTD_LIBS¶
C compiler and linker flags for
libzstd
, used bycompression.zstd
module, overridingpkg-config
.Added in version 3.14.
- PANEL_CFLAGS¶
- PANEL_LIBS¶
C compiler and linker flags for PANEL, overriding
pkg-config
.C compiler and linker flags for
libpanel
orlibpanelw
, used bycurses.panel
module, overridingpkg-config
.
- TCLTK_CFLAGS¶
- TCLTK_LIBS¶
C compiler and linker flags for TCLTK, overriding
pkg-config
.
- ZLIB_CFLAGS¶
3.3.5. WebAssembly Options¶
- --enable-wasm-dynamic-linking¶
Turn on dynamic linking support for WASM.
Dynamic linking enables
dlopen
. File size of the executable increases due to limited dead code elimination and additional features.Added in version 3.11.
- --enable-wasm-pthreads¶
Turn on pthreads support for WASM.
Added in version 3.11.
3.3.6. Install Options¶
- --prefix=PREFIX¶
Install architecture-independent files in PREFIX. On Unix, it defaults to
/usr/local
.This value can be retrieved at runtime using
sys.prefix
.As an example, one can use
--prefix="$HOME/.local/"
to install a Python in its home directory.
- --exec-prefix=EPREFIX¶
Install architecture-dependent files in EPREFIX, defaults to
--prefix
.This value can be retrieved at runtime using
sys.exec_prefix
.
3.3.7. Performance options¶
Configuring Python using --enable-optimizations --with-lto
(PGO + LTO) is
recommended for best performance. The experimental --enable-bolt
flag can
also be used to improve performance.
- --enable-optimizations¶
Enable Profile Guided Optimization (PGO) using
PROFILE_TASK
(disabled by default).The C compiler Clang requires
llvm-profdata
program for PGO. On macOS, GCC also requires it: GCC is just an alias to Clang on macOS.Disable also semantic interposition in libpython if
--enable-shared
and GCC is used: add-fno-semantic-interposition
to the compiler and linker flags.Note
During the build, you may encounter compiler warnings about profile data not being available for some source files. These warnings are harmless, as only a subset of the code is exercised during profile data acquisition. To disable these warnings on Clang, manually suppress them by adding
-Wno-profile-instr-unprofiled
toCFLAGS
.Added in version 3.6.
Changed in version 3.10: Use
-fno-semantic-interposition
on GCC.
- PROFILE_TASK¶
Environment variable used in the Makefile: Python command line arguments for the PGO generation task.
Default:
-m test --pgo --timeout=$(TESTTIMEOUT)
.Added in version 3.8.
Changed in version 3.13: Task failure is no longer ignored silently.
- --with-lto=[full|thin|no|yes]¶
Enable Link Time Optimization (LTO) in any build (disabled by default).
The C compiler Clang requires
llvm-ar
for LTO (ar
on macOS), as well as an LTO-aware linker (ld.gold
orlld
).Added in version 3.6.
Added in version 3.11: To use ThinLTO feature, use
--with-lto=thin
on Clang.Changed in version 3.12: Use ThinLTO as the default optimization policy on Clang if the compiler accepts the flag.
- --enable-bolt¶
Enable usage of the BOLT post-link binary optimizer (disabled by default).
BOLT is part of the LLVM project but is not always included in their binary distributions. This flag requires that
llvm-bolt
andmerge-fdata
are available.BOLT is still a fairly new project so this flag should be considered experimental for now. Because this tool operates on machine code its success is dependent on a combination of the build environment + the other optimization configure args + the CPU architecture, and not all combinations are supported. BOLT versions before LLVM 16 are known to crash BOLT under some scenarios. Use of LLVM 16 or newer for BOLT optimization is strongly encouraged.
The
BOLT_INSTRUMENT_FLAGS
andBOLT_APPLY_FLAGS
configure variables can be defined to override the default set of arguments for llvm-bolt to instrument and apply BOLT data to binaries, respectively.Added in version 3.12.
- BOLT_APPLY_FLAGS¶
Arguments to
llvm-bolt
when creating a BOLT optimized binary.Added in version 3.12.
- BOLT_INSTRUMENT_FLAGS¶
Arguments to
llvm-bolt
when instrumenting binaries.Added in version 3.12.
- --with-computed-gotos¶
Enable computed gotos in evaluation loop (enabled by default on supported compilers).
- --with-tail-call-interp¶
Enable interpreters using tail calls in CPython. If enabled, enabling PGO (
--enable-optimizations
) is highly recommended. This option specifically requires a C compiler with proper tail call support, and the preserve_none calling convention. For example, Clang 19 and newer supports this feature.Added in version 3.14.
- --without-mimalloc¶
Disable the fast mimalloc allocator (enabled by default).
See also
PYTHONMALLOC
environment variable.
- --without-pymalloc¶
Disable the specialized Python memory allocator pymalloc (enabled by default).
See also
PYTHONMALLOC
environment variable.
- --without-doc-strings¶
Disable static documentation strings to reduce the memory footprint (enabled by default). Documentation strings defined in Python are not affected.
Don’t define the
WITH_DOC_STRINGS
macro.See the
PyDoc_STRVAR()
macro.
- --enable-profiling¶
Enable C-level code profiling with
gprof
(disabled by default).
- --with-strict-overflow¶
Add
-fstrict-overflow
to the C compiler flags (by default we add-fno-strict-overflow
instead).
- --without-remote-debug¶
Deactivate remote debugging support described in PEP 768 (enabled by default). When this flag is provided the code that allows the interpreter to schedule the execution of a Python file in a separate process as described in PEP 768 is not compiled. This includes both the functionality to schedule code to be executed and the functionality to receive code to be executed.
-
Py_REMOTE_DEBUG¶
This macro is defined by default, unless Python is configured with
--without-remote-debug
.Note that even if the macro is defined, remote debugging may not be available (for example, on an incompatible platform).
Added in version 3.14.
-
Py_REMOTE_DEBUG¶
3.3.8. Python Debug Build¶
A debug build is Python built with the --with-pydebug
configure
option.
Effects of a debug build:
Display all warnings by default: the list of default warning filters is empty in the
warnings
module.Add
d
tosys.abiflags
.Add
sys.gettotalrefcount()
function.Add
-X showrefcount
command line option.Add
-d
command line option andPYTHONDEBUG
environment variable to debug the parser.Add support for the
__lltrace__
variable: enable low-level tracing in the bytecode evaluation loop if the variable is defined.Install debug hooks on memory allocators to detect buffer overflow and other memory errors.
Define
Py_DEBUG
andPy_REF_DEBUG
macros.Add runtime checks: code surrounded by
#ifdef Py_DEBUG
and#endif
. Enableassert(...)
and_PyObject_ASSERT(...)
assertions: don’t set theNDEBUG
macro (see also the--with-assertions
configure option). Main runtime checks:Add sanity checks on the function arguments.
Unicode and int objects are created with their memory filled with a pattern to detect usage of uninitialized objects.
Ensure that functions which can clear or replace the current exception are not called with an exception raised.
Check that deallocator functions don’t change the current exception.
The garbage collector (
gc.collect()
function) runs some basic checks on objects consistency.The
Py_SAFE_DOWNCAST()
macro checks for integer underflow and overflow when downcasting from wide types to narrow types.
See also the Python Development Mode and the
--with-trace-refs
configure option.
Changed in version 3.8: Release builds and debug builds are now ABI compatible: defining the
Py_DEBUG
macro no longer implies the Py_TRACE_REFS
macro (see the
--with-trace-refs
option).
3.3.9. Debug options¶
- --with-pydebug¶
Build Python in debug mode: define the
Py_DEBUG
macro (disabled by default).
- --with-trace-refs¶
Enable tracing references for debugging purpose (disabled by default).
Effects:
Define the
Py_TRACE_REFS
macro.Add
sys.getobjects()
function.Add
PYTHONDUMPREFS
environment variable.
The
PYTHONDUMPREFS
environment variable can be used to dump objects and reference counts still alive at Python exit.Statically allocated objects are not traced.
Added in version 3.8.
Changed in version 3.13: This build is now ABI compatible with release build and debug build.
- --with-assertions¶
Build with C assertions enabled (default is no):
assert(...);
and_PyObject_ASSERT(...);
.If set, the
NDEBUG
macro is not defined in theOPT
compiler variable.See also the
--with-pydebug
option (debug build) which also enables assertions.Added in version 3.6.
- --with-valgrind¶
Enable Valgrind support (default is no).
- --with-dtrace¶
Enable DTrace support (default is no).
See Instrumenting CPython with DTrace and SystemTap.
Added in version 3.6.
- --with-address-sanitizer¶
Enable AddressSanitizer memory error detector,
asan
(default is no). To improve ASan detection capabilities you may also want to combine this with--without-pymalloc
to disable the specialized small-object allocator whose allocations are not tracked by ASan.Added in version 3.6.
- --with-memory-sanitizer¶
Enable MemorySanitizer allocation error detector,
msan
(default is no).Added in version 3.6.
- --with-undefined-behavior-sanitizer¶
Enable UndefinedBehaviorSanitizer undefined behaviour detector,
ubsan
(default is no).Added in version 3.6.
- --with-thread-sanitizer¶
Enable ThreadSanitizer data race detector,
tsan
(default is no).Added in version 3.13.
3.3.10. Linker options¶
Enable building a shared Python library:
libpython
(default is no).
- --without-static-libpython¶
Do not build
libpythonMAJOR.MINOR.a
and do not installpython.o
(built and enabled by default).Added in version 3.10.
3.3.11. Libraries options¶
- --with-libs='lib1 ...'¶
Link against additional libraries (default is no).
- --with-system-expat¶
Build the
pyexpat
module using an installedexpat
library (default is no).
- --with-system-libmpdec¶
Build the
_decimal
extension module using an installedmpdecimal
library, see thedecimal
module (default is yes).Added in version 3.3.
Changed in version 3.13: Default to using the installed
mpdecimal
library.Changed in version 3.15: A bundled copy of the library will no longer be selected implicitly if an installed
mpdecimal
library is not found. In Python 3.15 only, it can still be selected explicitly using--with-system-libmpdec=no
or--without-system-libmpdec
.Deprecated since version 3.13, will be removed in version 3.16: A copy of the
mpdecimal
library sources will no longer be distributed with Python 3.16.See also
- --with-readline=readline|editline¶
Designate a backend library for the
readline
module.readline: Use readline as the backend.
editline: Use editline as the backend.
Added in version 3.10.
- --without-readline¶
Don’t build the
readline
module (built by default).Don’t define the
HAVE_LIBREADLINE
macro.Added in version 3.10.
- --with-libm=STRING¶
Override
libm
math library to STRING (default is system-dependent).
- --with-libc=STRING¶
Override
libc
C library to STRING (default is system-dependent).
- --with-openssl=DIR¶
Root of the OpenSSL directory.
Added in version 3.7.
- --with-openssl-rpath=[no|auto|DIR]¶
Set runtime library directory (rpath) for OpenSSL libraries:
no
(default): don’t set rpath;auto
: auto-detect rpath from--with-openssl
andpkg-config
;DIR: set an explicit rpath.
Added in version 3.10.
3.3.12. Security Options¶
- --with-hash-algorithm=[fnv|siphash13|siphash24]¶
Select hash algorithm for use in
Python/pyhash.c
:siphash13
(default);siphash24
;fnv
.
Added in version 3.4.
Added in version 3.11:
siphash13
is added and it is the new default.
- --with-builtin-hashlib-hashes=md5,sha1,sha256,sha512,sha3,blake2¶
Built-in hash modules:
md5
;sha1
;sha256
;sha512
;sha3
(with shake);blake2
.
Added in version 3.9.
- --with-ssl-default-suites=[python|openssl|STRING]¶
Override the OpenSSL default cipher suites string:
python
(default): use Python’s preferred selection;openssl
: leave OpenSSL’s defaults untouched;STRING: use a custom string
See the
ssl
module.Added in version 3.7.
Changed in version 3.10: The settings
python
and STRING also set TLS 1.2 as minimum protocol version.
- --disable-safety¶
Disable compiler options that are recommended by OpenSSF for security reasons with no performance overhead. If this option is not enabled, CPython will be built based on safety compiler options with no slow down. When this option is enabled, CPython will not be built with the compiler options listed below.
The following compiler options are disabled with
--disable-safety
:-fstack-protector-strong: Enable run-time checks for stack-based buffer overflows.
-Wtrampolines: Enable warnings about trampolines that require executable stacks.
Added in version 3.14.
- --enable-slower-safety¶
Enable compiler options that are recommended by OpenSSF for security reasons which require overhead. If this option is not enabled, CPython will not be built based on safety compiler options which performance impact. When this option is enabled, CPython will be built with the compiler options listed below.
The following compiler options are enabled with
--enable-slower-safety
:-D_FORTIFY_SOURCE=3: Fortify sources with compile- and run-time checks for unsafe libc usage and buffer overflows.
Added in version 3.14.
3.3.13. macOS Options¶
See Mac/README.rst.
- --enable-universalsdk¶
- --enable-universalsdk=SDKDIR¶
Create a universal binary build. SDKDIR specifies which macOS SDK should be used to perform the build (default is no).
- --enable-framework¶
- --enable-framework=INSTALLDIR¶
Create a Python.framework rather than a traditional Unix install. Optional INSTALLDIR specifies the installation path (default is no).
- --with-universal-archs=ARCH¶
Specify the kind of universal binary that should be created. This option is only valid when
--enable-universalsdk
is set.Options:
universal2
(x86-64 and arm64);32-bit
(PPC and i386);64-bit
(PPC64 and x86-64);3-way
(i386, PPC and x86-64);intel
(i386 and x86-64);intel-32
(i386);intel-64
(x86-64);all
(PPC, i386, PPC64 and x86-64).
Note that values for this configuration item are not the same as the identifiers used for universal binary wheels on macOS. See the Python Packaging User Guide for details on the packaging platform compatibility tags used on macOS
- --with-framework-name=FRAMEWORK¶
Specify the name for the python framework on macOS only valid when
--enable-framework
is set (default:Python
).
- --with-app-store-compliance¶
- --with-app-store-compliance=PATCH-FILE¶
The Python standard library contains strings that are known to trigger automated inspection tool errors when submitted for distribution by the macOS and iOS App Stores. If enabled, this option will apply the list of patches that are known to correct app store compliance. A custom patch file can also be specified. This option is disabled by default.
Added in version 3.13.
3.3.14. iOS Options¶
See iOS/README.rst.
- --enable-framework=INSTALLDIR¶
Create a Python.framework. Unlike macOS, the INSTALLDIR argument specifying the installation path is mandatory.
- --with-framework-name=FRAMEWORK¶
Specify the name for the framework (default:
Python
).
3.3.15. Cross Compiling Options¶
Cross compiling, also known as cross building, can be used to build Python for another CPU architecture or platform. Cross compiling requires a Python interpreter for the build platform. The version of the build Python must match the version of the cross compiled host Python.
- --build=BUILD¶
configure for building on BUILD, usually guessed by config.guess.
- --host=HOST¶
cross-compile to build programs to run on HOST (target platform)
- --with-build-python=path/to/python¶
path to build
python
binary for cross compilingAdded in version 3.11.
- CONFIG_SITE=file¶
An environment variable that points to a file with configure overrides.
Example config.site file:
# config.site-aarch64 ac_cv_buggy_getaddrinfo=no ac_cv_file__dev_ptmx=yes ac_cv_file__dev_ptc=no
- HOSTRUNNER¶
Program to run CPython for the host platform for cross-compilation.
Added in version 3.11.
Cross compiling example:
CONFIG_SITE=config.site-aarch64 ../configure \
--build=x86_64-pc-linux-gnu \
--host=aarch64-unknown-linux-gnu \
--with-build-python=../x86_64/python
3.4. Python Build System¶
3.4.1. Main files of the build system¶
configure.ac
=>configure
;Makefile.pre.in
=>Makefile
(created byconfigure
);pyconfig.h
(created byconfigure
);Modules/Setup
: C extensions built by the Makefile usingModule/makesetup
shell script;
3.4.2. Main build steps¶
C files (
.c
) are built as object files (.o
).A static
libpython
library (.a
) is created from objects files.python.o
and the staticlibpython
library are linked into the finalpython
program.C extensions are built by the Makefile (see
Modules/Setup
).
3.4.3. Main Makefile targets¶
3.4.3.1. make¶
For the most part, when rebuilding after editing some code or
refreshing your checkout from upstream, all you need to do is execute
make
, which (per Make’s semantics) builds the default target, the
first one defined in the Makefile. By tradition (including in the
CPython project) this is usually the all
target. The
configure
script expands an autoconf
variable,
@DEF_MAKE_ALL_RULE@
to describe precisely which targets make
all
will build. The three choices are:
profile-opt
(configured with--enable-optimizations
)build_wasm
(chosen if the host platform matcheswasm32-wasi*
orwasm32-emscripten
)build_all
(configured without explicitly using either of the others)
Depending on the most recent source file changes, Make will rebuild
any targets (object files and executables) deemed out-of-date,
including running configure
again if necessary. Source/target
dependencies are many and maintained manually however, so Make
sometimes doesn’t have all the information necessary to correctly
detect all targets which need to be rebuilt. Depending on which
targets aren’t rebuilt, you might experience a number of problems. If
you have build or test problems which you can’t otherwise explain,
make clean && make
should work around most dependency problems, at
the expense of longer build times.
3.4.3.2. make platform¶
Build the python
program, but don’t build the standard library
extension modules. This generates a file named platform
which
contains a single line describing the details of the build platform,
e.g., macosx-14.3-arm64-3.12
or linux-x86_64-3.13
.
3.4.3.3. make profile-opt¶
Build Python using profile-guided optimization (PGO). You can use the
configure --enable-optimizations
option to make this the
default target of the make
command (make all
or just
make
).
3.4.3.4. make clean¶
Remove built files.
3.4.3.5. make distclean¶
In addition to the work done by make clean
, remove files
created by the configure script. configure
will have to be run
before building again. [6]
3.4.3.6. make install¶
Build the all
target and install Python.
3.4.3.7. make test¶
Build the all
target and run the Python test suite with the
--fast-ci
option without GUI tests. Variables:
TESTOPTS
: additional regrtest command-line options.TESTPYTHONOPTS
: additional Python command-line options.TESTTIMEOUT
: timeout in seconds (default: 10 minutes).
3.4.3.8. make ci¶
This is similar to make test
, but uses the -ugui
to also run GUI tests.
Added in version 3.14.
3.4.3.9. make buildbottest¶
This is similar to make test
, but uses the --slow-ci
option and default timeout of 20 minutes, instead of --fast-ci
option.
3.4.3.10. make regen-all¶
Regenerate (almost) all generated files. These include (but are not
limited to) bytecode cases, and parser generator file.
make regen-stdlib-module-names
and autoconf
must be run
separately for the remaining generated files.
3.4.4. C extensions¶
Some C extensions are built as built-in modules, like the sys
module.
They are built with the Py_BUILD_CORE_BUILTIN
macro defined.
Built-in modules have no __file__
attribute:
>>> import sys
>>> sys
<module 'sys' (built-in)>
>>> sys.__file__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'sys' has no attribute '__file__'
Other C extensions are built as dynamic libraries, like the _asyncio
module.
They are built with the Py_BUILD_CORE_MODULE
macro defined.
Example on Linux x86-64:
>>> import _asyncio
>>> _asyncio
<module '_asyncio' from '/usr/lib64/python3.9/lib-dynload/_asyncio.cpython-39-x86_64-linux-gnu.so'>
>>> _asyncio.__file__
'/usr/lib64/python3.9/lib-dynload/_asyncio.cpython-39-x86_64-linux-gnu.so'
Modules/Setup
is used to generate Makefile targets to build C extensions.
At the beginning of the files, C extensions are built as built-in modules.
Extensions defined after the *shared*
marker are built as dynamic libraries.
The PyAPI_FUNC()
, PyAPI_DATA()
and
PyMODINIT_FUNC
macros of Include/exports.h
are defined
differently depending if the Py_BUILD_CORE_MODULE
macro is defined:
Use
Py_EXPORTED_SYMBOL
if thePy_BUILD_CORE_MODULE
is definedUse
Py_IMPORTED_SYMBOL
otherwise.
If the Py_BUILD_CORE_BUILTIN
macro is used by mistake on a C extension
built as a shared library, its PyInit_xxx()
function is not exported,
causing an ImportError
on import.
3.5. Compiler and linker flags¶
Options set by the ./configure
script and environment variables and used by
Makefile
.
3.5.1. Preprocessor flags¶
- CONFIGURE_CPPFLAGS¶
Value of
CPPFLAGS
variable passed to the./configure
script.Added in version 3.6.
- CPPFLAGS¶
(Objective) C/C++ preprocessor flags, e.g.
-Iinclude_dir
if you have headers in a nonstandard directory include_dir.Both
CPPFLAGS
andLDFLAGS
need to contain the shell’s value to be able to build extension modules using the directories specified in the environment variables.
- BASECPPFLAGS¶
Added in version 3.4.
- PY_CPPFLAGS¶
Extra preprocessor flags added for building the interpreter object files.
Default:
$(BASECPPFLAGS) -I. -I$(srcdir)/Include $(CONFIGURE_CPPFLAGS) $(CPPFLAGS)
.Added in version 3.2.
3.5.2. Compiler flags¶
- CC¶
C compiler command.
Example:
gcc -pthread
.
- CXX¶
C++ compiler command.
Example:
g++ -pthread
.
- CFLAGS¶
C compiler flags.
- CFLAGS_NODIST¶
CFLAGS_NODIST
is used for building the interpreter and stdlib C extensions. Use it when a compiler flag should not be part ofCFLAGS
once Python is installed (gh-65320).In particular,
CFLAGS
should not contain:the compiler flag
-I
(for setting the search path for include files). The-I
flags are processed from left to right, and any flags inCFLAGS
would take precedence over user- and package-supplied-I
flags.hardening flags such as
-Werror
because distributions cannot control whether packages installed by users conform to such heightened standards.
Added in version 3.5.
- COMPILEALL_OPTS¶
Options passed to the
compileall
command line when building PYC files inmake install
. Default:-j0
.Added in version 3.12.
- EXTRA_CFLAGS¶
Extra C compiler flags.
- CONFIGURE_CFLAGS_NODIST¶
Value of
CFLAGS_NODIST
variable passed to the./configure
script.Added in version 3.5.
- BASECFLAGS¶
Base compiler flags.
- OPT¶
Optimization flags.
- CFLAGS_ALIASING¶
Strict or non-strict aliasing flags used to compile
Python/dtoa.c
.Added in version 3.7.
- CCSHARED¶
Compiler flags used to build a shared library.
For example,
-fPIC
is used on Linux and on BSD.
- CFLAGSFORSHARED¶
Extra C flags added for building the interpreter object files.
Default:
$(CCSHARED)
when--enable-shared
is used, or an empty string otherwise.
- PY_CFLAGS¶
Default:
$(BASECFLAGS) $(OPT) $(CONFIGURE_CFLAGS) $(CFLAGS) $(EXTRA_CFLAGS)
.
- PY_CFLAGS_NODIST¶
Default:
$(CONFIGURE_CFLAGS_NODIST) $(CFLAGS_NODIST) -I$(srcdir)/Include/internal
.Added in version 3.5.
- PY_STDMODULE_CFLAGS¶
C flags used for building the interpreter object files.
Default:
$(PY_CFLAGS) $(PY_CFLAGS_NODIST) $(PY_CPPFLAGS) $(CFLAGSFORSHARED)
.Added in version 3.7.
- PY_CORE_CFLAGS¶
Default:
$(PY_STDMODULE_CFLAGS) -DPy_BUILD_CORE
.Added in version 3.2.
- PY_BUILTIN_MODULE_CFLAGS¶
Compiler flags to build a standard library extension module as a built-in module, like the
posix
module.Default:
$(PY_STDMODULE_CFLAGS) -DPy_BUILD_CORE_BUILTIN
.Added in version 3.8.
- PURIFY¶
Purify command. Purify is a memory debugger program.
Default: empty string (not used).
3.5.3. Linker flags¶
- LINKCC¶
Linker command used to build programs like
python
and_testembed
.Default:
$(PURIFY) $(CC)
.
- CONFIGURE_LDFLAGS¶
Value of
LDFLAGS
variable passed to the./configure
script.Avoid assigning
CFLAGS
,LDFLAGS
, etc. so users can use them on the command line to append to these values without stomping the pre-set values.Added in version 3.2.
- LDFLAGS_NODIST¶
LDFLAGS_NODIST
is used in the same manner asCFLAGS_NODIST
. Use it when a linker flag should not be part ofLDFLAGS
once Python is installed (gh-65320).In particular,
LDFLAGS
should not contain:the compiler flag
-L
(for setting the search path for libraries). The-L
flags are processed from left to right, and any flags inLDFLAGS
would take precedence over user- and package-supplied-L
flags.
- CONFIGURE_LDFLAGS_NODIST¶
Value of
LDFLAGS_NODIST
variable passed to the./configure
script.Added in version 3.8.
- LDFLAGS¶
Linker flags, e.g.
-Llib_dir
if you have libraries in a nonstandard directory lib_dir.Both
CPPFLAGS
andLDFLAGS
need to contain the shell’s value to be able to build extension modules using the directories specified in the environment variables.
- LIBS¶
Linker flags to pass libraries to the linker when linking the Python executable.
Example:
-lrt
.
- LDSHARED¶
Command to build a shared library.
Default:
@LDSHARED@ $(PY_LDFLAGS)
.
- BLDSHARED¶
Command to build
libpython
shared library.Default:
@BLDSHARED@ $(PY_CORE_LDFLAGS)
.
- PY_LDFLAGS¶
Default:
$(CONFIGURE_LDFLAGS) $(LDFLAGS)
.
- PY_LDFLAGS_NODIST¶
Default:
$(CONFIGURE_LDFLAGS_NODIST) $(LDFLAGS_NODIST)
.Added in version 3.8.
- PY_CORE_LDFLAGS¶
Linker flags used for building the interpreter object files.
Added in version 3.8.
Footnotes
git clean -fdx
is an even more extreme way to “clean” your
checkout. It removes all files not known to Git.
When bug hunting using git bisect
, this is
recommended between probes
to guarantee a completely clean build. Use with care, as it
will delete all files not checked into Git, including your
new, uncommitted work.


Transports and Protocols

Transports and Protocols¶
Preface
Transports and Protocols are used by the low-level event loop
APIs such as loop.create_connection()
. They use
callback-based programming style and enable high-performance
implementations of network or IPC protocols (e.g. HTTP).
Essentially, transports and protocols should only be used in libraries and frameworks and never in high-level asyncio applications.
This documentation page covers both Transports and Protocols.
Introduction
At the highest level, the transport is concerned with how bytes are transmitted, while the protocol determines which bytes to transmit (and to some extent when).
A different way of saying the same thing: a transport is an abstraction for a socket (or similar I/O endpoint) while a protocol is an abstraction for an application, from the transport’s point of view.
Yet another view is the transport and protocol interfaces together define an abstract interface for using network I/O and interprocess I/O.
There is always a 1:1 relationship between transport and protocol objects: the protocol calls transport methods to send data, while the transport calls protocol methods to pass it data that has been received.
Most of connection oriented event loop methods
(such as loop.create_connection()
) usually accept a
protocol_factory argument used to create a Protocol object
for an accepted connection, represented by a Transport object.
Such methods usually return a tuple of (transport, protocol)
.
Contents
This documentation page contains the following sections:
The Transports section documents asyncio
BaseTransport
,ReadTransport
,WriteTransport
,Transport
,DatagramTransport
, andSubprocessTransport
classes.The Protocols section documents asyncio
BaseProtocol
,Protocol
,BufferedProtocol
,DatagramProtocol
, andSubprocessProtocol
classes.The Examples section showcases how to work with transports, protocols, and low-level event loop APIs.
Transports¶
Source code: Lib/asyncio/transports.py
Transports are classes provided by asyncio
in order to abstract
various kinds of communication channels.
Transport objects are always instantiated by an asyncio event loop.
asyncio implements transports for TCP, UDP, SSL, and subprocess pipes. The methods available on a transport depend on the transport’s kind.
The transport classes are not thread safe.
Transports Hierarchy¶
- class asyncio.BaseTransport¶
Base class for all transports. Contains methods that all asyncio transports share.
- class asyncio.WriteTransport(BaseTransport)¶
A base transport for write-only connections.
Instances of the WriteTransport class are returned from the
loop.connect_write_pipe()
event loop method and are also used by subprocess-related methods likeloop.subprocess_exec()
.
- class asyncio.ReadTransport(BaseTransport)¶
A base transport for read-only connections.
Instances of the ReadTransport class are returned from the
loop.connect_read_pipe()
event loop method and are also used by subprocess-related methods likeloop.subprocess_exec()
.
- class asyncio.Transport(WriteTransport, ReadTransport)¶
Interface representing a bidirectional transport, such as a TCP connection.
The user does not instantiate a transport directly; they call a utility function, passing it a protocol factory and other information necessary to create the transport and protocol.
Instances of the Transport class are returned from or used by event loop methods like
loop.create_connection()
,loop.create_unix_connection()
,loop.create_server()
,loop.sendfile()
, etc.
- class asyncio.DatagramTransport(BaseTransport)¶
A transport for datagram (UDP) connections.
Instances of the DatagramTransport class are returned from the
loop.create_datagram_endpoint()
event loop method.
- class asyncio.SubprocessTransport(BaseTransport)¶
An abstraction to represent a connection between a parent and its child OS process.
Instances of the SubprocessTransport class are returned from event loop methods
loop.subprocess_shell()
andloop.subprocess_exec()
.
Base Transport¶
- BaseTransport.close()¶
Close the transport.
If the transport has a buffer for outgoing data, buffered data will be flushed asynchronously. No more data will be received. After all buffered data is flushed, the protocol’s
protocol.connection_lost()
method will be called withNone
as its argument. The transport should not be used once it is closed.
- BaseTransport.is_closing()¶
Return
True
if the transport is closing or is closed.
- BaseTransport.get_extra_info(name, default=None)¶
Return information about the transport or underlying resources it uses.
name is a string representing the piece of transport-specific information to get.
default is the value to return if the information is not available, or if the transport does not support querying it with the given third-party event loop implementation or on the current platform.
For example, the following code attempts to get the underlying socket object of the transport:
sock = transport.get_extra_info('socket') if sock is not None: print(sock.getsockopt(...))
Categories of information that can be queried on some transports:
socket:
'peername'
: the remote address to which the socket is connected, result ofsocket.socket.getpeername()
(None
on error)'socket'
:socket.socket
instance'sockname'
: the socket’s own address, result ofsocket.socket.getsockname()
SSL socket:
'compression'
: the compression algorithm being used as a string, orNone
if the connection isn’t compressed; result ofssl.SSLSocket.compression()
'cipher'
: a three-value tuple containing the name of the cipher being used, the version of the SSL protocol that defines its use, and the number of secret bits being used; result ofssl.SSLSocket.cipher()
'peercert'
: peer certificate; result ofssl.SSLSocket.getpeercert()
'sslcontext'
:ssl.SSLContext
instance'ssl_object'
:ssl.SSLObject
orssl.SSLSocket
instance
pipe:
'pipe'
: pipe object
subprocess:
'subprocess'
:subprocess.Popen
instance
- BaseTransport.set_protocol(protocol)¶
Set a new protocol.
Switching protocol should only be done when both protocols are documented to support the switch.
- BaseTransport.get_protocol()¶
Return the current protocol.
Read-only Transports¶
- ReadTransport.is_reading()¶
Return
True
if the transport is receiving new data.Added in version 3.7.
- ReadTransport.pause_reading()¶
Pause the receiving end of the transport. No data will be passed to the protocol’s
protocol.data_received()
method untilresume_reading()
is called.Changed in version 3.7: The method is idempotent, i.e. it can be called when the transport is already paused or closed.
- ReadTransport.resume_reading()¶
Resume the receiving end. The protocol’s
protocol.data_received()
method will be called once again if some data is available for reading.Changed in version 3.7: The method is idempotent, i.e. it can be called when the transport is already reading.
Write-only Transports¶
- WriteTransport.abort()¶
Close the transport immediately, without waiting for pending operations to complete. Buffered data will be lost. No more data will be received. The protocol’s
protocol.connection_lost()
method will eventually be called withNone
as its argument.
- WriteTransport.can_write_eof()¶
Return
True
if the transport supportswrite_eof()
,False
if not.
- WriteTransport.get_write_buffer_size()¶
Return the current size of the output buffer used by the transport.
- WriteTransport.get_write_buffer_limits()¶
Get the high and low watermarks for write flow control. Return a tuple
(low, high)
where low and high are positive number of bytes.Use
set_write_buffer_limits()
to set the limits.Added in version 3.4.2.
- WriteTransport.set_write_buffer_limits(high=None, low=None)¶
Set the high and low watermarks for write flow control.
These two values (measured in number of bytes) control when the protocol’s
protocol.pause_writing()
andprotocol.resume_writing()
methods are called. If specified, the low watermark must be less than or equal to the high watermark. Neither high nor low can be negative.pause_writing()
is called when the buffer size becomes greater than or equal to the high value. If writing has been paused,resume_writing()
is called when the buffer size becomes less than or equal to the low value.The defaults are implementation-specific. If only the high watermark is given, the low watermark defaults to an implementation-specific value less than or equal to the high watermark. Setting high to zero forces low to zero as well, and causes
pause_writing()
to be called whenever the buffer becomes non-empty. Setting low to zero causesresume_writing()
to be called only once the buffer is empty. Use of zero for either limit is generally sub-optimal as it reduces opportunities for doing I/O and computation concurrently.Use
get_write_buffer_limits()
to get the limits.
- WriteTransport.write(data)¶
Write some data bytes to the transport.
This method does not block; it buffers the data and arranges for it to be sent out asynchronously.
- WriteTransport.writelines(list_of_data)¶
Write a list (or any iterable) of data bytes to the transport. This is functionally equivalent to calling
write()
on each element yielded by the iterable, but may be implemented more efficiently.
- WriteTransport.write_eof()¶
Close the write end of the transport after flushing all buffered data. Data may still be received.
This method can raise
NotImplementedError
if the transport (e.g. SSL) doesn’t support half-closed connections.
Datagram Transports¶
- DatagramTransport.sendto(data, addr=None)¶
Send the data bytes to the remote peer given by addr (a transport-dependent target address). If addr is
None
, the data is sent to the target address given on transport creation.This method does not block; it buffers the data and arranges for it to be sent out asynchronously.
Changed in version 3.13: This method can be called with an empty bytes object to send a zero-length datagram. The buffer size calculation used for flow control is also updated to account for the datagram header.
- DatagramTransport.abort()¶
Close the transport immediately, without waiting for pending operations to complete. Buffered data will be lost. No more data will be received. The protocol’s
protocol.connection_lost()
method will eventually be called withNone
as its argument.
Subprocess Transports¶
- SubprocessTransport.get_pid()¶
Return the subprocess process id as an integer.
- SubprocessTransport.get_pipe_transport(fd)¶
Return the transport for the communication pipe corresponding to the integer file descriptor fd:
0
: writable streaming transport of the standard input (stdin), orNone
if the subprocess was not created withstdin=PIPE
1
: readable streaming transport of the standard output (stdout), orNone
if the subprocess was not created withstdout=PIPE
2
: readable streaming transport of the standard error (stderr), orNone
if the subprocess was not created withstderr=PIPE
other fd:
None
- SubprocessTransport.get_returncode()¶
Return the subprocess return code as an integer or
None
if it hasn’t returned, which is similar to thesubprocess.Popen.returncode
attribute.
- SubprocessTransport.kill()¶
Kill the subprocess.
On POSIX systems, the function sends SIGKILL to the subprocess. On Windows, this method is an alias for
terminate()
.See also
subprocess.Popen.kill()
.
- SubprocessTransport.send_signal(signal)¶
Send the signal number to the subprocess, as in
subprocess.Popen.send_signal()
.
- SubprocessTransport.terminate()¶
Stop the subprocess.
On POSIX systems, this method sends
SIGTERM
to the subprocess. On Windows, the Windows API functionTerminateProcess()
is called to stop the subprocess.See also
subprocess.Popen.terminate()
.
Protocols¶
Source code: Lib/asyncio/protocols.py
asyncio provides a set of abstract base classes that should be used to implement network protocols. Those classes are meant to be used together with transports.
Subclasses of abstract base protocol classes may implement some or all methods. All these methods are callbacks: they are called by transports on certain events, for example when some data is received. A base protocol method should be called by the corresponding transport.
Base Protocols¶
- class asyncio.BaseProtocol¶
Base protocol with methods that all protocols share.
- class asyncio.Protocol(BaseProtocol)¶
The base class for implementing streaming protocols (TCP, Unix sockets, etc).
- class asyncio.BufferedProtocol(BaseProtocol)¶
A base class for implementing streaming protocols with manual control of the receive buffer.
- class asyncio.DatagramProtocol(BaseProtocol)¶
The base class for implementing datagram (UDP) protocols.
- class asyncio.SubprocessProtocol(BaseProtocol)¶
The base class for implementing protocols communicating with child processes (unidirectional pipes).
Base Protocol¶
All asyncio protocols can implement Base Protocol callbacks.
Connection Callbacks
Connection callbacks are called on all protocols, exactly once per a successful connection. All other protocol callbacks can only be called between those two methods.
- BaseProtocol.connection_made(transport)¶
Called when a connection is made.
The transport argument is the transport representing the connection. The protocol is responsible for storing the reference to its transport.
- BaseProtocol.connection_lost(exc)¶
Called when the connection is lost or closed.
The argument is either an exception object or
None
. The latter means a regular EOF is received, or the connection was aborted or closed by this side of the connection.
Flow Control Callbacks
Flow control callbacks can be called by transports to pause or resume writing performed by the protocol.
See the documentation of the set_write_buffer_limits()
method for more details.
- BaseProtocol.pause_writing()¶
Called when the transport’s buffer goes over the high watermark.
- BaseProtocol.resume_writing()¶
Called when the transport’s buffer drains below the low watermark.
If the buffer size equals the high watermark,
pause_writing()
is not called: the buffer size must
go strictly over.
Conversely, resume_writing()
is called when the
buffer size is equal or lower than the low watermark. These end
conditions are important to ensure that things go as expected when
either mark is zero.
Streaming Protocols¶
Event methods, such as loop.create_server()
,
loop.create_unix_server()
, loop.create_connection()
,
loop.create_unix_connection()
, loop.connect_accepted_socket()
,
loop.connect_read_pipe()
, and loop.connect_write_pipe()
accept factories that return streaming protocols.
- Protocol.data_received(data)¶
Called when some data is received. data is a non-empty bytes object containing the incoming data.
Whether the data is buffered, chunked or reassembled depends on the transport. In general, you shouldn’t rely on specific semantics and instead make your parsing generic and flexible. However, data is always received in the correct order.
The method can be called an arbitrary number of times while a connection is open.
However,
protocol.eof_received()
is called at most once. Onceeof_received()
is called,data_received()
is not called anymore.
- Protocol.eof_received()¶
Called when the other end signals it won’t send any more data (for example by calling
transport.write_eof()
, if the other end also uses asyncio).This method may return a false value (including
None
), in which case the transport will close itself. Conversely, if this method returns a true value, the protocol used determines whether to close the transport. Since the default implementation returnsNone
, it implicitly closes the connection.Some transports, including SSL, don’t support half-closed connections, in which case returning true from this method will result in the connection being closed.
State machine:
start -> connection_made
[-> data_received]*
[-> eof_received]?
-> connection_lost -> end
Buffered Streaming Protocols¶
Added in version 3.7.
Buffered Protocols can be used with any event loop method that supports Streaming Protocols.
BufferedProtocol
implementations allow explicit manual allocation
and control of the receive buffer. Event loops can then use the buffer
provided by the protocol to avoid unnecessary data copies. This
can result in noticeable performance improvement for protocols that
receive big amounts of data. Sophisticated protocol implementations
can significantly reduce the number of buffer allocations.
The following callbacks are called on BufferedProtocol
instances:
- BufferedProtocol.get_buffer(sizehint)¶
Called to allocate a new receive buffer.
sizehint is the recommended minimum size for the returned buffer. It is acceptable to return smaller or larger buffers than what sizehint suggests. When set to -1, the buffer size can be arbitrary. It is an error to return a buffer with a zero size.
get_buffer()
must return an object implementing the buffer protocol.
- BufferedProtocol.buffer_updated(nbytes)¶
Called when the buffer was updated with the received data.
nbytes is the total number of bytes that were written to the buffer.
- BufferedProtocol.eof_received()¶
See the documentation of the
protocol.eof_received()
method.
get_buffer()
can be called an arbitrary number
of times during a connection. However, protocol.eof_received()
is called at most once
and, if called, get_buffer()
and
buffer_updated()
won’t be called after it.
State machine:
start -> connection_made
[-> get_buffer
[-> buffer_updated]?
]*
[-> eof_received]?
-> connection_lost -> end
Datagram Protocols¶
Datagram Protocol instances should be constructed by protocol
factories passed to the loop.create_datagram_endpoint()
method.
- DatagramProtocol.datagram_received(data, addr)¶
Called when a datagram is received. data is a bytes object containing the incoming data. addr is the address of the peer sending the data; the exact format depends on the transport.
- DatagramProtocol.error_received(exc)¶
Called when a previous send or receive operation raises an
OSError
. exc is theOSError
instance.This method is called in rare conditions, when the transport (e.g. UDP) detects that a datagram could not be delivered to its recipient. In many conditions though, undeliverable datagrams will be silently dropped.
Note
On BSD systems (macOS, FreeBSD, etc.) flow control is not supported for datagram protocols, because there is no reliable way to detect send failures caused by writing too many packets.
The socket always appears ‘ready’ and excess packets are dropped. An
OSError
with errno
set to errno.ENOBUFS
may
or may not be raised; if it is raised, it will be reported to
DatagramProtocol.error_received()
but otherwise ignored.
Subprocess Protocols¶
Subprocess Protocol instances should be constructed by protocol
factories passed to the loop.subprocess_exec()
and
loop.subprocess_shell()
methods.
- SubprocessProtocol.pipe_data_received(fd, data)¶
Called when the child process writes data into its stdout or stderr pipe.
fd is the integer file descriptor of the pipe.
data is a non-empty bytes object containing the received data.
- SubprocessProtocol.pipe_connection_lost(fd, exc)¶
Called when one of the pipes communicating with the child process is closed.
fd is the integer file descriptor that was closed.
- SubprocessProtocol.process_exited()¶
Called when the child process has exited.
It can be called before
pipe_data_received()
andpipe_connection_lost()
methods.
Examples¶
TCP Echo Server¶
Create a TCP echo server using the loop.create_server()
method, send back
received data, and close the connection:
import asyncio
class EchoServerProtocol(asyncio.Protocol):
def connection_made(self, transport):
peername = transport.get_extra_info('peername')
print('Connection from {}'.format(peername))
self.transport = transport
def data_received(self, data):
message = data.decode()
print('Data received: {!r}'.format(message))
print('Send: {!r}'.format(message))
self.transport.write(data)
print('Close the client socket')
self.transport.close()
async def main():
# Get a reference to the event loop as we plan to use
# low-level APIs.
loop = asyncio.get_running_loop()
server = await loop.create_server(
EchoServerProtocol,
'127.0.0.1', 8888)
async with server:
await server.serve_forever()
asyncio.run(main())
See also
The TCP echo server using streams
example uses the high-level asyncio.start_server()
function.
TCP Echo Client¶
A TCP echo client using the loop.create_connection()
method, sends
data, and waits until the connection is closed:
import asyncio
class EchoClientProtocol(asyncio.Protocol):
def __init__(self, message, on_con_lost):
self.message = message
self.on_con_lost = on_con_lost
def connection_made(self, transport):
transport.write(self.message.encode())
print('Data sent: {!r}'.format(self.message))
def data_received(self, data):
print('Data received: {!r}'.format(data.decode()))
def connection_lost(self, exc):
print('The server closed the connection')
self.on_con_lost.set_result(True)
async def main():
# Get a reference to the event loop as we plan to use
# low-level APIs.
loop = asyncio.get_running_loop()
on_con_lost = loop.create_future()
message = 'Hello World!'
transport, protocol = await loop.create_connection(
lambda: EchoClientProtocol(message, on_con_lost),
'127.0.0.1', 8888)
# Wait until the protocol signals that the connection
# is lost and close the transport.
try:
await on_con_lost
finally:
transport.close()
asyncio.run(main())
See also
The TCP echo client using streams
example uses the high-level asyncio.open_connection()
function.
UDP Echo Server¶
A UDP echo server, using the loop.create_datagram_endpoint()
method, sends back received data:
import asyncio
class EchoServerProtocol:
def connection_made(self, transport):
self.transport = transport
def datagram_received(self, data, addr):
message = data.decode()
print('Received %r from %s' % (message, addr))
print('Send %r to %s' % (message, addr))
self.transport.sendto(data, addr)
async def main():
print("Starting UDP server")
# Get a reference to the event loop as we plan to use
# low-level APIs.
loop = asyncio.get_running_loop()
# One protocol instance will be created to serve all
# client requests.
transport, protocol = await loop.create_datagram_endpoint(
EchoServerProtocol,
local_addr=('127.0.0.1', 9999))
try:
await asyncio.sleep(3600) # Serve for 1 hour.
finally:
transport.close()
asyncio.run(main())
UDP Echo Client¶
A UDP echo client, using the loop.create_datagram_endpoint()
method, sends data and closes the transport when it receives the answer:
import asyncio
class EchoClientProtocol:
def __init__(self, message, on_con_lost):
self.message = message
self.on_con_lost = on_con_lost
self.transport = None
def connection_made(self, transport):
self.transport = transport
print('Send:', self.message)
self.transport.sendto(self.message.encode())
def datagram_received(self, data, addr):
print("Received:", data.decode())
print("Close the socket")
self.transport.close()
def error_received(self, exc):
print('Error received:', exc)
def connection_lost(self, exc):
print("Connection closed")
self.on_con_lost.set_result(True)
async def main():
# Get a reference to the event loop as we plan to use
# low-level APIs.
loop = asyncio.get_running_loop()
on_con_lost = loop.create_future()
message = "Hello World!"
transport, protocol = await loop.create_datagram_endpoint(
lambda: EchoClientProtocol(message, on_con_lost),
remote_addr=('127.0.0.1', 9999))
try:
await on_con_lost
finally:
transport.close()
asyncio.run(main())
Connecting Existing Sockets¶
Wait until a socket receives data using the
loop.create_connection()
method with a protocol:
import asyncio
import socket
class MyProtocol(asyncio.Protocol):
def __init__(self, on_con_lost):
self.transport = None
self.on_con_lost = on_con_lost
def connection_made(self, transport):
self.transport = transport
def data_received(self, data):
print("Received:", data.decode())
# We are done: close the transport;
# connection_lost() will be called automatically.
self.transport.close()
def connection_lost(self, exc):
# The socket has been closed
self.on_con_lost.set_result(True)
async def main():
# Get a reference to the event loop as we plan to use
# low-level APIs.
loop = asyncio.get_running_loop()
on_con_lost = loop.create_future()
# Create a pair of connected sockets
rsock, wsock = socket.socketpair()
# Register the socket to wait for data.
transport, protocol = await loop.create_connection(
lambda: MyProtocol(on_con_lost), sock=rsock)
# Simulate the reception of data from the network.
loop.call_soon(wsock.send, 'abc'.encode())
try:
await protocol.on_con_lost
finally:
transport.close()
wsock.close()
asyncio.run(main())
See also
The watch a file descriptor for read events example uses the low-level
loop.add_reader()
method to register an FD.
The register an open socket to wait for data using streams example uses high-level streams
created by the open_connection()
function in a coroutine.
loop.subprocess_exec() and SubprocessProtocol¶
An example of a subprocess protocol used to get the output of a subprocess and to wait for the subprocess exit.
The subprocess is created by the loop.subprocess_exec()
method:
import asyncio
import sys
class DateProtocol(asyncio.SubprocessProtocol):
def __init__(self, exit_future):
self.exit_future = exit_future
self.output = bytearray()
self.pipe_closed = False
self.exited = False
def pipe_connection_lost(self, fd, exc):
self.pipe_closed = True
self.check_for_exit()
def pipe_data_received(self, fd, data):
self.output.extend(data)
def process_exited(self):
self.exited = True
# process_exited() method can be called before
# pipe_connection_lost() method: wait until both methods are
# called.
self.check_for_exit()
def check_for_exit(self):
if self.pipe_closed and self.exited:
self.exit_future.set_result(True)
async def get_date():
# Get a reference to the event loop as we plan to use
# low-level APIs.
loop = asyncio.get_running_loop()
code = 'import datetime; print(datetime.datetime.now())'
exit_future = asyncio.Future(loop=loop)
# Create the subprocess controlled by DateProtocol;
# redirect the standard output into a pipe.
transport, protocol = await loop.subprocess_exec(
lambda: DateProtocol(exit_future),
sys.executable, '-c', code,
stdin=None, stderr=None)
# Wait for the subprocess exit using the process_exited()
# method of the protocol.
await exit_future
# Close the stdout pipe.
transport.close()
# Read the output which was collected by the
# pipe_data_received() method of the protocol.
data = bytes(protocol.output)
return data.decode('ascii').rstrip()
date = asyncio.run(get_date())
print(f"Current date: {date}")
See also the same example written using high-level APIs.