Code Objects

Code Objects¶
Code objects are a low-level detail of the CPython implementation. Each one represents a chunk of executable code that hasn’t yet been bound into a function.
-
type PyCodeObject¶
The C structure of the objects used to describe code objects. The fields of this type are subject to change at any time.
-
PyTypeObject PyCode_Type¶
This is an instance of
PyTypeObject
representing the Python code object.
-
int PyCode_Check(PyObject *co)¶
Return true if co is a code object. This function always succeeds.
-
Py_ssize_t PyCode_GetNumFree(PyCodeObject *co)¶
Return the number of free (closure) variables in a code object.
-
int PyUnstable_Code_GetFirstFree(PyCodeObject *co)¶
- This is Unstable API. It may change without warning in minor releases.
Return the position of the first free (closure) variable in a code object.
Changed in version 3.13: Renamed from
PyCode_GetFirstFree
as part of Unstable C API. The old name is deprecated, but will remain available until the signature changes again.
-
PyCodeObject *PyUnstable_Code_New(int argcount, int kwonlyargcount, int nlocals, int stacksize, int flags, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, PyObject *qualname, int firstlineno, PyObject *linetable, PyObject *exceptiontable)¶
- This is Unstable API. It may change without warning in minor releases.
Return a new code object. If you need a dummy code object to create a frame, use
PyCode_NewEmpty()
instead.Since the definition of the bytecode changes often, calling
PyUnstable_Code_New()
directly can bind you to a precise Python version.The many arguments of this function are inter-dependent in complex ways, meaning that subtle changes to values are likely to result in incorrect execution or VM crashes. Use this function only with extreme care.
Changed in version 3.11: Added
qualname
andexceptiontable
parameters.Changed in version 3.12: Renamed from
PyCode_New
as part of Unstable C API. The old name is deprecated, but will remain available until the signature changes again.
-
PyCodeObject *PyUnstable_Code_NewWithPosOnlyArgs(int argcount, int posonlyargcount, int kwonlyargcount, int nlocals, int stacksize, int flags, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, PyObject *qualname, int firstlineno, PyObject *linetable, PyObject *exceptiontable)¶
- This is Unstable API. It may change without warning in minor releases.
Similar to
PyUnstable_Code_New()
, but with an extra “posonlyargcount” for positional-only arguments. The same caveats that apply toPyUnstable_Code_New
also apply to this function.Added in version 3.8: as
PyCode_NewWithPosOnlyArgs
Changed in version 3.11: Added
qualname
andexceptiontable
parameters.Changed in version 3.12: Renamed to
PyUnstable_Code_NewWithPosOnlyArgs
. The old name is deprecated, but will remain available until the signature changes again.
-
PyCodeObject *PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno)¶
- Return value: New reference.
Return a new empty code object with the specified filename, function name, and first line number. The resulting code object will raise an
Exception
if executed.
-
int PyCode_Addr2Line(PyCodeObject *co, int byte_offset)¶
Return the line number of the instruction that occurs on or before
byte_offset
and ends after it. If you just need the line number of a frame, usePyFrame_GetLineNumber()
instead.For efficiently iterating over the line numbers in a code object, use the API described in PEP 626.
-
int PyCode_Addr2Location(PyObject *co, int byte_offset, int *start_line, int *start_column, int *end_line, int *end_column)¶
Sets the passed
int
pointers to the source code line and column numbers for the instruction atbyte_offset
. Sets the value to0
when information is not available for any particular element.Returns
1
if the function succeeds and 0 otherwise.Added in version 3.11.
-
PyObject *PyCode_GetCode(PyCodeObject *co)¶
Equivalent to the Python code
getattr(co, 'co_code')
. Returns a strong reference to aPyBytesObject
representing the bytecode in a code object. On error,NULL
is returned and an exception is raised.This
PyBytesObject
may be created on-demand by the interpreter and does not necessarily represent the bytecode actually executed by CPython. The primary use case for this function is debuggers and profilers.Added in version 3.11.
-
PyObject *PyCode_GetVarnames(PyCodeObject *co)¶
Equivalent to the Python code
getattr(co, 'co_varnames')
. Returns a new reference to aPyTupleObject
containing the names of the local variables. On error,NULL
is returned and an exception is raised.Added in version 3.11.
-
PyObject *PyCode_GetCellvars(PyCodeObject *co)¶
Equivalent to the Python code
getattr(co, 'co_cellvars')
. Returns a new reference to aPyTupleObject
containing the names of the local variables that are referenced by nested functions. On error,NULL
is returned and an exception is raised.Added in version 3.11.
-
PyObject *PyCode_GetFreevars(PyCodeObject *co)¶
Equivalent to the Python code
getattr(co, 'co_freevars')
. Returns a new reference to aPyTupleObject
containing the names of the free (closure) variables. On error,NULL
is returned and an exception is raised.Added in version 3.11.
-
int PyCode_AddWatcher(PyCode_WatchCallback callback)¶
Register callback as a code object watcher for the current interpreter. Return an ID which may be passed to
PyCode_ClearWatcher()
. In case of error (e.g. no more watcher IDs available), return-1
and set an exception.Added in version 3.12.
-
int PyCode_ClearWatcher(int watcher_id)¶
Clear watcher identified by watcher_id previously returned from
PyCode_AddWatcher()
for the current interpreter. Return0
on success, or-1
and set an exception on error (e.g. if the given watcher_id was never registered.)Added in version 3.12.
-
type PyCodeEvent¶
Enumeration of possible code object watcher events: -
PY_CODE_EVENT_CREATE
-PY_CODE_EVENT_DESTROY
Added in version 3.12.
-
typedef int (*PyCode_WatchCallback)(PyCodeEvent event, PyCodeObject *co)¶
Type of a code object watcher callback function.
If event is
PY_CODE_EVENT_CREATE
, then the callback is invoked after co has been fully initialized. Otherwise, the callback is invoked before the destruction of co takes place, so the prior state of co can be inspected.If event is
PY_CODE_EVENT_DESTROY
, taking a reference in the callback to the about-to-be-destroyed code object will resurrect it and prevent it from being freed at this time. When the resurrected object is destroyed later, any watcher callbacks active at that time will be called again.Users of this API should not rely on internal runtime implementation details. Such details may include, but are not limited to, the exact order and timing of creation and destruction of code objects. While changes in these details may result in differences observable by watchers (including whether a callback is invoked or not), it does not change the semantics of the Python code being executed.
If the callback sets an exception, it must return
-1
; this exception will be printed as an unraisable exception usingPyErr_WriteUnraisable()
. Otherwise it should return0
.There may already be a pending exception set on entry to the callback. In this case, the callback should return
0
with the same exception still set. This means the callback may not call any other API that can set an exception unless it saves and clears the exception state first, and restores it before returning.Added in version 3.12.
Code Object Flags¶
Code objects contain a bit-field of flags, which can be retrieved as the
co_flags
Python attribute (for example using
PyObject_GetAttrString()
), and set using a flags argument to
PyUnstable_Code_New()
and similar functions.
Flags whose names start with CO_FUTURE_
correspond to features normally
selectable by future statements. These flags can be used in
PyCompilerFlags.cf_flags
.
Note that many CO_FUTURE_
flags are mandatory in current versions of
Python, and setting them has no effect.
The following flags are available. For their meaning, see the linked documentation of their Python equivalents.
Flag |
Meaning |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
no effect ( |
|
no effect ( |
|
no effect ( |
|
no effect ( |
|
no effect ( |
|
no effect ( |
|
Extra information¶
To support low-level extensions to frame evaluation, such as external just-in-time compilers, it is possible to attach arbitrary extra data to code objects.
These functions are part of the unstable C API tier: this functionality is a CPython implementation detail, and the API may change without deprecation warnings.
-
Py_ssize_t PyUnstable_Eval_RequestCodeExtraIndex(freefunc free)¶
- This is Unstable API. It may change without warning in minor releases.
Return a new opaque index value used to adding data to code objects.
You generally call this function once (per interpreter) and use the result with
PyCode_GetExtra
andPyCode_SetExtra
to manipulate data on individual code objects.If free is not
NULL
: when a code object is deallocated, free will be called on non-NULL
data stored under the new index. UsePy_DecRef()
when storingPyObject
.Added in version 3.6: as
_PyEval_RequestCodeExtraIndex
Changed in version 3.12: Renamed to
PyUnstable_Eval_RequestCodeExtraIndex
. The old private name is deprecated, but will be available until the API changes.
-
int PyUnstable_Code_GetExtra(PyObject *code, Py_ssize_t index, void **extra)¶
- This is Unstable API. It may change without warning in minor releases.
Set extra to the extra data stored under the given index. Return 0 on success. Set an exception and return -1 on failure.
If no data was set under the index, set extra to
NULL
and return 0 without setting an exception.Added in version 3.6: as
_PyCode_GetExtra
Changed in version 3.12: Renamed to
PyUnstable_Code_GetExtra
. The old private name is deprecated, but will be available until the API changes.
-
int PyUnstable_Code_SetExtra(PyObject *code, Py_ssize_t index, void *extra)¶
- This is Unstable API. It may change without warning in minor releases.
Set the extra data stored under the given index to extra. Return 0 on success. Set an exception and return -1 on failure.
Added in version 3.6: as
_PyCode_SetExtra
Changed in version 3.12: Renamed to
PyUnstable_Code_SetExtra
. The old private name is deprecated, but will be available until the API changes.


Importing Modules

Importing Modules¶
-
PyObject *PyImport_ImportModule(const char *name)¶
- Return value: New reference. Part of the Stable ABI.
This is a wrapper around
PyImport_Import()
which takes a const char* as an argument instead of a PyObject*.
-
PyObject *PyImport_ImportModuleNoBlock(const char *name)¶
- Return value: New reference. Part of the Stable ABI.
This function is a deprecated alias of
PyImport_ImportModule()
.Changed in version 3.3: This function used to fail immediately when the import lock was held by another thread. In Python 3.3 though, the locking scheme switched to per-module locks for most purposes, so this function’s special behaviour isn’t needed anymore.
Deprecated since version 3.13, will be removed in version 3.15: Use
PyImport_ImportModule()
instead.
-
PyObject *PyImport_ImportModuleEx(const char *name, PyObject *globals, PyObject *locals, PyObject *fromlist)¶
- Return value: New reference.
Import a module. This is best described by referring to the built-in Python function
__import__()
.The return value is a new reference to the imported module or top-level package, or
NULL
with an exception set on failure. Like for__import__()
, the return value when a submodule of a package was requested is normally the top-level package, unless a non-empty fromlist was given.Failing imports remove incomplete module objects, like with
PyImport_ImportModule()
.
-
PyObject *PyImport_ImportModuleLevelObject(PyObject *name, PyObject *globals, PyObject *locals, PyObject *fromlist, int level)¶
- Return value: New reference. Part of the Stable ABI since version 3.7.
Import a module. This is best described by referring to the built-in Python function
__import__()
, as the standard__import__()
function calls this function directly.The return value is a new reference to the imported module or top-level package, or
NULL
with an exception set on failure. Like for__import__()
, the return value when a submodule of a package was requested is normally the top-level package, unless a non-empty fromlist was given.Added in version 3.3.
-
PyObject *PyImport_ImportModuleLevel(const char *name, PyObject *globals, PyObject *locals, PyObject *fromlist, int level)¶
- Return value: New reference. Part of the Stable ABI.
Similar to
PyImport_ImportModuleLevelObject()
, but the name is a UTF-8 encoded string instead of a Unicode object.Changed in version 3.3: Negative values for level are no longer accepted.
-
PyObject *PyImport_Import(PyObject *name)¶
- Return value: New reference. Part of the Stable ABI.
This is a higher-level interface that calls the current “import hook function” (with an explicit level of 0, meaning absolute import). It invokes the
__import__()
function from the__builtins__
of the current globals. This means that the import is done using whatever import hooks are installed in the current environment.This function always uses absolute imports.
-
PyObject *PyImport_ReloadModule(PyObject *m)¶
- Return value: New reference. Part of the Stable ABI.
Reload a module. Return a new reference to the reloaded module, or
NULL
with an exception set on failure (the module still exists in this case).
-
PyObject *PyImport_AddModuleRef(const char *name)¶
- Return value: New reference. Part of the Stable ABI since version 3.13.
Return the module object corresponding to a module name.
The name argument may be of the form
package.module
. First check the modules dictionary if there’s one there, and if not, create a new one and insert it in the modules dictionary.Return a strong reference to the module on success. Return
NULL
with an exception set on failure.The module name name is decoded from UTF-8.
This function does not load or import the module; if the module wasn’t already loaded, you will get an empty module object. Use
PyImport_ImportModule()
or one of its variants to import a module. Package structures implied by a dotted name for name are not created if not already present.Added in version 3.13.
-
PyObject *PyImport_AddModuleObject(PyObject *name)¶
- Return value: Borrowed reference. Part of the Stable ABI since version 3.7.
Similar to
PyImport_AddModuleRef()
, but return a borrowed reference and name is a Pythonstr
object.Added in version 3.3.
-
PyObject *PyImport_AddModule(const char *name)¶
- Return value: Borrowed reference. Part of the Stable ABI.
Similar to
PyImport_AddModuleRef()
, but return a borrowed reference.
-
PyObject *PyImport_ExecCodeModule(const char *name, PyObject *co)¶
- Return value: New reference. Part of the Stable ABI.
Given a module name (possibly of the form
package.module
) and a code object read from a Python bytecode file or obtained from the built-in functioncompile()
, load the module. Return a new reference to the module object, orNULL
with an exception set if an error occurred. name is removed fromsys.modules
in error cases, even if name was already insys.modules
on entry toPyImport_ExecCodeModule()
. Leaving incompletely initialized modules insys.modules
is dangerous, as imports of such modules have no way to know that the module object is an unknown (and probably damaged with respect to the module author’s intents) state.The module’s
__spec__
and__loader__
will be set, if not set already, with the appropriate values. The spec’s loader will be set to the module’s__loader__
(if set) and to an instance ofSourceFileLoader
otherwise.The module’s
__file__
attribute will be set to the code object’sco_filename
. If applicable,__cached__
will also be set.This function will reload the module if it was already imported. See
PyImport_ReloadModule()
for the intended way to reload a module.If name points to a dotted name of the form
package.module
, any package structures not already created will still not be created.See also
PyImport_ExecCodeModuleEx()
andPyImport_ExecCodeModuleWithPathnames()
.Changed in version 3.12: The setting of
__cached__
and__loader__
is deprecated. SeeModuleSpec
for alternatives.
-
PyObject *PyImport_ExecCodeModuleEx(const char *name, PyObject *co, const char *pathname)¶
- Return value: New reference. Part of the Stable ABI.
Like
PyImport_ExecCodeModule()
, but the__file__
attribute of the module object is set to pathname if it is non-NULL
.See also
PyImport_ExecCodeModuleWithPathnames()
.
-
PyObject *PyImport_ExecCodeModuleObject(PyObject *name, PyObject *co, PyObject *pathname, PyObject *cpathname)¶
- Return value: New reference. Part of the Stable ABI since version 3.7.
Like
PyImport_ExecCodeModuleEx()
, but the__cached__
attribute of the module object is set to cpathname if it is non-NULL
. Of the three functions, this is the preferred one to use.Added in version 3.3.
Changed in version 3.12: Setting
__cached__
is deprecated. SeeModuleSpec
for alternatives.
-
PyObject *PyImport_ExecCodeModuleWithPathnames(const char *name, PyObject *co, const char *pathname, const char *cpathname)¶
- Return value: New reference. Part of the Stable ABI.
Like
PyImport_ExecCodeModuleObject()
, but name, pathname and cpathname are UTF-8 encoded strings. Attempts are also made to figure out what the value for pathname should be from cpathname if the former is set toNULL
.Added in version 3.2.
Changed in version 3.3: Uses
imp.source_from_cache()
in calculating the source path if only the bytecode path is provided.Changed in version 3.12: No longer uses the removed
imp
module.
-
long PyImport_GetMagicNumber()¶
- Part of the Stable ABI.
Return the magic number for Python bytecode files (a.k.a.
.pyc
file). The magic number should be present in the first four bytes of the bytecode file, in little-endian byte order. Returns-1
on error.Changed in version 3.3: Return value of
-1
upon failure.
-
const char *PyImport_GetMagicTag()¶
- Part of the Stable ABI.
Return the magic tag string for PEP 3147 format Python bytecode file names. Keep in mind that the value at
sys.implementation.cache_tag
is authoritative and should be used instead of this function.Added in version 3.2.
-
PyObject *PyImport_GetModuleDict()¶
- Return value: Borrowed reference. Part of the Stable ABI.
Return the dictionary used for the module administration (a.k.a.
sys.modules
). Note that this is a per-interpreter variable.
-
PyObject *PyImport_GetModule(PyObject *name)¶
- Return value: New reference. Part of the Stable ABI since version 3.8.
Return the already imported module with the given name. If the module has not been imported yet then returns
NULL
but does not set an error. ReturnsNULL
and sets an error if the lookup failed.Added in version 3.7.
-
PyObject *PyImport_GetImporter(PyObject *path)¶
- Return value: New reference. Part of the Stable ABI.
Return a finder object for a
sys.path
/pkg.__path__
item path, possibly by fetching it from thesys.path_importer_cache
dict. If it wasn’t yet cached, traversesys.path_hooks
until a hook is found that can handle the path item. ReturnNone
if no hook could; this tells our caller that the path based finder could not find a finder for this path item. Cache the result insys.path_importer_cache
. Return a new reference to the finder object.
-
int PyImport_ImportFrozenModuleObject(PyObject *name)¶
- Part of the Stable ABI since version 3.7.
Load a frozen module named name. Return
1
for success,0
if the module is not found, and-1
with an exception set if the initialization failed. To access the imported module on a successful load, usePyImport_ImportModule()
. (Note the misnomer — this function would reload the module if it was already imported.)Added in version 3.3.
Changed in version 3.4: The
__file__
attribute is no longer set on the module.
-
int PyImport_ImportFrozenModule(const char *name)¶
- Part of the Stable ABI.
Similar to
PyImport_ImportFrozenModuleObject()
, but the name is a UTF-8 encoded string instead of a Unicode object.
-
struct _frozen¶
This is the structure type definition for frozen module descriptors, as generated by the freeze utility (see
Tools/freeze/
in the Python source distribution). Its definition, found inInclude/import.h
, is:struct _frozen { const char *name; const unsigned char *code; int size; bool is_package; };
Changed in version 3.11: The new
is_package
field indicates whether the module is a package or not. This replaces setting thesize
field to a negative value.
-
const struct _frozen *PyImport_FrozenModules¶
This pointer is initialized to point to an array of
_frozen
records, terminated by one whose members are allNULL
or zero. When a frozen module is imported, it is searched in this table. Third-party code could play tricks with this to provide a dynamically created collection of frozen modules.
-
int PyImport_AppendInittab(const char *name, PyObject *(*initfunc)(void))¶
- Part of the Stable ABI.
Add a single module to the existing table of built-in modules. This is a convenience wrapper around
PyImport_ExtendInittab()
, returning-1
if the table could not be extended. The new module can be imported by the name name, and uses the function initfunc as the initialization function called on the first attempted import. This should be called beforePy_Initialize()
.
-
struct _inittab¶
Structure describing a single entry in the list of built-in modules. Programs which embed Python may use an array of these structures in conjunction with
PyImport_ExtendInittab()
to provide additional built-in modules. The structure consists of two members:-
const char *name¶
The module name, as an ASCII encoded string.
-
const char *name¶
-
int PyImport_ExtendInittab(struct _inittab *newtab)¶
Add a collection of modules to the table of built-in modules. The newtab array must end with a sentinel entry which contains
NULL
for thename
field; failure to provide the sentinel value can result in a memory fault. Returns0
on success or-1
if insufficient memory could be allocated to extend the internal table. In the event of failure, no modules are added to the internal table. This must be called beforePy_Initialize()
.If Python is initialized multiple times,
PyImport_AppendInittab()
orPyImport_ExtendInittab()
must be called before each Python initialization.
-
struct _inittab *PyImport_Inittab¶
The table of built-in modules used by Python initialization. Do not use this directly; use
PyImport_AppendInittab()
andPyImport_ExtendInittab()
instead.
-
PyObject *PyImport_ImportModuleAttr(PyObject *mod_name, PyObject *attr_name)¶
- Return value: New reference.
Import the module mod_name and get its attribute attr_name.
Names must be Python
str
objects.Helper function combining
PyImport_Import()
andPyObject_GetAttr()
. For example, it can raiseImportError
if the module is not found, andAttributeError
if the attribute doesn’t exist.Added in version 3.14.
-
PyObject *PyImport_ImportModuleAttrString(const char *mod_name, const char *attr_name)¶
- Return value: New reference.
Similar to
PyImport_ImportModuleAttr()
, but names are UTF-8 encoded strings instead of Pythonstr
objects.Added in version 3.14.


Reference Counting

Reference Counting¶
The functions and macros in this section are used for managing reference counts of Python objects.
-
Py_ssize_t Py_REFCNT(PyObject *o)¶
- Part of the Stable ABI since version 3.14.
Get the reference count of the Python object o.
Note that the returned value may not actually reflect how many references to the object are actually held. For example, some objects are immortal and have a very high refcount that does not reflect the actual number of references. Consequently, do not rely on the returned value to be accurate, other than a value of 0 or 1.
Use the
Py_SET_REFCNT()
function to set an object reference count.Note
On free-threaded builds of Python, returning 1 isn’t sufficient to determine if it’s safe to treat o as having no access by other threads. Use
PyUnstable_Object_IsUniquelyReferenced()
for that instead.See also the function
PyUnstable_Object_IsUniqueReferencedTemporary()
.Changed in version 3.10:
Py_REFCNT()
is changed to the inline static function.Changed in version 3.11: The parameter type is no longer const PyObject*.
-
void Py_SET_REFCNT(PyObject *o, Py_ssize_t refcnt)¶
Set the object o reference counter to refcnt.
On Python build with Free Threading, if refcnt is larger than
UINT32_MAX
, the object is made immortal.This function has no effect on immortal objects.
Added in version 3.9.
Changed in version 3.12: Immortal objects are not modified.
-
void Py_INCREF(PyObject *o)¶
Indicate taking a new strong reference to object o, indicating it is in use and should not be destroyed.
This function has no effect on immortal objects.
This function is usually used to convert a borrowed reference to a strong reference in-place. The
Py_NewRef()
function can be used to create a new strong reference.When done using the object, release is by calling
Py_DECREF()
.The object must not be
NULL
; if you aren’t sure that it isn’tNULL
, usePy_XINCREF()
.Do not expect this function to actually modify o in any way. For at least some objects, this function has no effect.
Changed in version 3.12: Immortal objects are not modified.
-
void Py_XINCREF(PyObject *o)¶
Similar to
Py_INCREF()
, but the object o can beNULL
, in which case this has no effect.See also
Py_XNewRef()
.
-
PyObject *Py_NewRef(PyObject *o)¶
- Part of the Stable ABI since version 3.10.
Create a new strong reference to an object: call
Py_INCREF()
on o and return the object o.When the strong reference is no longer needed,
Py_DECREF()
should be called on it to release the reference.The object o must not be
NULL
; usePy_XNewRef()
if o can beNULL
.For example:
Py_INCREF(obj); self->attr = obj;
can be written as:
self->attr = Py_NewRef(obj);
See also
Py_INCREF()
.Added in version 3.10.
-
PyObject *Py_XNewRef(PyObject *o)¶
- Part of the Stable ABI since version 3.10.
Similar to
Py_NewRef()
, but the object o can be NULL.If the object o is
NULL
, the function just returnsNULL
.Added in version 3.10.
-
void Py_DECREF(PyObject *o)¶
Release a strong reference to object o, indicating the reference is no longer used.
This function has no effect on immortal objects.
Once the last strong reference is released (i.e. the object’s reference count reaches 0), the object’s type’s deallocation function (which must not be
NULL
) is invoked.This function is usually used to delete a strong reference before exiting its scope.
The object must not be
NULL
; if you aren’t sure that it isn’tNULL
, usePy_XDECREF()
.Do not expect this function to actually modify o in any way. For at least some objects, this function has no effect.
Warning
The deallocation function can cause arbitrary Python code to be invoked (e.g. when a class instance with a
__del__()
method is deallocated). While exceptions in such code are not propagated, the executed code has free access to all Python global variables. This means that any object that is reachable from a global variable should be in a consistent state beforePy_DECREF()
is invoked. For example, code to delete an object from a list should copy a reference to the deleted object in a temporary variable, update the list data structure, and then callPy_DECREF()
for the temporary variable.Changed in version 3.12: Immortal objects are not modified.
-
void Py_XDECREF(PyObject *o)¶
Similar to
Py_DECREF()
, but the object o can beNULL
, in which case this has no effect. The same warning fromPy_DECREF()
applies here as well.
-
void Py_CLEAR(PyObject *o)¶
Release a strong reference for object o. The object may be
NULL
, in which case the macro has no effect; otherwise the effect is the same as forPy_DECREF()
, except that the argument is also set toNULL
. The warning forPy_DECREF()
does not apply with respect to the object passed because the macro carefully uses a temporary variable and sets the argument toNULL
before releasing the reference.It is a good idea to use this macro whenever releasing a reference to an object that might be traversed during garbage collection.
Changed in version 3.12: The macro argument is now only evaluated once. If the argument has side effects, these are no longer duplicated.
-
void Py_IncRef(PyObject *o)¶
- Part of the Stable ABI.
Indicate taking a new strong reference to object o. A function version of
Py_XINCREF()
. It can be used for runtime dynamic embedding of Python.
-
void Py_DecRef(PyObject *o)¶
- Part of the Stable ABI.
Release a strong reference to object o. A function version of
Py_XDECREF()
. It can be used for runtime dynamic embedding of Python.
-
Py_SETREF(dst, src)¶
Macro safely releasing a strong reference to object dst and setting dst to src.
As in case of
Py_CLEAR()
, “the obvious” code can be deadly:Py_DECREF(dst); dst = src;
The safe way is:
Py_SETREF(dst, src);
That arranges to set dst to src before releasing the reference to the old value of dst, so that any code triggered as a side-effect of dst getting torn down no longer believes dst points to a valid object.
Added in version 3.6.
Changed in version 3.12: The macro arguments are now only evaluated once. If an argument has side effects, these are no longer duplicated.
-
Py_XSETREF(dst, src)¶
Variant of
Py_SETREF
macro that usesPy_XDECREF()
instead ofPy_DECREF()
.Added in version 3.6.
Changed in version 3.12: The macro arguments are now only evaluated once. If an argument has side effects, these are no longer duplicated.


Extending

Extending¶
The main direction for asyncio
extending is writing custom event loop
classes. Asyncio has helpers that could be used to simplify this task.
Note
Third-parties should reuse existing asyncio code with caution, a new Python version is free to break backward compatibility in internal part of API.
Writing a Custom Event Loop¶
asyncio.AbstractEventLoop
declares very many methods. Implementing all them
from scratch is a tedious job.
A loop can get many common methods implementation for free by inheriting from
asyncio.BaseEventLoop
.
In turn, the successor should implement a bunch of private methods declared but not
implemented in asyncio.BaseEventLoop
.
For example, loop.create_connection()
checks arguments, resolves DNS addresses, and
calls loop._make_socket_transport()
that should be implemented by inherited class.
The _make_socket_transport()
method is not documented and is considered as an
internal API.
Future and Task private constructors¶
asyncio.Future
and asyncio.Task
should be never created directly,
please use corresponding loop.create_future()
and loop.create_task()
,
or asyncio.create_task()
factories instead.
However, third-party event loops may reuse built-in future and task implementations for the sake of getting a complex and highly optimized code for free.
For this purpose the following, private constructors are listed:
- Future.__init__(*, loop=None)¶
Create a built-in future instance.
loop is an optional event loop instance.
- Task.__init__(coro, *, loop=None, name=None, context=None)¶
Create a built-in task instance.
loop is an optional event loop instance. The rest of arguments are described in
loop.create_task()
description.Changed in version 3.11: context argument is added.
Task lifetime support¶
A third party task implementation should call the following functions to keep a task
visible by asyncio.all_tasks()
and asyncio.current_task()
:
- asyncio._register_task(task)¶
Register a new task as managed by asyncio.
Call the function from a task constructor.
- asyncio._unregister_task(task)¶
Unregister a task from asyncio internal structures.
The function should be called when a task is about to finish.
- asyncio._enter_task(loop, task)¶
Switch the current task to the task argument.
Call the function just before executing a portion of embedded coroutine (
coroutine.send()
orcoroutine.throw()
).
- asyncio._leave_task(loop, task)¶
Switch the current task back from task to
None
.Call the function just after
coroutine.send()
orcoroutine.throw()
execution.


ossaudiodev
— Access to OSS-compatible audio devices¶
Deprecated since version 3.11, removed in version 3.13.
This module is no longer part of the Python standard library. It was removed in Python 3.13 after being deprecated in Python 3.11. The removal was decided in PEP 594.
The last version of Python that provided the ossaudiodev
module was
Python 3.12.


Objects for Type Hinting

Objects for Type Hinting¶
Various built-in types for type hinting are provided. Currently,
two types exist – GenericAlias and
Union. Only GenericAlias
is exposed to C.
-
PyObject *Py_GenericAlias(PyObject *origin, PyObject *args)¶
- Part of the Stable ABI since version 3.9.
Create a GenericAlias object. Equivalent to calling the Python class
types.GenericAlias
. The origin and args arguments set theGenericAlias
‘s__origin__
and__args__
attributes respectively. origin should be a PyTypeObject*, and args can be a PyTupleObject* or anyPyObject*
. If args passed is not a tuple, a 1-tuple is automatically constructed and__args__
is set to(args,)
. Minimal checking is done for the arguments, so the function will succeed even if origin is not a type. TheGenericAlias
‘s__parameters__
attribute is constructed lazily from__args__
. On failure, an exception is raised andNULL
is returned.Here’s an example of how to make an extension type generic:
... static PyMethodDef my_obj_methods[] = { // Other methods. ... {"__class_getitem__", Py_GenericAlias, METH_O|METH_CLASS, "See PEP 585"} ... }
See also
The data model method
__class_getitem__()
.Added in version 3.9.
-
PyTypeObject Py_GenericAliasType¶
- Part of the Stable ABI since version 3.9.
The C type of the object returned by
Py_GenericAlias()
. Equivalent totypes.GenericAlias
in Python.Added in version 3.9.


Installing Python Modules

Installing Python Modules¶
- Email:
As a popular open source development project, Python has an active supporting community of contributors and users that also make their software available for other Python developers to use under open source license terms.
This allows Python users to share and collaborate effectively, benefiting from the solutions others have already created to common (and sometimes even rare!) problems, as well as potentially contributing their own solutions to the common pool.
This guide covers the installation part of the process. For a guide to creating and sharing your own Python projects, refer to the Python packaging user guide.
Note
For corporate and other institutional users, be aware that many organisations have their own policies around using and contributing to open source software. Please take such policies into account when making use of the distribution and installation tools provided with Python.
Key terms¶
pip
is the preferred installer program. Starting with Python 3.4, it is included by default with the Python binary installers.A virtual environment is a semi-isolated Python environment that allows packages to be installed for use by a particular application, rather than being installed system wide.
venv
is the standard tool for creating virtual environments, and has been part of Python since Python 3.3. Starting with Python 3.4, it defaults to installingpip
into all created virtual environments.virtualenv
is a third party alternative (and predecessor) tovenv
. It allows virtual environments to be used on versions of Python prior to 3.4, which either don’t providevenv
at all, or aren’t able to automatically installpip
into created environments.The Python Package Index is a public repository of open source licensed packages made available for use by other Python users.
the Python Packaging Authority is the group of developers and documentation authors responsible for the maintenance and evolution of the standard packaging tools and the associated metadata and file format standards. They maintain a variety of tools, documentation, and issue trackers on GitHub.
distutils
is the original build and distribution system first added to the Python standard library in 1998. While direct use ofdistutils
is being phased out, it still laid the foundation for the current packaging and distribution infrastructure, and it not only remains part of the standard library, but its name lives on in other ways (such as the name of the mailing list used to coordinate Python packaging standards development).
Changed in version 3.5: The use of venv
is now recommended for creating virtual environments.
Basic usage¶
The standard packaging tools are all designed to be used from the command line.
The following command will install the latest version of a module and its dependencies from the Python Package Index:
python -m pip install SomePackage
Note
For POSIX users (including macOS and Linux users), the examples in this guide assume the use of a virtual environment.
For Windows users, the examples in this guide assume that the option to adjust the system PATH environment variable was selected when installing Python.
It’s also possible to specify an exact or minimum version directly on the
command line. When using comparator operators such as >
, <
or some other
special character which get interpreted by shell, the package name and the
version should be enclosed within double quotes:
python -m pip install SomePackage==1.0.4 # specific version
python -m pip install "SomePackage>=1.0.4" # minimum version
Normally, if a suitable module is already installed, attempting to install it again will have no effect. Upgrading existing modules must be requested explicitly:
python -m pip install --upgrade SomePackage
More information and resources regarding pip
and its capabilities can be
found in the Python Packaging User Guide.
Creation of virtual environments is done through the venv
module.
Installing packages into an active virtual environment uses the commands shown
above.
How do I …?¶
These are quick answers or links for some common tasks.
… install pip
in versions of Python prior to Python 3.4?¶
Python only started bundling pip
with Python 3.4. For earlier versions,
pip
needs to be “bootstrapped” as described in the Python Packaging
User Guide.
… install packages just for the current user?¶
Passing the --user
option to python -m pip install
will install a
package just for the current user, rather than for all users of the system.
… install scientific Python packages?¶
A number of scientific Python packages have complex binary dependencies, and
aren’t currently easy to install using pip
directly. At this point in
time, it will often be easier for users to install these packages by
other means
rather than attempting to install them with pip
.
… work with multiple versions of Python installed in parallel?¶
On Linux, macOS, and other POSIX systems, use the versioned Python commands
in combination with the -m
switch to run the appropriate copy of
pip
:
python2 -m pip install SomePackage # default Python 2
python2.7 -m pip install SomePackage # specifically Python 2.7
python3 -m pip install SomePackage # default Python 3
python3.4 -m pip install SomePackage # specifically Python 3.4
Appropriately versioned pip
commands may also be available.
On Windows, use the py
Python launcher in combination with the -m
switch:
py -2 -m pip install SomePackage # default Python 2
py -2.7 -m pip install SomePackage # specifically Python 2.7
py -3 -m pip install SomePackage # default Python 3
py -3.4 -m pip install SomePackage # specifically Python 3.4
Common installation issues¶
Installing into the system Python on Linux¶
On Linux systems, a Python installation will typically be included as part
of the distribution. Installing into this Python installation requires
root access to the system, and may interfere with the operation of the
system package manager and other components of the system if a component
is unexpectedly upgraded using pip
.
On such systems, it is often better to use a virtual environment or a
per-user installation when installing packages with pip
.
Pip not installed¶
It is possible that pip
does not get installed by default. One potential fix is:
python -m ensurepip --default-pip
There are also additional resources for installing pip.
Installing binary extensions¶
Python has typically relied heavily on source based distribution, with end users being expected to compile extension modules from source as part of the installation process.
With the introduction of support for the binary wheel
format, and the
ability to publish wheels for at least Windows and macOS through the
Python Package Index, this problem is expected to diminish over time,
as users are more regularly able to install pre-built extensions rather
than needing to build them themselves.
Some of the solutions for installing scientific software
that are not yet available as pre-built wheel
files may also help with
obtaining other binary extensions without needing to build them locally.


Boolean Objects

Boolean Objects¶
Booleans in Python are implemented as a subclass of integers. There are only
two booleans, Py_False
and Py_True
. As such, the normal
creation and deletion functions don’t apply to booleans. The following macros
are available, however.
-
PyTypeObject PyBool_Type¶
- Part of the Stable ABI.
This instance of
PyTypeObject
represents the Python boolean type; it is the same object asbool
in the Python layer.
-
int PyBool_Check(PyObject *o)¶
Return true if o is of type
PyBool_Type
. This function always succeeds.
-
PyObject *PyBool_FromLong(long v)¶
- Return value: New reference. Part of the Stable ABI.
Return
Py_True
orPy_False
, depending on the truth value of v.


Errors and Exceptions

8. Errors and Exceptions¶
Until now error messages haven’t been more than mentioned, but if you have tried out the examples you have probably seen some. There are (at least) two distinguishable kinds of errors: syntax errors and exceptions.
8.1. Syntax Errors¶
Syntax errors, also known as parsing errors, are perhaps the most common kind of complaint you get while you are still learning Python:
>>> while True print('Hello world')
File "<stdin>", line 1
while True print('Hello world')
^^^^^
SyntaxError: invalid syntax
The parser repeats the offending line and displays little arrows pointing
at the place where the error was detected. Note that this is not always the
place that needs to be fixed. In the example, the error is detected at the
function print()
, since a colon (':'
) is missing just before it.
The file name (<stdin>
in our example) and line number are printed so you
know where to look in case the input came from a file.
8.2. Exceptions¶
Even if a statement or expression is syntactically correct, it may cause an error when an attempt is made to execute it. Errors detected during execution are called exceptions and are not unconditionally fatal: you will soon learn how to handle them in Python programs. Most exceptions are not handled by programs, however, and result in error messages as shown here:
>>> 10 * (1/0)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
10 * (1/0)
~^~
ZeroDivisionError: division by zero
>>> 4 + spam*3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
4 + spam*3
^^^^
NameError: name 'spam' is not defined
>>> '2' + 2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
'2' + 2
~~~~^~~
TypeError: can only concatenate str (not "int") to str
The last line of the error message indicates what happened. Exceptions come in
different types, and the type is printed as part of the message: the types in
the example are ZeroDivisionError
, NameError
and TypeError
.
The string printed as the exception type is the name of the built-in exception
that occurred. This is true for all built-in exceptions, but need not be true
for user-defined exceptions (although it is a useful convention). Standard
exception names are built-in identifiers (not reserved keywords).
The rest of the line provides detail based on the type of exception and what caused it.
The preceding part of the error message shows the context where the exception occurred, in the form of a stack traceback. In general it contains a stack traceback listing source lines; however, it will not display lines read from standard input.
Built-in Exceptions lists the built-in exceptions and their meanings.
8.3. Handling Exceptions¶
It is possible to write programs that handle selected exceptions. Look at the
following example, which asks the user for input until a valid integer has been
entered, but allows the user to interrupt the program (using Control-C or
whatever the operating system supports); note that a user-generated interruption
is signalled by raising the KeyboardInterrupt
exception.
>>> while True:
... try:
... x = int(input("Please enter a number: "))
... break
... except ValueError:
... print("Oops! That was no valid number. Try again...")
...
The try
statement works as follows.
First, the try clause (the statement(s) between the
try
andexcept
keywords) is executed.If no exception occurs, the except clause is skipped and execution of the
try
statement is finished.If an exception occurs during execution of the
try
clause, the rest of the clause is skipped. Then, if its type matches the exception named after theexcept
keyword, the except clause is executed, and then execution continues after the try/except block.If an exception occurs which does not match the exception named in the except clause, it is passed on to outer
try
statements; if no handler is found, it is an unhandled exception and execution stops with an error message.
A try
statement may have more than one except clause, to specify
handlers for different exceptions. At most one handler will be executed.
Handlers only handle exceptions that occur in the corresponding try clause,
not in other handlers of the same try
statement. An except clause
may name multiple exceptions as a parenthesized tuple, for example:
... except (RuntimeError, TypeError, NameError):
... pass
A class in an except
clause matches exceptions which are instances of the
class itself or one of its derived classes (but not the other way around — an
except clause listing a derived class does not match instances of its base classes).
For example, the following code will print B, C, D in that order:
class B(Exception):
pass
class C(B):
pass
class D(C):
pass
for cls in [B, C, D]:
try:
raise cls()
except D:
print("D")
except C:
print("C")
except B:
print("B")
Note that if the except clauses were reversed (with except B
first), it
would have printed B, B, B — the first matching except clause is triggered.
When an exception occurs, it may have associated values, also known as the exception’s arguments. The presence and types of the arguments depend on the exception type.
The except clause may specify a variable after the exception name. The
variable is bound to the exception instance which typically has an args
attribute that stores the arguments. For convenience, builtin exception
types define __str__()
to print all the arguments without explicitly
accessing .args
.
>>> try:
... raise Exception('spam', 'eggs')
... except Exception as inst:
... print(type(inst)) # the exception type
... print(inst.args) # arguments stored in .args
... print(inst) # __str__ allows args to be printed directly,
... # but may be overridden in exception subclasses
... x, y = inst.args # unpack args
... print('x =', x)
... print('y =', y)
...
<class 'Exception'>
('spam', 'eggs')
('spam', 'eggs')
x = spam
y = eggs
The exception’s __str__()
output is printed as the last part (‘detail’)
of the message for unhandled exceptions.
BaseException
is the common base class of all exceptions. One of its
subclasses, Exception
, is the base class of all the non-fatal exceptions.
Exceptions which are not subclasses of Exception
are not typically
handled, because they are used to indicate that the program should terminate.
They include SystemExit
which is raised by sys.exit()
and
KeyboardInterrupt
which is raised when a user wishes to interrupt
the program.
Exception
can be used as a wildcard that catches (almost) everything.
However, it is good practice to be as specific as possible with the types
of exceptions that we intend to handle, and to allow any unexpected
exceptions to propagate on.
The most common pattern for handling Exception
is to print or log
the exception and then re-raise it (allowing a caller to handle the
exception as well):
import sys
try:
f = open('myfile.txt')
s = f.readline()
i = int(s.strip())
except OSError as err:
print("OS error:", err)
except ValueError:
print("Could not convert data to an integer.")
except Exception as err:
print(f"Unexpected {err=}, {type(err)=}")
raise
The try
… except
statement has an optional else
clause, which, when present, must follow all except clauses. It is useful
for code that must be executed if the try clause does not raise an exception.
For example:
for arg in sys.argv[1:]:
try:
f = open(arg, 'r')
except OSError:
print('cannot open', arg)
else:
print(arg, 'has', len(f.readlines()), 'lines')
f.close()
The use of the else
clause is better than adding additional code to
the try
clause because it avoids accidentally catching an exception
that wasn’t raised by the code being protected by the try
…
except
statement.
Exception handlers do not handle only exceptions that occur immediately in the try clause, but also those that occur inside functions that are called (even indirectly) in the try clause. For example:
>>> def this_fails():
... x = 1/0
...
>>> try:
... this_fails()
... except ZeroDivisionError as err:
... print('Handling run-time error:', err)
...
Handling run-time error: division by zero
8.4. Raising Exceptions¶
The raise
statement allows the programmer to force a specified
exception to occur. For example:
>>> raise NameError('HiThere')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
raise NameError('HiThere')
NameError: HiThere
The sole argument to raise
indicates the exception to be raised.
This must be either an exception instance or an exception class (a class that
derives from BaseException
, such as Exception
or one of its
subclasses). If an exception class is passed, it will be implicitly
instantiated by calling its constructor with no arguments:
raise ValueError # shorthand for 'raise ValueError()'
If you need to determine whether an exception was raised but don’t intend to
handle it, a simpler form of the raise
statement allows you to
re-raise the exception:
>>> try:
... raise NameError('HiThere')
... except NameError:
... print('An exception flew by!')
... raise
...
An exception flew by!
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
raise NameError('HiThere')
NameError: HiThere
8.5. Exception Chaining¶
If an unhandled exception occurs inside an except
section, it will
have the exception being handled attached to it and included in the error
message:
>>> try:
... open("database.sqlite")
... except OSError:
... raise RuntimeError("unable to handle error")
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
open("database.sqlite")
~~~~^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'database.sqlite'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
raise RuntimeError("unable to handle error")
RuntimeError: unable to handle error
To indicate that an exception is a direct consequence of another, the
raise
statement allows an optional from
clause:
# exc must be exception instance or None.
raise RuntimeError from exc
This can be useful when you are transforming exceptions. For example:
>>> def func():
... raise ConnectionError
...
>>> try:
... func()
... except ConnectionError as exc:
... raise RuntimeError('Failed to open database') from exc
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
func()
~~~~^^
File "<stdin>", line 2, in func
ConnectionError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
raise RuntimeError('Failed to open database') from exc
RuntimeError: Failed to open database
It also allows disabling automatic exception chaining using the from None
idiom:
>>> try:
... open('database.sqlite')
... except OSError:
... raise RuntimeError from None
...
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
raise RuntimeError from None
RuntimeError
For more information about chaining mechanics, see Built-in Exceptions.
8.6. User-defined Exceptions¶
Programs may name their own exceptions by creating a new exception class (see
Classes for more about Python classes). Exceptions should typically
be derived from the Exception
class, either directly or indirectly.
Exception classes can be defined which do anything any other class can do, but are usually kept simple, often only offering a number of attributes that allow information about the error to be extracted by handlers for the exception.
Most exceptions are defined with names that end in “Error”, similar to the naming of the standard exceptions.
Many standard modules define their own exceptions to report errors that may occur in functions they define.
8.7. Defining Clean-up Actions¶
The try
statement has another optional clause which is intended to
define clean-up actions that must be executed under all circumstances. For
example:
>>> try:
... raise KeyboardInterrupt
... finally:
... print('Goodbye, world!')
...
Goodbye, world!
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
raise KeyboardInterrupt
KeyboardInterrupt
If a finally
clause is present, the finally
clause will execute as the last task before the try
statement completes. The finally
clause runs whether or
not the try
statement produces an exception. The following
points discuss more complex cases when an exception occurs:
If an exception occurs during execution of the
try
clause, the exception may be handled by anexcept
clause. If the exception is not handled by anexcept
clause, the exception is re-raised after thefinally
clause has been executed.An exception could occur during execution of an
except
orelse
clause. Again, the exception is re-raised after thefinally
clause has been executed.If the
finally
clause executes abreak
,continue
orreturn
statement, exceptions are not re-raised. This can be confusing and is therefore discouraged. From version 3.14 the compiler emits aSyntaxWarning
for it (see PEP 765).If the
try
statement reaches abreak
,continue
orreturn
statement, thefinally
clause will execute just prior to thebreak
,continue
orreturn
statement’s execution.If a
finally
clause includes areturn
statement, the returned value will be the one from thefinally
clause’sreturn
statement, not the value from thetry
clause’sreturn
statement. This can be confusing and is therefore discouraged. From version 3.14 the compiler emits aSyntaxWarning
for it (see PEP 765).
For example:
>>> def bool_return():
... try:
... return True
... finally:
... return False
...
>>> bool_return()
False
A more complicated example:
>>> def divide(x, y):
... try:
... result = x / y
... except ZeroDivisionError:
... print("division by zero!")
... else:
... print("result is", result)
... finally:
... print("executing finally clause")
...
>>> divide(2, 1)
result is 2.0
executing finally clause
>>> divide(2, 0)
division by zero!
executing finally clause
>>> divide("2", "1")
executing finally clause
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
divide("2", "1")
~~~~~~^^^^^^^^^^
File "<stdin>", line 3, in divide
result = x / y
~~^~~
TypeError: unsupported operand type(s) for /: 'str' and 'str'
As you can see, the finally
clause is executed in any event. The
TypeError
raised by dividing two strings is not handled by the
except
clause and therefore re-raised after the finally
clause has been executed.
In real world applications, the finally
clause is useful for
releasing external resources (such as files or network connections), regardless
of whether the use of the resource was successful.
8.8. Predefined Clean-up Actions¶
Some objects define standard clean-up actions to be undertaken when the object is no longer needed, regardless of whether or not the operation using the object succeeded or failed. Look at the following example, which tries to open a file and print its contents to the screen.
for line in open("myfile.txt"):
print(line, end="")
The problem with this code is that it leaves the file open for an indeterminate
amount of time after this part of the code has finished executing.
This is not an issue in simple scripts, but can be a problem for larger
applications. The with
statement allows objects like files to be
used in a way that ensures they are always cleaned up promptly and correctly.
with open("myfile.txt") as f:
for line in f:
print(line, end="")
After the statement is executed, the file f is always closed, even if a problem was encountered while processing the lines. Objects which, like files, provide predefined clean-up actions will indicate this in their documentation.
8.10. Enriching Exceptions with Notes¶
When an exception is created in order to be raised, it is usually initialized
with information that describes the error that has occurred. There are cases
where it is useful to add information after the exception was caught. For this
purpose, exceptions have a method add_note(note)
that accepts a string and
adds it to the exception’s notes list. The standard traceback rendering
includes all notes, in the order they were added, after the exception.
>>> try:
... raise TypeError('bad type')
... except Exception as e:
... e.add_note('Add some information')
... e.add_note('Add some more information')
... raise
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
raise TypeError('bad type')
TypeError: bad type
Add some information
Add some more information
>>>
For example, when collecting exceptions into an exception group, we may want to add context information for the individual errors. In the following each exception in the group has a note indicating when this error has occurred.
>>> def f():
... raise OSError('operation failed')
...
>>> excs = []
>>> for i in range(3):
... try:
... f()
... except Exception as e:
... e.add_note(f'Happened in Iteration {i+1}')
... excs.append(e)
...
>>> raise ExceptionGroup('We have some problems', excs)
+ Exception Group Traceback (most recent call last):
| File "<stdin>", line 1, in <module>
| raise ExceptionGroup('We have some problems', excs)
| ExceptionGroup: We have some problems (3 sub-exceptions)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "<stdin>", line 3, in <module>
| f()
| ~^^
| File "<stdin>", line 2, in f
| raise OSError('operation failed')
| OSError: operation failed
| Happened in Iteration 1
+---------------- 2 ----------------
| Traceback (most recent call last):
| File "<stdin>", line 3, in <module>
| f()
| ~^^
| File "<stdin>", line 2, in f
| raise OSError('operation failed')
| OSError: operation failed
| Happened in Iteration 2
+---------------- 3 ----------------
| Traceback (most recent call last):
| File "<stdin>", line 3, in <module>
| f()
| ~^^
| File "<stdin>", line 2, in f
| raise OSError('operation failed')
| OSError: operation failed
| Happened in Iteration 3
+------------------------------------
>>>

File
, line
:
File
, line
, in
:
File
, line
, in
:
File
, line
, in
:
File
, line
, in
:
File
, line
, in
:
File
, line
, in
:
File
, line
, in
:
File
, line
, in
File
, line
, in
File
, line
, in
:
File
, line
, in
File
, line
, in
File
, line
, in
File
, line
, in
:
File
, line
, in
:


Python support for free threading

Python support for free threading¶
Starting with the 3.13 release, CPython has support for a build of Python called free threading where the global interpreter lock (GIL) is disabled. Free-threaded execution allows for full utilization of the available processing power by running threads in parallel on available CPU cores. While not all software will benefit from this automatically, programs designed with threading in mind will run faster on multi-core hardware.
Some third-party packages, in particular ones with an extension module, may not be ready for use in a free-threaded build, and will re-enable the GIL.
This document describes the implications of free threading for Python code. See C API Extension Support for Free Threading for information on how to write C extensions that support the free-threaded build.
See also
PEP 703 – Making the Global Interpreter Lock Optional in CPython for an overall description of free-threaded Python.
Installation¶
Starting with Python 3.13, the official macOS and Windows installers optionally support installing free-threaded Python binaries. The installers are available at https://www.python.org/downloads/.
For information on other platforms, see the Installing a Free-Threaded Python, a community-maintained installation guide for installing free-threaded Python.
When building CPython from source, the --disable-gil
configure option
should be used to build a free-threaded Python interpreter.
Identifying free-threaded Python¶
To check if the current interpreter supports free-threading, python -VV
and sys.version
contain “free-threading build”.
The new sys._is_gil_enabled()
function can be used to check whether
the GIL is actually disabled in the running process.
The sysconfig.get_config_var("Py_GIL_DISABLED")
configuration variable can
be used to determine whether the build supports free threading. If the variable
is set to 1
, then the build supports free threading. This is the recommended
mechanism for decisions related to the build configuration.
The global interpreter lock in free-threaded Python¶
Free-threaded builds of CPython support optionally running with the GIL enabled
at runtime using the environment variable PYTHON_GIL
or
the command-line option -X gil
.
The GIL may also automatically be enabled when importing a C-API extension module that is not explicitly marked as supporting free threading. A warning will be printed in this case.
In addition to individual package documentation, the following websites track the status of popular packages support for free threading:
Thread safety¶
The free-threaded build of CPython aims to provide similar thread-safety
behavior at the Python level to the default GIL-enabled build. Built-in
types like dict
, list
, and set
use internal locks
to protect against concurrent modifications in ways that behave similarly to
the GIL. However, Python has not historically guaranteed specific behavior for
concurrent modifications to these built-in types, so this should be treated
as a description of the current implementation, not a guarantee of current or
future behavior.
Note
It’s recommended to use the threading.Lock
or other synchronization
primitives instead of relying on the internal locks of built-in types, when
possible.
Known limitations¶
This section describes known limitations of the free-threaded CPython build.
Immortalization¶
In the free-threaded build, some objects are immortal. Immortal objects are not deallocated and have reference counts that are never modified. This is done to avoid reference count contention that would prevent efficient multi-threaded scaling.
As of the 3.14 release, immortalization is limited to:
Code constants: numeric literals, string literals, and tuple literals composed of other constants.
Strings interned by
sys.intern()
.
Frame objects¶
It is not safe to access frame.f_locals
from a frame
object if that frame is currently executing in another thread, and doing so may
crash the interpreter.
Iterators¶
It is generally not thread-safe to access the same iterator object from multiple threads concurrently, and threads may see duplicate or missing elements.
Single-threaded performance¶
The free-threaded build has additional overhead when executing Python code compared to the default GIL-enabled build. The amount of overhead depends on the workload and hardware. On the pyperformance benchmark suite, the average overhead ranges from about 1% on macOS aarch64 to 8% on x86-64 Linux systems.
Behavioral changes¶
This section describes CPython behavioural changes with the free-threaded build.
Context variables¶
In the free-threaded build, the flag thread_inherit_context
is set to true by default which causes threads created with
threading.Thread
to start with a copy of the
Context()
of the caller of
start()
. In the default GIL-enabled build, the flag
defaults to false so threads start with an
empty Context()
.
Warning filters¶
In the free-threaded build, the flag context_aware_warnings
is set to true by default. In the default GIL-enabled build, the flag defaults
to false. If the flag is true then the warnings.catch_warnings
context manager uses a context variable for warning filters. If the flag is
false then catch_warnings
modifies the global filters list,
which is not thread-safe. See the warnings
module for more details.


: Internationalized headers

email.header
: Internationalized headers¶
Source code: Lib/email/header.py
This module is part of the legacy (Compat32
) email API. In the current API
encoding and decoding of headers is handled transparently by the
dictionary-like API of the EmailMessage
class. In
addition to uses in legacy code, this module can be useful in applications that
need to completely control the character sets used when encoding headers.
The remaining text in this section is the original documentation of the module.
RFC 2822 is the base standard that describes the format of email messages. It derives from the older RFC 822 standard which came into widespread use at a time when most email was composed of ASCII characters only. RFC 2822 is a specification written assuming email contains only 7-bit ASCII characters.
Of course, as email has been deployed worldwide, it has become
internationalized, such that language specific character sets can now be used in
email messages. The base standard still requires email messages to be
transferred using only 7-bit ASCII characters, so a slew of RFCs have been
written describing how to encode email containing non-ASCII characters into
RFC 2822-compliant format. These RFCs include RFC 2045, RFC 2046,
RFC 2047, and RFC 2231. The email
package supports these standards
in its email.header
and email.charset
modules.
If you want to include non-ASCII characters in your email headers, say in the
Subject or To fields, you should use the
Header
class and assign the field in the Message
object to an instance of Header
instead of using a string for the header
value. Import the Header
class from the email.header
module.
For example:
>>> from email.message import Message
>>> from email.header import Header
>>> msg = Message()
>>> h = Header('p\xf6stal', 'iso-8859-1')
>>> msg['Subject'] = h
>>> msg.as_string()
'Subject: =?iso-8859-1?q?p=F6stal?=\n\n'
Notice here how we wanted the Subject field to contain a non-ASCII
character? We did this by creating a Header
instance and passing in
the character set that the byte string was encoded in. When the subsequent
Message
instance was flattened, the Subject
field was properly RFC 2047 encoded. MIME-aware mail readers would show this
header using the embedded ISO-8859-1 character.
Here is the Header
class description:
- class email.header.Header(s=None, charset=None, maxlinelen=None, header_name=None, continuation_ws=' ', errors='strict')¶
Create a MIME-compliant header that can contain strings in different character sets.
Optional s is the initial header value. If
None
(the default), the initial header value is not set. You can later append to the header withappend()
method calls. s may be an instance ofbytes
orstr
, but see theappend()
documentation for semantics.Optional charset serves two purposes: it has the same meaning as the charset argument to the
append()
method. It also sets the default character set for all subsequentappend()
calls that omit the charset argument. If charset is not provided in the constructor (the default), theus-ascii
character set is used both as s’s initial charset and as the default for subsequentappend()
calls.The maximum line length can be specified explicitly via maxlinelen. For splitting the first line to a shorter value (to account for the field header which isn’t included in s, e.g. Subject) pass in the name of the field in header_name. The default maxlinelen is 78, and the default value for header_name is
None
, meaning it is not taken into account for the first line of a long, split header.Optional continuation_ws must be RFC 2822-compliant folding whitespace, and is usually either a space or a hard tab character. This character will be prepended to continuation lines. continuation_ws defaults to a single space character.
Optional errors is passed straight through to the
append()
method.- append(s, charset=None, errors='strict')¶
Append the string s to the MIME header.
Optional charset, if given, should be a
Charset
instance (seeemail.charset
) or the name of a character set, which will be converted to aCharset
instance. A value ofNone
(the default) means that the charset given in the constructor is used.s may be an instance of
bytes
orstr
. If it is an instance ofbytes
, then charset is the encoding of that byte string, and aUnicodeError
will be raised if the string cannot be decoded with that character set.If s is an instance of
str
, then charset is a hint specifying the character set of the characters in the string.In either case, when producing an RFC 2822-compliant header using RFC 2047 rules, the string will be encoded using the output codec of the charset. If the string cannot be encoded using the output codec, a UnicodeError will be raised.
Optional errors is passed as the errors argument to the decode call if s is a byte string.
- encode(splitchars=';, \t', maxlinelen=None, linesep='\n')¶
Encode a message header into an RFC-compliant format, possibly wrapping long lines and encapsulating non-ASCII parts in base64 or quoted-printable encodings.
Optional splitchars is a string containing characters which should be given extra weight by the splitting algorithm during normal header wrapping. This is in very rough support of RFC 2822's ‘higher level syntactic breaks’: split points preceded by a splitchar are preferred during line splitting, with the characters preferred in the order in which they appear in the string. Space and tab may be included in the string to indicate whether preference should be given to one over the other as a split point when other split chars do not appear in the line being split. Splitchars does not affect RFC 2047 encoded lines.
maxlinelen, if given, overrides the instance’s value for the maximum line length.
linesep specifies the characters used to separate the lines of the folded header. It defaults to the most useful value for Python application code (
\n
), but\r\n
can be specified in order to produce headers with RFC-compliant line separators.Changed in version 3.2: Added the linesep argument.
The
Header
class also provides a number of methods to support standard operators and built-in functions.- __str__()¶
Returns an approximation of the
Header
as a string, using an unlimited line length. All pieces are converted to unicode using the specified encoding and joined together appropriately. Any pieces with a charset of'unknown-8bit'
are decoded as ASCII using the'replace'
error handler.Changed in version 3.2: Added handling for the
'unknown-8bit'
charset.
The email.header
module also provides the following convenient functions.
- email.header.decode_header(header)¶
Decode a message header value without converting the character set. The header value is in header.
For historical reasons, this function may return either:
A list of pairs containing each of the decoded parts of the header,
(decoded_bytes, charset)
, where decoded_bytes is always an instance ofbytes
, and charset is either:A lower case string containing the name of the character set specified.
None
for non-encoded parts of the header.
A list of length 1 containing a pair
(string, None)
, where string is always an instance ofstr
.
An
email.errors.HeaderParseError
may be raised when certain decoding errors occur (e.g. a base64 decoding exception).Here are examples:
>>> from email.header import decode_header >>> decode_header('=?iso-8859-1?q?p=F6stal?=') [(b'p\xf6stal', 'iso-8859-1')] >>> decode_header('unencoded_string') [('unencoded_string', None)] >>> decode_header('bar =?utf-8?B?ZsOzbw==?=') [(b'bar ', None), (b'f\xc3\xb3o', 'utf-8')]
Note
This function exists for backwards compatibility only. For new code, we recommend using
email.headerregistry.HeaderRegistry
.
- email.header.make_header(decoded_seq, maxlinelen=None, header_name=None, continuation_ws=' ')¶
Create a
Header
instance from a sequence of pairs as returned bydecode_header()
.decode_header()
takes a header value string and returns a sequence of pairs of the format(decoded_string, charset)
where charset is the name of the character set.This function takes one of those sequence of pairs and returns a
Header
instance. Optional maxlinelen, header_name, and continuation_ws are as in theHeader
constructor.Note
This function exists for backwards compatibility only, and is not recommended for use in new code.