— Byte-compile Python libraries

compileall
— Byte-compile Python libraries¶
Source code: Lib/compileall.py
This module provides some utility functions to support installing Python libraries. These functions compile Python source files in a directory tree. This module can be used to create the cached byte-code files at library installation time, which makes them available for use even by users who don’t have write permission to the library directories.
Availability: not WASI.
This module does not work or is not available on WebAssembly. See WebAssembly platforms for more information.
Command-line use¶
This module can work as a script (using python -m compileall) to compile Python sources.
- directory ...¶
- file ...¶
Positional arguments are files to compile or directories that contain source files, traversed recursively. If no argument is given, behave as if the command line was
-l <directories from sys.path>
.
- -l¶
Do not recurse into subdirectories, only compile source code files directly contained in the named or implied directories.
- -f¶
Force rebuild even if timestamps are up-to-date.
- -q¶
Do not print the list of files compiled. If passed once, error messages will still be printed. If passed twice (
-qq
), all output is suppressed.
- -d destdir¶
Directory prepended to the path to each file being compiled. This will appear in compilation time tracebacks, and is also compiled in to the byte-code file, where it will be used in tracebacks and other messages in cases where the source file does not exist at the time the byte-code file is executed.
- -s strip_prefix¶
Remove the given prefix from paths recorded in the
.pyc
files. Paths are made relative to the prefix.This option can be used with
-p
but not with-d
.
- -p prepend_prefix¶
Prepend the given prefix to paths recorded in the
.pyc
files. Use-p /
to make the paths absolute.This option can be used with
-s
but not with-d
.
- -x regex¶
regex is used to search the full path to each file considered for compilation, and if the regex produces a match, the file is skipped.
- -i list¶
Read the file
list
and add each line that it contains to the list of files and directories to compile. Iflist
is-
, read lines fromstdin
.
- -b¶
Write the byte-code files to their legacy locations and names, which may overwrite byte-code files created by another version of Python. The default is to write files to their PEP 3147 locations and names, which allows byte-code files from multiple versions of Python to coexist.
- -r¶
Control the maximum recursion level for subdirectories. If this is given, then
-l
option will not be taken into account. python -m compileall <directory> -r 0 is equivalent to python -m compileall <directory> -l.
- -j N¶
Use N workers to compile the files within the given directory. If
0
is used, then the result ofos.process_cpu_count()
will be used.
- --invalidation-mode [timestamp|checked-hash|unchecked-hash]¶
Control how the generated byte-code files are invalidated at runtime. The
timestamp
value, means that.pyc
files with the source timestamp and size embedded will be generated. Thechecked-hash
andunchecked-hash
values cause hash-based pycs to be generated. Hash-based pycs embed a hash of the source file contents rather than a timestamp. See Cached bytecode invalidation for more information on how Python validates bytecode cache files at runtime. The default istimestamp
if theSOURCE_DATE_EPOCH
environment variable is not set, andchecked-hash
if theSOURCE_DATE_EPOCH
environment variable is set.
- -o level¶
Compile with the given optimization level. May be used multiple times to compile for multiple levels at a time (for example,
compileall -o 1 -o 2
).
- -e dir¶
Ignore symlinks pointing outside the given directory.
- --hardlink-dupes¶
If two
.pyc
files with different optimization level have the same content, use hard links to consolidate duplicate files.
Changed in version 3.2: Added the -i
, -b
and -h
options.
Changed in version 3.5: Added the -j
, -r
, and -qq
options. -q
option
was changed to a multilevel value. -b
will always produce a
byte-code file ending in .pyc
, never .pyo
.
Changed in version 3.7: Added the --invalidation-mode
option.
Changed in version 3.9: Added the -s
, -p
, -e
and --hardlink-dupes
options.
Raised the default recursion limit from 10 to
sys.getrecursionlimit()
.
Added the possibility to specify the -o
option multiple times.
There is no command-line option to control the optimization level used by the
compile()
function, because the Python interpreter itself already
provides the option: python -O -m compileall.
Similarly, the compile()
function respects the sys.pycache_prefix
setting. The generated bytecode cache will only be useful if compile()
is
run with the same sys.pycache_prefix
(if any) that will be used at
runtime.
Public functions¶
- compileall.compile_dir(dir, maxlevels=sys.getrecursionlimit(), ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False)¶
Recursively descend the directory tree named by dir, compiling all
.py
files along the way. Return a true value if all the files compiled successfully, and a false value otherwise.The maxlevels parameter is used to limit the depth of the recursion; it defaults to
sys.getrecursionlimit()
.If ddir is given, it is prepended to the path to each file being compiled for use in compilation time tracebacks, and is also compiled in to the byte-code file, where it will be used in tracebacks and other messages in cases where the source file does not exist at the time the byte-code file is executed.
If force is true, modules are re-compiled even if the timestamps are up to date.
If rx is given, its
search
method is called on the complete path to each file considered for compilation, and if it returns a true value, the file is skipped. This can be used to exclude files matching a regular expression, given as a re.Pattern object.If quiet is
False
or0
(the default), the filenames and other information are printed to standard out. Set to1
, only errors are printed. Set to2
, all output is suppressed.If legacy is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of Python. The default is to write files to their PEP 3147 locations and names, which allows byte-code files from multiple versions of Python to coexist.
optimize specifies the optimization level for the compiler. It is passed to the built-in
compile()
function. Accepts also a sequence of optimization levels which lead to multiple compilations of one.py
file in one call.The argument workers specifies how many workers are used to compile files in parallel. The default is to not use multiple workers. If the platform can’t use multiple workers and workers argument is given, then sequential compilation will be used as a fallback. If workers is 0, the number of cores in the system is used. If workers is lower than
0
, aValueError
will be raised.invalidation_mode should be a member of the
py_compile.PycInvalidationMode
enum and controls how the generated pycs are invalidated at runtime.The stripdir, prependdir and limit_sl_dest arguments correspond to the
-s
,-p
and-e
options described above. They may be specified asstr
oros.PathLike
.If hardlink_dupes is true and two
.pyc
files with different optimization level have the same content, use hard links to consolidate duplicate files.Changed in version 3.2: Added the legacy and optimize parameter.
Changed in version 3.5: Added the workers parameter.
Changed in version 3.5: quiet parameter was changed to a multilevel value.
Changed in version 3.5: The legacy parameter only writes out
.pyc
files, not.pyo
files no matter what the value of optimize is.Changed in version 3.6: Accepts a path-like object.
Changed in version 3.7: The invalidation_mode parameter was added.
Changed in version 3.7.2: The invalidation_mode parameter’s default value is updated to
None
.Changed in version 3.8: Setting workers to 0 now chooses the optimal number of cores.
Changed in version 3.9: Added stripdir, prependdir, limit_sl_dest and hardlink_dupes arguments. Default value of maxlevels was changed from
10
tosys.getrecursionlimit()
- compileall.compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=None, *, stripdir=None, prependdir=None, limit_sl_dest=None, hardlink_dupes=False)¶
Compile the file with path fullname. Return a true value if the file compiled successfully, and a false value otherwise.
If ddir is given, it is prepended to the path to the file being compiled for use in compilation time tracebacks, and is also compiled in to the byte-code file, where it will be used in tracebacks and other messages in cases where the source file does not exist at the time the byte-code file is executed.
If rx is given, its
search
method is passed the full path name to the file being compiled, and if it returns a true value, the file is not compiled andTrue
is returned. This can be used to exclude files matching a regular expression, given as a re.Pattern object.If quiet is
False
or0
(the default), the filenames and other information are printed to standard out. Set to1
, only errors are printed. Set to2
, all output is suppressed.If legacy is true, byte-code files are written to their legacy locations and names, which may overwrite byte-code files created by another version of Python. The default is to write files to their PEP 3147 locations and names, which allows byte-code files from multiple versions of Python to coexist.
optimize specifies the optimization level for the compiler. It is passed to the built-in
compile()
function. Accepts also a sequence of optimization levels which lead to multiple compilations of one.py
file in one call.invalidation_mode should be a member of the
py_compile.PycInvalidationMode
enum and controls how the generated pycs are invalidated at runtime.The stripdir, prependdir and limit_sl_dest arguments correspond to the
-s
,-p
and-e
options described above. They may be specified asstr
oros.PathLike
.If hardlink_dupes is true and two
.pyc
files with different optimization level have the same content, use hard links to consolidate duplicate files.Added in version 3.2.
Changed in version 3.5: quiet parameter was changed to a multilevel value.
Changed in version 3.5: The legacy parameter only writes out
.pyc
files, not.pyo
files no matter what the value of optimize is.Changed in version 3.7: The invalidation_mode parameter was added.
Changed in version 3.7.2: The invalidation_mode parameter’s default value is updated to
None
.Changed in version 3.9: Added stripdir, prependdir, limit_sl_dest and hardlink_dupes arguments.
- compileall.compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1, invalidation_mode=None)¶
Byte-compile all the
.py
files found alongsys.path
. Return a true value if all the files compiled successfully, and a false value otherwise.If skip_curdir is true (the default), the current directory is not included in the search. All other parameters are passed to the
compile_dir()
function. Note that unlike the other compile functions,maxlevels
defaults to0
.Changed in version 3.2: Added the legacy and optimize parameter.
Changed in version 3.5: quiet parameter was changed to a multilevel value.
Changed in version 3.5: The legacy parameter only writes out
.pyc
files, not.pyo
files no matter what the value of optimize is.Changed in version 3.7: The invalidation_mode parameter was added.
Changed in version 3.7.2: The invalidation_mode parameter’s default value is updated to
None
.
To force a recompile of all the .py
files in the Lib/
subdirectory and all its subdirectories:
import compileall
compileall.compile_dir('Lib/', force=True)
# Perform same compilation, excluding files in .svn directories.
import re
compileall.compile_dir('Lib/', rx=re.compile(r'[/\\][.]svn'), force=True)
# pathlib.Path objects can also be used.
import pathlib
compileall.compile_dir(pathlib.Path('Lib/'), force=True)
See also
- Module
py_compile
Byte-compile a single source file.


Pickle buffer objects

Pickle buffer objects¶
Added in version 3.8.
A pickle.PickleBuffer
object wraps a buffer-providing object for out-of-band data transfer with the pickle
module.
-
PyTypeObject PyPickleBuffer_Type¶
This instance of
PyTypeObject
represents the Python pickle buffer type. This is the same object aspickle.PickleBuffer
in the Python layer.
-
int PyPickleBuffer_Check(PyObject *op)¶
Return true if op is a pickle buffer instance. This function always succeeds.
-
PyObject *PyPickleBuffer_FromObject(PyObject *obj)¶
Create a pickle buffer from the object obj.
This function will fail if obj doesn’t support the buffer protocol.
On success, return a new pickle buffer instance. On failure, set an exception and return
NULL
.Analogous to calling
pickle.PickleBuffer
with obj in Python.
-
const Py_buffer *PyPickleBuffer_GetBuffer(PyObject *picklebuf)¶
Get a pointer to the underlying
Py_buffer
that the pickle buffer wraps.The returned pointer is valid as long as picklebuf is alive and has not been released. The caller must not modify or free the returned
Py_buffer
. If the pickle buffer has been released, raiseValueError
.On success, return a pointer to the buffer view. On failure, set an exception and return
NULL
.
-
int PyPickleBuffer_Release(PyObject *picklebuf)¶
Release the underlying buffer held by the pickle buffer.
Return
0
on success. On failure, set an exception and return-1
.Analogous to calling
pickle.PickleBuffer.release()
in Python.


Coroutine Objects

Coroutine Objects¶
Added in version 3.5.
Coroutine objects are what functions declared with an async
keyword
return.
-
type PyCoroObject¶
The C structure used for coroutine objects.
-
PyTypeObject PyCoro_Type¶
The type object corresponding to coroutine objects.
-
int PyCoro_CheckExact(PyObject *ob)¶
Return true if ob’s type is
PyCoro_Type
; ob must not beNULL
. This function always succeeds.
-
PyObject *PyCoro_New(PyFrameObject *frame, PyObject *name, PyObject *qualname)¶
- Return value: New reference.
Create and return a new coroutine object based on the frame object, with
__name__
and__qualname__
set to name and qualname. A reference to frame is stolen by this function. The frame argument must not beNULL
.


— Read and write AIFF and AIFC files

aifc
— Read and write AIFF and AIFC files¶
Deprecated since version 3.11, removed in version 3.13.
This module is no longer part of the Python standard library. It was removed in Python 3.13 after being deprecated in Python 3.11. The removal was decided in PEP 594.
The last version of Python that provided the aifc
module was
Python 3.12.


Copyright

Copyright¶
Python and this documentation is:
Copyright © 2001 Python Software Foundation. All rights reserved.
Copyright © 2000 BeOpen.com. All rights reserved.
Copyright © 1995-2000 Corporation for National Research Initiatives. All rights reserved.
Copyright © 1991-1995 Stichting Mathematisch Centrum. All rights reserved.
See History and License for complete license and permissions information.


— Detection of ambiguous indentation

tabnanny
— Detection of ambiguous indentation¶
Source code: Lib/tabnanny.py
For the time being this module is intended to be called as a script. However it
is possible to import it into an IDE and use the function check()
described below.
Note
The API provided by this module is likely to change in future releases; such changes may not be backward compatible.
- tabnanny.check(file_or_dir)¶
If file_or_dir is a directory and not a symbolic link, then recursively descend the directory tree named by file_or_dir, checking all
.py
files along the way. If file_or_dir is an ordinary Python source file, it is checked for whitespace related problems. The diagnostic messages are written to standard output using theprint()
function.
- tabnanny.verbose¶
Flag indicating whether to print verbose messages. This is incremented by the
-v
option if called as a script.
- tabnanny.filename_only¶
Flag indicating whether to print only the filenames of files containing whitespace related problems. This is set to true by the
-q
option if called as a script.
- exception tabnanny.NannyNag¶
Raised by
process_tokens()
if detecting an ambiguous indent. Captured and handled incheck()
.
- tabnanny.process_tokens(tokens)¶
This function is used by
check()
to process tokens generated by thetokenize
module.
See also
- Module
tokenize
Lexical scanner for Python source code.


C API Extension Support for Free Threading¶
Starting with the 3.13 release, CPython has support for running with the global interpreter lock (GIL) disabled in a configuration called free threading. This document describes how to adapt C API extensions to support free threading.
Identifying the Free-Threaded Build in C¶
The CPython C API exposes the Py_GIL_DISABLED
macro: in the free-threaded
build it’s defined to 1
, and in the regular build it’s not defined.
You can use it to enable code that only runs under the free-threaded build:
#ifdef Py_GIL_DISABLED
/* code that only runs in the free-threaded build */
#endif
Note
On Windows, this macro is not defined automatically, but must be specified
to the compiler when building. The sysconfig.get_config_var()
function
can be used to determine whether the current running interpreter had the
macro defined.
Module Initialization¶
Extension modules need to explicitly indicate that they support running with the GIL disabled; otherwise importing the extension will raise a warning and enable the GIL at runtime.
There are two ways to indicate that an extension module supports running with the GIL disabled depending on whether the extension uses multi-phase or single-phase initialization.
Multi-Phase Initialization¶
Extensions that use multi-phase initialization (i.e.,
PyModuleDef_Init()
) should add a Py_mod_gil
slot in the
module definition. If your extension supports older versions of CPython,
you should guard the slot with a PY_VERSION_HEX
check.
static struct PyModuleDef_Slot module_slots[] = {
...
#if PY_VERSION_HEX >= 0x030D0000
{Py_mod_gil, Py_MOD_GIL_NOT_USED},
#endif
{0, NULL}
};
static struct PyModuleDef moduledef = {
PyModuleDef_HEAD_INIT,
.m_slots = module_slots,
...
};
Single-Phase Initialization¶
Extensions that use single-phase initialization (i.e.,
PyModule_Create()
) should call PyUnstable_Module_SetGIL()
to
indicate that they support running with the GIL disabled. The function is
only defined in the free-threaded build, so you should guard the call with
#ifdef Py_GIL_DISABLED
to avoid compilation errors in the regular build.
static struct PyModuleDef moduledef = {
PyModuleDef_HEAD_INIT,
...
};
PyMODINIT_FUNC
PyInit_mymodule(void)
{
PyObject *m = PyModule_Create(&moduledef);
if (m == NULL) {
return NULL;
}
#ifdef Py_GIL_DISABLED
PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED);
#endif
return m;
}
General API Guidelines¶
Most of the C API is thread-safe, but there are some exceptions.
Struct Fields: Accessing fields in Python C API objects or structs directly is not thread-safe if the field may be concurrently modified.
Macros: Accessor macros like
PyList_GET_ITEM
,PyList_SET_ITEM
, and macros likePySequence_Fast_GET_SIZE
that use the object returned byPySequence_Fast()
do not perform any error checking or locking. These macros are not thread-safe if the container object may be modified concurrently.Borrowed References: C API functions that return borrowed references may not be thread-safe if the containing object is modified concurrently. See the section on borrowed references for more information.
Container Thread Safety¶
Containers like PyListObject
,
PyDictObject
, and PySetObject
perform internal locking
in the free-threaded build. For example, the PyList_Append()
will
lock the list before appending an item.
PyDict_Next
¶
A notable exception is PyDict_Next()
, which does not lock the
dictionary. You should use Py_BEGIN_CRITICAL_SECTION
to protect
the dictionary while iterating over it if the dictionary may be concurrently
modified:
Py_BEGIN_CRITICAL_SECTION(dict);
PyObject *key, *value;
Py_ssize_t pos = 0;
while (PyDict_Next(dict, &pos, &key, &value)) {
...
}
Py_END_CRITICAL_SECTION();
Borrowed References¶
Some C API functions return borrowed references.
These APIs are not thread-safe if the containing object is modified
concurrently. For example, it’s not safe to use PyList_GetItem()
if the list may be modified concurrently.
The following table lists some borrowed reference APIs and their replacements that return strong references.
Borrowed reference API |
Strong reference API |
|---|---|
none (see PyDict_Next) |
|
Not all APIs that return borrowed references are problematic. For
example, PyTuple_GetItem()
is safe because tuples are immutable.
Similarly, not all uses of the above APIs are problematic. For example,
PyDict_GetItem()
is often used for parsing keyword argument
dictionaries in function calls; those keyword argument dictionaries are
effectively private (not accessible by other threads), so using borrowed
references in that context is safe.
Some of these functions were added in Python 3.13. You can use the pythoncapi-compat package to provide implementations of these functions for older Python versions.
Memory Allocation APIs¶
Python’s memory management C API provides functions in three different allocation domains: “raw”, “mem”, and “object”. For thread-safety, the free-threaded build requires that only Python objects are allocated using the object domain, and that all Python objects are allocated using that domain. This differs from the prior Python versions, where this was only a best practice and not a hard requirement.
Note
Search for uses of PyObject_Malloc()
in your
extension and check that the allocated memory is used for Python objects.
Use PyMem_Malloc()
to allocate buffers instead of
PyObject_Malloc()
.
Thread State and GIL APIs¶
Python provides a set of functions and macros to manage thread state and the GIL, such as:
These functions should still be used in the free-threaded build to manage
thread state even when the GIL is disabled. For example, if you
create a thread outside of Python, you must call PyGILState_Ensure()
before calling into the Python API to ensure that the thread has a valid
Python thread state.
You should continue to call PyEval_SaveThread()
or
Py_BEGIN_ALLOW_THREADS
around blocking operations, such as I/O or
lock acquisitions, to allow other threads to run the
cyclic garbage collector.
Protecting Internal Extension State¶
Your extension may have internal state that was previously protected by the GIL. You may need to add locking to protect this state. The approach will depend on your extension, but some common patterns include:
Caches: global caches are a common source of shared state. Consider using a lock to protect the cache or disabling it in the free-threaded build if the cache is not critical for performance.
Global State: global state may need to be protected by a lock or moved to thread local storage. C11 and C++11 provide the
thread_local
or_Thread_local
for thread-local storage.
Critical Sections¶
In the free-threaded build, CPython provides a mechanism called “critical sections” to protect data that would otherwise be protected by the GIL. While extension authors may not interact with the internal critical section implementation directly, understanding their behavior is crucial when using certain C API functions or managing shared state in the free-threaded build.
What Are Critical Sections?¶
Conceptually, critical sections act as a deadlock avoidance layer built on
top of simple mutexes. Each thread maintains a stack of active critical
sections. When a thread needs to acquire a lock associated with a critical
section (e.g., implicitly when calling a thread-safe C API function like
PyDict_SetItem()
, or explicitly using macros), it attempts to acquire
the underlying mutex.
Using Critical Sections¶
The primary APIs for using critical sections are:
Py_BEGIN_CRITICAL_SECTION
andPy_END_CRITICAL_SECTION
- For locking a single objectPy_BEGIN_CRITICAL_SECTION2
andPy_END_CRITICAL_SECTION2
- For locking two objects simultaneously
These macros must be used in matching pairs and must appear in the same C scope, since they establish a new local scope. These macros are no-ops in non-free-threaded builds, so they can be safely added to code that needs to support both build types.
A common use of a critical section would be to lock an object while accessing an internal attribute of it. For example, if an extension type has an internal count field, you could use a critical section while reading or writing that field:
// read the count, returns new reference to internal count value
PyObject *result;
Py_BEGIN_CRITICAL_SECTION(obj);
result = Py_NewRef(obj->count);
Py_END_CRITICAL_SECTION();
return result;
// write the count, consumes reference from new_count
Py_BEGIN_CRITICAL_SECTION(obj);
obj->count = new_count;
Py_END_CRITICAL_SECTION();
How Critical Sections Work¶
Unlike traditional locks, critical sections do not guarantee exclusive access throughout their entire duration. If a thread would block while holding a critical section (e.g., by acquiring another lock or performing I/O), the critical section is temporarily suspended—all locks are released—and then resumed when the blocking operation completes.
This behavior is similar to what happens with the GIL when a thread makes a blocking call. The key differences are:
Critical sections operate on a per-object basis rather than globally
Critical sections follow a stack discipline within each thread (the “begin” and “end” macros enforce this since they must be paired and within the same scope)
Critical sections automatically release and reacquire locks around potential blocking operations
Deadlock Avoidance¶
Critical sections help avoid deadlocks in two ways:
If a thread tries to acquire a lock that’s already held by another thread, it first suspends all of its active critical sections, temporarily releasing their locks
When the blocking operation completes, only the top-most critical section is reacquired first
This means you cannot rely on nested critical sections to lock multiple objects
at once, as the inner critical section may suspend the outer ones. Instead, use
Py_BEGIN_CRITICAL_SECTION2
to lock two objects simultaneously.
Note that the locks described above are only PyMutex
based locks.
The critical section implementation does not know about or affect other locking
mechanisms that might be in use, like POSIX mutexes. Also note that while
blocking on any PyMutex
causes the critical sections to be
suspended, only the mutexes that are part of the critical sections are
released. If PyMutex
is used without a critical section, it will
not be released and therefore does not get the same deadlock avoidance.
Important Considerations¶
Critical sections may temporarily release their locks, allowing other threads to modify the protected data. Be careful about making assumptions about the state of the data after operations that might block.
Because locks can be temporarily released (suspended), entering a critical section does not guarantee exclusive access to the protected resource throughout the section’s duration. If code within a critical section calls another function that blocks (e.g., acquires another lock, performs blocking I/O), all locks held by the thread via critical sections will be released. This is similar to how the GIL can be released during blocking calls.
Only the lock(s) associated with the most recently entered (top-most) critical section are guaranteed to be held at any given time. Locks for outer, nested critical sections might have been suspended.
You can lock at most two objects simultaneously with these APIs. If you need to lock more objects, you’ll need to restructure your code.
While critical sections will not deadlock if you attempt to lock the same object twice, they are less efficient than purpose-built reentrant locks for this use case.
When using
Py_BEGIN_CRITICAL_SECTION2
, the order of the objects doesn’t affect correctness (the implementation handles deadlock avoidance), but it’s good practice to always lock objects in a consistent order.Remember that the critical section macros are primarily for protecting access to Python objects that might be involved in internal CPython operations susceptible to the deadlock scenarios described above. For protecting purely internal extension state, standard mutexes or other synchronization primitives might be more appropriate.
Building Extensions for the Free-Threaded Build¶
C API extensions need to be built specifically for the free-threaded build.
The wheels, shared libraries, and binaries are indicated by a t
suffix.
pypa/manylinux supports the free-threaded build, with the
t
suffix, such aspython3.13t
.pypa/cibuildwheel supports the free-threaded build on Python 3.13 and 3.14. On Python 3.14, free-threaded wheels will be built by default. On Python 3.13, you will need to set CIBW_ENABLE to cpython-freethreading.
Limited C API and Stable ABI¶
The free-threaded build does not currently support the
Limited C API or the stable ABI. If you use
setuptools to build
your extension and currently set py_limited_api=True
you can use
py_limited_api=not sysconfig.get_config_var("Py_GIL_DISABLED")
to opt out
of the limited API when building with the free-threaded build.
Note
You will need to build separate wheels specifically for the free-threaded build. If you currently use the stable ABI, you can continue to build a single wheel for multiple non-free-threaded Python versions.
Windows¶
Due to a limitation of the official Windows installer, you will need to
manually define Py_GIL_DISABLED=1
when building extensions from source.
See also
Porting Extension Modules to Support Free-Threading: A community-maintained porting guide for extension authors.


: Encoders

email.encoders
: Encoders¶
Source code: Lib/email/encoders.py
This module is part of the legacy (Compat32
) email API. In the
new API the functionality is provided by the cte parameter of
the set_content()
method.
This module is deprecated in Python 3. The functions provided here
should not be called explicitly since the MIMEText
class sets the content type and CTE header using the _subtype and _charset
values passed during the instantiation of that class.
The remaining text in this section is the original documentation of the module.
When creating Message
objects from scratch, you often
need to encode the payloads for transport through compliant mail servers. This
is especially true for image/* and text/* type messages
containing binary data.
The email
package provides some convenient encoders in its
encoders
module. These encoders are actually used by the
MIMEAudio
and MIMEImage
class constructors to provide default encodings. All encoder functions take
exactly one argument, the message object to encode. They usually extract the
payload, encode it, and reset the payload to this newly encoded value. They
should also set the Content-Transfer-Encoding header as appropriate.
Note that these functions are not meaningful for a multipart message. They
must be applied to individual subparts instead, and will raise a
TypeError
if passed a message whose type is multipart.
Here are the encoding functions provided:
- email.encoders.encode_quopri(msg)¶
Encodes the payload into quoted-printable form and sets the Content-Transfer-Encoding header to
quoted-printable
[1]. This is a good encoding to use when most of your payload is normal printable data, but contains a few unprintable characters.
- email.encoders.encode_base64(msg)¶
Encodes the payload into base64 form and sets the Content-Transfer-Encoding header to
base64
. This is a good encoding to use when most of your payload is unprintable data since it is a more compact form than quoted-printable. The drawback of base64 encoding is that it renders the text non-human readable.
- email.encoders.encode_7or8bit(msg)¶
This doesn’t actually modify the message’s payload, but it does set the Content-Transfer-Encoding header to either
7bit
or8bit
as appropriate, based on the payload data.
- email.encoders.encode_noop(msg)¶
This does nothing; it doesn’t even set the Content-Transfer-Encoding header.
Footnotes


: Custom Header Objects

email.headerregistry
: Custom Header Objects¶
Source code: Lib/email/headerregistry.py
Added in version 3.6: [1]
Headers are represented by customized subclasses of str
. The
particular class used to represent a given header is determined by the
header_factory
of the policy
in
effect when the headers are created. This section documents the particular
header_factory
implemented by the email package for handling RFC 5322
compliant email messages, which not only provides customized header objects for
various header types, but also provides an extension mechanism for applications
to add their own custom header types.
When using any of the policy objects derived from
EmailPolicy
, all headers are produced by
HeaderRegistry
and have BaseHeader
as their last base
class. Each header class has an additional base class that is determined by
the type of the header. For example, many headers have the class
UnstructuredHeader
as their other base class. The specialized second
class for a header is determined by the name of the header, using a lookup
table stored in the HeaderRegistry
. All of this is managed
transparently for the typical application program, but interfaces are provided
for modifying the default behavior for use by more complex applications.
The sections below first document the header base classes and their attributes,
followed by the API for modifying the behavior of HeaderRegistry
, and
finally the support classes used to represent the data parsed from structured
headers.
- class email.headerregistry.BaseHeader(name, value)¶
name and value are passed to
BaseHeader
from theheader_factory
call. The string value of any header object is the value fully decoded to unicode.This base class defines the following read-only properties:
- name¶
The name of the header (the portion of the field before the ‘:’). This is exactly the value passed in the
header_factory
call for name; that is, case is preserved.
- defects¶
A tuple of
HeaderDefect
instances reporting any RFC compliance problems found during parsing. The email package tries to be complete about detecting compliance issues. See theerrors
module for a discussion of the types of defects that may be reported.
- max_count¶
The maximum number of headers of this type that can have the same
name
. A value ofNone
means unlimited. TheBaseHeader
value for this attribute isNone
; it is expected that specialized header classes will override this value as needed.
BaseHeader
also provides the following method, which is called by the email library code and should not in general be called by application programs:- fold(*, policy)¶
Return a string containing
linesep
characters as required to correctly fold the header according to policy. Acte_type
of8bit
will be treated as if it were7bit
, since headers may not contain arbitrary binary data. Ifutf8
isFalse
, non-ASCII data will be RFC 2047 encoded.
BaseHeader
by itself cannot be used to create a header object. It defines a protocol that each specialized header cooperates with in order to produce the header object. Specifically,BaseHeader
requires that the specialized class provide aclassmethod()
namedparse
. This method is called as follows:parse(string, kwds)
kwds
is a dictionary containing one pre-initialized key,defects
.defects
is an empty list. The parse method should append any detected defects to this list. On return, thekwds
dictionary must contain values for at least the keysdecoded
anddefects
.decoded
should be the string value for the header (that is, the header value fully decoded to unicode). The parse method should assume that string may contain content-transfer-encoded parts, but should correctly handle all valid unicode characters as well so that it can parse un-encoded header values.BaseHeader
’s__new__
then creates the header instance, and calls itsinit
method. The specialized class only needs to provide aninit
method if it wishes to set additional attributes beyond those provided byBaseHeader
itself. Such aninit
method should look like this:def init(self, /, *args, **kw): self._myattr = kw.pop('myattr') super().init(*args, **kw)
That is, anything extra that the specialized class puts in to the
kwds
dictionary should be removed and handled, and the remaining contents ofkw
(andargs
) passed to theBaseHeader
init
method.
- class email.headerregistry.UnstructuredHeader¶
An “unstructured” header is the default type of header in RFC 5322. Any header that does not have a specified syntax is treated as unstructured. The classic example of an unstructured header is the Subject header.
In RFC 5322, an unstructured header is a run of arbitrary text in the ASCII character set. RFC 2047, however, has an RFC 5322 compatible mechanism for encoding non-ASCII text as ASCII characters within a header value. When a value containing encoded words is passed to the constructor, the
UnstructuredHeader
parser converts such encoded words into unicode, following the RFC 2047 rules for unstructured text. The parser uses heuristics to attempt to decode certain non-compliant encoded words. Defects are registered in such cases, as well as defects for issues such as invalid characters within the encoded words or the non-encoded text.This header type provides no additional attributes.
- class email.headerregistry.DateHeader¶
RFC 5322 specifies a very specific format for dates within email headers. The
DateHeader
parser recognizes that date format, as well as recognizing a number of variant forms that are sometimes found “in the wild”.This header type provides the following additional attributes:
- datetime¶
If the header value can be recognized as a valid date of one form or another, this attribute will contain a
datetime
instance representing that date. If the timezone of the input date is specified as-0000
(indicating it is in UTC but contains no information about the source timezone), thendatetime
will be a naivedatetime
. If a specific timezone offset is found (including+0000
), thendatetime
will contain an awaredatetime
that usesdatetime.timezone
to record the timezone offset.
The
decoded
value of the header is determined by formatting thedatetime
according to the RFC 5322 rules; that is, it is set to:email.utils.format_datetime(self.datetime)
When creating a
DateHeader
, value may bedatetime
instance. This means, for example, that the following code is valid and does what one would expect:msg['Date'] = datetime(2011, 7, 15, 21)
Because this is a naive
datetime
it will be interpreted as a UTC timestamp, and the resulting value will have a timezone of-0000
. Much more useful is to use thelocaltime()
function from theutils
module:msg['Date'] = utils.localtime()
This example sets the date header to the current time and date using the current timezone offset.
- class email.headerregistry.AddressHeader¶
Address headers are one of the most complex structured header types. The
AddressHeader
class provides a generic interface to any address header.This header type provides the following additional attributes:
- groups¶
A tuple of
Group
objects encoding the addresses and groups found in the header value. Addresses that are not part of a group are represented in this list as single-addressGroups
whosedisplay_name
isNone
.
- addresses¶
A tuple of
Address
objects encoding all of the individual addresses from the header value. If the header value contains any groups, the individual addresses from the group are included in the list at the point where the group occurs in the value (that is, the list of addresses is “flattened” into a one dimensional list).
The
decoded
value of the header will have all encoded words decoded to unicode.idna
encoded domain names are also decoded to unicode. Thedecoded
value is set by joining thestr
value of the elements of thegroups
attribute with', '
.A list of
Address
andGroup
objects in any combination may be used to set the value of an address header.Group
objects whosedisplay_name
isNone
will be interpreted as single addresses, which allows an address list to be copied with groups intact by using the list obtained from thegroups
attribute of the source header.
- class email.headerregistry.SingleAddressHeader¶
A subclass of
AddressHeader
that adds one additional attribute:- address¶
The single address encoded by the header value. If the header value actually contains more than one address (which would be a violation of the RFC under the default
policy
), accessing this attribute will result in aValueError
.
Many of the above classes also have a Unique
variant (for example,
UniqueUnstructuredHeader
). The only difference is that in the Unique
variant, max_count
is set to 1.
- class email.headerregistry.MIMEVersionHeader¶
There is really only one valid value for the MIME-Version header, and that is
1.0
. For future proofing, this header class supports other valid version numbers. If a version number has a valid value per RFC 2045, then the header object will have non-None
values for the following attributes:- version¶
The version number as a string, with any whitespace and/or comments removed.
- major¶
The major version number as an integer
- minor¶
The minor version number as an integer
- class email.headerregistry.ParameterizedMIMEHeader¶
MIME headers all start with the prefix ‘Content-’. Each specific header has a certain value, described under the class for that header. Some can also take a list of supplemental parameters, which have a common format. This class serves as a base for all the MIME headers that take parameters.
- params¶
A dictionary mapping parameter names to parameter values.
- class email.headerregistry.ContentTypeHeader¶
A
ParameterizedMIMEHeader
class that handles the Content-Type header.- content_type¶
The content type string, in the form
maintype/subtype
.
- maintype¶
- subtype¶
- class email.headerregistry.ContentDispositionHeader¶
A
ParameterizedMIMEHeader
class that handles the Content-Disposition header.- content_disposition¶
inline
andattachment
are the only valid values in common use.
- class email.headerregistry.ContentTransferEncodingHeader¶
Handles the Content-Transfer-Encoding header.
- class email.headerregistry.HeaderRegistry(base_class=BaseHeader, default_class=UnstructuredHeader, use_default_map=True)¶
This is the factory used by
EmailPolicy
by default.HeaderRegistry
builds the class used to create a header instance dynamically, using base_class and a specialized class retrieved from a registry that it holds. When a given header name does not appear in the registry, the class specified by default_class is used as the specialized class. When use_default_map isTrue
(the default), the standard mapping of header names to classes is copied in to the registry during initialization. base_class is always the last class in the generated class’s__bases__
list.The default mappings are:
- subject:
UniqueUnstructuredHeader
- date:
UniqueDateHeader
- resent-date:
DateHeader
- orig-date:
UniqueDateHeader
- sender:
UniqueSingleAddressHeader
- resent-sender:
SingleAddressHeader
- to:
UniqueAddressHeader
- resent-to:
AddressHeader
- cc:
UniqueAddressHeader
- resent-cc:
AddressHeader
- bcc:
UniqueAddressHeader
- resent-bcc:
AddressHeader
- from:
UniqueAddressHeader
- resent-from:
AddressHeader
- reply-to:
UniqueAddressHeader
- mime-version:
MIMEVersionHeader
- content-type:
ContentTypeHeader
- content-disposition:
ContentDispositionHeader
- content-transfer-encoding:
ContentTransferEncodingHeader
- message-id:
MessageIDHeader
HeaderRegistry
has the following methods:- map_to_type(self, name, cls)¶
name is the name of the header to be mapped. It will be converted to lower case in the registry. cls is the specialized class to be used, along with base_class, to create the class used to instantiate headers that match name.
- __getitem__(name)¶
Construct and return a class to handle creating a name header.
- __call__(name, value)¶
Retrieves the specialized header associated with name from the registry (using default_class if name does not appear in the registry) and composes it with base_class to produce a class, calls the constructed class’s constructor, passing it the same argument list, and finally returns the class instance created thereby.
The following classes are the classes used to represent data parsed from structured headers and can, in general, be used by an application program to construct structured values to assign to specific headers.
- class email.headerregistry.Address(display_name='', username='', domain='', addr_spec=None)¶
The class used to represent an email address. The general form of an address is:
[display_name] <username@domain>
or:
username@domain
where each part must conform to specific syntax rules spelled out in RFC 5322.
As a convenience addr_spec can be specified instead of username and domain, in which case username and domain will be parsed from the addr_spec. An addr_spec must be a properly RFC quoted string; if it is not
Address
will raise an error. Unicode characters are allowed and will be property encoded when serialized. However, per the RFCs, unicode is not allowed in the username portion of the address.- display_name¶
The display name portion of the address, if any, with all quoting removed. If the address does not have a display name, this attribute will be an empty string.
- username¶
The
username
portion of the address, with all quoting removed.
- domain¶
The
domain
portion of the address.
- addr_spec¶
The
username@domain
portion of the address, correctly quoted for use as a bare address (the second form shown above). This attribute is not mutable.
- __str__()¶
The
str
value of the object is the address quoted according to RFC 5322 rules, but with no Content Transfer Encoding of any non-ASCII characters.
To support SMTP (RFC 5321),
Address
handles one special case: ifusername
anddomain
are both the empty string (orNone
), then the string value of theAddress
is<>
.
- class email.headerregistry.Group(display_name=None, addresses=None)¶
The class used to represent an address group. The general form of an address group is:
display_name: [address-list];
As a convenience for processing lists of addresses that consist of a mixture of groups and single addresses, a
Group
may also be used to represent single addresses that are not part of a group by setting display_name toNone
and providing a list of the single address as addresses.- display_name¶
The
display_name
of the group. If it isNone
and there is exactly oneAddress
inaddresses
, then theGroup
represents a single address that is not in a group.
Footnotes


Brief Tour of the Standard Library

10. Brief Tour of the Standard Library¶
10.1. Operating System Interface¶
The os
module provides dozens of functions for interacting with the
operating system:
>>> import os
>>> os.getcwd() # Return the current working directory
'C:\\Python314'
>>> os.chdir('/server/accesslogs') # Change current working directory
>>> os.system('mkdir today') # Run the command mkdir in the system shell
0
Be sure to use the import os
style instead of from os import *
. This
will keep os.open()
from shadowing the built-in open()
function which
operates much differently.
The built-in dir()
and help()
functions are useful as interactive
aids for working with large modules like os
:
>>> import os
>>> dir(os)
<returns a list of all module functions>
>>> help(os)
<returns an extensive manual page created from the module's docstrings>
For daily file and directory management tasks, the shutil
module provides
a higher level interface that is easier to use:
>>> import shutil
>>> shutil.copyfile('data.db', 'archive.db')
'archive.db'
>>> shutil.move('/build/executables', 'installdir')
'installdir'
10.2. File Wildcards¶
The glob
module provides a function for making file lists from directory
wildcard searches:
>>> import glob
>>> glob.glob('*.py')
['primes.py', 'random.py', 'quote.py']
10.3. Command Line Arguments¶
Common utility scripts often need to process command line arguments. These
arguments are stored in the sys
module’s argv attribute as a list. For
instance, let’s take the following demo.py
file:
# File demo.py
import sys
print(sys.argv)
Here is the output from running python demo.py one two three
at the command
line:
['demo.py', 'one', 'two', 'three']
The argparse
module provides a more sophisticated mechanism to process
command line arguments. The following script extracts one or more filenames
and an optional number of lines to be displayed:
import argparse
parser = argparse.ArgumentParser(
prog='top',
description='Show top lines from each file')
parser.add_argument('filenames', nargs='+')
parser.add_argument('-l', '--lines', type=int, default=10)
args = parser.parse_args()
print(args)
When run at the command line with python top.py --lines=5 alpha.txt
beta.txt
, the script sets args.lines
to 5
and args.filenames
to ['alpha.txt', 'beta.txt']
.
10.4. Error Output Redirection and Program Termination¶
The sys
module also has attributes for stdin, stdout, and stderr.
The latter is useful for emitting warnings and error messages to make them
visible even when stdout has been redirected:
>>> sys.stderr.write('Warning, log file not found starting a new one\n')
Warning, log file not found starting a new one
The most direct way to terminate a script is to use sys.exit()
.
10.5. String Pattern Matching¶
The re
module provides regular expression tools for advanced string
processing. For complex matching and manipulation, regular expressions offer
succinct, optimized solutions:
>>> import re
>>> re.findall(r'\bf[a-z]*', 'which foot or hand fell fastest')
['foot', 'fell', 'fastest']
>>> re.sub(r'(\b[a-z]+) \1', r'\1', 'cat in the the hat')
'cat in the hat'
When only simple capabilities are needed, string methods are preferred because they are easier to read and debug:
>>> 'tea for too'.replace('too', 'two')
'tea for two'
10.6. Mathematics¶
The math
module gives access to the underlying C library functions for
floating-point math:
>>> import math
>>> math.cos(math.pi / 4)
0.70710678118654757
>>> math.log(1024, 2)
10.0
The random
module provides tools for making random selections:
>>> import random
>>> random.choice(['apple', 'pear', 'banana'])
'apple'
>>> random.sample(range(100), 10) # sampling without replacement
[30, 83, 16, 4, 8, 81, 41, 50, 18, 33]
>>> random.random() # random float from the interval [0.0, 1.0)
0.17970987693706186
>>> random.randrange(6) # random integer chosen from range(6)
4
The statistics
module calculates basic statistical properties
(the mean, median, variance, etc.) of numeric data:
>>> import statistics
>>> data = [2.75, 1.75, 1.25, 0.25, 0.5, 1.25, 3.5]
>>> statistics.mean(data)
1.6071428571428572
>>> statistics.median(data)
1.25
>>> statistics.variance(data)
1.3720238095238095
The SciPy project <https://scipy.org> has many other modules for numerical computations.
10.7. Internet Access¶
There are a number of modules for accessing the internet and processing internet
protocols. Two of the simplest are urllib.request
for retrieving data
from URLs and smtplib
for sending mail:
>>> from urllib.request import urlopen
>>> with urlopen('https://docs.python.org/3/') as response:
... for line in response:
... line = line.decode() # Convert bytes to a str
... if 'updated' in line:
... print(line.rstrip()) # Remove trailing newline
...
Last updated on Nov 11, 2025 (20:11 UTC).
>>> import smtplib
>>> server = smtplib.SMTP('localhost')
>>> server.sendmail('soothsayer@example.org', 'jcaesar@example.org',
... """To: jcaesar@example.org
... From: soothsayer@example.org
...
... Beware the Ides of March.
... """)
>>> server.quit()
(Note that the second example needs a mailserver running on localhost.)
10.8. Dates and Times¶
The datetime
module supplies classes for manipulating dates and times in
both simple and complex ways. While date and time arithmetic is supported, the
focus of the implementation is on efficient member extraction for output
formatting and manipulation. The module also supports objects that are timezone
aware.
>>> # dates are easily constructed and formatted
>>> from datetime import date
>>> now = date.today()
>>> now
datetime.date(2003, 12, 2)
>>> now.strftime("%m-%d-%y. %d %b %Y is a %A on the %d day of %B.")
'12-02-03. 02 Dec 2003 is a Tuesday on the 02 day of December.'
>>> # dates support calendar arithmetic
>>> birthday = date(1964, 7, 31)
>>> age = now - birthday
>>> age.days
14368
10.9. Data Compression¶
Common data archiving and compression formats are directly supported by modules
including: zlib
, gzip
, bz2
, lzma
, zipfile
and
tarfile
.
>>> import zlib
>>> s = b'witch which has which witches wrist watch'
>>> len(s)
41
>>> t = zlib.compress(s)
>>> len(t)
37
>>> zlib.decompress(t)
b'witch which has which witches wrist watch'
>>> zlib.crc32(s)
226805979
10.10. Performance Measurement¶
Some Python users develop a deep interest in knowing the relative performance of different approaches to the same problem. Python provides a measurement tool that answers those questions immediately.
For example, it may be tempting to use the tuple packing and unpacking feature
instead of the traditional approach to swapping arguments. The timeit
module quickly demonstrates a modest performance advantage:
>>> from timeit import Timer
>>> Timer('t=a; a=b; b=t', 'a=1; b=2').timeit()
0.57535828626024577
>>> Timer('a,b = b,a', 'a=1; b=2').timeit()
0.54962537085770791
In contrast to timeit
’s fine level of granularity, the profile
and
pstats
modules provide tools for identifying time critical sections in
larger blocks of code.
10.11. Quality Control¶
One approach for developing high quality software is to write tests for each function as it is developed and to run those tests frequently during the development process.
The doctest
module provides a tool for scanning a module and validating
tests embedded in a program’s docstrings. Test construction is as simple as
cutting-and-pasting a typical call along with its results into the docstring.
This improves the documentation by providing the user with an example and it
allows the doctest module to make sure the code remains true to the
documentation:
def average(values):
"""Computes the arithmetic mean of a list of numbers.
>>> print(average([20, 30, 70]))
40.0
"""
return sum(values) / len(values)
import doctest
doctest.testmod() # automatically validate the embedded tests
The unittest
module is not as effortless as the doctest
module,
but it allows a more comprehensive set of tests to be maintained in a separate
file:
import unittest
class TestStatisticalFunctions(unittest.TestCase):
def test_average(self):
self.assertEqual(average([20, 30, 70]), 40.0)
self.assertEqual(round(average([1, 5, 7]), 1), 4.3)
with self.assertRaises(ZeroDivisionError):
average([])
with self.assertRaises(TypeError):
average(20, 30, 70)
unittest.main() # Calling from the command line invokes all tests
10.12. Batteries Included¶
Python has a “batteries included” philosophy. This is best seen through the sophisticated and robust capabilities of its larger packages. For example:
The
xmlrpc.client
andxmlrpc.server
modules make implementing remote procedure calls into an almost trivial task. Despite the modules’ names, no direct knowledge or handling of XML is needed.The
email
package is a library for managing email messages, including MIME and other RFC 5322-based message documents. Unlikesmtplib
andpoplib
which actually send and receive messages, the email package has a complete toolset for building or decoding complex message structures (including attachments) and for implementing internet encoding and header protocols.The
json
package provides robust support for parsing this popular data interchange format. Thecsv
module supports direct reading and writing of files in Comma-Separated Value format, commonly supported by databases and spreadsheets. XML processing is supported by thexml.etree.ElementTree
,xml.dom
andxml.sax
packages. Together, these modules and packages greatly simplify data interchange between Python applications and other tools.The
sqlite3
module is a wrapper for the SQLite database library, providing a persistent database that can be updated and accessed using slightly nonstandard SQL syntax.Internationalization is supported by a number of modules including
gettext
,locale
, and thecodecs
package.