Defining extension modules

Defining extension modules¶
A C extension for CPython is a shared library (for example, a .so
file
on Linux, .pyd
DLL on Windows), which is loadable into the Python process
(for example, it is compiled with compatible compiler settings), and which
exports an initialization function.
To be importable by default (that is, by
importlib.machinery.ExtensionFileLoader
),
the shared library must be available on sys.path
,
and must be named after the module name plus an extension listed in
importlib.machinery.EXTENSION_SUFFIXES
.
Note
Building, packaging and distributing extension modules is best done with third-party tools, and is out of scope of this document. One suitable tool is Setuptools, whose documentation can be found at https://setuptools.pypa.io/en/latest/setuptools.html.
Normally, the initialization function returns a module definition initialized
using PyModuleDef_Init()
.
This allows splitting the creation process into several phases:
Before any substantial code is executed, Python can determine which capabilities the module supports, and it can adjust the environment or refuse loading an incompatible extension.
By default, Python itself creates the module object – that is, it does the equivalent of
object.__new__()
for classes. It also sets initial attributes like__package__
and__loader__
.Afterwards, the module object is initialized using extension-specific code – the equivalent of
__init__()
on classes.
This is called multi-phase initialization to distinguish it from the legacy (but still supported) single-phase initialization scheme, where the initialization function returns a fully constructed module. See the single-phase-initialization section below for details.
Changed in version 3.5: Added support for multi-phase initialization (PEP 489).
Multiple module instances¶
By default, extension modules are not singletons.
For example, if the sys.modules
entry is removed and the module
is re-imported, a new module object is created, and typically populated with
fresh method and type objects.
The old module is subject to normal garbage collection.
This mirrors the behavior of pure-Python modules.
Additional module instances may be created in
sub-interpreters
or after Python runtime reinitialization
(Py_Finalize()
and Py_Initialize()
).
In these cases, sharing Python objects between module instances would likely
cause crashes or undefined behavior.
To avoid such issues, each instance of an extension module should be isolated: changes to one instance should not implicitly affect the others, and all state owned by the module, including references to Python objects, should be specific to a particular module instance. See Isolating Extension Modules for more details and a practical guide.
A simpler way to avoid these issues is raising an error on repeated initialization.
All modules are expected to support
sub-interpreters, or otherwise explicitly
signal a lack of support.
This is usually achieved by isolation or blocking repeated initialization,
as above.
A module may also be limited to the main interpreter using
the Py_mod_multiple_interpreters
slot.
Initialization function¶
The initialization function defined by an extension module has the following signature:
Its name should be PyInit_<name>
, with <name>
replaced by the
name of the module.
For modules with ASCII-only names, the function must instead be named
PyInit_<name>
, with <name>
replaced by the name of the module.
When using Multi-phase initialization, non-ASCII module names
are allowed. In this case, the initialization function name is
PyInitU_<name>
, with <name>
encoded using Python’s
punycode encoding with hyphens replaced by underscores. In Python:
def initfunc_name(name):
try:
suffix = b'_' + name.encode('ascii')
except UnicodeEncodeError:
suffix = b'U_' + name.encode('punycode').replace(b'-', b'_')
return b'PyInit' + suffix
It is recommended to define the initialization function using a helper macro:
-
PyMODINIT_FUNC¶
Declare an extension module initialization function. This macro:
specifies the PyObject* return type,
adds any special linkage declarations required by the platform, and
for C++, declares the function as
extern "C"
.
For example, a module called spam
would be defined like this:
static struct PyModuleDef spam_module = {
.m_base = PyModuleDef_HEAD_INIT,
.m_name = "spam",
...
};
PyMODINIT_FUNC
PyInit_spam(void)
{
return PyModuleDef_Init(&spam_module);
}
It is possible to export multiple modules from a single shared library by defining multiple initialization functions. However, importing them requires using symbolic links or a custom importer, because by default only the function corresponding to the filename is found. See the Multiple modules in one library section in PEP 489 for details.
The initialization function is typically the only non-static
item defined in the module’s C source.
Multi-phase initialization¶
Normally, the initialization function
(PyInit_modulename
) returns a PyModuleDef
instance with
non-NULL
m_slots
.
Before it is returned, the PyModuleDef
instance must be initialized
using the following function:
-
PyObject *PyModuleDef_Init(PyModuleDef *def)¶
- Part of the Stable ABI since version 3.5.
Ensure a module definition is a properly initialized Python object that correctly reports its type and a reference count.
Return def cast to
PyObject*
, orNULL
if an error occurred.Calling this function is required for Multi-phase initialization. It should not be used in other contexts.
Note that Python assumes that
PyModuleDef
structures are statically allocated. This function may return either a new reference or a borrowed one; this reference must not be released.Added in version 3.5.
Legacy single-phase initialization¶
Attention
Single-phase initialization is a legacy mechanism to initialize extension modules, with known drawbacks and design flaws. Extension module authors are encouraged to use multi-phase initialization instead.
In single-phase initialization, the
initialization function (PyInit_modulename
)
should create, populate and return a module object.
This is typically done using PyModule_Create()
and functions like
PyModule_AddObjectRef()
.
Single-phase initialization differs from the default in the following ways:
Single-phase modules are, or rather contain, “singletons”.
When the module is first initialized, Python saves the contents of the module’s
__dict__
(that is, typically, the module’s functions and types).For subsequent imports, Python does not call the initialization function again. Instead, it creates a new module object with a new
__dict__
, and copies the saved contents to it. For example, given a single-phase module_testsinglephase
[1] that defines a functionsum
and an exception classerror
:>>> import sys >>> import _testsinglephase as one >>> del sys.modules['_testsinglephase'] >>> import _testsinglephase as two >>> one is two False >>> one.__dict__ is two.__dict__ False >>> one.sum is two.sum True >>> one.error is two.error True
The exact behavior should be considered a CPython implementation detail.
To work around the fact that
PyInit_modulename
does not take a spec argument, some state of the import machinery is saved and applied to the first suitable module created during thePyInit_modulename
call. Specifically, when a sub-module is imported, this mechanism prepends the parent package name to the name of the module.A single-phase
PyInit_modulename
function should create “its” module object as soon as possible, before any other module objects can be created.Non-ASCII module names (
PyInitU_modulename
) are not supported.Single-phase modules support module lookup functions like
PyState_FindModule()
.


Annotations Best Practices

Annotations Best Practices¶
- author:
Larry Hastings
Accessing The Annotations Dict Of An Object In Python 3.10 And Newer¶
Python 3.10 adds a new function to the standard library:
inspect.get_annotations()
. In Python versions 3.10
through 3.13, calling this function is the best practice for
accessing the annotations dict of any object that supports
annotations. This function can also “un-stringize”
stringized annotations for you.
In Python 3.14, there is a new annotationlib
module
with functionality for working with annotations. This
includes a annotationlib.get_annotations()
function,
which supersedes inspect.get_annotations()
.
If for some reason inspect.get_annotations()
isn’t
viable for your use case, you may access the
__annotations__
data member manually. Best practice
for this changed in Python 3.10 as well: as of Python 3.10,
o.__annotations__
is guaranteed to always work
on Python functions, classes, and modules. If you’re
certain the object you’re examining is one of these three
specific objects, you may simply use o.__annotations__
to get at the object’s annotations dict.
However, other types of callables–for example,
callables created by functools.partial()
–may
not have an __annotations__
attribute defined. When
accessing the __annotations__
of a possibly unknown
object, best practice in Python versions 3.10 and
newer is to call getattr()
with three arguments,
for example getattr(o, '__annotations__', None)
.
Before Python 3.10, accessing __annotations__
on a class that
defines no annotations but that has a parent class with
annotations would return the parent’s __annotations__
.
In Python 3.10 and newer, the child class’s annotations
will be an empty dict instead.
Accessing The Annotations Dict Of An Object In Python 3.9 And Older¶
In Python 3.9 and older, accessing the annotations dict of an object is much more complicated than in newer versions. The problem is a design flaw in these older versions of Python, specifically to do with class annotations.
Best practice for accessing the annotations dict of other
objects–functions, other callables, and modules–is the same
as best practice for 3.10, assuming you aren’t calling
inspect.get_annotations()
: you should use three-argument
getattr()
to access the object’s __annotations__
attribute.
Unfortunately, this isn’t best practice for classes. The problem
is that, since __annotations__
is optional on classes, and
because classes can inherit attributes from their base classes,
accessing the __annotations__
attribute of a class may
inadvertently return the annotations dict of a base class.
As an example:
class Base:
a: int = 3
b: str = 'abc'
class Derived(Base):
pass
print(Derived.__annotations__)
This will print the annotations dict from Base
, not
Derived
.
Your code will have to have a separate code path if the object
you’re examining is a class (isinstance(o, type)
).
In that case, best practice relies on an implementation detail
of Python 3.9 and before: if a class has annotations defined,
they are stored in the class’s __dict__
dictionary. Since
the class may or may not have annotations defined, best practice
is to call the get()
method on the class dict.
To put it all together, here is some sample code that safely
accesses the __annotations__
attribute on an arbitrary
object in Python 3.9 and before:
if isinstance(o, type):
ann = o.__dict__.get('__annotations__', None)
else:
ann = getattr(o, '__annotations__', None)
After running this code, ann
should be either a
dictionary or None
. You’re encouraged to double-check
the type of ann
using isinstance()
before further
examination.
Note that some exotic or malformed type objects may not have
a __dict__
attribute, so for extra safety you may also wish
to use getattr()
to access __dict__
.
Manually Un-Stringizing Stringized Annotations¶
In situations where some annotations may be “stringized”,
and you wish to evaluate those strings to produce the
Python values they represent, it really is best to
call inspect.get_annotations()
to do this work
for you.
If you’re using Python 3.9 or older, or if for some reason
you can’t use inspect.get_annotations()
, you’ll need
to duplicate its logic. You’re encouraged to examine the
implementation of inspect.get_annotations()
in the
current Python version and follow a similar approach.
In a nutshell, if you wish to evaluate a stringized annotation
on an arbitrary object o
:
If
o
is a module, useo.__dict__
as theglobals
when callingeval()
.If
o
is a class, usesys.modules[o.__module__].__dict__
as theglobals
, anddict(vars(o))
as thelocals
, when callingeval()
.If
o
is a wrapped callable usingfunctools.update_wrapper()
,functools.wraps()
, orfunctools.partial()
, iteratively unwrap it by accessing eithero.__wrapped__
oro.func
as appropriate, until you have found the root unwrapped function.If
o
is a callable (but not a class), useo.__globals__
as the globals when callingeval()
.
However, not all string values used as annotations can
be successfully turned into Python values by eval()
.
String values could theoretically contain any valid string,
and in practice there are valid use cases for type hints that
require annotating with string values that specifically
can’t be evaluated. For example:
PEP 604 union types using
|
, before support for this was added to Python 3.10.Definitions that aren’t needed at runtime, only imported when
typing.TYPE_CHECKING
is true.
If eval()
attempts to evaluate such values, it will
fail and raise an exception. So, when designing a library
API that works with annotations, it’s recommended to only
attempt to evaluate string values when explicitly requested
to by the caller.
Best Practices For __annotations__
In Any Python Version¶
You should avoid assigning to the
__annotations__
member of objects directly. Let Python manage setting__annotations__
.If you do assign directly to the
__annotations__
member of an object, you should always set it to adict
object.You should avoid accessing
__annotations__
directly on any object. Instead, useannotationlib.get_annotations()
(Python 3.14+) orinspect.get_annotations()
(Python 3.10+).If you do directly access the
__annotations__
member of an object, you should ensure that it’s a dictionary before attempting to examine its contents.You should avoid modifying
__annotations__
dicts.You should avoid deleting the
__annotations__
attribute of an object.
__annotations__
Quirks¶
In all versions of Python 3, function
objects lazy-create an annotations dict if no annotations
are defined on that object. You can delete the __annotations__
attribute using del fn.__annotations__
, but if you then
access fn.__annotations__
the object will create a new empty dict
that it will store and return as its annotations. Deleting the
annotations on a function before it has lazily created its annotations
dict will throw an AttributeError
; using del fn.__annotations__
twice in a row is guaranteed to always throw an AttributeError
.
Everything in the above paragraph also applies to class and module objects in Python 3.10 and newer.
In all versions of Python 3, you can set __annotations__
on a function object to None
. However, subsequently
accessing the annotations on that object using fn.__annotations__
will lazy-create an empty dictionary as per the first paragraph of
this section. This is not true of modules and classes, in any Python
version; those objects permit setting __annotations__
to any
Python value, and will retain whatever value is set.
If Python stringizes your annotations for you
(using from __future__ import annotations
), and you
specify a string as an annotation, the string will
itself be quoted. In effect the annotation is quoted
twice. For example:
from __future__ import annotations
def foo(a: "str"): pass
print(foo.__annotations__)
This prints {'a': "'str'"}
. This shouldn’t really be considered
a “quirk”; it’s mentioned here simply because it might be surprising.
If you use a class with a custom metaclass and access __annotations__
on the class, you may observe unexpected behavior; see
749 for some examples. You can avoid these
quirks by using annotationlib.get_annotations()
on Python 3.14+ or
inspect.get_annotations()
on Python 3.10+. On earlier versions of
Python, you can avoid these bugs by accessing the annotations from the
class’s __dict__
(for example, cls.__dict__.get('__annotations__', None)
).
In some versions of Python, instances of classes may have an __annotations__
attribute. However, this is not supported functionality. If you need the
annotations of an instance, you can use type()
to access its class
(for example, annotationlib.get_annotations(type(myinstance))
on Python 3.14+).


How to port Python 2 Code to Python 3

How to port Python 2 Code to Python 3¶
- author:
Brett Cannon
Python 2 reached its official end-of-life at the start of 2020. This means that no new bug reports, fixes, or changes will be made to Python 2 - it’s no longer supported: see PEP 373 and status of Python versions.
If you are looking to port an extension module instead of pure Python code, please see Porting Extension Modules to Python 3.
The archived python-porting mailing list may contain some useful guidance.
Since Python 3.11 the original porting guide was discontinued. You can find the old guide in the archive.
Third-party guides¶
There are also multiple third-party guides that might be useful: