Python/C API Reference Manual

node1.html
node1.html
node46.html
---
# Python/C API Reference Manual
Guido van Rossum
February 17, 1998
Copyright © 1991-1995 by Stichting Mathematisch Centrum,
Amsterdam, The Netherlands.
All Rights Reserved
Permission to use, copy, modify, and distribute this software and its
documentation for any purpose and without fee is hereby granted,
provided that the above copyright notice appear in all copies and that
both that copyright notice and this permission notice appear in
supporting documentation, and that the names of Stichting Mathematisch
Centrum or CWI or Corporation for National Research Initiatives or
CNRI not be used in advertising or publicity pertaining to
distribution of the software without specific, written prior
permission.
While CWI is the initial source for this software, a modified version
is made available by the Corporation for National Research Initiatives
(CNRI) at the Internet address ftp://ftp.python.org.
STICHTING MATHEMATISCH CENTRUM AND CNRI DISCLAIM ALL WARRANTIES WITH
REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH
CENTRUM OR CNRI BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL
DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
### Abstract:
This manual documents the API used by C (or C++) programmers who
want to write extension modules or embed Python. It is a companion to
Extending and Embedding the Python Interpreter, which describes
the general principles of extension writing but does not document the
API functions in detail.
Warning: The current version of this document is incomplete.
I hope that it is nevertheless useful. I will continue to work on it,
and release new versions from time to time, independent from Python
source code releases.
---
- Contents (node1.html#SECTION001000000000000000000)
1 Introduction (node2.html#SECTION002000000000000000000)
- 1.1 Include Files (node3.html#SECTION002100000000000000000)
1.2 Objects, Types and Reference Counts (node4.html#SECTION002200000000000000000)
- Reference Counts (node5.html#SECTION002210000000000000000)
Types (node7.html#SECTION002220000000000000000)
1.3 Exceptions (node8.html#SECTION002300000000000000000)
1.4 Embedding Python (node9.html#SECTION002400000000000000000)
2 The Very High Level Layer (node10.html#SECTION003000000000000000000)
3 Reference Counting (node11.html#SECTION004000000000000000000)
4 Exception Handling (node12.html#SECTION005000000000000000000)
- 4.1 Standard Exceptions (node13.html#SECTION005100000000000000000)
5 Utilities (node14.html#SECTION006000000000000000000)
- 5.1 OS Utilities (node15.html#SECTION006100000000000000000)
5.2 Process Control (node16.html#SECTION006200000000000000000)
5.3 Importing Modules (node17.html#SECTION006300000000000000000)
6 Abstract Objects Layer (node18.html#SECTION007000000000000000000)
- 6.1 Object Protocol (node19.html#SECTION007100000000000000000)
6.2 Number Protocol (node20.html#SECTION007200000000000000000)
6.3 Sequence Protocol (node21.html#SECTION007300000000000000000)
6.4 Mapping Protocol (node22.html#SECTION007400000000000000000)
6.5 Constructors (node23.html#SECTION007500000000000000000)
7 Concrete Objects Layer (node24.html#SECTION008000000000000000000)
- 7.1 Fundamental Objects (node25.html#SECTION008100000000000000000)
- Type Objects (node26.html#SECTION008110000000000000000)
The None Object (node27.html#SECTION008120000000000000000)
7.2 Sequence Objects (node28.html#SECTION008200000000000000000)
- String Objects (node29.html#SECTION008210000000000000000)
Tuple Objects (node30.html#SECTION008220000000000000000)
List Objects (node31.html#SECTION008230000000000000000)
7.3 Mapping Objects (node32.html#SECTION008300000000000000000)
- Dictionary Objects (node33.html#SECTION008310000000000000000)
7.4 Numeric Objects (node34.html#SECTION008400000000000000000)
- Plain Integer Objects (node35.html#SECTION008410000000000000000)
Long Integer Objects (node36.html#SECTION008420000000000000000)
Floating Point Objects (node37.html#SECTION008430000000000000000)
Complex Number Objects (node38.html#SECTION008440000000000000000)
7.5 Other Objects (node39.html#SECTION008500000000000000000)
- File Objects (node40.html#SECTION008510000000000000000)
CObjects (node41.html#SECTION008520000000000000000)
8 Initialization, Finalization, and Threads (node42.html#SECTION009000000000000000000)
- 8.1 Thread State and the Global Interpreter Lock (node43.html#SECTION009100000000000000000)
9 Defining New Object Types (node44.html#SECTION0010000000000000000000)
10 Debugging (node45.html#SECTION0011000000000000000000)
Index (node46.html#SECTION0012000000000000000000)
About this document ... (node47.html#SECTION0013000000000000000000)
---
node1.html
node1.html
node46.html
guido@python.org


Python/C API Reference Manual


Contents

node2.html
api.html
api.html
node46.html
---
## Contents
- 1 Introduction (node2.html#SECTION002000000000000000000)
- 1.1 Include Files (node3.html#SECTION002100000000000000000)
1.2 Objects, Types and Reference Counts (node4.html#SECTION002200000000000000000)
- Reference Counts (node5.html#SECTION002210000000000000000)
- Reference Count Details (node6.html#SECTION002211000000000000000)
Types (node7.html#SECTION002220000000000000000)
1.3 Exceptions (node8.html#SECTION002300000000000000000)
1.4 Embedding Python (node9.html#SECTION002400000000000000000)
2 The Very High Level Layer (node10.html#SECTION003000000000000000000)
3 Reference Counting (node11.html#SECTION004000000000000000000)
4 Exception Handling (node12.html#SECTION005000000000000000000)
- 4.1 Standard Exceptions (node13.html#SECTION005100000000000000000)
5 Utilities (node14.html#SECTION006000000000000000000)
- 5.1 OS Utilities (node15.html#SECTION006100000000000000000)
5.2 Process Control (node16.html#SECTION006200000000000000000)
5.3 Importing Modules (node17.html#SECTION006300000000000000000)
6 Abstract Objects Layer (node18.html#SECTION007000000000000000000)
- 6.1 Object Protocol (node19.html#SECTION007100000000000000000)
6.2 Number Protocol (node20.html#SECTION007200000000000000000)
6.3 Sequence Protocol (node21.html#SECTION007300000000000000000)
6.4 Mapping Protocol (node22.html#SECTION007400000000000000000)
6.5 Constructors (node23.html#SECTION007500000000000000000)
7 Concrete Objects Layer (node24.html#SECTION008000000000000000000)
- 7.1 Fundamental Objects (node25.html#SECTION008100000000000000000)
- Type Objects (node26.html#SECTION008110000000000000000)
The None Object (node27.html#SECTION008120000000000000000)
7.2 Sequence Objects (node28.html#SECTION008200000000000000000)
- String Objects (node29.html#SECTION008210000000000000000)
Tuple Objects (node30.html#SECTION008220000000000000000)
List Objects (node31.html#SECTION008230000000000000000)
7.3 Mapping Objects (node32.html#SECTION008300000000000000000)
- Dictionary Objects (node33.html#SECTION008310000000000000000)
7.4 Numeric Objects (node34.html#SECTION008400000000000000000)
- Plain Integer Objects (node35.html#SECTION008410000000000000000)
Long Integer Objects (node36.html#SECTION008420000000000000000)
Floating Point Objects (node37.html#SECTION008430000000000000000)
Complex Number Objects (node38.html#SECTION008440000000000000000)
7.5 Other Objects (node39.html#SECTION008500000000000000000)
- File Objects (node40.html#SECTION008510000000000000000)
CObjects (node41.html#SECTION008520000000000000000)
8 Initialization, Finalization, and Threads (node42.html#SECTION009000000000000000000)
- 8.1 Thread State and the Global Interpreter Lock (node43.html#SECTION009100000000000000000)
9 Defining New Object Types (node44.html#SECTION0010000000000000000000)
10 Debugging (node45.html#SECTION0011000000000000000000)
Index (node46.html#SECTION0012000000000000000000)
About this document ... (node47.html#SECTION0013000000000000000000)
---
guido@python.org


2 The Very High Level Layer

node11.html
api.html
node9.html
node1.html
node46.html
---
# 2 The Very High Level Layer
The functions in this chapter will let you execute Python source code
given in a file or a buffer, but they will not let you interact in a
more detailed way with the interpreter.
---
guido@python.org


3 Reference Counting

node12.html
api.html
node10.html
node1.html
node46.html
---
# 3 Reference Counting
The macros in this section are used for managing reference counts
of Python objects.
The following functions or macros are only for internal use:
_Py_Dealloc(), _Py_ForgetReference(),
_Py_NewReference(), as well as the global variable
_Py_RefTotal.
XXX Should mention Py_Malloc(), Py_Realloc(), Py_Free(),
PyMem_Malloc(), PyMem_Realloc(), PyMem_Free(), PyMem_NEW(),
PyMem_RESIZE(), PyMem_DEL(), PyMem_XDEL().
---
guido@python.org


4 Exception Handling

node13.html
api.html
node11.html
node1.html
node46.html
---
# 4 Exception Handling
The functions in this chapter will let you handle and raise Python
exceptions. It is important to understand some of the basics of
Python exception handling. It works somewhat like the Unix
errno variable: there is a global indicator (per thread) of the
last error that occurred. Most functions don't clear this on success,
but will set it to indicate the cause of the error on failure. Most
functions also return an error indicator, usually NULL if they are
supposed to return a pointer, or -1 if they return an integer
(exception: the PyArg_Parse*() functions return 1 for
success and 0 for failure). When a function must fail because
some function it called failed, it generally doesn't set the error
indicator; the function it called already set it.
The error indicator consists of three Python objects corresponding to
the Python variables sys.exc_type, sys.exc_value and
sys.exc_traceback. API functions exist to interact with the
error indicator in various ways. There is a separate error indicator
for each thread.
---
- 4.1 Standard Exceptions (node13.html#SECTION005100000000000000000)
---
node13.html
api.html
node11.html
node1.html
node46.html
guido@python.org


4.1 Standard Exceptions

node14.html
node12.html
node12.html
node1.html
node46.html
---
# 4.1 Standard Exceptions
All standard Python exceptions are available as global variables whose
names are `PyExc_' followed by the Python exception name.
These have the type PyObject *; they are all either class
objects or string objects, depending on the use of the -X
option to the interpreter. For completeness, here are all the
variables:
PyExc_Exception,
PyExc_StandardError,
PyExc_ArithmeticError,
PyExc_LookupError,
PyExc_AssertionError,
PyExc_AttributeError,
PyExc_EOFError,
PyExc_FloatingPointError,
PyExc_IOError,
PyExc_ImportError,
PyExc_IndexError,
PyExc_KeyError,
PyExc_KeyboardInterrupt,
PyExc_MemoryError,
PyExc_NameError,
PyExc_OverflowError,
PyExc_RuntimeError,
PyExc_SyntaxError,
PyExc_SystemError,
PyExc_SystemExit,
PyExc_TypeError,
PyExc_ValueError,
PyExc_ZeroDivisionError.
---
guido@python.org


5 Utilities

node15.html
api.html
node13.html
node1.html
node46.html
---
# 5 Utilities
The functions in this chapter perform various utility tasks, such as
parsing function arguments and constructing Python values from C
values.
---
- 5.1 OS Utilities (node15.html#SECTION006100000000000000000)
5.2 Process Control (node16.html#SECTION006200000000000000000)
5.3 Importing Modules (node17.html#SECTION006300000000000000000)
---
guido@python.org


5.1 OS Utilities

node16.html
node14.html
node14.html
node1.html
node46.html
---
# 5.1 OS Utilities
---
guido@python.org


5.2 Process Control

node17.html
node14.html
node15.html
node1.html
node46.html
---
# 5.2 Process Control
---
guido@python.org


5.3 Importing Modules

node18.html
node14.html
node16.html
node1.html
node46.html
---
# 5.3 Importing Modules
---
node18.html
node14.html
node16.html
node1.html
node46.html
guido@python.org


6 Abstract Objects Layer

node19.html
api.html
node17.html
node1.html
node46.html
---
# 6 Abstract Objects Layer
The functions in this chapter interact with Python objects regardless
of their type, or with wide classes of object types (e.g. all
numerical types, or all sequence types). When used on object types
for which they do not apply, they will flag a Python exception.
---
- 6.1 Object Protocol (node19.html#SECTION007100000000000000000)
6.2 Number Protocol (node20.html#SECTION007200000000000000000)
6.3 Sequence Protocol (node21.html#SECTION007300000000000000000)
6.4 Mapping Protocol (node22.html#SECTION007400000000000000000)
6.5 Constructors (node23.html#SECTION007500000000000000000)
---
guido@python.org


6.1 Object Protocol

node20.html
node18.html
node18.html
node1.html
node46.html
---
# 6.1 Object Protocol
---
node20.html
node18.html
node18.html
node1.html
node46.html
guido@python.org


1 Introduction

node3.html
api.html
node1.html
node1.html
node46.html
---
# 1 Introduction
The Application Programmer's Interface to Python gives C and C++
programmers access to the Python interpreter at a variety of levels.
The API is equally usable from C++, but for brevity it is generally
referred to as the Python/C API. There are two fundamentally
different reasons for using the Python/C API. The first reason is
to write extension modules for specific purposes; these are
C modules that extend the Python interpreter. This is probably the
most common use. The second reason is to use Python as a component in
a larger application; this technique is generally referred to as
embedding Python in an application.
Writing an extension module is a relatively well-understood process,
where a ``cookbook'' approach works well. There are several tools
that automate the process to some extent. While people have embedded
Python in other applications since its early existence, the process of
embedding Python is less straightforward that writing an extension.
Python 1.5 introduces a number of new API functions as well as some
changes to the build process that make embedding much simpler.
This manual describes the , state of affair.
Many API functions are useful independent of whether you're embedding
or extending Python; moreover, most applications that embed Python
will need to provide a custom extension as well, so it's probably a
good idea to become familiar with writing an extension before
attempting to embed Python in a real application.
---
- 1.1 Include Files (node3.html#SECTION002100000000000000000)
1.2 Objects, Types and Reference Counts (node4.html#SECTION002200000000000000000)
- Reference Counts (node5.html#SECTION002210000000000000000)
- Reference Count Details (node6.html#SECTION002211000000000000000)
Types (node7.html#SECTION002220000000000000000)
1.3 Exceptions (node8.html#SECTION002300000000000000000)
1.4 Embedding Python (node9.html#SECTION002400000000000000000)
---
guido@python.org


6.2 Number Protocol

node21.html
node18.html
node19.html
node1.html
node46.html
---
# 6.2 Number Protocol
---
node21.html
node18.html
node19.html
node1.html
node46.html
guido@python.org


6.3 Sequence Protocol

node22.html
node18.html
node20.html
node1.html
node46.html
---
# 6.3 Sequence Protocol
---
node22.html
node18.html
node20.html
node1.html
node46.html
guido@python.org


6.4 Mapping Protocol

node23.html
node18.html
node21.html
node1.html
node46.html
---
# 6.4 Mapping Protocol
---
node23.html
node18.html
node21.html
node1.html
node46.html
guido@python.org


6.5 Constructors

node24.html
node18.html
node22.html
node1.html
node46.html
---
# 6.5 Constructors
---
node24.html
node18.html
node22.html
node1.html
node46.html
guido@python.org


7 Concrete Objects Layer

node25.html
api.html
node23.html
node1.html
node46.html
---
# 7 Concrete Objects Layer
The functions in this chapter are specific to certain Python object
types. Passing them an object of the wrong type is not a good idea;
if you receive an object from a Python program and you are not sure
that it has the right type, you must perform a type check first;
e.g. to check that an object is a dictionary, use
PyDict_Check(). The chapter is structured like the
``family tree'' of Python object types.
---
- 7.1 Fundamental Objects (node25.html#SECTION008100000000000000000)
- Type Objects (node26.html#SECTION008110000000000000000)
The None Object (node27.html#SECTION008120000000000000000)
7.2 Sequence Objects (node28.html#SECTION008200000000000000000)
- String Objects (node29.html#SECTION008210000000000000000)
Tuple Objects (node30.html#SECTION008220000000000000000)
List Objects (node31.html#SECTION008230000000000000000)
7.3 Mapping Objects (node32.html#SECTION008300000000000000000)
- Dictionary Objects (node33.html#SECTION008310000000000000000)
7.4 Numeric Objects (node34.html#SECTION008400000000000000000)
- Plain Integer Objects (node35.html#SECTION008410000000000000000)
Long Integer Objects (node36.html#SECTION008420000000000000000)
Floating Point Objects (node37.html#SECTION008430000000000000000)
Complex Number Objects (node38.html#SECTION008440000000000000000)
7.5 Other Objects (node39.html#SECTION008500000000000000000)
- File Objects (node40.html#SECTION008510000000000000000)
CObjects (node41.html#SECTION008520000000000000000)
---
guido@python.org


7.1 Fundamental Objects

node26.html
node24.html
node24.html
node1.html
node46.html
---
# 7.1 Fundamental Objects
This section describes Python type objects and the singleton object
None.
---
- Type Objects (node26.html#SECTION008110000000000000000)
The None Object (node27.html#SECTION008120000000000000000)
---
guido@python.org


Type Objects

node27.html
node25.html
node25.html
node1.html
node46.html
---
## Type Objects
---
guido@python.org


The None Object

node28.html
node25.html
node26.html
node1.html
node46.html
---
## The None Object
---
guido@python.org


7.2 Sequence Objects

node29.html
node24.html
node27.html
node1.html
node46.html
---
# 7.2 Sequence Objects
Generic operations on sequence objects were discussed in the previous
chapter; this section deals with the specific kinds of sequence
objects that are intrinsic to the Python language.
---
- String Objects (node29.html#SECTION008210000000000000000)
Tuple Objects (node30.html#SECTION008220000000000000000)
List Objects (node31.html#SECTION008230000000000000000)
---
guido@python.org


String Objects

node30.html
node28.html
node28.html
node1.html
node46.html
---
## String Objects
---
guido@python.org


1.1 Include Files

node4.html
node2.html
node2.html
node1.html
node46.html
---
# 1.1 Include Files
All function, type and macro definitions needed to use the Python/C
API are included in your code by the following line:
```text
#include "Python.h"
```
This implies inclusion of the following standard headers:
<stdio.h>, <string.h>, <errno.h>, and
<stdlib.h> (if available).
All user visible names defined by Python.h (except those defined by
the included standard headers) have one of the prefixes `Py' or
`_Py'. Names beginning with `_Py' are for internal use
only. Structure member names do not have a reserved prefix.
Important: user code should never define names that begin
with `Py' or `_Py'. This confuses the reader, and
jeopardizes the portability of the user code to future Python
versions, which may define additional names beginning with one of
these prefixes.
---
guido@python.org


Tuple Objects

node31.html
node28.html
node29.html
node1.html
node46.html
---
## Tuple Objects
---
node31.html
node28.html
node29.html
node1.html
node46.html
guido@python.org


List Objects

node32.html
node28.html
node30.html
node1.html
node46.html
---
## List Objects
---
guido@python.org


7.3 Mapping Objects

node33.html
node24.html
node31.html
node1.html
node46.html
---
# 7.3 Mapping Objects
---
- Dictionary Objects (node33.html#SECTION008310000000000000000)
---
guido@python.org


Dictionary Objects

node34.html
node32.html
node32.html
node1.html
node46.html
---
## Dictionary Objects
---
node34.html
node32.html
node32.html
node1.html
node46.html
guido@python.org


7.4 Numeric Objects

node35.html
node24.html
node33.html
node1.html
node46.html
---
# 7.4 Numeric Objects
---
- Plain Integer Objects (node35.html#SECTION008410000000000000000)
Long Integer Objects (node36.html#SECTION008420000000000000000)
Floating Point Objects (node37.html#SECTION008430000000000000000)
Complex Number Objects (node38.html#SECTION008440000000000000000)
---
guido@python.org


Plain Integer Objects

node36.html
node34.html
node34.html
node1.html
node46.html
---
## Plain Integer Objects
---
guido@python.org


Long Integer Objects

node37.html
node34.html
node35.html
node1.html
node46.html
---
## Long Integer Objects
---
guido@python.org


Floating Point Objects

node38.html
node34.html
node36.html
node1.html
node46.html
---
## Floating Point Objects
---
guido@python.org


Complex Number Objects

node39.html
node34.html
node37.html
node1.html
node46.html
---
## Complex Number Objects
---
guido@python.org


7.5 Other Objects

node40.html
node24.html
node38.html
node1.html
node46.html
---
# 7.5 Other Objects
---
- File Objects (node40.html#SECTION008510000000000000000)
CObjects (node41.html#SECTION008520000000000000000)
---
guido@python.org


1.2 Objects, Types and Reference Counts

node5.html
node2.html
node3.html
node1.html
node46.html
---
# 1.2 Objects, Types and Reference Counts
Most Python/C API functions have one or more arguments as well as a
return value of type PyObject *. This type is a pointer
to an opaque data type representing an arbitrary Python
object. Since all Python object types are treated the same way by the
Python language in most situations (e.g., assignments, scope rules,
and argument passing), it is only fitting that they should be
represented by a single C type. All Python objects live on the heap:
you never declare an automatic or static variable of type
PyObject, only pointer variables of type PyObject * can
be declared.
All Python objects (even Python integers) have a type and a
reference count. An object's type determines what kind of object
it is (e.g., an integer, a list, or a user-defined function; there are
many more as explained in the Python Reference Manual). For
each of the well-known types there is a macro to check whether an
object is of that type; for instance, `PyList_Check(a)' is
true iff the object pointed to by a is a Python list.
---
- Reference Counts (node5.html#SECTION002210000000000000000)
- Reference Count Details (node6.html#SECTION002211000000000000000)
Types (node7.html#SECTION002220000000000000000)
---
guido@python.org


File Objects

node41.html
node39.html
node39.html
node1.html
node46.html
---
## File Objects
---
guido@python.org


CObjects

node42.html
node39.html
node40.html
node1.html
node46.html
---
## CObjects
XXX
---
guido@python.org


node43.html
api.html
node41.html
node1.html
node46.html
---
# 8 Initialization, Finalization, and Threads
---
- 8.1 Thread State and the Global Interpreter Lock (node43.html#SECTION009100000000000000000)
---
node43.html
api.html
node41.html
node1.html
node46.html
guido@python.org


node44.html
node42.html
node42.html
node1.html
node46.html
---
# 8.1 Thread State and the Global Interpreter Lock
The Python interpreter is not fully thread safe. In order to support
multi-threaded Python programs, there's a global lock that must be
held by the current thread before it can safely access Python objects.
Without the lock, even the simplest operations could cause problems in
a multi-threaded proram: for example, when two threads simultaneously
increment the reference count of the same object, the reference count
could end up being incremented only once instead of twice.
Therefore, the rule exists that only the thread that has acquired the
global interpreter lock may operate on Python objects or call Python/C
API functions. In order to support multi-threaded Python programs,
the interpreter regularly release and reacquires the lock -- by
default, every ten bytecode instructions (this can be changed with
sys.setcheckinterval()). The lock is also released and
reacquired around potentially blocking I/O operations like reading or
writing a file, so that other threads can run while the thread that
requests the I/O is waiting for the I/O operation to complete.
The Python interpreter needs to keep some bookkeeping information
separate per thread -- for this it uses a data structure called
PyThreadState. This is new in Python 1.5; in earlier versions,
such state was stored in global variables, and switching threads could
cause problems. In particular, exception handling is now thread safe,
when the application uses sys.exc_info() to access the
exception last raised in the current thread.
There's one global variable left, however: the pointer to the current
PyThreadState structure. While most thread packages have a way
to store ``per-thread global data,'' Python's internal platform
independent thread abstraction doesn't support this yet. Therefore,
the current thread state must be manipulated explicitly.
This is easy enough in most cases. Most code manipulating the global
interpreter lock has the following simple structure:
```text
Save the thread state in a local variable.
Release the interpreter lock.
...Do some blocking I/O operation...
Reacquire the interpreter lock.
Restore the thread state from the local variable.
```
This is so common that a pair of macros exists to simplify it:
```text
Py_BEGIN_ALLOW_THREADS
...Do some blocking I/O operation...
Py_END_ALLOW_THREADS
```
The Py_BEGIN_ALLOW_THREADS macro opens a new block and declares
a hidden local variable; the Py_END_ALLOW_THREADS macro closes
the block. Another advantage of using these two macros is that when
Python is compiled without thread support, they are defined empty,
thus saving the thread state and lock manipulations.
When thread support is enabled, the block above expands to the
following code:
```text
{
PyThreadState *_save;
_save = PyEval_SaveThread();
...Do some blocking I/O operation...
PyEval_RestoreThread(_save);
}
```
Using even lower level primitives, we can get roughly the same effect
as follows:
```text
{
PyThreadState *_save;
_save = PyThreadState_Swap(NULL);
PyEval_ReleaseLock();
...Do some blocking I/O operation...
PyEval_AcquireLock();
PyThreadState_Swap(_save);
}
```
There are some subtle differences; in particular,
PyEval_RestoreThread() saves and restores the value of the
global variable errno, since the lock manipulation does not
guarantee that errno is left alone. Also, when thread support
is disabled, PyEval_SaveThread() and
PyEval_RestoreThread() don't manipulate the lock; in this
case, PyEval_ReleaseLock() and
PyEval_AcquireLock() are not available. This is done so
that dynamically loaded extensions compiled with thread support
enabled can be loaded by an interpreter that was compiled with
disabled thread support.
The global interpreter lock is used to protect the pointer to the
current thread state. When releasing the lock and saving the thread
state, the current thread state pointer must be retrieved before the
lock is released (since another thread could immediately acquire the
lock and store its own thread state in the global variable).
Reversely, when acquiring the lock and restoring the thread state, the
lock must be acquired before storing the thread state pointer.
Why am I going on with so much detail about this? Because when
threads are created from C, they don't have the global interpreter
lock, nor is there a thread state data structure for them. Such
threads must bootstrap themselves into existence, by first creating a
thread state data structure, then acquiring the lock, and finally
storing their thread state pointer, before they can start using the
Python/C API. When they are done, they should reset the thread state
pointer, release the lock, and finally free their thread state data
structure.
When creating a thread data structure, you need to provide an
interpreter state data structure. The interpreter state data
structure hold global data that is shared by all threads in an
interpreter, for example the module administration
(sys.modules). Depending on your needs, you can either create
a new interpreter state data structure, or share the interpreter state
data structure used by the Python main thread (to access the latter,
you must obtain the thread state and access its interp member;
this must be done by a thread that is created by Python or by the main
thread after Python is initialized).
XXX More?
All of the following functions are only available when thread support
is enabled at compile time, and must be called only when the
interpreter lock has been created.
---
node44.html
node42.html
node42.html
node1.html
node46.html
guido@python.org


9 Defining New Object Types

node45.html
api.html
node43.html
node1.html
node46.html
---
# 9 Defining New Object Types
PyObject, PyVarObject
PyObject_HEAD, PyObject_HEAD_INIT, PyObject_VAR_HEAD
Typedefs:
unaryfunc, binaryfunc, ternaryfunc, inquiry, coercion, intargfunc,
intintargfunc, intobjargproc, intintobjargproc, objobjargproc,
getreadbufferproc, getwritebufferproc, getsegcountproc,
destructor, printfunc, getattrfunc, getattrofunc, setattrfunc,
setattrofunc, cmpfunc, reprfunc, hashfunc
PyNumberMethods
PySequenceMethods
PyMappingMethods
PyBufferProcs
PyTypeObject
DL_IMPORT
PyType_Type
Py*_Check
Py_None, _Py_NoneStruct
---
guido@python.org


10 Debugging

node46.html
api.html
node44.html
node1.html
node46.html
---
# 10 Debugging
XXX Explain Py_DEBUG, Py_TRACE_REFS, Py_REF_DEBUG.
---
guido@python.org


Index

node47.html
api.html
node45.html
node1.html
---
## Index
---
guido@python.org


About this document ...

api.html
node46.html
node1.html
node46.html
---
# About this document ...
Python/C API Reference Manual
This document was generated using the LaTeX2HTML (http://www-dsed.llnl.gov/files/programs/unix/latex2html/manual/) translator Version 96.1 (Feb 5, 1996) Copyright © 1993, 1994, 1995, 1996, Nikos Drakos (http://cbl.leeds.ac.uk/nikos/personal.html), Computer Based Learning Unit, University of Leeds.
The command line arguments were:
latex2html -address guido@python.org api.tex.
The translation was initiated by Fred L. Drake on Tue Feb 17 10:32:21 EST 1998
---
guido@python.org


Reference Counts

node6.html
node4.html
node4.html
node1.html
node46.html
---
## Reference Counts
The reference count is important because today's computers have a
finite (and often severly limited) memory size; it counts how many
different places there are that have a reference to an object. Such a
place could be another object, or a global (or static) C variable, or
a local variable in some C function. When an object's reference count
becomes zero, the object is deallocated. If it contains references to
other objects, their reference count is decremented. Those other
objects may be deallocated in turn, if this decrement makes their
reference count become zero, and so on. (There's an obvious problem
with objects that reference each other here; for now, the solution is
``don't do that''.)
Reference counts are always manipulated explicitly. The normal way is
to use the macro Py_INCREF() to increment an object's
reference count by one, and Py_DECREF() to decrement it by
one. The decref macro is considerably more complex than the incref one,
since it must check whether the reference count becomes zero and then
cause the object's deallocator, which is a function pointer contained
in the object's type structure. The type-specific deallocator takes
care of decrementing the reference counts for other objects contained
in the object, and so on, if this is a compound object type such as a
list. There's no chance that the reference count can overflow; at
least as many bits are used to hold the reference count as there are
distinct memory locations in virtual memory (assuming
sizeof(long) >= sizeof(char *)). Thus, the reference count
increment is a simple operation.
It is not necessary to increment an object's reference count for every
local variable that contains a pointer to an object. In theory, the
object's reference count goes up by one when the variable is made to
point to it and it goes down by one when the variable goes out of
scope. However, these two cancel each other out, so at the end the
reference count hasn't changed. The only real reason to use the
reference count is to prevent the object from being deallocated as
long as our variable is pointing to it. If we know that there is at
least one other reference to the object that lives at least as long as
our variable, there is no need to increment the reference count
temporarily. An important situation where this arises is in objects
that are passed as arguments to C functions in an extension module
that are called from Python; the call mechanism guarantees to hold a
reference to every argument for the duration of the call.
However, a common pitfall is to extract an object from a list and
hold on to it for a while without incrementing its reference count.
Some other operation might conceivably remove the object from the
list, decrementing its reference count and possible deallocating it.
The real danger is that innocent-looking operations may invoke
arbitrary Python code which could do this; there is a code path which
allows control to flow back to the user from a Py_DECREF(),
so almost any operation is potentially dangerous.
A safe approach is to always use the generic operations (functions
whose name begins with `PyObject_', `PyNumber_',
`PySequence_' or `PyMapping_'). These operations always
increment the reference count of the object they return. This leaves
the caller with the responsibility to call Py_DECREF()
when they are done with the result; this soon becomes second nature.
---
- Reference Count Details (node6.html#SECTION002211000000000000000)
---
node6.html
node4.html
node4.html
node1.html
node46.html
guido@python.org


Reference Count Details

node7.html
node5.html
node5.html
node1.html
node46.html
---
### Reference Count Details
The reference count behavior of functions in the Python/C API is best
expelained in terms of ownership of references. Note that we
talk of owning references, never of owning objects; objects are always
shared! When a function owns a reference, it has to dispose of it
properly -- either by passing ownership on (usually to its caller) or
by calling Py_DECREF() or Py_XDECREF(). When
a function passes ownership of a reference on to its caller, the
caller is said to receive a new reference. When no ownership
is transferred, the caller is said to borrow the reference.
Nothing needs to be done for a borrowed reference.
Conversely, when calling a function passes it a reference to an
object, there are two possibilities: the function steals a
reference to the object, or it does not. Few functions steal
references; the two notable exceptions are
PyList_SetItem() and PyTuple_SetItem(), which
steal a reference to the item (but not to the tuple or list into which
the item it put!). These functions were designed to steal a reference
because of a common idiom for populating a tuple or list with newly
created objects; for example, the code to create the tuple (1,
2, "three") could look like this (forgetting about error handling for
the moment; a better way to code this is shown below anyway):
```text
PyObject *t;
t = PyTuple_New(3);
PyTuple_SetItem(t, 0, PyInt_FromLong(1L));
PyTuple_SetItem(t, 1, PyInt_FromLong(2L));
PyTuple_SetItem(t, 2, PyString_FromString("three"));
```
Incidentally, PyTuple_SetItem() is the only way to
set tuple items; PySequence_SetItem() and
PyObject_SetItem() refuse to do this since tuples are an
immutable data type. You should only use
PyTuple_SetItem() for tuples that you are creating
yourself.
Equivalent code for populating a list can be written using
PyList_New() and PyList_SetItem(). Such code
can also use PySequence_SetItem(); this illustrates the
difference between the two (the extra Py_DECREF() calls):
```text
PyObject *l, *x;
l = PyList_New(3);
x = PyInt_FromLong(1L);
PySequence_SetItem(l, 0, x); Py_DECREF(x);
x = PyInt_FromLong(2L);
PySequence_SetItem(l, 1, x); Py_DECREF(x);
x = PyString_FromString("three");
PySequence_SetItem(l, 2, x); Py_DECREF(x);
```
You might find it strange that the ``recommended'' approach takes more
code. However, in practice, you will rarely use these ways of
creating and populating a tuple or list. There's a generic function,
Py_BuildValue(), that can create most common objects from
C values, directed by a format string. For example, the
above two blocks of code could be replaced by the following (which
also takes care of the error checking):
```text
PyObject *t, *l;
t = Py_BuildValue("(iis)", 1, 2, "three");
l = Py_BuildValue("[iis]", 1, 2, "three");
```
It is much more common to use PyObject_SetItem() and
friends with items whose references you are only borrowing, like
arguments that were passed in to the function you are writing. In
that case, their behaviour regarding reference counts is much saner,
since you don't have to increment a reference count so you can give a
reference away (``have it be stolen''). For example, this function
sets all items of a list (actually, any mutable sequence) to a given
item:
```text
int set_all(PyObject *target, PyObject *item)
{
int i, n;
n = PyObject_Length(target);
if (n < 0)
return -1;
for (i = 0; i < n; i++) {
if (PyObject_SetItem(target, i, item) < 0)
return -1;
}
return 0;
}
```
The situation is slightly different for function return values.
While passing a reference to most functions does not change your
ownership responsibilities for that reference, many functions that
return a referece to an object give you ownership of the reference.
The reason is simple: in many cases, the returned object is created
on the fly, and the reference you get is the only reference to the
object. Therefore, the generic functions that return object
references, like PyObject_GetItem() and
PySequence_GetItem(), always return a new reference (i.e.,
the caller becomes the owner of the reference).
It is important to realize that whether you own a reference returned
by a function depends on which function you call only -- the
plumage (i.e., the type of the type of the object passed as an
argument to the function) doesn't enter into it! Thus, if you
extract an item from a list using PyList_GetItem(), you
don't own the reference -- but if you obtain the same item from the
same list using PySequence_GetItem() (which happens to
take exactly the same arguments), you do own a reference to the
returned object.
Here is an example of how you could write a function that computes the
sum of the items in a list of integers; once using
PyList_GetItem(), once using
PySequence_GetItem().
```text
long sum_list(PyObject *list)
{
int i, n;
long total = 0;
PyObject *item;
n = PyList_Size(list);
if (n < 0)
return -1; /* Not a list */
for (i = 0; i < n; i++) {
item = PyList_GetItem(list, i); /* Can't fail */
if (!PyInt_Check(item)) continue; /* Skip non-integers */
total += PyInt_AsLong(item);
}
return total;
}
```
```text
long sum_sequence(PyObject *sequence)
{
int i, n;
long total = 0;
PyObject *item;
n = PyObject_Size(list);
if (n < 0)
return -1; /* Has no length */
for (i = 0; i < n; i++) {
item = PySequence_GetItem(list, i);
if (item == NULL)
return -1; /* Not a sequence, or other failure */
if (PyInt_Check(item))
total += PyInt_AsLong(item);
Py_DECREF(item); /* Discard reference ownership */
}
return total;
}
```
---
node7.html
node5.html
node5.html
node1.html
node46.html
guido@python.org


Types

node8.html
node4.html
node6.html
node1.html
node46.html
---
## Types
There are few other data types that play a significant role in
the Python/C API; most are simple C types such as int,
long, double and char *. A few structure types
are used to describe static tables used to list the functions exported
by a module or the data attributes of a new object type. These will
be discussed together with the functions that use them.
---
guido@python.org


1.3 Exceptions

node9.html
node2.html
node7.html
node1.html
node46.html
---
# 1.3 Exceptions
The Python programmer only needs to deal with exceptions if specific
error handling is required; unhandled exceptions are automatically
propagated to the caller, then to the caller's caller, and so on, till
they reach the top-level interpreter, where they are reported to the
user accompanied by a stack traceback.
For C programmers, however, error checking always has to be explicit.
All functions in the Python/C API can raise exceptions, unless an
explicit claim is made otherwise in a function's documentation. In
general, when a function encounters an error, it sets an exception,
discards any object references that it owns, and returns an
error indicator -- usually NULL or -1. A few functions
return a Boolean true/false result, with false indicating an error.
Very few functions return no explicit error indicator or have an
ambiguous return value, and require explicit testing for errors with
PyErr_Occurred().
Exception state is maintained in per-thread storage (this is
equivalent to using global storage in an unthreaded application). A
thread can be on one of two states: an exception has occurred, or not.
The function PyErr_Occurred() can be used to check for
this: it returns a borrowed reference to the exception type object
when an exception has occurred, and NULL otherwise. There are a
number of functions to set the exception state:
PyErr_SetString() is the most common (though not the most
general) function to set the exception state, and
PyErr_Clear() clears the exception state.
The full exception state consists of three objects (all of which can
be NULL): the exception type, the corresponding exception
value, and the traceback. These have the same meanings as the Python
object sys.exc_type, sys.exc_value,
sys.exc_traceback; however, they are not the same: the Python
objects represent the last exception being handled by a Python
try ... except statement, while the C level
exception state only exists while an exception is being passed on
between C functions until it reaches the Python interpreter, which
takes care of transferring it to sys.exc_type and friends.
(Note that starting with Python 1.5, the preferred, thread-safe way to
access the exception state from Python code is to call the function
sys.exc_info(), which returns the per-thread exception state
for Python code. Also, the semantics of both ways to access the
exception state have changed so that a function which catches an
exception will save and restore its thread's exception state so as to
preserve the exception state of its caller. This prevents common bugs
in exception handling code caused by an innocent-looking function
overwriting the exception being handled; it also reduces the often
unwanted lifetime extension for objects that are referenced by the
stack frames in the traceback.)
As a general principle, a function that calls another function to
perform some task should check whether the called function raised an
exception, and if so, pass the exception state on to its caller. It
should discard any object references that it owns, and returns an
error indicator, but it should not set another exception --
that would overwrite the exception that was just raised, and lose
important information about the exact cause of the error.
A simple example of detecting exceptions and passing them on is shown
in the sum_sequence() example above. It so happens that
that example doesn't need to clean up any owned references when it
detects an error. The following example function shows some error
cleanup. First, to remind you why you like Python, we show the
equivalent Python code:
```text
def incr_item(dict, key):
try:
item = dict[key]
except KeyError:
item = 0
return item + 1
```
Here is the corresponding C code, in all its glory:
```text
int incr_item(PyObject *dict, PyObject *key)
{
/* Objects all initialized to NULL for Py_XDECREF */
PyObject *item = NULL, *const_one = NULL, *incremented_item = NULL;
int rv = -1; /* Return value initialized to -1 (failure) */

item = PyObject_GetItem(dict, key);
if (item == NULL) {
/* Handle keyError only: */
if (!PyErr_ExceptionMatches(PyExc_keyError)) goto error;

/* Clear the error and use zero: */
PyErr_Clear();
item = PyInt_FromLong(0L);
if (item == NULL) goto error;
}

const_one = PyInt_FromLong(1L);
if (const_one == NULL) goto error;

incremented_item = PyNumber_Add(item, const_one);
if (incremented_item == NULL) goto error;

if (PyObject_SetItem(dict, key, incremented_item) < 0) goto error;
rv = 0; /* Success */
/* Continue with cleanup code */

error:
/* Cleanup code, shared by success and failure path */

/* Use Py_XDECREF() to ignore NULL references */
Py_XDECREF(item);
Py_XDECREF(const_one);
Py_XDECREF(incremented_item);

return rv; /* -1 for error, 0 for success */
}
```
This example represents an endorsed use of the goto statement
in C! It illustrates the use of
PyErr_ExceptionMatches() and PyErr_Clear() to
handle specific exceptions, and the use of Py_XDECREF() to
dispose of owned references that may be NULL (note the `X' in
the name; Py_DECREF() would crash when confronted with a
NULL reference). It is important that the variables used to hold
owned references are initialized to NULL for this to work;
likewise, the proposed return value is initialized to -1
(failure) and only set to success after the final call made is
successful.
---
node9.html
node2.html
node7.html
node1.html
node46.html
guido@python.org


1.4 Embedding Python

node10.html
node2.html
node8.html
node1.html
node46.html
---
# 1.4 Embedding Python
The one important task that only embedders (as opposed to extension
writers) of the Python interpreter have to worry about is the
initialization, and possibly the finalization, of the Python
interpreter. Most functionality of the interpreter can only be used
after the interpreter has been initialized.
The basic initialization function is Py_Initialize().
This initializes the table of loaded modules, and creates the
fundamental modules __builtin__,
__main__ and
sys. It also initializes the module
search path (sys.path).
Py_Initialize() does not set the ``script argument list''
(sys.argv). If this variable is needed by Python code that
will be executed later, it must be set explicitly with a call to
PySys_SetArgv(argc, argv) subsequent to the call
to Py_Initialize().
On most systems (in particular, on Unix and Windows, although the
details are slightly different), Py_Initialize()
calculates the module search path based upon its best guess for the
location of the standard Python interpreter executable, assuming that
the Python library is found in a fixed location relative to the Python
interpreter executable. In particular, it looks for a directory named
`lib/python,' (replacing `,' with the current
interpreter version) relative to the parent directory where the
executable named `python' is found on the shell command search
path (the environment variable $PATH).
For instance, if the Python executable is found in
`/usr/local/bin/python', it will assume that the libraries are in
`/usr/local/lib/python,'. (In fact, this particular path
is also the ``fallback'' location, used when no executable file named
`python' is found along $PATH.) The user can override
this behavior by setting the environment variable $PYTHONHOME,
or insert additional directories in front of the standard path by
setting $PYTHONPATH.
The embedding application can steer the search by calling
Py_SetProgramName(file) before calling
Py_Initialize(). Note that $PYTHONHOME still
overrides this and $PYTHONPATH is still inserted in front of
the standard path. An application that requires total control has to
provide its own implementation of Py_GetPath(),
Py_GetPrefix(), Py_GetExecPrefix(),
Py_GetProgramFullPath() (all defined in
`Modules/getpath.c').
Sometimes, it is desirable to ``uninitialize'' Python. For instance,
the application may want to start over (make another call to
Py_Initialize()) or the application is simply done with its
use of Python and wants to free all memory allocated by Python. This
can be accomplished by calling Py_Finalize(). The function
Py_IsInitialized() returns true iff Python is currently in the
initialized state. More information about these functions is given in
a later chapter.
---
node10.html
node2.html
node8.html
node1.html
node46.html
guido@python.org


node1.html
node1.html
---
# Extending and Embedding the Python Interpreter
Guido van Rossum
February 17, 1998
Copyright © 1991-1995 by Stichting Mathematisch Centrum,
Amsterdam, The Netherlands.
All Rights Reserved
Permission to use, copy, modify, and distribute this software and its
documentation for any purpose and without fee is hereby granted,
provided that the above copyright notice appear in all copies and that
both that copyright notice and this permission notice appear in
supporting documentation, and that the names of Stichting Mathematisch
Centrum or CWI or Corporation for National Research Initiatives or
CNRI not be used in advertising or publicity pertaining to
distribution of the software without specific, written prior
permission.
While CWI is the initial source for this software, a modified version
is made available by the Corporation for National Research Initiatives
(CNRI) at the Internet address ftp://ftp.python.org.
STICHTING MATHEMATISCH CENTRUM AND CNRI DISCLAIM ALL WARRANTIES WITH
REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH
CENTRUM OR CNRI BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL
DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
PERFORMANCE OF THIS SOFTWARE.
### Abstract:
Python is an interpreted, object-oriented programming language. This
document describes how to write modules in C or C++ to extend the
Python interpreter with new modules. Those modules can define new
functions but also new object types and their methods. The document
also describes how to embed the Python interpreter in another
application, for use as an extension language. Finally, it shows how
to compile and link extension modules so that they can be loaded
dynamically (at run time) into the interpreter, if the underlying
operating system supports this feature.
This document assumes basic knowledge about Python. For an informal
introduction to the language, see the Python Tutorial. The Python
Reference Manual gives a more formal definition of the language. The
Python Library Reference documents the existing object types,
functions and modules (both built-in and written in Python) that give
the language its wide application range.
For a detailed description of the whole Python/C API, see the separate
Python/C API Reference Manual. Note: While that manual is
still in a state of flux, it is safe to say that it is much more up to
date than the manual you're reading currently (which has been in need
for an upgrade for some time now).
---
- Contents (node1.html#SECTION001000000000000000000)
Extending Python with C or C++ code (node2.html#SECTION002000000000000000000)
- 1.1 Introduction (node3.html#SECTION002100000000000000000)
1.2 A Simple Example (node4.html#SECTION002200000000000000000)
1.3 Intermezzo: Errors and Exceptions (node5.html#SECTION002300000000000000000)
1.4 Back to the Example (node6.html#SECTION002400000000000000000)
1.5 The Module's Method Table and Initialization Function (node7.html#SECTION002500000000000000000)
1.6 Compilation and Linkage (node8.html#SECTION002600000000000000000)
1.7 Calling Python Functions From C (node9.html#SECTION002700000000000000000)
Format Strings for PyArg_ParseTuple() (node10.html#SECTION002800000000000000000)
Keyword Parsing with PyArg_ParseTupleAndKeywords() (node11.html#SECTION002900000000000000000)
The Py_BuildValue() Function (node12.html#SECTION0021000000000000000000)
1.11 Reference Counts (node13.html#SECTION0021100000000000000000)
- Introduction (node14.html#SECTION0021110000000000000000)
Reference Counting in Python (node15.html#SECTION0021120000000000000000)
Ownership Rules (node16.html#SECTION0021130000000000000000)
Thin Ice (node17.html#SECTION0021140000000000000000)
NULL Pointers (node18.html#SECTION0021150000000000000000)
Writing Extensions in C++ (node19.html#SECTION0021200000000000000000)
2 Embedding Python in another application (node20.html#SECTION003000000000000000000)
- Embedding Python in C++ (node21.html#SECTION003100000000000000000)
3 Dynamic Loading (node22.html#SECTION004000000000000000000)
- 3.1 Configuring and Building the Interpreter for Dynamic Loading (node23.html#SECTION004100000000000000000)
- Shared Libraries (node24.html#SECTION004110000000000000000)
SGI IRIX 4 Dynamic Loading (node25.html#SECTION004120000000000000000)
GNU Dynamic Loading (node26.html#SECTION004130000000000000000)
3.2 Building a Dynamically Loadable Module (node27.html#SECTION004200000000000000000)
- Shared Libraries (node28.html#SECTION004210000000000000000)
SGI IRIX 4 Dynamic Loading (node29.html#SECTION004220000000000000000)
GNU Dynamic Loading (node30.html#SECTION004230000000000000000)
About this document ... (node31.html#SECTION005000000000000000000)
---
node1.html
node1.html
guido@python.org


Footnotes

guido@python.org


Contents

node2.html
ext.html
ext.html
---
## Contents
- Extending Python with C or C++ code (node2.html#SECTION002000000000000000000)
- 1.1 Introduction (node3.html#SECTION002100000000000000000)
1.2 A Simple Example (node4.html#SECTION002200000000000000000)
1.3 Intermezzo: Errors and Exceptions (node5.html#SECTION002300000000000000000)
1.4 Back to the Example (node6.html#SECTION002400000000000000000)
1.5 The Module's Method Table and Initialization Function (node7.html#SECTION002500000000000000000)
1.6 Compilation and Linkage (node8.html#SECTION002600000000000000000)
1.7 Calling Python Functions From C (node9.html#SECTION002700000000000000000)
Format Strings for PyArg_ParseTuple() (node10.html#SECTION002800000000000000000)
Keyword Parsing with PyArg_ParseTupleAndKeywords() (node11.html#SECTION002900000000000000000)
The Py_BuildValue() Function (node12.html#SECTION0021000000000000000000)
1.11 Reference Counts (node13.html#SECTION0021100000000000000000)
- Introduction (node14.html#SECTION0021110000000000000000)
Reference Counting in Python (node15.html#SECTION0021120000000000000000)
Ownership Rules (node16.html#SECTION0021130000000000000000)
Thin Ice (node17.html#SECTION0021140000000000000000)
NULL Pointers (node18.html#SECTION0021150000000000000000)
Writing Extensions in C++ (node19.html#SECTION0021200000000000000000)
2 Embedding Python in another application (node20.html#SECTION003000000000000000000)
- Embedding Python in C++ (node21.html#SECTION003100000000000000000)
3 Dynamic Loading (node22.html#SECTION004000000000000000000)
- 3.1 Configuring and Building the Interpreter for Dynamic Loading (node23.html#SECTION004100000000000000000)
- Shared Libraries (node24.html#SECTION004110000000000000000)
SGI IRIX 4 Dynamic Loading (node25.html#SECTION004120000000000000000)
GNU Dynamic Loading (node26.html#SECTION004130000000000000000)
3.2 Building a Dynamically Loadable Module (node27.html#SECTION004200000000000000000)
- Shared Libraries (node28.html#SECTION004210000000000000000)
SGI IRIX 4 Dynamic Loading (node29.html#SECTION004220000000000000000)
GNU Dynamic Loading (node30.html#SECTION004230000000000000000)
About this document ... (node31.html#SECTION005000000000000000000)
---
guido@python.org


Format Strings for PyArg_ParseTuple()

node11.html
node2.html
node9.html
node1.html
---
# Format Strings for PyArg_ParseTuple()
The PyArg_ParseTuple() function is declared as follows:
```text
int PyArg_ParseTuple(PyObject *arg, char *format, ...);
```
The arg argument must be a tuple object containing an argument
list passed from Python to a C function. The format argument
must be a format string, whose syntax is explained below. The
remaining arguments must be addresses of variables whose type is
determined by the format string. For the conversion to succeed, the
arg object must match the format and the format must be
exhausted.
Note that while PyArg_ParseTuple() checks that the Python
arguments have the required types, it cannot check the validity of the
addresses of C variables passed to the call: if you make mistakes
there, your code will probably crash or at least overwrite random bits
in memory. So be careful!
A format string consists of zero or more ``format units''. A format
unit describes one Python object; it is usually a single character or
a parenthesized sequence of format units. With a few exceptions, a
format unit that is not a parenthesized sequence normally corresponds
to a single address argument to PyArg_ParseTuple(). In the
following description, the quoted form is the format unit; the entry
in (round) parentheses is the Python object type that matches the
format unit; and the entry in [square] brackets is the type of the C
variable(s) whose address should be passed. (Use the `&'
operator to pass a variable's address.)
It is possible to pass Python long integers where integers are
requested; however no proper range checking is done -- the most
significant bits are silently truncated when the receiving field is
too small to receive the value (actually, the semantics are inherited
from downcasts in C -- your milage may vary).
A few other characters have a meaning in a format string. These may
not occur inside nested parentheses. They are:
Some example calls:
```text
int ok;
int i, j;
long k, l;
char *s;
int size;

ok = PyArg_ParseTuple(args, ""); /* No arguments */
/* Python call: f() */

ok = PyArg_ParseTuple(args, "s", &s); /* A string */
/* Possible Python call: f('whoops!') */

ok = PyArg_ParseTuple(args, "lls", &k, &l, &s); /* Two longs and a string */
/* Possible Python call: f(1, 2, 'three') */

ok = PyArg_ParseTuple(args, "(ii)s#", &i, &j, &s, &size);
/* A pair of ints and a string, whose size is also returned */
/* Possible Python call: f((1, 2), 'three') */

{
char *file;
char *mode = "r";
int bufsize = 0;
ok = PyArg_ParseTuple(args, "s|si", &file, &mode, &bufsize);
/* A string, and optionally another string and an integer */
/* Possible Python calls:
f('spam')
f('spam', 'w')
f('spam', 'wb', 100000) */
}

{
int left, top, right, bottom, h, v;
ok = PyArg_ParseTuple(args, "((ii)(ii))(ii)",
&left, &top, &right, &bottom, &h, &v);
/* A rectangle and a point */
/* Possible Python call:
f(((0, 0), (400, 300)), (10, 10)) */
}

{
Py_complex c;
ok = PyArg_ParseTuple(args, "D:myfunction", &c);
/* a complex, also providing a function name for errors */
/* Possible Python call: myfunction(1+2j) */
}
```
---
node11.html
node2.html
node9.html
node1.html
guido@python.org


node12.html
node2.html
node10.html
node1.html
---
# Keyword Parsing with PyArg_ParseTupleAndKeywords()
The PyArg_ParseTupleAndKeywords() function is declared as
follows:
```text
int PyArg_ParseTupleAndKeywords(PyObject *arg, PyObject *kwdict,
char *format, char **kwlist, ...);
```
The arg and format parameters are identical to those of the
PyArg_ParseTuple() function. The kwdict parameter
is the dictionary of keywords received as the third parameter from the
Python runtime. The kwlist parameter is a NULL-terminated
list of strings which identify the parameters; the names are matched
with the type information from format from left to right.
Note: Nested tuples cannot be parsed when using keyword
arguments! Keyword parameters passed in which are not present in the
kwlist will cause a TypeError to be raised.
Here is an example module which uses keywords, based on an example by
Geoff Philbrick (philbrick@hks.com):
```text
#include <stdio.h>
#include "Python.h"

static PyObject *
keywdarg_parrot(self, args, keywds)
PyObject *self;
PyObject *args;
PyObject *keywds;
{
int voltage;
char *state = "a stiff";
char *action = "voom";
char *type = "Norwegian Blue";

static char *kwlist[] = {"voltage", "state", "action", "type", NULL};

if (!PyArg_ParseTupleAndKeywords(args, keywds, "i|sss", kwlist,
&voltage, &state, &action, &type))
return NULL;

printf("-- This parrot wouldn't %s if you put %i Volts through it.\n",
action, voltage);
printf("-- Lovely plumage, the %s -- It's %s!\n", type, state);

Py_INCREF(Py_None);

return Py_None;
}

static PyMethodDef keywdarg_methods[] = {
{"parrot", (PyCFunction)keywdarg_parrot, METH_VARARGS|METH_KEYWORDS},
{NULL, NULL} /* sentinel */
};

void
initkeywdarg()
{
/* Create the module and add the functions */
Py_InitModule("keywdarg", keywdarg_methods);

}
```
---
guido@python.org


The Py_BuildValue() Function

node13.html
node2.html
node11.html
node1.html
---
# The Py_BuildValue() Function
This function is the counterpart to PyArg_ParseTuple(). It is
declared as follows:
```text
PyObject *Py_BuildValue(char *format, ...);
```
It recognizes a set of format units similar to the ones recognized by
PyArg_ParseTuple(), but the arguments (which are input to the
function, not output) must not be pointers, just values. It returns a
new Python object, suitable for returning from a C function called
from Python.
One difference with PyArg_ParseTuple(): while the latter
requires its first argument to be a tuple (since Python argument lists
are always represented as tuples internally), BuildValue() does
not always build a tuple. It builds a tuple only if its format string
contains two or more format units. If the format string is empty, it
returns None; if it contains exactly one format unit, it
returns whatever object is described by that format unit. To force it
to return a tuple of size 0 or one, parenthesize the format string.
In the following description, the quoted form is the format unit; the
entry in (round) parentheses is the Python object type that the format
unit will return; and the entry in [square] brackets is the type of
the C value(s) to be passed.
The characters space, tab, colon and comma are ignored in format
strings (but not within format units such as `s#'). This can be
used to make long format strings a tad more readable.
If there is an error in the format string, the
PyExc_SystemError exception is raised and NULL returned.
Examples (to the left the call, to the right the resulting Python value):
```text
Py_BuildValue("") None
Py_BuildValue("i", 123) 123
Py_BuildValue("iii", 123, 456, 789) (123, 456, 789)
Py_BuildValue("s", "hello") 'hello'
Py_BuildValue("ss", "hello", "world") ('hello', 'world')
Py_BuildValue("s#", "hello", 4) 'hell'
Py_BuildValue("()") ()
Py_BuildValue("(i)", 123) (123,)
Py_BuildValue("(ii)", 123, 456) (123, 456)
Py_BuildValue("(i,i)", 123, 456) (123, 456)
Py_BuildValue("[i,i]", 123, 456) [123, 456]
Py_BuildValue("{s:i,s:i}",
"abc", 123, "def", 456) {'abc': 123, 'def': 456}
Py_BuildValue("((ii)(ii)) (ii)",
1, 2, 3, 4, 5, 6) (((1, 2), (3, 4)), (5, 6))
```
---
node13.html
node2.html
node11.html
node1.html
guido@python.org


1.11 Reference Counts

node14.html
node2.html
node12.html
node1.html
---
# 1.11 Reference Counts
---
- Introduction (node14.html#SECTION0021110000000000000000)
Reference Counting in Python (node15.html#SECTION0021120000000000000000)
Ownership Rules (node16.html#SECTION0021130000000000000000)
Thin Ice (node17.html#SECTION0021140000000000000000)
NULL Pointers (node18.html#SECTION0021150000000000000000)
---
guido@python.org


Introduction

node15.html
node13.html
node13.html
node1.html
---
## Introduction
In languages like C or C++, the programmer is responsible for
dynamic allocation and deallocation of memory on the heap. In C, this
is done using the functions malloc() and free(). In
C++, the operators new and delete are used with
essentially the same meaning; they are actually implemented using
malloc() and free(), so we'll restrict the following
discussion to the latter.
Every block of memory allocated with malloc() should eventually
be returned to the pool of available memory by exactly one call to
free(). It is important to call free() at the right
time. If a block's address is forgotten but free() is not
called for it, the memory it occupies cannot be reused until the
program terminates. This is called a memory leak. On the other
hand, if a program calls free() for a block and then continues
to use the block, it creates a conflict with re-use of the block
through another malloc() call. This is called using freed
memory. It has the same bad consequences as referencing uninitialized
data -- core dumps, wrong results, mysterious crashes.
Common causes of memory leaks are unusual paths through the code. For
instance, a function may allocate a block of memory, do some
calculation, and then free the block again. Now a change in the
requirements for the function may add a test to the calculation that
detects an error condition and can return prematurely from the
function. It's easy to forget to free the allocated memory block when
taking this premature exit, especially when it is added later to the
code. Such leaks, once introduced, often go undetected for a long
time: the error exit is taken only in a small fraction of all calls,
and most modern machines have plenty of virtual memory, so the leak
only becomes apparent in a long-running process that uses the leaking
function frequently. Therefore, it's important to prevent leaks from
happening by having a coding convention or strategy that minimizes
this kind of errors.
Since Python makes heavy use of malloc() and free(), it
needs a strategy to avoid memory leaks as well as the use of freed
memory. The chosen method is called reference counting. The
principle is simple: every object contains a counter, which is
incremented when a reference to the object is stored somewhere, and
which is decremented when a reference to it is deleted. When the
counter reaches zero, the last reference to the object has been
deleted and the object is freed.
An alternative strategy is called automatic garbage collection.
(Sometimes, reference counting is also referred to as a garbage
collection strategy, hence my use of ``automatic'' to distinguish the
two.) The big advantage of automatic garbage collection is that the
user doesn't need to call free() explicitly. (Another claimed
advantage is an improvement in speed or memory usage -- this is no
hard fact however.) The disadvantage is that for C, there is no
truly portable automatic garbage collector, while reference counting
can be implemented portably (as long as the functions malloc()
and free() are available -- which the C Standard guarantees).
Maybe some day a sufficiently portable automatic garbage collector
will be available for C. Until then, we'll have to live with
reference counts.
---
node15.html
node13.html
node13.html
node1.html
guido@python.org


Reference Counting in Python

node16.html
node13.html
node14.html
node1.html
---
## Reference Counting in Python
There are two macros, Py_INCREF(x) and Py_DECREF(x),
which handle the incrementing and decrementing of the reference count.
Py_DECREF() also frees the object when the count reaches zero.
For flexibility, it doesn't call free() directly -- rather, it
makes a call through a function pointer in the object's type
object. For this purpose (and others), every object also contains a
pointer to its type object.
The big question now remains: when to use Py_INCREF(x) and
Py_DECREF(x)? Let's first introduce some terms. Nobody
``owns'' an object; however, you can own a reference to an
object. An object's reference count is now defined as the number of
owned references to it. The owner of a reference is responsible for
calling Py_DECREF() when the reference is no longer needed.
Ownership of a reference can be transferred. There are three ways to
dispose of an owned reference: pass it on, store it, or call
Py_DECREF(). Forgetting to dispose of an owned reference creates
a memory leak.
It is also possible to borrowfootnode.html#477 a reference to an object. The borrower
of a reference should not call Py_DECREF(). The borrower must
not hold on to the object longer than the owner from which it was
borrowed. Using a borrowed reference after the owner has disposed of
it risks using freed memory and should be avoided
completely.footnode.html#655
The advantage of borrowing over owning a reference is that you don't
need to take care of disposing of the reference on all possible paths
through the code -- in other words, with a borrowed reference you
don't run the risk of leaking when a premature exit is taken. The
disadvantage of borrowing over leaking is that there are some subtle
situations where in seemingly correct code a borrowed reference can be
used after the owner from which it was borrowed has in fact disposed
of it.
A borrowed reference can be changed into an owned reference by calling
Py_INCREF(). This does not affect the status of the owner from
which the reference was borrowed -- it creates a new owned reference,
and gives full owner responsibilities (i.e., the new owner must
dispose of the reference properly, as well as the previous owner).
---
node16.html
node13.html
node14.html
node1.html
guido@python.org


Ownership Rules

node17.html
node13.html
node15.html
node1.html
---
## Ownership Rules
Whenever an object reference is passed into or out of a function, it
is part of the function's interface specification whether ownership is
transferred with the reference or not.
Most functions that return a reference to an object pass on ownership
with the reference. In particular, all functions whose function it is
to create a new object, e.g. PyInt_FromLong() and
Py_BuildValue(), pass ownership to the receiver. Even if in
fact, in some cases, you don't receive a reference to a brand new
object, you still receive ownership of the reference. For instance,
PyInt_FromLong() maintains a cache of popular values and can
return a reference to a cached item.
Many functions that extract objects from other objects also transfer
ownership with the reference, for instance
PyObject_GetAttrString(). The picture is less clear, here,
however, since a few common routines are exceptions:
PyTuple_GetItem(), PyList_GetItem() and
PyDict_GetItem() (and PyDict_GetItemString()) all return
references that you borrow from the tuple, list or dictionary.
The function PyImport_AddModule() also returns a borrowed
reference, even though it may actually create the object it returns:
this is possible because an owned reference to the object is stored in
sys.modules.
When you pass an object reference into another function, in general,
the function borrows the reference from you -- if it needs to store
it, it will use Py_INCREF() to become an independent owner.
There are exactly two important exceptions to this rule:
PyTuple_SetItem() and PyList_SetItem(). These functions
take over ownership of the item passed to them -- even if they fail!
(Note that PyDict_SetItem() and friends don't take over
ownership -- they are ``normal''.)
When a C function is called from Python, it borrows references to its
arguments from the caller. The caller owns a reference to the object,
so the borrowed reference's lifetime is guaranteed until the function
returns. Only when such a borrowed reference must be stored or passed
on, it must be turned into an owned reference by calling
Py_INCREF().
The object reference returned from a C function that is called from
Python must be an owned reference -- ownership is tranferred from the
function to its caller.
---
node17.html
node13.html
node15.html
node1.html
guido@python.org


Thin Ice

node18.html
node13.html
node16.html
node1.html
---
## Thin Ice
There are a few situations where seemingly harmless use of a borrowed
reference can lead to problems. These all have to do with implicit
invocations of the interpreter, which can cause the owner of a
reference to dispose of it.
The first and most important case to know about is using
Py_DECREF() on an unrelated object while borrowing a reference
to a list item. For instance:
```text
bug(PyObject *list) {
PyObject *item = PyList_GetItem(list, 0);
PyList_SetItem(list, 1, PyInt_FromLong(0L));
PyObject_Print(item, stdout, 0); /* BUG! */
}
```
This function first borrows a reference to list[0], then
replaces list[1] with the value 0, and finally prints
the borrowed reference. Looks harmless, right? But it's not!
Let's follow the control flow into PyList_SetItem(). The list
owns references to all its items, so when item 1 is replaced, it has
to dispose of the original item 1. Now let's suppose the original
item 1 was an instance of a user-defined class, and let's further
suppose that the class defined a __del__() method. If this
class instance has a reference count of 1, disposing of it will call
its __del__() method.
Since it is written in Python, the __del__() method can execute
arbitrary Python code. Could it perhaps do something to invalidate
the reference to item in bug()? You bet! Assuming that
the list passed into bug() is accessible to the
__del__() method, it could execute a statement to the effect of
del list[0], and assuming this was the last reference to that
object, it would free the memory associated with it, thereby
invalidating item.
The solution, once you know the source of the problem, is easy:
temporarily increment the reference count. The correct version of the
function reads:
```text
no_bug(PyObject *list) {
PyObject *item = PyList_GetItem(list, 0);
Py_INCREF(item);
PyList_SetItem(list, 1, PyInt_FromLong(0L));
PyObject_Print(item, stdout, 0);
Py_DECREF(item);
}
```
This is a true story. An older version of Python contained variants
of this bug and someone spent a considerable amount of time in a C
debugger to figure out why his __del__() methods would fail...
The second case of problems with a borrowed reference is a variant
involving threads. Normally, multiple threads in the Python
interpreter can't get in each other's way, because there is a global
lock protecting Python's entire object space. However, it is possible
to temporarily release this lock using the macro
Py_BEGIN_ALLOW_THREADS, and to re-acquire it using
Py_END_ALLOW_THREADS. This is common around blocking I/O
calls, to let other threads use the CPU while waiting for the I/O to
complete. Obviously, the following function has the same problem as
the previous one:
```text
bug(PyObject *list) {
PyObject *item = PyList_GetItem(list, 0);
Py_BEGIN_ALLOW_THREADS
...some blocking I/O call...
Py_END_ALLOW_THREADS
PyObject_Print(item, stdout, 0); /* BUG! */
}
```
---
node18.html
node13.html
node16.html
node1.html
guido@python.org


NULL Pointers

node19.html
node13.html
node17.html
node1.html
---
## NULL Pointers
In general, functions that take object references as arguments don't
expect you to pass them NULL pointers, and will dump core (or
cause later core dumps) if you do so. Functions that return object
references generally return NULL only to indicate that an
exception occurred. The reason for not testing for NULL
arguments is that functions often pass the objects they receive on to
other function -- if each function were to test for NULL,
there would be a lot of redundant tests and the code would run slower.
It is better to test for NULL only at the ``source'', i.e.\
when a pointer that may be NULL is received, e.g. from
malloc() or from a function that may raise an exception.
The macros Py_INCREF() and Py_DECREF()
don't check for NULL pointers -- however, their variants
Py_XINCREF() and Py_XDECREF() do.
The macros for checking for a particular object type
(Pytype_Check()) don't check for NULL pointers --
again, there is much code that calls several of these in a row to test
an object against various different expected types, and this would
generate redundant tests. There are no variants with NULL
checking.
The C function calling mechanism guarantees that the argument list
passed to C functions (args in the examples) is never
NULL -- in fact it guarantees that it is always a tuple.footnode.html#538
It is a severe error to ever let a NULL pointer ``escape'' to
the Python user.
---
guido@python.org


Writing Extensions in C++

node20.html
node2.html
node18.html
node1.html
---
# Writing Extensions in C++
It is possible to write extension modules in C++. Some restrictions
apply. If the main program (the Python interpreter) is compiled and
linked by the C compiler, global or static objects with constructors
cannot be used. This is not a problem if the main program is linked
by the C++ compiler. Functions that will be called by the
Python interpreter (in particular, module initalization functions)
have to be declared using extern "C".
It is unnecessary to enclose the Python header files in
extern "C" {...} -- they use this form already if the symbol
`__cplusplus' is defined (all recent C++ compilers define this
symbol).
---
guido@python.org


Extending Python with C or C++ code

node3.html
ext.html
node1.html
node1.html
---
# Extending Python with C or C++ code
---
- 1.1 Introduction (node3.html#SECTION002100000000000000000)
1.2 A Simple Example (node4.html#SECTION002200000000000000000)
1.3 Intermezzo: Errors and Exceptions (node5.html#SECTION002300000000000000000)
1.4 Back to the Example (node6.html#SECTION002400000000000000000)
1.5 The Module's Method Table and Initialization Function (node7.html#SECTION002500000000000000000)
1.6 Compilation and Linkage (node8.html#SECTION002600000000000000000)
1.7 Calling Python Functions From C (node9.html#SECTION002700000000000000000)
Format Strings for PyArg_ParseTuple() (node10.html#SECTION002800000000000000000)
Keyword Parsing with PyArg_ParseTupleAndKeywords() (node11.html#SECTION002900000000000000000)
The Py_BuildValue() Function (node12.html#SECTION0021000000000000000000)
1.11 Reference Counts (node13.html#SECTION0021100000000000000000)
- Introduction (node14.html#SECTION0021110000000000000000)
Reference Counting in Python (node15.html#SECTION0021120000000000000000)
Ownership Rules (node16.html#SECTION0021130000000000000000)
Thin Ice (node17.html#SECTION0021140000000000000000)
NULL Pointers (node18.html#SECTION0021150000000000000000)
Writing Extensions in C++ (node19.html#SECTION0021200000000000000000)
---
guido@python.org


node21.html
ext.html
node19.html
node1.html
---
# 2 Embedding Python in another application
Embedding Python is similar to extending it, but not quite. The
difference is that when you extend Python, the main program of the
application is still the Python interpreter, while if you embed
Python, the main program may have nothing to do with Python --
instead, some parts of the application occasionally call the Python
interpreter to run some Python code.
So if you are embedding Python, you are providing your own main
program. One of the things this main program has to do is initialize
the Python interpreter. At the very least, you have to call the
function Py_Initialize(). There are optional calls to pass command
line arguments to Python. Then later you can call the interpreter
from any part of the application.
There are several different ways to call the interpreter: you can pass
a string containing Python statements to PyRun_SimpleString(),
or you can pass a stdio file pointer and a file name (for
identification in error messages only) to PyRun_SimpleFile(). You
can also call the lower-level operations described in the previous
chapters to construct and use Python objects.
A simple demo of embedding Python can be found in the directory
`Demo/embed'.
---
- Embedding Python in C++ (node21.html#SECTION003100000000000000000)
---
guido@python.org


Embedding Python in C++

node22.html
node20.html
node20.html
node1.html
---
# Embedding Python in C++
It is also possible to embed Python in a C++ program; precisely how this
is done will depend on the details of the C++ system used; in general you
will need to write the main program in C++, and use the C++ compiler
to compile and link your program. There is no need to recompile Python
itself using C++.
---
guido@python.org


3 Dynamic Loading

node23.html
ext.html
node21.html
node1.html
---
# 3 Dynamic Loading
On most modern systems it is possible to configure Python to support
dynamic loading of extension modules implemented in C. When shared
libraries are used dynamic loading is configured automatically;
otherwise you have to select it as a build option (see below). Once
configured, dynamic loading is trivial to use: when a Python program
executes import spam, the search for modules tries to find a
file `spammodule.o' (`spammodule.so' when using shared
libraries) in the module search path, and if one is found, it is
loaded into the executing binary and executed. Once loaded, the
module acts just like a built-in extension module.
The advantages of dynamic loading are twofold: the ``core'' Python
binary gets smaller, and users can extend Python with their own
modules implemented in C without having to build and maintain their
own copy of the Python interpreter. There are also disadvantages:
dynamic loading isn't available on all systems (this just means that
on some systems you have to use static loading), and dynamically
loading a module that was compiled for a different version of Python
(e.g. with a different representation of objects) may dump core.
---
- 3.1 Configuring and Building the Interpreter for Dynamic Loading (node23.html#SECTION004100000000000000000)
- Shared Libraries (node24.html#SECTION004110000000000000000)
SGI IRIX 4 Dynamic Loading (node25.html#SECTION004120000000000000000)
GNU Dynamic Loading (node26.html#SECTION004130000000000000000)
3.2 Building a Dynamically Loadable Module (node27.html#SECTION004200000000000000000)
- Shared Libraries (node28.html#SECTION004210000000000000000)
SGI IRIX 4 Dynamic Loading (node29.html#SECTION004220000000000000000)
GNU Dynamic Loading (node30.html#SECTION004230000000000000000)
---
guido@python.org


node24.html
node22.html
node22.html
node1.html
---
# 3.1 Configuring and Building the Interpreter for Dynamic Loading
There are three styles of dynamic loading: one using shared libraries,
one using SGI IRIX 4 dynamic loading, and one using GNU dynamic
loading.
---
- Shared Libraries (node24.html#SECTION004110000000000000000)
SGI IRIX 4 Dynamic Loading (node25.html#SECTION004120000000000000000)
GNU Dynamic Loading (node26.html#SECTION004130000000000000000)
---
guido@python.org


Shared Libraries

node25.html
node23.html
node23.html
node1.html
---
## Shared Libraries
The following systems support dynamic loading using shared libraries:
SunOS 4; Solaris 2; SGI IRIX 5 (but not SGI IRIX 4!); and probably all
systems derived from SVR4, or at least those SVR4 derivatives that
support shared libraries (are there any that don't?).
You don't need to do anything to configure dynamic loading on these
systems -- the `configure' detects the presence of the
`<dlfcn.h>' header file and automatically configures dynamic
loading.
---
guido@python.org


SGI IRIX 4 Dynamic Loading

node26.html
node23.html
node24.html
node1.html
---
## SGI IRIX 4 Dynamic Loading
Only SGI IRIX 4 supports dynamic loading of modules using SGI dynamic
loading. (SGI IRIX 5 might also support it but it is inferior to
using shared libraries so there is no reason to; a small test didn't
work right away so I gave up trying to support it.)
Before you build Python, you first need to fetch and build the dl
package written by Jack Jansen. This is available by anonymous ftp
from ftp://ftp.cwi.nl/pub/dynload, file
`dl-1.6.tar.Z'. (The version number may change.) Follow the
instructions in the package's `README' file to build it.
Once you have built dl, you can configure Python to use it. To
this end, you run the `configure' script with the option
-with-dl=directory where directory is the absolute
pathname of the dl directory.
Now build and install Python as you normally would (see the
`README' file in the toplevel Python directory.)
---
guido@python.org


GNU Dynamic Loading

node27.html
node23.html
node25.html
node1.html
---
## GNU Dynamic Loading
GNU dynamic loading supports (according to its `README' file) the
following hardware and software combinations: VAX (Ultrix), Sun 3
(SunOS 3.4 and 4.0), Sparc (SunOS 4.0), Sequent Symmetry (Dynix), and
Atari ST. There is no reason to use it on a Sparc; I haven't seen a
Sun 3 for years so I don't know if these have shared libraries or not.
You need to fetch and build two packages.
One is GNU DLD. All development of this code has been done with DLD
version 3.2.3, which is available by anonymous ftp from
ftp://ftp.cwi.nl/pub/dynload, file
`dld-3.2.3.tar.Z'. (A more recent version of DLD is available
via http://www-swiss.ai.mit.edu/ jaffer/DLD.html but this has
not been tested.)
The other package needed is an
emulation of Jack Jansen's dl package that I wrote on top of
GNU DLD 3.2.3. This is available from the same host and directory,
file `dl-dld-1.1.tar.Z'. (The version number may change -- but I doubt
it will.) Follow the instructions in each package's `README'
file to configure and build them.
Now configure Python. Run the `configure' script with the option
-with-dl-dld=dl-directory,dld-directory where
dl-directory is the absolute pathname of the directory where you
have built the `dl-dld' package, and dld-directory is that
of the GNU DLD package. The Python interpreter you build hereafter
will support GNU dynamic loading.
---
guido@python.org


node28.html
node22.html
node26.html
node1.html
---
# 3.2 Building a Dynamically Loadable Module
Since there are three styles of dynamic loading, there are also three
groups of instructions for building a dynamically loadable module.
Instructions common for all three styles are given first. Assuming
your module is called spam, the source filename must be
`spammodule.c', so the object name is `spammodule.o'. The
module must be written as a normal Python extension module (as
described earlier).
Note that in all cases you will have to create your own Makefile that
compiles your module file(s). This Makefile will have to pass two
`-I' arguments to the C compiler which will make it find the
Python header files. If the Make variable PYTHONTOP points to
the toplevel Python directory, your CFLAGS Make variable should
contain the options `-I$(PYTHONTOP) -I$(PYTHONTOP)/Include'.
(Most header files are in the `Include' subdirectory, but the
`config.h' header lives in the toplevel directory.)
---
- Shared Libraries (node28.html#SECTION004210000000000000000)
SGI IRIX 4 Dynamic Loading (node29.html#SECTION004220000000000000000)
GNU Dynamic Loading (node30.html#SECTION004230000000000000000)
---
guido@python.org


Shared Libraries

node29.html
node27.html
node27.html
node1.html
---
## Shared Libraries
You must link the `.o' file to produce a shared library. This is
done using a special invocation of the Unix loader/linker,
ld(1). Unfortunately the invocation differs slightly per
system.
On SunOS 4, use
```text
ld spammodule.o -o spammodule.so
```
On Solaris 2, use
```text
ld -G spammodule.o -o spammodule.so
```
On SGI IRIX 5, use
```text
ld -shared spammodule.o -o spammodule.so
```
On other systems, consult the manual page for ld(1) to find what
flags, if any, must be used.
If your extension module uses system libraries that haven't already
been linked with Python (e.g. a windowing system), these must be
passed to the ld command as `-l' options after the
`.o' file.
The resulting file `spammodule.so' must be copied into a directory
along the Python module search path.
---
guido@python.org


SGI IRIX 4 Dynamic Loading

node30.html
node27.html
node28.html
node1.html
---
## SGI IRIX 4 Dynamic Loading
IMPORTANT: You must compile your extension module with the
additional C flag `-G0' (or `-G 0'). This instruct the
assembler to generate position-independent code.
You don't need to link the resulting `spammodule.o' file; just
copy it into a directory along the Python module search path.
The first time your extension is loaded, it takes some extra time and
a few messages may be printed. This creates a file
`spammodule.ld' which is an image that can be loaded quickly into
the Python interpreter process. When a new Python interpreter is
installed, the dl package detects this and rebuilds
`spammodule.ld'. The file `spammodule.ld' is placed in the
directory where `spammodule.o' was found, unless this directory is
unwritable; in that case it is placed in a temporary
directory.footnode.html#661
If your extension modules uses additional system libraries, you must
create a file `spammodule.libs' in the same directory as the
`spammodule.o'. This file should contain one or more lines with
whitespace-separated options that will be passed to the linker --
normally only `-l' options or absolute pathnames of libraries
(`.a' files) should be used.
---
guido@python.org


1.1 Introduction

node4.html
node2.html
node2.html
node1.html
---
# 1.1 Introduction
It is quite easy to add new built-in modules to Python, if you know
how to program in C. Such extension modules can do two things
that can't be done directly in Python: they can implement new built-in
object types, and they can call C library functions and system calls.
To support extensions, the Python API (Application Programmers
Interface) defines a set of functions, macros and variables that
provide access to most aspects of the Python run-time system. The
Python API is incorporated in a C source file by including the header
"Python.h".
The compilation of an extension module depends on its intended use as
well as on your system setup; details are given in a later section.
---
guido@python.org


GNU Dynamic Loading

node31.html
node27.html
node29.html
node1.html
---
## GNU Dynamic Loading
Just copy `spammodule.o' into a directory along the Python module
search path.
If your extension modules uses additional system libraries, you must
create a file `spammodule.libs' in the same directory as the
`spammodule.o'. This file should contain one or more lines with
whitespace-separated absolute pathnames of libraries (`.a'
files). No `-l' options can be used.
---
guido@python.org


About this document ...

ext.html
node30.html
node1.html
---
# About this document ...
Extending and Embedding the Python Interpreter
This document was generated using the LaTeX2HTML (http://www-dsed.llnl.gov/files/programs/unix/latex2html/manual/) translator Version 96.1 (Feb 5, 1996) Copyright © 1993, 1994, 1995, 1996, Nikos Drakos (http://cbl.leeds.ac.uk/nikos/personal.html), Computer Based Learning Unit, University of Leeds.
The command line arguments were:
latex2html -address guido@python.org ext.tex.
The translation was initiated by Fred L. Drake on Tue Feb 17 10:28:43 EST 1998
---
guido@python.org


1.2 A Simple Example

node5.html
node2.html
node3.html
node1.html
---
# 1.2 A Simple Example
Let's create an extension module called `spam' (the favorite food
of Monty Python fans...) and let's say we want to create a Python
interface to the C library function system().footnode.html#637
This function takes a null-terminated character string as argument and
returns an integer. We want this function to be callable from Python
as follows:
```text
>>> import spam
>>> status = spam.system("ls -l")
```
Begin by creating a file `spammodule.c'. (In general, if a
module is called `spam', the C file containing its implementation
is called `spammodule.c'; if the module name is very long, like
`spammify', the module name can be just `spammify.c'.)
The first line of our file can be:
```text
#include "Python.h"
```
which pulls in the Python API (you can add a comment describing the
purpose of the module and a copyright notice if you like).
All user-visible symbols defined by "Python.h" have a prefix of
`Py' or `PY', except those defined in standard header files.
For convenience, and since they are used extensively by the Python
interpreter, "Python.h" includes a few standard header files:
<stdio.h>, <string.h>, <errno.h>, and
<stdlib.h>. If the latter header file does not exist on your
system, it declares the functions malloc(), free() and
realloc() directly.
The next thing we add to our module file is the C function that will
be called when the Python expression `spam.system(string)'
is evaluated (we'll see shortly how it ends up being called):
```text
static PyObject *
spam_system(self, args)
PyObject *self;
PyObject *args;
{
char *command;
int sts;
if (!PyArg_ParseTuple(args, "s", &command))
return NULL;
sts = system(command);
return Py_BuildValue("i", sts);
}
```
There is a straightforward translation from the argument list in
Python (e.g. the single expression "ls -l") to the arguments
passed to the C function. The C function always has two arguments,
conventionally named self and args.
The self argument is only used when the C function implements a
builtin method. This will be discussed later. In the example,
self will always be a NULL pointer, since we are defining
a function, not a method. (This is done so that the interpreter
doesn't have to understand two different types of C functions.)
The args argument will be a pointer to a Python tuple object
containing the arguments. Each item of the tuple corresponds to an
argument in the call's argument list. The arguments are Python
objects -- in order to do anything with them in our C function we have
to convert them to C values. The function PyArg_ParseTuple()
in the Python API checks the argument types and converts them to C
values. It uses a template string to determine the required types of
the arguments as well as the types of the C variables into which to
store the converted values. More about this later.
PyArg_ParseTuple() returns true (nonzero) if all arguments have
the right type and its components have been stored in the variables
whose addresses are passed. It returns false (zero) if an invalid
argument list was passed. In the latter case it also raises an
appropriate exception by so the calling function can return
NULL immediately (as we saw in the example).
---
node5.html
node2.html
node3.html
node1.html
guido@python.org


1.3 Intermezzo: Errors and Exceptions

node6.html
node2.html
node4.html
node1.html
---
# 1.3 Intermezzo: Errors and Exceptions
An important convention throughout the Python interpreter is the
following: when a function fails, it should set an exception condition
and return an error value (usually a NULL pointer). Exceptions
are stored in a static global variable inside the interpreter; if this
variable is NULL no exception has occurred. A second global
variable stores the ``associated value'' of the exception (the second
argument to raise). A third variable contains the stack
traceback in case the error originated in Python code. These three
variables are the C equivalents of the Python variables
sys.exc_type, sys.exc_value and sys.exc_traceback
(see the section on module sys in the Library Reference
Manual). It is important to know about them to understand how errors
are passed around.
The Python API defines a number of functions to set various types of
exceptions.
The most common one is PyErr_SetString(). Its arguments are an
exception object and a C string. The exception object is usually a
predefined object like PyExc_ZeroDivisionError. The C string
indicates the cause of the error and is converted to a Python string
object and stored as the ``associated value'' of the exception.
Another useful function is PyErr_SetFromErrno(), which only
takes an exception argument and constructs the associated value by
inspection of the (Unix) global variable errno. The most
general function is PyErr_SetObject(), which takes two object
arguments, the exception and its associated value. You don't need to
Py_INCREF() the objects passed to any of these functions.
You can test non-destructively whether an exception has been set with
PyErr_Occurred(). This returns the current exception object,
or NULL if no exception has occurred. You normally don't need
to call PyErr_Occurred() to see whether an error occurred in a
function call, since you should be able to tell from the return value.
When a function f that calls another function g detects
that the latter fails, f should itself return an error value
(e.g. NULL or -1). It should not call one of the
PyErr_*() functions -- one has already been called by g.
f's caller is then supposed to also return an error indication
to its caller, again without calling PyErr_*(),
and so on -- the most detailed cause of the error was already
reported by the function that first detected it. Once the error
reaches the Python interpreter's main loop, this aborts the currently
executing Python code and tries to find an exception handler specified
by the Python programmer.
(There are situations where a module can actually give a more detailed
error message by calling another PyErr_*() function, and in
such cases it is fine to do so. As a general rule, however, this is
not necessary, and can cause information about the cause of the error
to be lost: most operations can fail for a variety of reasons.)
To ignore an exception set by a function call that failed, the exception
condition must be cleared explicitly by calling PyErr_Clear().
The only time C code should call PyErr_Clear() is if it doesn't
want to pass the error on to the interpreter but wants to handle it
completely by itself (e.g. by trying something else or pretending
nothing happened).
Note that a failing malloc() call must be turned into an
exception -- the direct caller of malloc() (or
realloc()) must call PyErr_NoMemory() and return a
failure indicator itself. All the object-creating functions
(PyInt_FromLong() etc.) already do this, so only if you call
malloc() directly this note is of importance.
Also note that, with the important exception of
PyArg_ParseTuple() and friends, functions that return an
integer status usually return a positive value or zero for success and
-1 for failure, like Unix system calls.
Finally, be careful to clean up garbage (by making Py_XDECREF()
or Py_DECREF() calls for objects you have already created) when
you return an error indicator!
The choice of which exception to raise is entirely yours. There are
predeclared C objects corresponding to all built-in Python exceptions,
e.g. PyExc_ZeroDevisionError which you can use directly. Of
course, you should choose exceptions wisely -- don't use
PyExc_TypeError to mean that a file couldn't be opened (that
should probably be PyExc_IOError). If something's wrong with
the argument list, the PyArg_ParseTuple() function usually
raises PyExc_TypeError. If you have an argument whose value
which must be in a particular range or must satisfy other conditions,
PyExc_ValueError is appropriate.
You can also define a new exception that is unique to your module.
For this, you usually declare a static object variable at the
beginning of your file, e.g.
```text
static PyObject *SpamError;
```
and initialize it in your module's initialization function
(initspam()) with a string object, e.g. (leaving out the error
checking for now):
```text
void
initspam()
{
PyObject *m, *d;
m = Py_InitModule("spam", SpamMethods);
d = PyModule_GetDict(m);
SpamError = PyString_FromString("spam.error");
PyDict_SetItemString(d, "error", SpamError);
}
```
Note that the Python name for the exception object is
spam.error. It is conventional for module and exception names
to be spelled in lower case. It is also conventional that the
value of the exception object is the same as its name, e.g.\
the string "spam.error".
---
node6.html
node2.html
node4.html
node1.html
guido@python.org