Python support for the Linux

Python support for the Linux perf
profiler¶
- author:
Pablo Galindo
The Linux perf profiler
is a very powerful tool that allows you to profile and obtain
information about the performance of your application.
perf
also has a very vibrant ecosystem of tools
that aid with the analysis of the data that it produces.
The main problem with using the perf
profiler with Python applications is that
perf
only gets information about native symbols, that is, the names of
functions and procedures written in C. This means that the names and file names
of Python functions in your code will not appear in the output of perf
.
Since Python 3.12, the interpreter can run in a special mode that allows Python
functions to appear in the output of the perf
profiler. When this mode is
enabled, the interpreter will interpose a small piece of code compiled on the
fly before the execution of every Python function and it will teach perf
the
relationship between this piece of code and the associated Python function using
perf map files.
Note
Support for the perf
profiler is currently only available for Linux on
select architectures. Check the output of the configure
build step or
check the output of python -m sysconfig | grep HAVE_PERF_TRAMPOLINE
to see if your system is supported.
For example, consider the following script:
def foo(n):
result = 0
for _ in range(n):
result += 1
return result
def bar(n):
foo(n)
def baz(n):
bar(n)
if __name__ == "__main__":
baz(1000000)
We can run perf
to sample CPU stack traces at 9999 hertz:
$ perf record -F 9999 -g -o perf.data python my_script.py
Then we can use perf report
to analyze the data:
$ perf report --stdio -n -g
# Children Self Samples Command Shared Object Symbol
# ........ ........ ............ .......... .................. ..........................................
#
91.08% 0.00% 0 python.exe python.exe [.] _start
|
---_start
|
--90.71%--__libc_start_main
Py_BytesMain
|
|--56.88%--pymain_run_python.constprop.0
| |
| |--56.13%--_PyRun_AnyFileObject
| | _PyRun_SimpleFileObject
| | |
| | |--55.02%--run_mod
| | | |
| | | --54.65%--PyEval_EvalCode
| | | _PyEval_EvalFrameDefault
| | | PyObject_Vectorcall
| | | _PyEval_Vector
| | | _PyEval_EvalFrameDefault
| | | PyObject_Vectorcall
| | | _PyEval_Vector
| | | _PyEval_EvalFrameDefault
| | | PyObject_Vectorcall
| | | _PyEval_Vector
| | | |
| | | |--51.67%--_PyEval_EvalFrameDefault
| | | | |
| | | | |--11.52%--_PyLong_Add
| | | | | |
| | | | | |--2.97%--_PyObject_Malloc
...
As you can see, the Python functions are not shown in the output, only _PyEval_EvalFrameDefault
(the function that evaluates the Python bytecode) shows up. Unfortunately that’s not very useful because all Python
functions use the same C function to evaluate bytecode so we cannot know which Python function corresponds to which
bytecode-evaluating function.
Instead, if we run the same experiment with perf
support enabled we get:
$ perf report --stdio -n -g
# Children Self Samples Command Shared Object Symbol
# ........ ........ ............ .......... .................. .....................................................................
#
90.58% 0.36% 1 python.exe python.exe [.] _start
|
---_start
|
--89.86%--__libc_start_main
Py_BytesMain
|
|--55.43%--pymain_run_python.constprop.0
| |
| |--54.71%--_PyRun_AnyFileObject
| | _PyRun_SimpleFileObject
| | |
| | |--53.62%--run_mod
| | | |
| | | --53.26%--PyEval_EvalCode
| | | py::<module>:/src/script.py
| | | _PyEval_EvalFrameDefault
| | | PyObject_Vectorcall
| | | _PyEval_Vector
| | | py::baz:/src/script.py
| | | _PyEval_EvalFrameDefault
| | | PyObject_Vectorcall
| | | _PyEval_Vector
| | | py::bar:/src/script.py
| | | _PyEval_EvalFrameDefault
| | | PyObject_Vectorcall
| | | _PyEval_Vector
| | | py::foo:/src/script.py
| | | |
| | | |--51.81%--_PyEval_EvalFrameDefault
| | | | |
| | | | |--13.77%--_PyLong_Add
| | | | | |
| | | | | |--3.26%--_PyObject_Malloc
How to enable perf
profiling support¶
perf
profiling support can be enabled either from the start using
the environment variable PYTHONPERFSUPPORT
or the
-X perf
option,
or dynamically using sys.activate_stack_trampoline()
and
sys.deactivate_stack_trampoline()
.
The sys
functions take precedence over the -X
option,
the -X
option takes precedence over the environment variable.
Example, using the environment variable:
$ PYTHONPERFSUPPORT=1 perf record -F 9999 -g -o perf.data python my_script.py
$ perf report -g -i perf.data
Example, using the -X
option:
$ perf record -F 9999 -g -o perf.data python -X perf my_script.py
$ perf report -g -i perf.data
Example, using the sys
APIs in file example.py
:
import sys
sys.activate_stack_trampoline("perf")
do_profiled_stuff()
sys.deactivate_stack_trampoline()
non_profiled_stuff()
…then:
$ perf record -F 9999 -g -o perf.data python ./example.py
$ perf report -g -i perf.data
How to obtain the best results¶
For best results, Python should be compiled with
CFLAGS="-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer"
as this allows
profilers to unwind using only the frame pointer and not on DWARF debug
information. This is because as the code that is interposed to allow perf
support is dynamically generated it doesn’t have any DWARF debugging information
available.
You can check if your system has been compiled with this flag by running:
$ python -m sysconfig | grep 'no-omit-frame-pointer'
If you don’t see any output it means that your interpreter has not been compiled with
frame pointers and therefore it may not be able to show Python functions in the output
of perf
.
How to work without frame pointers¶
If you are working with a Python interpreter that has been compiled without
frame pointers, you can still use the perf
profiler, but the overhead will be
a bit higher because Python needs to generate unwinding information for every
Python function call on the fly. Additionally, perf
will take more time to
process the data because it will need to use the DWARF debugging information to
unwind the stack and this is a slow process.
To enable this mode, you can use the environment variable
PYTHON_PERF_JIT_SUPPORT
or the -X perf_jit
option,
which will enable the JIT mode for the perf
profiler.
Note
Due to a bug in the perf
tool, only perf
versions higher than v6.8
will work with the JIT mode. The fix was also backported to the v6.7.2
version of the tool.
Note that when checking the version of the perf
tool (which can be done
by running perf version
) you must take into account that some distros
add some custom version numbers including a -
character. This means
that perf 6.7-3
is not necessarily perf 6.7.3
.
When using the perf JIT mode, you need an extra step before you can run perf
report
. You need to call the perf inject
command to inject the JIT
information into the perf.data
file.:
$ perf record -F 9999 -g -k 1 --call-graph dwarf -o perf.data python -Xperf_jit my_script.py
$ perf inject -i perf.data --jit --output perf.jit.data
$ perf report -g -i perf.jit.data
or using the environment variable:
$ PYTHON_PERF_JIT_SUPPORT=1 perf record -F 9999 -g --call-graph dwarf -o perf.data python my_script.py
$ perf inject -i perf.data --jit --output perf.jit.data
$ perf report -g -i perf.jit.data
perf inject --jit
command will read perf.data
,
automatically pick up the perf dump file that Python creates (in
/tmp/perf-$PID.dump
), and then create perf.jit.data
which merges all the
JIT information together. It should also create a lot of jitted-XXXX-N.so
files in the current directory which are ELF images for all the JIT trampolines
that were created by Python.
Warning
When using --call-graph dwarf
, the perf
tool will take
snapshots of the stack of the process being profiled and save the
information in the perf.data
file. By default, the size of the stack dump
is 8192 bytes, but you can change the size by passing it after
a comma like --call-graph dwarf,16384
.
The size of the stack dump is important because if the size is too small
perf
will not be able to unwind the stack and the output will be
incomplete. On the other hand, if the size is too big, then perf
won’t
be able to sample the process as frequently as it would like as the overhead
will be higher.
The stack size is particularly important when profiling Python code compiled
with low optimization levels (like -O0
), as these builds tend to have
larger stack frames. If you are compiling Python with -O0
and not seeing
Python functions in your profiling output, try increasing the stack dump
size to 65528 bytes (the maximum):
$ perf record -F 9999 -g -k 1 --call-graph dwarf,65528 -o perf.data python -Xperf_jit my_script.py
Different compilation flags can significantly impact stack sizes:
Builds with
-O0
typically have much larger stack frames than those with-O1
or higherAdding optimizations (
-O1
,-O2
, etc.) typically reduces stack sizeFrame pointers (
-fno-omit-frame-pointer
) generally provide more reliable stack unwinding


Cell Objects

Cell Objects¶
“Cell” objects are used to implement variables referenced by multiple scopes. For each such variable, a cell object is created to store the value; the local variables of each stack frame that references the value contain a reference to the cells from outer scopes which also use that variable. When the value is accessed, the value contained in the cell is used instead of the cell object itself. This de-referencing of the cell object requires support from the generated byte-code; these are not automatically de-referenced when accessed. Cell objects are not likely to be useful elsewhere.
-
type PyCellObject¶
The C structure used for cell objects.
-
PyTypeObject PyCell_Type¶
The type object corresponding to cell objects.
-
int PyCell_Check(PyObject *ob)¶
Return true if ob is a cell object; ob must not be
NULL
. This function always succeeds.
-
PyObject *PyCell_New(PyObject *ob)¶
- Return value: New reference.
Create and return a new cell object containing the value ob. The parameter may be
NULL
.
-
PyObject *PyCell_Get(PyObject *cell)¶
- Return value: New reference.
Return the contents of the cell cell, which can be
NULL
. If cell is not a cell object, returnsNULL
with an exception set.
-
PyObject *PyCell_GET(PyObject *cell)¶
- Return value: Borrowed reference.
Return the contents of the cell cell, but without checking that cell is non-
NULL
and a cell object.


Exceptions

Exceptions¶
Source code: Lib/asyncio/exceptions.py
- exception asyncio.TimeoutError¶
A deprecated alias of
TimeoutError
, raised when the operation has exceeded the given deadline.Changed in version 3.11: This class was made an alias of
TimeoutError
.
- exception asyncio.CancelledError¶
The operation has been cancelled.
This exception can be caught to perform custom operations when asyncio Tasks are cancelled. In almost all situations the exception must be re-raised.
Changed in version 3.8:
CancelledError
is now a subclass ofBaseException
rather thanException
.
- exception asyncio.InvalidStateError¶
Invalid internal state of
Task
orFuture
.Can be raised in situations like setting a result value for a Future object that already has a result value set.
- exception asyncio.SendfileNotAvailableError¶
The “sendfile” syscall is not available for the given socket or file type.
A subclass of
RuntimeError
.
- exception asyncio.IncompleteReadError¶
The requested read operation did not complete fully.
Raised by the asyncio stream APIs.
This exception is a subclass of
EOFError
.
- exception asyncio.LimitOverrunError¶
Reached the buffer size limit while looking for a separator.
Raised by the asyncio stream APIs.
- consumed¶
The total number of to be consumed bytes.


Python 3.14.3 documentation

Python 3.14.3 documentation
Welcome! This is the official documentation for Python 3.14.3.
Documentation sections:
|
What's new in Python 3.14?
Or all "What's new" documents since Python 2.0
Tutorial
Start here: a tour of Python's syntax and features
Library reference
Standard library and builtins
Language reference
Syntax and language elements
Python setup and usage
How to install, configure, and use Python
Python HOWTOs
In-depth topic manuals
|
Installing Python modules
Third-party modules and PyPI.org
Distributing Python modules
Publishing modules for use by other people
Extending and embedding
For C/C++ programmers
Python's C API
C API reference
FAQs
Frequently asked questions (with answers!)
Deprecations
Deprecated functionality
|
Indices, glossary, and search:
|
Global module index
All modules and libraries
General index
All functions, classes, and terms
Glossary
Terms explained
|
Search page
Search this documentation
Complete table of contents
Lists all sections and subsections
|
Project information:


Python Initialization Configuration

Python Initialization Configuration¶
PyInitConfig C API¶
Added in version 3.14.
Python can be initialized with Py_InitializeFromInitConfig()
.
The Py_RunMain()
function can be used to write a customized Python
program.
See also Initialization, Finalization, and Threads.
See also
PEP 741 “Python Configuration C API”.
Example¶
Example of customized Python always running with the Python Development
Mode enabled; return -1
on error:
int init_python(void)
{
PyInitConfig *config = PyInitConfig_Create();
if (config == NULL) {
printf("PYTHON INIT ERROR: memory allocation failed\n");
return -1;
}
// Enable the Python Development Mode
if (PyInitConfig_SetInt(config, "dev_mode", 1) < 0) {
goto error;
}
// Initialize Python with the configuration
if (Py_InitializeFromInitConfig(config) < 0) {
goto error;
}
PyInitConfig_Free(config);
return 0;
error:
{
// Display the error message.
//
// This uncommon braces style is used, because you cannot make
// goto targets point to variable declarations.
const char *err_msg;
(void)PyInitConfig_GetError(config, &err_msg);
printf("PYTHON INIT ERROR: %s\n", err_msg);
PyInitConfig_Free(config);
return -1;
}
}
Create Config¶
-
struct PyInitConfig¶
Opaque structure to configure the Python initialization.
-
PyInitConfig *PyInitConfig_Create(void)¶
Create a new initialization configuration using Isolated Configuration default values.
It must be freed by
PyInitConfig_Free()
.Return
NULL
on memory allocation failure.
-
void PyInitConfig_Free(PyInitConfig *config)¶
Free memory of the initialization configuration config.
If config is
NULL
, no operation is performed.
Error Handling¶
-
int PyInitConfig_GetError(PyInitConfig *config, const char **err_msg)¶
Get the config error message.
Set *err_msg and return
1
if an error is set.Set *err_msg to
NULL
and return0
otherwise.
An error message is a UTF-8 encoded string.
If config has an exit code, format the exit code as an error message.
The error message remains valid until another
PyInitConfig
function is called with config. The caller doesn’t have to free the error message.
-
int PyInitConfig_GetExitCode(PyInitConfig *config, int *exitcode)¶
Get the config exit code.
Set *exitcode and return
1
if config has an exit code set.Return
0
if config has no exit code set.
Only the
Py_InitializeFromInitConfig()
function can set an exit code if theparse_argv
option is non-zero.An exit code can be set when parsing the command line failed (exit code
2
) or when a command line option asks to display the command line help (exit code0
).
Get Options¶
The configuration option name parameter must be a non-NULL null-terminated UTF-8 encoded string. See Configuration Options.
-
int PyInitConfig_HasOption(PyInitConfig *config, const char *name)¶
Test if the configuration has an option called name.
Return
1
if the option exists, or return0
otherwise.
-
int PyInitConfig_GetInt(PyInitConfig *config, const char *name, int64_t *value)¶
Get an integer configuration option.
Set *value, and return
0
on success.Set an error in config and return
-1
on error.
-
int PyInitConfig_GetStr(PyInitConfig *config, const char *name, char **value)¶
Get a string configuration option as a null-terminated UTF-8 encoded string.
Set *value, and return
0
on success.Set an error in config and return
-1
on error.
*value can be set to
NULL
if the option is an optional string and the option is unset.On success, the string must be released with
free(value)
if it’s notNULL
.
-
int PyInitConfig_GetStrList(PyInitConfig *config, const char *name, size_t *length, char ***items)¶
Get a string list configuration option as an array of null-terminated UTF-8 encoded strings.
Set *length and *value, and return
0
on success.Set an error in config and return
-1
on error.
On success, the string list must be released with
PyInitConfig_FreeStrList(length, items)
.
-
void PyInitConfig_FreeStrList(size_t length, char **items)¶
Free memory of a string list created by
PyInitConfig_GetStrList()
.
Set Options¶
The configuration option name parameter must be a non-NULL null-terminated UTF-8 encoded string. See Configuration Options.
Some configuration options have side effects on other options. This logic is
only implemented when Py_InitializeFromInitConfig()
is called, not by the
“Set” functions below. For example, setting dev_mode
to 1
does not set
faulthandler
to 1
.
-
int PyInitConfig_SetInt(PyInitConfig *config, const char *name, int64_t value)¶
Set an integer configuration option.
Return
0
on success.Set an error in config and return
-1
on error.
-
int PyInitConfig_SetStr(PyInitConfig *config, const char *name, const char *value)¶
Set a string configuration option from a null-terminated UTF-8 encoded string. The string is copied.
Return
0
on success.Set an error in config and return
-1
on error.
-
int PyInitConfig_SetStrList(PyInitConfig *config, const char *name, size_t length, char *const *items)¶
Set a string list configuration option from an array of null-terminated UTF-8 encoded strings. The string list is copied.
Return
0
on success.Set an error in config and return
-1
on error.
Module¶
-
int PyInitConfig_AddModule(PyInitConfig *config, const char *name, PyObject *(*initfunc)(void))¶
Add a built-in extension module to the table of built-in modules.
The new module can be imported by the name name, and uses the function initfunc as the initialization function called on the first attempted import.
Return
0
on success.Set an error in config and return
-1
on error.
If Python is initialized multiple times,
PyInitConfig_AddModule()
must be called at each Python initialization.Similar to the
PyImport_AppendInittab()
function.
Initialize Python¶
-
int Py_InitializeFromInitConfig(PyInitConfig *config)¶
Initialize Python from the initialization configuration.
Return
0
on success.Set an error in config and return
-1
on error.Set an exit code in config and return
-1
if Python wants to exit.
See
PyInitConfig_GetExitcode()
for the exit code case.
Configuration Options¶
Option |
PyConfig/PyPreConfig member |
Type |
Visibility |
|---|---|---|---|
|
|
Read-only |
|
|
|
Public |
|
|
|
Public |
|
|
|
Public |
|
|
|
Public |
|
|
|
Read-only |
|
|
|
Public |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Public |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Public |
|
|
|
Public |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Public |
|
|
|
Read-only |
|
|
|
Public |
|
|
|
Public |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Public |
|
|
|
Public |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Public |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Public |
|
|
|
Public |
|
|
|
Read-only |
|
|
|
Public |
|
|
|
Public |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Public |
|
|
|
Read-only |
|
|
|
Public |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Read-only |
|
|
|
Public |
|
|
|
Read-only |
|
|
|
Public |
|
|
|
Public |
|
|
|
Public |
|
|
|
Read-only |
Visibility:
Public: Can be retrieved by
PyConfig_Get()
and set byPyConfig_Set()
.Read-only: Can be retrieved by
PyConfig_Get()
, but cannot be set byPyConfig_Set()
.
Runtime Python configuration API¶
At runtime, it’s possible to get and set configuration options using
PyConfig_Get()
and PyConfig_Set()
functions.
The configuration option name parameter must be a non-NULL null-terminated UTF-8 encoded string. See Configuration Options.
Some options are read from the sys
attributes. For example, the option
"argv"
is read from sys.argv
.
-
PyObject *PyConfig_Get(const char *name)¶
Get the current runtime value of a configuration option as a Python object.
Return a new reference on success.
Set an exception and return
NULL
on error.
The object type depends on the configuration option. It can be:
bool
int
str
list[str]
dict[str, str]
The caller must have an attached thread state. The function cannot be called before Python initialization nor after Python finalization.
Added in version 3.14.
-
int PyConfig_GetInt(const char *name, int *value)¶
Similar to
PyConfig_Get()
, but get the value as a C int.Return
0
on success.Set an exception and return
-1
on error.
Added in version 3.14.
-
PyObject *PyConfig_Names(void)¶
Get all configuration option names as a
frozenset
.Return a new reference on success.
Set an exception and return
NULL
on error.
The caller must have an attached thread state. The function cannot be called before Python initialization nor after Python finalization.
Added in version 3.14.
-
int PyConfig_Set(const char *name, PyObject *value)¶
Set the current runtime value of a configuration option.
Raise a
ValueError
if there is no option name.Raise a
ValueError
if value is an invalid value.Raise a
ValueError
if the option is read-only (cannot be set).Raise a
TypeError
if value has not the proper type.
The caller must have an attached thread state. The function cannot be called before Python initialization nor after Python finalization.
Raises an auditing event
cpython.PyConfig_Set
with argumentsname
,value
.Added in version 3.14.
PyConfig C API¶
Added in version 3.8.
Python can be initialized with Py_InitializeFromConfig()
and the
PyConfig
structure. It can be preinitialized with
Py_PreInitialize()
and the PyPreConfig
structure.
There are two kinds of configuration:
The Python Configuration can be used to build a customized Python which behaves as the regular Python. For example, environment variables and command line arguments are used to configure Python.
The Isolated Configuration can be used to embed Python into an application. It isolates Python from the system. For example, environment variables are ignored, the LC_CTYPE locale is left unchanged and no signal handler is registered.
The Py_RunMain()
function can be used to write a customized Python
program.
See also Initialization, Finalization, and Threads.
See also
PEP 587 “Python Initialization Configuration”.
Example¶
Example of customized Python always running in isolated mode:
int main(int argc, char **argv)
{
PyStatus status;
PyConfig config;
PyConfig_InitPythonConfig(&config);
config.isolated = 1;
/* Decode command line arguments.
Implicitly preinitialize Python (in isolated mode). */
status = PyConfig_SetBytesArgv(&config, argc, argv);
if (PyStatus_Exception(status)) {
goto exception;
}
status = Py_InitializeFromConfig(&config);
if (PyStatus_Exception(status)) {
goto exception;
}
PyConfig_Clear(&config);
return Py_RunMain();
exception:
PyConfig_Clear(&config);
if (PyStatus_IsExit(status)) {
return status.exitcode;
}
/* Display the error message and exit the process with
non-zero exit code */
Py_ExitStatusException(status);
}
PyWideStringList¶
-
type PyWideStringList¶
List of
wchar_t*
strings.If length is non-zero, items must be non-
NULL
and all strings must be non-NULL
.Methods:
-
PyStatus PyWideStringList_Append(PyWideStringList *list, const wchar_t *item)¶
Append item to list.
Python must be preinitialized to call this function.
-
PyStatus PyWideStringList_Insert(PyWideStringList *list, Py_ssize_t index, const wchar_t *item)¶
Insert item into list at index.
If index is greater than or equal to list length, append item to list.
index must be greater than or equal to
0
.Python must be preinitialized to call this function.
Structure fields:
-
Py_ssize_t length¶
List length.
-
wchar_t **items¶
List items.
-
PyStatus PyWideStringList_Append(PyWideStringList *list, const wchar_t *item)¶
PyStatus¶
-
type PyStatus¶
Structure to store an initialization function status: success, error or exit.
For an error, it can store the C function name which created the error.
Structure fields:
-
int exitcode¶
Exit code. Argument passed to
exit()
.
-
const char *err_msg¶
Error message.
-
const char *func¶
Name of the function which created an error, can be
NULL
.
Functions to create a status:
-
PyStatus PyStatus_Error(const char *err_msg)¶
Initialization error with a message.
err_msg must not be
NULL
.
Functions to handle a status:
-
int PyStatus_Exception(PyStatus status)¶
Is the status an error or an exit? If true, the exception must be handled; by calling
Py_ExitStatusException()
for example.
-
int exitcode¶
Note
Internally, Python uses macros which set PyStatus.func
,
whereas functions to create a status set func
to NULL
.
Example:
PyStatus alloc(void **ptr, size_t size)
{
*ptr = PyMem_RawMalloc(size);
if (*ptr == NULL) {
return PyStatus_NoMemory();
}
return PyStatus_Ok();
}
int main(int argc, char **argv)
{
void *ptr;
PyStatus status = alloc(&ptr, 16);
if (PyStatus_Exception(status)) {
Py_ExitStatusException(status);
}
PyMem_Free(ptr);
return 0;
}
PyPreConfig¶
-
type PyPreConfig¶
Structure used to preinitialize Python.
Function to initialize a preconfiguration:
-
void PyPreConfig_InitPythonConfig(PyPreConfig *preconfig)¶
Initialize the preconfiguration with Python Configuration.
-
void PyPreConfig_InitIsolatedConfig(PyPreConfig *preconfig)¶
Initialize the preconfiguration with Isolated Configuration.
Structure fields:
-
int allocator¶
Name of the Python memory allocators:
PYMEM_ALLOCATOR_NOT_SET
(0
): don’t change memory allocators (use defaults).PYMEM_ALLOCATOR_DEFAULT
(1
): default memory allocators.PYMEM_ALLOCATOR_DEBUG
(2
): default memory allocators with debug hooks.PYMEM_ALLOCATOR_MALLOC
(3
): usemalloc()
of the C library.PYMEM_ALLOCATOR_MALLOC_DEBUG
(4
): force usage ofmalloc()
with debug hooks.PYMEM_ALLOCATOR_PYMALLOC
(5
): Python pymalloc memory allocator.PYMEM_ALLOCATOR_PYMALLOC_DEBUG
(6
): Python pymalloc memory allocator with debug hooks.PYMEM_ALLOCATOR_MIMALLOC
(6
): usemimalloc
, a fast malloc replacement.PYMEM_ALLOCATOR_MIMALLOC_DEBUG
(7
): usemimalloc
, a fast malloc replacement with debug hooks.
PYMEM_ALLOCATOR_PYMALLOC
andPYMEM_ALLOCATOR_PYMALLOC_DEBUG
are not supported if Python isconfigured using --without-pymalloc
.PYMEM_ALLOCATOR_MIMALLOC
andPYMEM_ALLOCATOR_MIMALLOC_DEBUG
are not supported if Python isconfigured using --without-mimalloc
or if the underlying atomic support isn’t available.See Memory Management.
Default:
PYMEM_ALLOCATOR_NOT_SET
.
-
int configure_locale¶
Set the LC_CTYPE locale to the user preferred locale.
If equals to
0
, setcoerce_c_locale
andcoerce_c_locale_warn
members to0
.See the locale encoding.
Default:
1
in Python config,0
in isolated config.
-
int coerce_c_locale¶
If equals to
2
, coerce the C locale.If equals to
1
, read the LC_CTYPE locale to decide if it should be coerced.See the locale encoding.
Default:
-1
in Python config,0
in isolated config.
-
int coerce_c_locale_warn¶
If non-zero, emit a warning if the C locale is coerced.
Default:
-1
in Python config,0
in isolated config.
-
int dev_mode¶
Python Development Mode: see
PyConfig.dev_mode
.Default:
-1
in Python mode,0
in isolated mode.
-
int isolated¶
Isolated mode: see
PyConfig.isolated
.Default:
0
in Python mode,1
in isolated mode.
-
int legacy_windows_fs_encoding¶
If non-zero:
Set
PyPreConfig.utf8_mode
to0
,Set
PyConfig.filesystem_encoding
to"mbcs"
,Set
PyConfig.filesystem_errors
to"replace"
.
Initialized from the
PYTHONLEGACYWINDOWSFSENCODING
environment variable value.Only available on Windows.
#ifdef MS_WINDOWS
macro can be used for Windows specific code.Default:
0
.
-
int parse_argv¶
If non-zero,
Py_PreInitializeFromArgs()
andPy_PreInitializeFromBytesArgs()
parse theirargv
argument the same way the regular Python parses command line arguments: see Command Line Arguments.Default:
1
in Python config,0
in isolated config.
-
int use_environment¶
Use environment variables? See
PyConfig.use_environment
.Default:
1
in Python config and0
in isolated config.
-
int utf8_mode¶
If non-zero, enable the Python UTF-8 Mode.
Set to
0
or1
by the-X utf8
command line option and thePYTHONUTF8
environment variable.Also set to
1
if theLC_CTYPE
locale isC
orPOSIX
.Default:
-1
in Python config and0
in isolated config.
-
void PyPreConfig_InitPythonConfig(PyPreConfig *preconfig)¶
Preinitialize Python with PyPreConfig¶
The preinitialization of Python:
Set the Python memory allocators (
PyPreConfig.allocator
)Configure the LC_CTYPE locale (locale encoding)
Set the Python UTF-8 Mode (
PyPreConfig.utf8_mode
)
The current preconfiguration (PyPreConfig
type) is stored in
_PyRuntime.preconfig
.
Functions to preinitialize Python:
-
PyStatus Py_PreInitialize(const PyPreConfig *preconfig)¶
Preinitialize Python from preconfig preconfiguration.
preconfig must not be
NULL
.
-
PyStatus Py_PreInitializeFromBytesArgs(const PyPreConfig *preconfig, int argc, char *const *argv)¶
Preinitialize Python from preconfig preconfiguration.
Parse argv command line arguments (bytes strings) if
parse_argv
of preconfig is non-zero.preconfig must not be
NULL
.
-
PyStatus Py_PreInitializeFromArgs(const PyPreConfig *preconfig, int argc, wchar_t *const *argv)¶
Preinitialize Python from preconfig preconfiguration.
Parse argv command line arguments (wide strings) if
parse_argv
of preconfig is non-zero.preconfig must not be
NULL
.
The caller is responsible to handle exceptions (error or exit) using
PyStatus_Exception()
and Py_ExitStatusException()
.
For Python Configuration
(PyPreConfig_InitPythonConfig()
), if Python is initialized with
command line arguments, the command line arguments must also be passed to
preinitialize Python, since they have an effect on the pre-configuration
like encodings. For example, the -X utf8
command line option
enables the Python UTF-8 Mode.
PyMem_SetAllocator()
can be called after Py_PreInitialize()
and
before Py_InitializeFromConfig()
to install a custom memory allocator.
It can be called before Py_PreInitialize()
if
PyPreConfig.allocator
is set to PYMEM_ALLOCATOR_NOT_SET
.
Python memory allocation functions like PyMem_RawMalloc()
must not be
used before the Python preinitialization, whereas calling directly malloc()
and free()
is always safe. Py_DecodeLocale()
must not be called
before the Python preinitialization.
Example using the preinitialization to enable the Python UTF-8 Mode:
PyStatus status;
PyPreConfig preconfig;
PyPreConfig_InitPythonConfig(&preconfig);
preconfig.utf8_mode = 1;
status = Py_PreInitialize(&preconfig);
if (PyStatus_Exception(status)) {
Py_ExitStatusException(status);
}
/* at this point, Python speaks UTF-8 */
Py_Initialize();
/* ... use Python API here ... */
Py_Finalize();
PyConfig¶
-
type PyConfig¶
Structure containing most parameters to configure Python.
When done, the
PyConfig_Clear()
function must be used to release the configuration memory.Structure methods:
-
void PyConfig_InitPythonConfig(PyConfig *config)¶
Initialize configuration with the Python Configuration.
-
void PyConfig_InitIsolatedConfig(PyConfig *config)¶
Initialize configuration with the Isolated Configuration.
-
PyStatus PyConfig_SetString(PyConfig *config, wchar_t *const *config_str, const wchar_t *str)¶
Copy the wide character string str into
*config_str
.Preinitialize Python if needed.
-
PyStatus PyConfig_SetBytesString(PyConfig *config, wchar_t *const *config_str, const char *str)¶
Decode str using
Py_DecodeLocale()
and set the result into*config_str
.Preinitialize Python if needed.
-
PyStatus PyConfig_SetArgv(PyConfig *config, int argc, wchar_t *const *argv)¶
Set command line arguments (
argv
member of config) from the argv list of wide character strings.Preinitialize Python if needed.
-
PyStatus PyConfig_SetBytesArgv(PyConfig *config, int argc, char *const *argv)¶
Set command line arguments (
argv
member of config) from the argv list of bytes strings. Decode bytes usingPy_DecodeLocale()
.Preinitialize Python if needed.
-
PyStatus PyConfig_SetWideStringList(PyConfig *config, PyWideStringList *list, Py_ssize_t length, wchar_t **items)¶
Set the list of wide strings list to length and items.
Preinitialize Python if needed.
-
PyStatus PyConfig_Read(PyConfig *config)¶
Read all Python configuration.
Fields which are already initialized are left unchanged.
Fields for path configuration are no longer calculated or modified when calling this function, as of Python 3.11.
The
PyConfig_Read()
function only parsesPyConfig.argv
arguments once:PyConfig.parse_argv
is set to2
after arguments are parsed. Since Python arguments are stripped fromPyConfig.argv
, parsing arguments twice would parse the application options as Python options.Preinitialize Python if needed.
Changed in version 3.10: The
PyConfig.argv
arguments are now only parsed once,PyConfig.parse_argv
is set to2
after arguments are parsed, and arguments are only parsed ifPyConfig.parse_argv
equals1
.Changed in version 3.11:
PyConfig_Read()
no longer calculates all paths, and so fields listed under Python Path Configuration may no longer be updated untilPy_InitializeFromConfig()
is called.
Most
PyConfig
methods preinitialize Python if needed. In that case, the Python preinitialization configuration (PyPreConfig
) is based on thePyConfig
. If configuration fields which are in common withPyPreConfig
are tuned, they must be set before calling aPyConfig
method:Moreover, if
PyConfig_SetArgv()
orPyConfig_SetBytesArgv()
is used, this method must be called before other methods, since the preinitialization configuration depends on command line arguments (ifparse_argv
is non-zero).The caller of these methods is responsible to handle exceptions (error or exit) using
PyStatus_Exception()
andPy_ExitStatusException()
.Structure fields:
-
PyWideStringList argv¶
Set
sys.argv
command line arguments based onargv
. These parameters are similar to those passed to the program’smain()
function with the difference that the first entry should refer to the script file to be executed rather than the executable hosting the Python interpreter. If there isn’t a script that will be run, the first entry inargv
can be an empty string.Set
parse_argv
to1
to parseargv
the same way the regular Python parses Python command line arguments and then to strip Python arguments fromargv
.If
argv
is empty, an empty string is added to ensure thatsys.argv
always exists and is never empty.Default:
NULL
.See also the
orig_argv
member.
-
int safe_path¶
If equals to zero,
Py_RunMain()
prepends a potentially unsafe path tosys.path
at startup:If
argv[0]
is equal toL"-m"
(python -m module
), prepend the current working directory.If running a script (
python script.py
), prepend the script’s directory. If it’s a symbolic link, resolve symbolic links.Otherwise (
python -c code
andpython
), prepend an empty string, which means the current working directory.
Set to
1
by the-P
command line option and thePYTHONSAFEPATH
environment variable.Default:
0
in Python config,1
in isolated config.Added in version 3.11.
-
wchar_t *base_exec_prefix¶
-
Default:
NULL
.Part of the Python Path Configuration output.
See also
PyConfig.exec_prefix
.
-
wchar_t *base_executable¶
Python base executable:
sys._base_executable
.Set by the
__PYVENV_LAUNCHER__
environment variable.Set from
PyConfig.executable
ifNULL
.Default:
NULL
.Part of the Python Path Configuration output.
See also
PyConfig.executable
.
-
wchar_t *base_prefix¶
-
Default:
NULL
.Part of the Python Path Configuration output.
See also
PyConfig.prefix
.
-
int buffered_stdio¶
If equals to
0
andconfigure_c_stdio
is non-zero, disable buffering on the C streams stdout and stderr.Set to
0
by the-u
command line option and thePYTHONUNBUFFERED
environment variable.stdin is always opened in buffered mode.
Default:
1
.
-
int bytes_warning¶
If equals to
1
, issue a warning when comparingbytes
orbytearray
withstr
, or comparingbytes
withint
.If equal or greater to
2
, raise aBytesWarning
exception in these cases.Incremented by the
-b
command line option.Default:
0
.
-
int warn_default_encoding¶
If non-zero, emit a
EncodingWarning
warning whenio.TextIOWrapper
uses its default encoding. See Opt-in EncodingWarning for details.Default:
0
.Added in version 3.10.
-
int code_debug_ranges¶
If equals to
0
, disables the inclusion of the end line and column mappings in code objects. Also disables traceback printing carets to specific error locations.Set to
0
by thePYTHONNODEBUGRANGES
environment variable and by the-X no_debug_ranges
command line option.Default:
1
.Added in version 3.11.
-
wchar_t *check_hash_pycs_mode¶
Control the validation behavior of hash-based
.pyc
files: value of the--check-hash-based-pycs
command line option.Valid values:
L"always"
: Hash the source file for invalidation regardless of value of the ‘check_source’ flag.L"never"
: Assume that hash-based pycs always are valid.L"default"
: The ‘check_source’ flag in hash-based pycs determines invalidation.
Default:
L"default"
.See also PEP 552 “Deterministic pycs”.
-
int configure_c_stdio¶
If non-zero, configure C standard streams:
On Windows, set the binary mode (
O_BINARY
) on stdin, stdout and stderr.If
buffered_stdio
equals zero, disable buffering of stdin, stdout and stderr streams.If
interactive
is non-zero, enable stream buffering on stdin and stdout (only stdout on Windows).
Default:
1
in Python config,0
in isolated config.
-
int dev_mode¶
If non-zero, enable the Python Development Mode.
Set to
1
by the-X dev
option and thePYTHONDEVMODE
environment variable.Default:
-1
in Python mode,0
in isolated mode.
-
int dump_refs¶
Dump Python references?
If non-zero, dump all objects which are still alive at exit.
Set to
1
by thePYTHONDUMPREFS
environment variable.Needs a special build of Python with the
Py_TRACE_REFS
macro defined: see theconfigure --with-trace-refs option
.Default:
0
.
-
wchar_t *dump_refs_file¶
Filename where to dump Python references.
Set by the
PYTHONDUMPREFSFILE
environment variable.Default:
NULL
.Added in version 3.11.
-
wchar_t *exec_prefix¶
The site-specific directory prefix where the platform-dependent Python files are installed:
sys.exec_prefix
.Default:
NULL
.Part of the Python Path Configuration output.
See also
PyConfig.base_exec_prefix
.
-
wchar_t *executable¶
The absolute path of the executable binary for the Python interpreter:
sys.executable
.Default:
NULL
.Part of the Python Path Configuration output.
See also
PyConfig.base_executable
.
-
int faulthandler¶
Enable faulthandler?
If non-zero, call
faulthandler.enable()
at startup.Set to
1
by-X faulthandler
and thePYTHONFAULTHANDLER
environment variable.Default:
-1
in Python mode,0
in isolated mode.
-
wchar_t *filesystem_encoding¶
Filesystem encoding:
sys.getfilesystemencoding()
.On macOS, Android and VxWorks: use
"utf-8"
by default.On Windows: use
"utf-8"
by default, or"mbcs"
iflegacy_windows_fs_encoding
ofPyPreConfig
is non-zero.Default encoding on other platforms:
"utf-8"
ifPyPreConfig.utf8_mode
is non-zero."ascii"
if Python detects thatnl_langinfo(CODESET)
announces the ASCII encoding, whereas thembstowcs()
function decodes from a different encoding (usually Latin1)."utf-8"
ifnl_langinfo(CODESET)
returns an empty string.Otherwise, use the locale encoding:
nl_langinfo(CODESET)
result.
At Python startup, the encoding name is normalized to the Python codec name. For example,
"ANSI_X3.4-1968"
is replaced with"ascii"
.See also the
filesystem_errors
member.
-
wchar_t *filesystem_errors¶
Filesystem error handler:
sys.getfilesystemencodeerrors()
.On Windows: use
"surrogatepass"
by default, or"replace"
iflegacy_windows_fs_encoding
ofPyPreConfig
is non-zero.On other platforms: use
"surrogateescape"
by default.Supported error handlers:
"strict"
"surrogateescape"
"surrogatepass"
(only supported with the UTF-8 encoding)
See also the
filesystem_encoding
member.
-
int use_frozen_modules¶
If non-zero, use frozen modules.
Set by the
PYTHON_FROZEN_MODULES
environment variable.Default:
1
in a release build, or0
in a debug build.
-
unsigned long hash_seed¶
-
int use_hash_seed¶
Randomized hash function seed.
If
use_hash_seed
is zero, a seed is chosen randomly at Python startup, andhash_seed
is ignored.Set by the
PYTHONHASHSEED
environment variable.Default use_hash_seed value:
-1
in Python mode,0
in isolated mode.
-
wchar_t *home¶
Set the default Python “home” directory, that is, the location of the standard Python libraries (see
PYTHONHOME
).Set by the
PYTHONHOME
environment variable.Default:
NULL
.Part of the Python Path Configuration input.
-
int import_time¶
If
1
, profile import time. If2
, include additional output that indicates when an imported module has already been loaded.Set by the
-X importtime
option and thePYTHONPROFILEIMPORTTIME
environment variable.Default:
0
.Changed in version 3.14: Added support for
import_time = 2
-
int inspect¶
Enter interactive mode after executing a script or a command.
If greater than
0
, enable inspect: when a script is passed as first argument or the -c option is used, enter interactive mode after executing the script or the command, even whensys.stdin
does not appear to be a terminal.Incremented by the
-i
command line option. Set to1
if thePYTHONINSPECT
environment variable is non-empty.Default:
0
.
-
int install_signal_handlers¶
Install Python signal handlers?
Default:
1
in Python mode,0
in isolated mode.
-
int interactive¶
If greater than
0
, enable the interactive mode (REPL).Incremented by the
-i
command line option.Default:
0
.
-
int int_max_str_digits¶
Configures the integer string conversion length limitation. An initial value of
-1
means the value will be taken from the command line or environment or otherwise default to 4300 (sys.int_info.default_max_str_digits
). A value of0
disables the limitation. Values greater than zero but less than 640 (sys.int_info.str_digits_check_threshold
) are unsupported and will produce an error.Configured by the
-X int_max_str_digits
command line flag or thePYTHONINTMAXSTRDIGITS
environment variable.Default:
-1
in Python mode. 4300 (sys.int_info.default_max_str_digits
) in isolated mode.Added in version 3.12.
-
int cpu_count¶
If the value of
cpu_count
is not-1
then it will override the return values ofos.cpu_count()
,os.process_cpu_count()
, andmultiprocessing.cpu_count()
.Configured by the
-X cpu_count=n|default
command line flag or thePYTHON_CPU_COUNT
environment variable.Default:
-1
.Added in version 3.13.
-
int isolated¶
If greater than
0
, enable isolated mode:Set
safe_path
to1
: don’t prepend a potentially unsafe path tosys.path
at Python startup, such as the current directory, the script’s directory or an empty string.Set
use_environment
to0
: ignorePYTHON
environment variables.Set
user_site_directory
to0
: don’t add the user site directory tosys.path
.Python REPL doesn’t import
readline
nor enable default readline configuration on interactive prompts.
Set to
1
by the-I
command line option.Default:
0
in Python mode,1
in isolated mode.See also the Isolated Configuration and
PyPreConfig.isolated
.
-
int legacy_windows_stdio¶
If non-zero, use
io.FileIO
instead ofio._WindowsConsoleIO
forsys.stdin
,sys.stdout
andsys.stderr
.Set to
1
if thePYTHONLEGACYWINDOWSSTDIO
environment variable is set to a non-empty string.Only available on Windows.
#ifdef MS_WINDOWS
macro can be used for Windows specific code.Default:
0
.See also the PEP 528 (Change Windows console encoding to UTF-8).
-
int malloc_stats¶
If non-zero, dump statistics on Python pymalloc memory allocator at exit.
Set to
1
by thePYTHONMALLOCSTATS
environment variable.The option is ignored if Python is
configured using the --without-pymalloc option
.Default:
0
.
-
wchar_t *platlibdir¶
Platform library directory name:
sys.platlibdir
.Set by the
PYTHONPLATLIBDIR
environment variable.Default: value of the
PLATLIBDIR
macro which is set by theconfigure --with-platlibdir option
(default:"lib"
, or"DLLs"
on Windows).Part of the Python Path Configuration input.
Added in version 3.9.
Changed in version 3.11: This macro is now used on Windows to locate the standard library extension modules, typically under
DLLs
. However, for compatibility, note that this value is ignored for any non-standard layouts, including in-tree builds and virtual environments.
-
wchar_t *pythonpath_env¶
Module search paths (
sys.path
) as a string separated byDELIM
(os.pathsep
).Set by the
PYTHONPATH
environment variable.Default:
NULL
.Part of the Python Path Configuration input.
-
PyWideStringList module_search_paths¶
-
int module_search_paths_set¶
Module search paths:
sys.path
.If
module_search_paths_set
is equal to0
,Py_InitializeFromConfig()
will replacemodule_search_paths
and setsmodule_search_paths_set
to1
.Default: empty list (
module_search_paths
) and0
(module_search_paths_set
).Part of the Python Path Configuration output.
-
int optimization_level¶
Compilation optimization level:
0
: Peephole optimizer, set__debug__
toTrue
.1
: Level 0, remove assertions, set__debug__
toFalse
.2
: Level 1, strip docstrings.
Incremented by the
-O
command line option. Set to thePYTHONOPTIMIZE
environment variable value.Default:
0
.
-
PyWideStringList orig_argv¶
The list of the original command line arguments passed to the Python executable:
sys.orig_argv
.If
orig_argv
list is empty andargv
is not a list only containing an empty string,PyConfig_Read()
copiesargv
intoorig_argv
before modifyingargv
(ifparse_argv
is non-zero).See also the
argv
member and thePy_GetArgcArgv()
function.Default: empty list.
Added in version 3.10.
-
int parse_argv¶
Parse command line arguments?
If equals to
1
, parseargv
the same way the regular Python parses command line arguments, and strip Python arguments fromargv
.The
PyConfig_Read()
function only parsesPyConfig.argv
arguments once:PyConfig.parse_argv
is set to2
after arguments are parsed. Since Python arguments are stripped fromPyConfig.argv
, parsing arguments twice would parse the application options as Python options.Default:
1
in Python mode,0
in isolated mode.Changed in version 3.10: The
PyConfig.argv
arguments are now only parsed ifPyConfig.parse_argv
equals to1
.
-
int parser_debug¶
Parser debug mode. If greater than
0
, turn on parser debugging output (for expert only, depending on compilation options).Incremented by the
-d
command line option. Set to thePYTHONDEBUG
environment variable value.Needs a debug build of Python (the
Py_DEBUG
macro must be defined).Default:
0
.
-
int pathconfig_warnings¶
If non-zero, calculation of path configuration is allowed to log warnings into
stderr
. If equals to0
, suppress these warnings.Default:
1
in Python mode,0
in isolated mode.Part of the Python Path Configuration input.
Changed in version 3.11: Now also applies on Windows.
-
wchar_t *prefix¶
The site-specific directory prefix where the platform independent Python files are installed:
sys.prefix
.Default:
NULL
.Part of the Python Path Configuration output.
See also
PyConfig.base_prefix
.
-
wchar_t *program_name¶
Program name used to initialize
executable
and in early error messages during Python initialization.On macOS, use
PYTHONEXECUTABLE
environment variable if set.If the
WITH_NEXT_FRAMEWORK
macro is defined, use__PYVENV_LAUNCHER__
environment variable if set.Use
argv[0]
ofargv
if available and non-empty.Otherwise, use
L"python"
on Windows, orL"python3"
on other platforms.
Default:
NULL
.Part of the Python Path Configuration input.
-
wchar_t *pycache_prefix¶
Directory where cached
.pyc
files are written:sys.pycache_prefix
.Set by the
-X pycache_prefix=PATH
command line option and thePYTHONPYCACHEPREFIX
environment variable. The command-line option takes precedence.If
NULL
,sys.pycache_prefix
is set toNone
.Default:
NULL
.
-
int quiet¶
Quiet mode. If greater than
0
, don’t display the copyright and version at Python startup in interactive mode.Incremented by the
-q
command line option.Default:
0
.
-
wchar_t *run_command¶
Value of the
-c
command line option.Used by
Py_RunMain()
.Default:
NULL
.
-
wchar_t *run_filename¶
Filename passed on the command line: trailing command line argument without
-c
or-m
. It is used by thePy_RunMain()
function.For example, it is set to
script.py
by thepython3 script.py arg
command line.See also the
PyConfig.skip_source_first_line
option.Default:
NULL
.
-
wchar_t *run_module¶
Value of the
-m
command line option.Used by
Py_RunMain()
.Default:
NULL
.
-
wchar_t *run_presite¶
package.module
path to module that should be imported beforesite.py
is run.Set by the
-X presite=package.module
command-line option and thePYTHON_PRESITE
environment variable. The command-line option takes precedence.Needs a debug build of Python (the
Py_DEBUG
macro must be defined).Default:
NULL
.
-
int show_ref_count¶
Show total reference count at exit (excluding immortal objects)?
Set to
1
by-X showrefcount
command line option.Needs a debug build of Python (the
Py_REF_DEBUG
macro must be defined).Default:
0
.
-
int site_import¶
Import the
site
module at startup?If equal to zero, disable the import of the module site and the site-dependent manipulations of
sys.path
that it entails.Also disable these manipulations if the
site
module is explicitly imported later (callsite.main()
if you want them to be triggered).Set to
0
by the-S
command line option.sys.flags.no_site
is set to the inverted value ofsite_import
.Default:
1
.
-
int skip_source_first_line¶
If non-zero, skip the first line of the
PyConfig.run_filename
source.It allows the usage of non-Unix forms of
#!cmd
. This is intended for a DOS specific hack only.Set to
1
by the-x
command line option.Default:
0
.
-
wchar_t *stdio_encoding¶
-
wchar_t *stdio_errors¶
Encoding and encoding errors of
sys.stdin
,sys.stdout
andsys.stderr
(butsys.stderr
always uses"backslashreplace"
error handler).Use the
PYTHONIOENCODING
environment variable if it is non-empty.Default encoding:
"UTF-8"
ifPyPreConfig.utf8_mode
is non-zero.Otherwise, use the locale encoding.
Default error handler:
On Windows: use
"surrogateescape"
."surrogateescape"
ifPyPreConfig.utf8_mode
is non-zero, or if the LC_CTYPE locale is “C” or “POSIX”."strict"
otherwise.
See also
PyConfig.legacy_windows_stdio
.
-
int tracemalloc¶
Enable tracemalloc?
If non-zero, call
tracemalloc.start()
at startup.Set by
-X tracemalloc=N
command line option and by thePYTHONTRACEMALLOC
environment variable.Default:
-1
in Python mode,0
in isolated mode.
-
int perf_profiling¶
Enable the Linux
perf
profiler support?If equals to
1
, enable support for the Linuxperf
profiler.If equals to
2
, enable support for the Linuxperf
profiler with DWARF JIT support.Set to
1
by-X perf
command-line option and thePYTHONPERFSUPPORT
environment variable.Set to
2
by the-X perf_jit
command-line option and thePYTHON_PERF_JIT_SUPPORT
environment variable.Default:
-1
.See also
See Python support for the Linux perf profiler for more information.
Added in version 3.12.
-
wchar_t *stdlib_dir¶
Directory of the Python standard library.
Default:
NULL
.Added in version 3.11.
-
int use_environment¶
-
If equals to zero, ignore the environment variables.
Set to
0
by the-E
environment variable.Default:
1
in Python config and0
in isolated config.
-
int use_system_logger¶
If non-zero,
stdout
andstderr
will be redirected to the system log.Only available on macOS 10.12 and later, and on iOS.
Default:
0
(don’t use the system log) on macOS;1
on iOS (use the system log).Added in version 3.14.
-
int user_site_directory¶
If non-zero, add the user site directory to
sys.path
.Set to
0
by the-s
and-I
command line options.Set to
0
by thePYTHONNOUSERSITE
environment variable.Default:
1
in Python mode,0
in isolated mode.
-
int verbose¶
Verbose mode. If greater than
0
, print a message each time a module is imported, showing the place (filename or built-in module) from which it is loaded.If greater than or equal to
2
, print a message for each file that is checked for when searching for a module. Also provides information on module cleanup at exit.Incremented by the
-v
command line option.Set by the
PYTHONVERBOSE
environment variable value.Default:
0
.
-
PyWideStringList warnoptions¶
Options of the
warnings
module to build warnings filters, lowest to highest priority:sys.warnoptions
.The
warnings
module addssys.warnoptions
in the reverse order: the lastPyConfig.warnoptions
item becomes the first item ofwarnings.filters
which is checked first (highest priority).The
-W
command line options adds its value towarnoptions
, it can be used multiple times.The
PYTHONWARNINGS
environment variable can also be used to add warning options. Multiple options can be specified, separated by commas (,
).Default: empty list.
-
int write_bytecode¶
If equal to
0
, Python won’t try to write.pyc
files on the import of source modules.Set to
0
by the-B
command line option and thePYTHONDONTWRITEBYTECODE
environment variable.sys.dont_write_bytecode
is initialized to the inverted value ofwrite_bytecode
.Default:
1
.
-
PyWideStringList xoptions¶
Values of the
-X
command line options:sys._xoptions
.Default: empty list.
-
int _pystats¶
If non-zero, write performance statistics at Python exit.
Need a special build with the
Py_STATS
macro: see--enable-pystats
.Default:
0
.
-
void PyConfig_InitPythonConfig(PyConfig *config)¶
If parse_argv
is non-zero, argv
arguments are parsed the same way the regular Python parses command line
arguments, and Python arguments are stripped from
argv
.
The xoptions
options are parsed to set other options: see
the -X
command line option.
Changed in version 3.9: The show_alloc_count
field has been removed.
Initialization with PyConfig¶
Initializing the interpreter from a populated configuration struct is handled
by calling Py_InitializeFromConfig()
.
The caller is responsible to handle exceptions (error or exit) using
PyStatus_Exception()
and Py_ExitStatusException()
.
If PyImport_FrozenModules()
, PyImport_AppendInittab()
or
PyImport_ExtendInittab()
are used, they must be set or called after
Python preinitialization and before the Python initialization. If Python is
initialized multiple times, PyImport_AppendInittab()
or
PyImport_ExtendInittab()
must be called before each Python
initialization.
The current configuration (PyConfig
type) is stored in
PyInterpreterState.config
.
Example setting the program name:
void init_python(void)
{
PyStatus status;
PyConfig config;
PyConfig_InitPythonConfig(&config);
/* Set the program name. Implicitly preinitialize Python. */
status = PyConfig_SetString(&config, &config.program_name,
L"/path/to/my_program");
if (PyStatus_Exception(status)) {
goto exception;
}
status = Py_InitializeFromConfig(&config);
if (PyStatus_Exception(status)) {
goto exception;
}
PyConfig_Clear(&config);
return;
exception:
PyConfig_Clear(&config);
Py_ExitStatusException(status);
}
More complete example modifying the default configuration, read the configuration, and then override some parameters. Note that since 3.11, many parameters are not calculated until initialization, and so values cannot be read from the configuration structure. Any values set before initialize is called will be left unchanged by initialization:
PyStatus init_python(const char *program_name)
{
PyStatus status;
PyConfig config;
PyConfig_InitPythonConfig(&config);
/* Set the program name before reading the configuration
(decode byte string from the locale encoding).
Implicitly preinitialize Python. */
status = PyConfig_SetBytesString(&config, &config.program_name,
program_name);
if (PyStatus_Exception(status)) {
goto done;
}
/* Read all configuration at once */
status = PyConfig_Read(&config);
if (PyStatus_Exception(status)) {
goto done;
}
/* Specify sys.path explicitly */
/* If you want to modify the default set of paths, finish
initialization first and then use PySys_GetObject("path") */
config.module_search_paths_set = 1;
status = PyWideStringList_Append(&config.module_search_paths,
L"/path/to/stdlib");
if (PyStatus_Exception(status)) {
goto done;
}
status = PyWideStringList_Append(&config.module_search_paths,
L"/path/to/more/modules");
if (PyStatus_Exception(status)) {
goto done;
}
/* Override executable computed by PyConfig_Read() */
status = PyConfig_SetString(&config, &config.executable,
L"/path/to/my_executable");
if (PyStatus_Exception(status)) {
goto done;
}
status = Py_InitializeFromConfig(&config);
done:
PyConfig_Clear(&config);
return status;
}
Isolated Configuration¶
PyPreConfig_InitIsolatedConfig()
and
PyConfig_InitIsolatedConfig()
functions create a configuration to
isolate Python from the system. For example, to embed Python into an
application.
This configuration ignores global configuration variables, environment
variables, command line arguments (PyConfig.argv
is not parsed)
and user site directory. The C standard streams (ex: stdout
) and the
LC_CTYPE locale are left unchanged. Signal handlers are not installed.
Configuration files are still used with this configuration to determine
paths that are unspecified. Ensure PyConfig.home
is specified
to avoid computing the default path configuration.
Python Configuration¶
PyPreConfig_InitPythonConfig()
and PyConfig_InitPythonConfig()
functions create a configuration to build a customized Python which behaves as
the regular Python.
Environments variables and command line arguments are used to configure Python, whereas global configuration variables are ignored.
This function enables C locale coercion (PEP 538)
and Python UTF-8 Mode
(PEP 540) depending on the LC_CTYPE locale, PYTHONUTF8
and
PYTHONCOERCECLOCALE
environment variables.
Python Path Configuration¶
PyConfig
contains multiple fields for the path configuration:
Path configuration inputs:
current working directory: to get absolute paths
PATH
environment variable to get the program full path (fromPyConfig.program_name
)__PYVENV_LAUNCHER__
environment variable(Windows only) Application paths in the registry under “SoftwarePythonPythonCoreX.YPythonPath” of HKEY_CURRENT_USER and HKEY_LOCAL_MACHINE (where X.Y is the Python version).
Path configuration output fields:
If at least one “output field” is not set, Python calculates the path
configuration to fill unset fields. If
module_search_paths_set
is equal to 0
,
module_search_paths
is overridden and
module_search_paths_set
is set to 1
.
It is possible to completely ignore the function calculating the default
path configuration by setting explicitly all path configuration output
fields listed above. A string is considered as set even if it is non-empty.
module_search_paths
is considered as set if
module_search_paths_set
is set to 1
. In this case,
module_search_paths
will be used without modification.
Set pathconfig_warnings
to 0
to suppress warnings when
calculating the path configuration (Unix only, Windows does not log any warning).
If base_prefix
or base_exec_prefix
fields are not set, they inherit their value from prefix
and exec_prefix
respectively.
Py_RunMain()
and Py_Main()
modify sys.path
:
If
run_filename
is set and is a directory which contains a__main__.py
script, prependrun_filename
tosys.path
.If
isolated
is zero:If
run_module
is set, prepend the current directory tosys.path
. Do nothing if the current directory cannot be read.If
run_filename
is set, prepend the directory of the filename tosys.path
.Otherwise, prepend an empty string to
sys.path
.
If site_import
is non-zero, sys.path
can be
modified by the site
module. If
user_site_directory
is non-zero and the user’s
site-package directory exists, the site
module appends the user’s
site-package directory to sys.path
.
The following configuration files are used by the path configuration:
pyvenv.cfg
._pth
file (ex:python._pth
)pybuilddir.txt
(Unix only)
If a ._pth
file is present:
Set
isolated
to1
.Set
use_environment
to0
.Set
site_import
to0
.Set
safe_path
to1
.
If home
is not set and a pyvenv.cfg
file is present in
the same directory as executable
, or its parent,
prefix
and exec_prefix
are set that
location. When this happens, base_prefix
and
base_exec_prefix
still keep their value, pointing to the
base installation. See Virtual Environments for more
information.
The __PYVENV_LAUNCHER__
environment variable is used to set
PyConfig.base_executable
.
Changed in version 3.14: prefix
, and exec_prefix
, are now
set to the pyvenv.cfg
directory. This was previously done by site
,
therefore affected by -S
.
Py_GetArgcArgv()¶
-
void Py_GetArgcArgv(int *argc, wchar_t ***argv)¶
Get the original command line arguments, before Python modified them.
See also
PyConfig.orig_argv
member.
Delaying main module execution¶
In some embedding use cases, it may be desirable to separate interpreter initialization from the execution of the main module.
This separation can be achieved by setting PyConfig.run_command
to the empty
string during initialization (to prevent the interpreter from dropping into the
interactive prompt), and then subsequently executing the desired main module
code using __main__.__dict__
as the global namespace.


Tuple Objects

Tuple Objects¶
-
PyTypeObject PyTuple_Type¶
- Part of the Stable ABI.
This instance of
PyTypeObject
represents the Python tuple type; it is the same object astuple
in the Python layer.
-
int PyTuple_Check(PyObject *p)¶
Return true if p is a tuple object or an instance of a subtype of the tuple type. This function always succeeds.
-
int PyTuple_CheckExact(PyObject *p)¶
Return true if p is a tuple object, but not an instance of a subtype of the tuple type. This function always succeeds.
-
PyObject *PyTuple_New(Py_ssize_t len)¶
- Return value: New reference. Part of the Stable ABI.
Return a new tuple object of size len, or
NULL
with an exception set on failure.
-
PyObject *PyTuple_Pack(Py_ssize_t n, ...)¶
- Return value: New reference. Part of the Stable ABI.
Return a new tuple object of size n, or
NULL
with an exception set on failure. The tuple values are initialized to the subsequent n C arguments pointing to Python objects.PyTuple_Pack(2, a, b)
is equivalent toPy_BuildValue("(OO)", a, b)
.
-
Py_ssize_t PyTuple_Size(PyObject *p)¶
- Part of the Stable ABI.
Take a pointer to a tuple object, and return the size of that tuple. On error, return
-1
with an exception set.
-
Py_ssize_t PyTuple_GET_SIZE(PyObject *p)¶
Like
PyTuple_Size()
, but without error checking.
-
PyObject *PyTuple_GetItem(PyObject *p, Py_ssize_t pos)¶
- Return value: Borrowed reference. Part of the Stable ABI.
Return the object at position pos in the tuple pointed to by p. If pos is negative or out of bounds, return
NULL
and set anIndexError
exception.The returned reference is borrowed from the tuple p (that is: it is only valid as long as you hold a reference to p). To get a strong reference, use
Py_NewRef(PyTuple_GetItem(...))
orPySequence_GetItem()
.
-
PyObject *PyTuple_GET_ITEM(PyObject *p, Py_ssize_t pos)¶
- Return value: Borrowed reference.
Like
PyTuple_GetItem()
, but does no checking of its arguments.
-
PyObject *PyTuple_GetSlice(PyObject *p, Py_ssize_t low, Py_ssize_t high)¶
- Return value: New reference. Part of the Stable ABI.
Return the slice of the tuple pointed to by p between low and high, or
NULL
with an exception set on failure.This is the equivalent of the Python expression
p[low:high]
. Indexing from the end of the tuple is not supported.
-
int PyTuple_SetItem(PyObject *p, Py_ssize_t pos, PyObject *o)¶
- Part of the Stable ABI.
Insert a reference to object o at position pos of the tuple pointed to by p. Return
0
on success. If pos is out of bounds, return-1
and set anIndexError
exception.Note
This function “steals” a reference to o and discards a reference to an item already in the tuple at the affected position.
-
void PyTuple_SET_ITEM(PyObject *p, Py_ssize_t pos, PyObject *o)¶
Like
PyTuple_SetItem()
, but does no error checking, and should only be used to fill in brand new tuples.Bounds checking is performed as an assertion if Python is built in debug mode or
with assertions
.Note
This function “steals” a reference to o, and, unlike
PyTuple_SetItem()
, does not discard a reference to any item that is being replaced; any reference in the tuple at position pos will be leaked.Warning
This macro should only be used on tuples that are newly created. Using this macro on a tuple that is already in use (or in other words, has a refcount > 1) could lead to undefined behavior.
-
int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize)¶
Can be used to resize a tuple. newsize will be the new length of the tuple. Because tuples are supposed to be immutable, this should only be used if there is only one reference to the object. Do not use this if the tuple may already be known to some other part of the code. The tuple will always grow or shrink at the end. Think of this as destroying the old tuple and creating a new one, only more efficiently. Returns
0
on success. Client code should never assume that the resulting value of*p
will be the same as before calling this function. If the object referenced by*p
is replaced, the original*p
is destroyed. On failure, returns-1
and sets*p
toNULL
, and raisesMemoryError
orSystemError
.
Struct Sequence Objects¶
Struct sequence objects are the C equivalent of namedtuple()
objects, i.e. a sequence whose items can also be accessed through attributes.
To create a struct sequence, you first have to create a specific struct sequence
type.
-
PyTypeObject *PyStructSequence_NewType(PyStructSequence_Desc *desc)¶
- Return value: New reference. Part of the Stable ABI.
Create a new struct sequence type from the data in desc, described below. Instances of the resulting type can be created with
PyStructSequence_New()
.Return
NULL
with an exception set on failure.
-
void PyStructSequence_InitType(PyTypeObject *type, PyStructSequence_Desc *desc)¶
Initializes a struct sequence type type from desc in place.
-
int PyStructSequence_InitType2(PyTypeObject *type, PyStructSequence_Desc *desc)¶
Like
PyStructSequence_InitType()
, but returns0
on success and-1
with an exception set on failure.Added in version 3.4.
-
type PyStructSequence_Desc¶
- Part of the Stable ABI (including all members).
Contains the meta information of a struct sequence type to create.
-
const char *name¶
Fully qualified name of the type; null-terminated UTF-8 encoded. The name must contain the module name.
-
const char *doc¶
Pointer to docstring for the type or
NULL
to omit.
-
PyStructSequence_Field *fields¶
Pointer to
NULL
-terminated array with field names of the new type.
-
int n_in_sequence¶
Number of fields visible to the Python side (if used as tuple).
-
const char *name¶
-
type PyStructSequence_Field¶
- Part of the Stable ABI (including all members).
Describes a field of a struct sequence. As a struct sequence is modeled as a tuple, all fields are typed as PyObject*. The index in the
fields
array of thePyStructSequence_Desc
determines which field of the struct sequence is described.-
const char *name¶
Name for the field or
NULL
to end the list of named fields, set toPyStructSequence_UnnamedField
to leave unnamed.
-
const char *doc¶
Field docstring or
NULL
to omit.
-
const char *name¶
-
const char *const PyStructSequence_UnnamedField¶
- Part of the Stable ABI since version 3.11.
Special value for a field name to leave it unnamed.
Changed in version 3.9: The type was changed from
char *
.
-
PyObject *PyStructSequence_New(PyTypeObject *type)¶
- Return value: New reference. Part of the Stable ABI.
Creates an instance of type, which must have been created with
PyStructSequence_NewType()
.Return
NULL
with an exception set on failure.
-
PyObject *PyStructSequence_GetItem(PyObject *p, Py_ssize_t pos)¶
- Return value: Borrowed reference. Part of the Stable ABI.
Return the object at position pos in the struct sequence pointed to by p.
Bounds checking is performed as an assertion if Python is built in debug mode or
with assertions
.
-
PyObject *PyStructSequence_GET_ITEM(PyObject *p, Py_ssize_t pos)¶
- Return value: Borrowed reference.
Alias to
PyStructSequence_GetItem()
.Changed in version 3.13: Now implemented as an alias to
PyStructSequence_GetItem()
.
-
void PyStructSequence_SetItem(PyObject *p, Py_ssize_t pos, PyObject *o)¶
- Part of the Stable ABI.
Sets the field at index pos of the struct sequence p to value o. Like
PyTuple_SET_ITEM()
, this should only be used to fill in brand new instances.Bounds checking is performed as an assertion if Python is built in debug mode or
with assertions
.Note
This function “steals” a reference to o.
-
void PyStructSequence_SET_ITEM(PyObject *p, Py_ssize_t *pos, PyObject *o)¶
Alias to
PyStructSequence_SetItem()
.Changed in version 3.13: Now implemented as an alias to
PyStructSequence_SetItem()
.


Argparse Tutorial

Argparse Tutorial¶
- author:
Tshepang Mbambo
This tutorial is intended to be a gentle introduction to argparse
, the
recommended command-line parsing module in the Python standard library.
Note
The standard library includes two other libraries directly related
to command-line parameter processing: the lower level optparse
module (which may require more code to configure for a given application,
but also allows an application to request behaviors that argparse
doesn’t support), and the very low level getopt
(which specifically
serves as an equivalent to the getopt()
family of functions
available to C programmers).
While neither of those modules is covered directly in this guide, many of
the core concepts in argparse
first originated in optparse
, so
some aspects of this tutorial will also be relevant to optparse
users.
Concepts¶
Let’s show the sort of functionality that we are going to explore in this introductory tutorial by making use of the ls command:
$ ls
cpython devguide prog.py pypy rm-unused-function.patch
$ ls pypy
ctypes_configure demo dotviewer include lib_pypy lib-python ...
$ ls -l
total 20
drwxr-xr-x 19 wena wena 4096 Feb 18 18:51 cpython
drwxr-xr-x 4 wena wena 4096 Feb 8 12:04 devguide
-rwxr-xr-x 1 wena wena 535 Feb 19 00:05 prog.py
drwxr-xr-x 14 wena wena 4096 Feb 7 00:59 pypy
-rw-r--r-- 1 wena wena 741 Feb 18 01:01 rm-unused-function.patch
$ ls --help
Usage: ls [OPTION]... [FILE]...
List information about the FILEs (the current directory by default).
Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.
...
A few concepts we can learn from the four commands:
The ls command is useful when run without any options at all. It defaults to displaying the contents of the current directory.
If we want beyond what it provides by default, we tell it a bit more. In this case, we want it to display a different directory,
pypy
. What we did is specify what is known as a positional argument. It’s named so because the program should know what to do with the value, solely based on where it appears on the command line. This concept is more relevant to a command like cp, whose most basic usage iscp SRC DEST
. The first position is what you want copied, and the second position is where you want it copied to.Now, say we want to change behaviour of the program. In our example, we display more info for each file instead of just showing the file names. The
-l
in that case is known as an optional argument.That’s a snippet of the help text. It’s very useful in that you can come across a program you have never used before, and can figure out how it works simply by reading its help text.
The basics¶
Let us start with a very simple example which does (almost) nothing:
import argparse
parser = argparse.ArgumentParser()
parser.parse_args()
Following is a result of running the code:
$ python prog.py
$ python prog.py --help
usage: prog.py [-h]
options:
-h, --help show this help message and exit
$ python prog.py --verbose
usage: prog.py [-h]
prog.py: error: unrecognized arguments: --verbose
$ python prog.py foo
usage: prog.py [-h]
prog.py: error: unrecognized arguments: foo
Here is what is happening:
Running the script without any options results in nothing displayed to stdout. Not so useful.
The second one starts to display the usefulness of the
argparse
module. We have done almost nothing, but already we get a nice help message.The
--help
option, which can also be shortened to-h
, is the only option we get for free (i.e. no need to specify it). Specifying anything else results in an error. But even then, we do get a useful usage message, also for free.
Introducing Positional arguments¶
An example:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("echo")
args = parser.parse_args()
print(args.echo)
And running the code:
$ python prog.py
usage: prog.py [-h] echo
prog.py: error: the following arguments are required: echo
$ python prog.py --help
usage: prog.py [-h] echo
positional arguments:
echo
options:
-h, --help show this help message and exit
$ python prog.py foo
foo
Here is what’s happening:
We’ve added the
add_argument()
method, which is what we use to specify which command-line options the program is willing to accept. In this case, I’ve named itecho
so that it’s in line with its function.Calling our program now requires us to specify an option.
The
parse_args()
method actually returns some data from the options specified, in this case,echo
.The variable is some form of ‘magic’ that
argparse
performs for free (i.e. no need to specify which variable that value is stored in). You will also notice that its name matches the string argument given to the method,echo
.
Note however that, although the help display looks nice and all, it currently
is not as helpful as it can be. For example we see that we got echo
as a
positional argument, but we don’t know what it does, other than by guessing or
by reading the source code. So, let’s make it a bit more useful:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("echo", help="echo the string you use here")
args = parser.parse_args()
print(args.echo)
And we get:
$ python prog.py -h
usage: prog.py [-h] echo
positional arguments:
echo echo the string you use here
options:
-h, --help show this help message and exit
Now, how about doing something even more useful:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", help="display a square of a given number")
args = parser.parse_args()
print(args.square**2)
Following is a result of running the code:
$ python prog.py 4
Traceback (most recent call last):
File "prog.py", line 5, in <module>
print(args.square**2)
TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int'
That didn’t go so well. That’s because argparse
treats the options we
give it as strings, unless we tell it otherwise. So, let’s tell
argparse
to treat that input as an integer:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", help="display a square of a given number",
type=int)
args = parser.parse_args()
print(args.square**2)
Following is a result of running the code:
$ python prog.py 4
16
$ python prog.py four
usage: prog.py [-h] square
prog.py: error: argument square: invalid int value: 'four'
That went well. The program now even helpfully quits on bad illegal input before proceeding.
Introducing Optional arguments¶
So far we have been playing with positional arguments. Let us have a look on how to add optional ones:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--verbosity", help="increase output verbosity")
args = parser.parse_args()
if args.verbosity:
print("verbosity turned on")
And the output:
$ python prog.py --verbosity 1
verbosity turned on
$ python prog.py
$ python prog.py --help
usage: prog.py [-h] [--verbosity VERBOSITY]
options:
-h, --help show this help message and exit
--verbosity VERBOSITY
increase output verbosity
$ python prog.py --verbosity
usage: prog.py [-h] [--verbosity VERBOSITY]
prog.py: error: argument --verbosity: expected one argument
Here is what is happening:
The program is written so as to display something when
--verbosity
is specified and display nothing when not.To show that the option is actually optional, there is no error when running the program without it. Note that by default, if an optional argument isn’t used, the relevant variable, in this case
args.verbosity
, is givenNone
as a value, which is the reason it fails the truth test of theif
statement.The help message is a bit different.
When using the
--verbosity
option, one must also specify some value, any value.
The above example accepts arbitrary integer values for --verbosity
, but for
our simple program, only two values are actually useful, True
or False
.
Let’s modify the code accordingly:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--verbose", help="increase output verbosity",
action="store_true")
args = parser.parse_args()
if args.verbose:
print("verbosity turned on")
And the output:
$ python prog.py --verbose
verbosity turned on
$ python prog.py --verbose 1
usage: prog.py [-h] [--verbose]
prog.py: error: unrecognized arguments: 1
$ python prog.py --help
usage: prog.py [-h] [--verbose]
options:
-h, --help show this help message and exit
--verbose increase output verbosity
Here is what is happening:
The option is now more of a flag than something that requires a value. We even changed the name of the option to match that idea. Note that we now specify a new keyword,
action
, and give it the value"store_true"
. This means that, if the option is specified, assign the valueTrue
toargs.verbose
. Not specifying it impliesFalse
.It complains when you specify a value, in true spirit of what flags actually are.
Notice the different help text.
Short options¶
If you are familiar with command line usage, you will notice that I haven’t yet touched on the topic of short versions of the options. It’s quite simple:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("-v", "--verbose", help="increase output verbosity",
action="store_true")
args = parser.parse_args()
if args.verbose:
print("verbosity turned on")
And here goes:
$ python prog.py -v
verbosity turned on
$ python prog.py --help
usage: prog.py [-h] [-v]
options:
-h, --help show this help message and exit
-v, --verbose increase output verbosity
Note that the new ability is also reflected in the help text.
Combining Positional and Optional arguments¶
Our program keeps growing in complexity:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
help="display a square of a given number")
parser.add_argument("-v", "--verbose", action="store_true",
help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbose:
print(f"the square of {args.square} equals {answer}")
else:
print(answer)
And now the output:
$ python prog.py
usage: prog.py [-h] [-v] square
prog.py: error: the following arguments are required: square
$ python prog.py 4
16
$ python prog.py 4 --verbose
the square of 4 equals 16
$ python prog.py --verbose 4
the square of 4 equals 16
We’ve brought back a positional argument, hence the complaint.
Note that the order does not matter.
How about we give this program of ours back the ability to have multiple verbosity values, and actually get to use them:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
help="display a square of a given number")
parser.add_argument("-v", "--verbosity", type=int,
help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbosity == 2:
print(f"the square of {args.square} equals {answer}")
elif args.verbosity == 1:
print(f"{args.square}^2 == {answer}")
else:
print(answer)
And the output:
$ python prog.py 4
16
$ python prog.py 4 -v
usage: prog.py [-h] [-v VERBOSITY] square
prog.py: error: argument -v/--verbosity: expected one argument
$ python prog.py 4 -v 1
4^2 == 16
$ python prog.py 4 -v 2
the square of 4 equals 16
$ python prog.py 4 -v 3
16
These all look good except the last one, which exposes a bug in our program.
Let’s fix it by restricting the values the --verbosity
option can accept:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
help="display a square of a given number")
parser.add_argument("-v", "--verbosity", type=int, choices=[0, 1, 2],
help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbosity == 2:
print(f"the square of {args.square} equals {answer}")
elif args.verbosity == 1:
print(f"{args.square}^2 == {answer}")
else:
print(answer)
And the output:
$ python prog.py 4 -v 3
usage: prog.py [-h] [-v {0,1,2}] square
prog.py: error: argument -v/--verbosity: invalid choice: 3 (choose from 0, 1, 2)
$ python prog.py 4 -h
usage: prog.py [-h] [-v {0,1,2}] square
positional arguments:
square display a square of a given number
options:
-h, --help show this help message and exit
-v, --verbosity {0,1,2}
increase output verbosity
Note that the change also reflects both in the error message as well as the help string.
Now, let’s use a different approach of playing with verbosity, which is pretty
common. It also matches the way the CPython executable handles its own
verbosity argument (check the output of python --help
):
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
help="display the square of a given number")
parser.add_argument("-v", "--verbosity", action="count",
help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbosity == 2:
print(f"the square of {args.square} equals {answer}")
elif args.verbosity == 1:
print(f"{args.square}^2 == {answer}")
else:
print(answer)
We have introduced another action, “count”, to count the number of occurrences of specific options.
$ python prog.py 4
16
$ python prog.py 4 -v
4^2 == 16
$ python prog.py 4 -vv
the square of 4 equals 16
$ python prog.py 4 --verbosity --verbosity
the square of 4 equals 16
$ python prog.py 4 -v 1
usage: prog.py [-h] [-v] square
prog.py: error: unrecognized arguments: 1
$ python prog.py 4 -h
usage: prog.py [-h] [-v] square
positional arguments:
square display a square of a given number
options:
-h, --help show this help message and exit
-v, --verbosity increase output verbosity
$ python prog.py 4 -vvv
16
Yes, it’s now more of a flag (similar to
action="store_true"
) in the previous version of our script. That should explain the complaint.It also behaves similar to “store_true” action.
Now here’s a demonstration of what the “count” action gives. You’ve probably seen this sort of usage before.
And if you don’t specify the
-v
flag, that flag is considered to haveNone
value.As should be expected, specifying the long form of the flag, we should get the same output.
Sadly, our help output isn’t very informative on the new ability our script has acquired, but that can always be fixed by improving the documentation for our script (e.g. via the
help
keyword argument).That last output exposes a bug in our program.
Let’s fix:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
help="display a square of a given number")
parser.add_argument("-v", "--verbosity", action="count",
help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
# bugfix: replace == with >=
if args.verbosity >= 2:
print(f"the square of {args.square} equals {answer}")
elif args.verbosity >= 1:
print(f"{args.square}^2 == {answer}")
else:
print(answer)
And this is what it gives:
$ python prog.py 4 -vvv
the square of 4 equals 16
$ python prog.py 4 -vvvv
the square of 4 equals 16
$ python prog.py 4
Traceback (most recent call last):
File "prog.py", line 11, in <module>
if args.verbosity >= 2:
TypeError: '>=' not supported between instances of 'NoneType' and 'int'
First output went well, and fixes the bug we had before. That is, we want any value >= 2 to be as verbose as possible.
Third output not so good.
Let’s fix that bug:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("square", type=int,
help="display a square of a given number")
parser.add_argument("-v", "--verbosity", action="count", default=0,
help="increase output verbosity")
args = parser.parse_args()
answer = args.square**2
if args.verbosity >= 2:
print(f"the square of {args.square} equals {answer}")
elif args.verbosity >= 1:
print(f"{args.square}^2 == {answer}")
else:
print(answer)
We’ve just introduced yet another keyword, default
.
We’ve set it to 0
in order to make it comparable to the other int values.
Remember that by default,
if an optional argument isn’t specified,
it gets the None
value, and that cannot be compared to an int value
(hence the TypeError
exception).
And:
$ python prog.py 4
16
You can go quite far just with what we’ve learned so far,
and we have only scratched the surface.
The argparse
module is very powerful,
and we’ll explore a bit more of it before we end this tutorial.
Getting a little more advanced¶
What if we wanted to expand our tiny program to perform other powers, not just squares:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("x", type=int, help="the base")
parser.add_argument("y", type=int, help="the exponent")
parser.add_argument("-v", "--verbosity", action="count", default=0)
args = parser.parse_args()
answer = args.x**args.y
if args.verbosity >= 2:
print(f"{args.x} to the power {args.y} equals {answer}")
elif args.verbosity >= 1:
print(f"{args.x}^{args.y} == {answer}")
else:
print(answer)
Output:
$ python prog.py
usage: prog.py [-h] [-v] x y
prog.py: error: the following arguments are required: x, y
$ python prog.py -h
usage: prog.py [-h] [-v] x y
positional arguments:
x the base
y the exponent
options:
-h, --help show this help message and exit
-v, --verbosity
$ python prog.py 4 2 -v
4^2 == 16
Notice that so far we’ve been using verbosity level to change the text that gets displayed. The following example instead uses verbosity level to display more text instead:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("x", type=int, help="the base")
parser.add_argument("y", type=int, help="the exponent")
parser.add_argument("-v", "--verbosity", action="count", default=0)
args = parser.parse_args()
answer = args.x**args.y
if args.verbosity >= 2:
print(f"Running '{__file__}'")
if args.verbosity >= 1:
print(f"{args.x}^{args.y} == ", end="")
print(answer)
Output:
$ python prog.py 4 2
16
$ python prog.py 4 2 -v
4^2 == 16
$ python prog.py 4 2 -vv
Running 'prog.py'
4^2 == 16
Specifying ambiguous arguments¶
When there is ambiguity in deciding whether an argument is positional or for an
argument, --
can be used to tell parse_args()
that
everything after that is a positional argument:
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('-n', nargs='+')
>>> parser.add_argument('args', nargs='*')
>>> # ambiguous, so parse_args assumes it's an option
>>> parser.parse_args(['-f'])
usage: PROG [-h] [-n N [N ...]] [args ...]
PROG: error: unrecognized arguments: -f
>>> parser.parse_args(['--', '-f'])
Namespace(args=['-f'], n=None)
>>> # ambiguous, so the -n option greedily accepts arguments
>>> parser.parse_args(['-n', '1', '2', '3'])
Namespace(args=[], n=['1', '2', '3'])
>>> parser.parse_args(['-n', '1', '--', '2', '3'])
Namespace(args=['2', '3'], n=['1'])
Conflicting options¶
So far, we have been working with two methods of an
argparse.ArgumentParser
instance. Let’s introduce a third one,
add_mutually_exclusive_group()
. It allows for us to specify options that
conflict with each other. Let’s also change the rest of the program so that
the new functionality makes more sense:
we’ll introduce the --quiet
option,
which will be the opposite of the --verbose
one:
import argparse
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-v", "--verbose", action="store_true")
group.add_argument("-q", "--quiet", action="store_true")
parser.add_argument("x", type=int, help="the base")
parser.add_argument("y", type=int, help="the exponent")
args = parser.parse_args()
answer = args.x**args.y
if args.quiet:
print(answer)
elif args.verbose:
print(f"{args.x} to the power {args.y} equals {answer}")
else:
print(f"{args.x}^{args.y} == {answer}")
Our program is now simpler, and we’ve lost some functionality for the sake of demonstration. Anyways, here’s the output:
$ python prog.py 4 2
4^2 == 16
$ python prog.py 4 2 -q
16
$ python prog.py 4 2 -v
4 to the power 2 equals 16
$ python prog.py 4 2 -vq
usage: prog.py [-h] [-v | -q] x y
prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose
$ python prog.py 4 2 -v --quiet
usage: prog.py [-h] [-v | -q] x y
prog.py: error: argument -q/--quiet: not allowed with argument -v/--verbose
That should be easy to follow. I’ve added that last output so you can see the sort of flexibility you get, i.e. mixing long form options with short form ones.
Before we conclude, you probably want to tell your users the main purpose of your program, just in case they don’t know:
import argparse
parser = argparse.ArgumentParser(description="calculate X to the power of Y")
group = parser.add_mutually_exclusive_group()
group.add_argument("-v", "--verbose", action="store_true")
group.add_argument("-q", "--quiet", action="store_true")
parser.add_argument("x", type=int, help="the base")
parser.add_argument("y", type=int, help="the exponent")
args = parser.parse_args()
answer = args.x**args.y
if args.quiet:
print(answer)
elif args.verbose:
print(f"{args.x} to the power {args.y} equals {answer}")
else:
print(f"{args.x}^{args.y} == {answer}")
Note that slight difference in the usage text. Note the [-v | -q]
,
which tells us that we can either use -v
or -q
,
but not both at the same time:
$ python prog.py --help
usage: prog.py [-h] [-v | -q] x y
calculate X to the power of Y
positional arguments:
x the base
y the exponent
options:
-h, --help show this help message and exit
-v, --verbose
-q, --quiet
How to translate the argparse output¶
The output of the argparse
module such as its help text and error
messages are all made translatable using the gettext
module. This
allows applications to easily localize messages produced by
argparse
. See also Internationalizing your programs and modules.
For instance, in this argparse
output:
$ python prog.py --help
usage: prog.py [-h] [-v | -q] x y
calculate X to the power of Y
positional arguments:
x the base
y the exponent
options:
-h, --help show this help message and exit
-v, --verbose
-q, --quiet
The strings usage:
, positional arguments:
, options:
and
show this help message and exit
are all translatable.
In order to translate these strings, they must first be extracted
into a .po
file. For example, using Babel,
run this command:
$ pybabel extract -o messages.po /usr/lib/python3.12/argparse.py
This command will extract all translatable strings from the argparse
module and output them into a file named messages.po
. This command assumes
that your Python installation is in /usr/lib
.
You can find out the location of the argparse
module on your system
using this script:
import argparse
print(argparse.__file__)
Once the messages in the .po
file are translated and the translations are
installed using gettext
, argparse
will be able to display the
translated messages.
To translate your own strings in the argparse
output, use gettext
.
Custom type converters¶
The argparse
module allows you to specify custom type converters for
your command-line arguments. This allows you to modify user input before it’s
stored in the argparse.Namespace
. This can be useful when you need to
pre-process the input before it is used in your program.
When using a custom type converter, you can use any callable that takes a single string argument (the argument value) and returns the converted value. However, if you need to handle more complex scenarios, you can use a custom action class with the action parameter instead.
For example, let’s say you want to handle arguments with different prefixes and process them accordingly:
import argparse
parser = argparse.ArgumentParser(prefix_chars='-+')
parser.add_argument('-a', metavar='<value>', action='append',
type=lambda x: ('-', x))
parser.add_argument('+a', metavar='<value>', action='append',
type=lambda x: ('+', x))
args = parser.parse_args()
print(args)
Output:
$ python prog.py -a value1 +a value2
Namespace(a=[('-', 'value1'), ('+', 'value2')])
In this example, we:
Created a parser with custom prefix characters using the
prefix_chars
parameter.Defined two arguments,
-a
and+a
, which used thetype
parameter to create custom type converters to store the value in a tuple with the prefix.
Without the custom type converters, the arguments would have treated the -a
and +a
as the same argument, which would have been undesirable. By using custom
type converters, we were able to differentiate between the two arguments.
Conclusion¶
The argparse
module offers a lot more than shown here.
Its docs are quite detailed and thorough, and full of examples.
Having gone through this tutorial, you should easily digest them
without feeling overwhelmed.