python_doc_md/Beginner/whatsnew/3.2.md

What's New In Python 3.2

Author : Raymond Hettinger

This article explains the new features in Python 3.2 as compared to 3.1. Python 3.2 was released on February 20, 2011. It focuses on a few highlights and gives a few examples. For full details, see the Misc/NEWS file.

::: seealso 392{.interpreted-text role="pep"} - Python 3.2 Release Schedule :::

PEP 384: Defining a Stable ABI

In the past, extension modules built for one Python version were often not usable with other Python versions. Particularly on Windows, every feature release of Python required rebuilding all extension modules that one wanted to use. This requirement was the result of the free access to Python interpreter internals that extension modules could use.

With Python 3.2, an alternative approach becomes available: extension modules which restrict themselves to a limited API (by defining Py_LIMITED_API) cannot use many of the internals, but are constrained to a set of API functions that are promised to be stable for several releases. As a consequence, extension modules built for 3.2 in that mode will also work with 3.3, 3.4, and so on. Extension modules that make use of details of memory structures can still be built, but will need to be recompiled for every feature release.

::: seealso

384{.interpreted-text role="pep"} - Defining a Stable ABI

: PEP written by Martin von Löwis. :::

PEP 389: Argparse Command Line Parsing Module

A new module for command line parsing, argparse{.interpreted-text role="mod"}, was introduced to overcome the limitations of optparse{.interpreted-text role="mod"} which did not provide support for positional arguments (not just options), subcommands, required options and other common patterns of specifying and validating options.

This module has already had widespread success in the community as a third-party module. Being more fully featured than its predecessor, the argparse{.interpreted-text role="mod"} module is now the preferred module for command-line processing. The older module is still being kept available because of the substantial amount of legacy code that depends on it.

Here's an annotated example parser showing features like limiting results to a set of choices, specifying a metavar in the help screen, validating that one or more positional arguments is present, and making a required option:

import argparse
parser = argparse.ArgumentParser(
            description = 'Manage servers',         # main description for help
            epilog = 'Tested on Solaris and Linux') # displayed after help
parser.add_argument('action',                       # argument name
            choices = ['deploy', 'start', 'stop'],  # three allowed values
            help = 'action on each target')         # help msg
parser.add_argument('targets',
            metavar = 'HOSTNAME',                   # var name used in help msg
            nargs = '+',                            # require one or more targets
            help = 'url for target machines')       # help msg explanation
parser.add_argument('-u', '--user',                 # -u or --user option
            required = True,                        # make it a required argument
            help = 'login as user')

Example of calling the parser on a command string:

>>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain'
>>> result = parser.parse_args(cmd.split())
>>> result.action
'deploy'
>>> result.targets
['sneezy.example.com', 'sleepy.example.com']
>>> result.user
'skycaptain'

Example of the parser's automatically generated help:

>>> parser.parse_args('-h'.split())

usage: manage_cloud.py [-h] -u USER
                       {deploy,start,stop} HOSTNAME [HOSTNAME ...]

Manage servers

positional arguments:
  {deploy,start,stop}   action on each target
  HOSTNAME              url for target machines

optional arguments:
  -h, --help            show this help message and exit
  -u USER, --user USER  login as user

Tested on Solaris and Linux

An especially nice argparse{.interpreted-text role="mod"} feature is the ability to define subparsers, each with their own argument patterns and help displays:

import argparse
parser = argparse.ArgumentParser(prog='HELM')
subparsers = parser.add_subparsers()

parser_l = subparsers.add_parser('launch', help='Launch Control')   # first subgroup
parser_l.add_argument('-m', '--missiles', action='store_true')
parser_l.add_argument('-t', '--torpedos', action='store_true')

parser_m = subparsers.add_parser('move', help='Move Vessel',        # second subgroup
                                 aliases=('steer', 'turn'))         # equivalent names
parser_m.add_argument('-c', '--course', type=int, required=True)
parser_m.add_argument('-s', '--speed', type=int, default=0)

$ ./helm.py --help                         # top level help (launch and move)
$ ./helm.py launch --help                  # help for launch options
$ ./helm.py launch --missiles              # set missiles=True and torpedos=False
$ ./helm.py steer --course 180 --speed 5   # set movement parameters

::: seealso

389{.interpreted-text role="pep"} - New Command Line Parsing Module

: PEP written by Steven Bethard.

upgrading-optparse-code{.interpreted-text role="ref"} for details on the differences from optparse{.interpreted-text role="mod"}. :::

PEP 391: Dictionary Based Configuration for Logging

The logging{.interpreted-text role="mod"} module provided two kinds of configuration, one style with function calls for each option or another style driven by an external file saved in a configparser{.interpreted-text role="mod"} format. Those options did not provide the flexibility to create configurations from JSON or YAML files, nor did they support incremental configuration, which is needed for specifying logger options from a command line.

To support a more flexible style, the module now offers logging.config.dictConfig{.interpreted-text role="func"} for specifying logging configuration with plain Python dictionaries. The configuration options include formatters, handlers, filters, and loggers. Here's a working example of a configuration dictionary:

{"version": 1,
 "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"},
                "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"}
                },
 "handlers": {"console": {
                   "class": "logging.StreamHandler",
                   "formatter": "brief",
                   "level": "INFO",
                   "stream": "ext://sys.stdout"},
              "console_priority": {
                   "class": "logging.StreamHandler",
                   "formatter": "full",
                   "level": "ERROR",
                   "stream": "ext://sys.stderr"}
              },
 "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}}

If that dictionary is stored in a file called conf.json{.interpreted-text role="file"}, it can be loaded and called with code like this:

>>> import json, logging.config
>>> with open('conf.json') as f:
...     conf = json.load(f)
...
>>> logging.config.dictConfig(conf)
>>> logging.info("Transaction completed normally")
INFO    : root           : Transaction completed normally
>>> logging.critical("Abnormal termination")
2011-02-17 11:14:36,694 root            CRITICAL Abnormal termination

::: seealso

391{.interpreted-text role="pep"} - Dictionary Based Configuration for Logging

: PEP written by Vinay Sajip. :::

PEP 3148: The concurrent.futures module

Code for creating and managing concurrency is being collected in a new top-level namespace, concurrent. Its first member is a futures package which provides a uniform high-level interface for managing threads and processes.

The design for concurrent.futures{.interpreted-text role="mod"} was inspired by the java.util.concurrent package. In that model, a running call and its result are represented by a ~concurrent.futures.Future{.interpreted-text role="class"} object that abstracts features common to threads, processes, and remote procedure calls. That object supports status checks (running or done), timeouts, cancellations, adding callbacks, and access to results or exceptions.

The primary offering of the new module is a pair of executor classes for launching and managing calls. The goal of the executors is to make it easier to use existing tools for making parallel calls. They save the effort needed to setup a pool of resources, launch the calls, create a results queue, add time-out handling, and limit the total number of threads, processes, or remote procedure calls.

Ideally, each application should share a single executor across multiple components so that process and thread limits can be centrally managed. This solves the design challenge that arises when each component has its own competing strategy for resource management.

Both classes share a common interface with three methods: ~concurrent.futures.Executor.submit{.interpreted-text role="meth"} for scheduling a callable and returning a ~concurrent.futures.Future{.interpreted-text role="class"} object; ~concurrent.futures.Executor.map{.interpreted-text role="meth"} for scheduling many asynchronous calls at a time, and ~concurrent.futures.Executor.shutdown{.interpreted-text role="meth"} for freeing resources. The class is a context manager{.interpreted-text role="term"} and can be used in a with{.interpreted-text role="keyword"} statement to assure that resources are automatically released when currently pending futures are done executing.

A simple of example of ~concurrent.futures.ThreadPoolExecutor{.interpreted-text role="class"} is a launch of four parallel threads for copying files:

import concurrent.futures, shutil
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e:
    e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
    e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest4.txt')

::: seealso

3148{.interpreted-text role="pep"} - Futures -- Execute Computations Asynchronously

: PEP written by Brian Quinlan.

Code for Threaded Parallel URL reads<threadpoolexecutor-example>{.interpreted-text role="ref"}, an example using threads to fetch multiple web pages in parallel.

Code for computing prime numbers in parallel<processpoolexecutor-example>{.interpreted-text role="ref"}, an example demonstrating ~concurrent.futures.ProcessPoolExecutor{.interpreted-text role="class"}. :::

PEP 3147: PYC Repository Directories

Python's scheme for caching bytecode in .pyc files did not work well in environments with multiple Python interpreters. If one interpreter encountered a cached file created by another interpreter, it would recompile the source and overwrite the cached file, thus losing the benefits of caching.

The issue of "pyc fights" has become more pronounced as it has become commonplace for Linux distributions to ship with multiple versions of Python. These conflicts also arise with CPython alternatives such as Unladen Swallow.

To solve this problem, Python's import machinery has been extended to use distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and Unladen Swallow each competing for a file called "mymodule.pyc", they will now look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and "mymodule.unladen10.pyc". And to prevent all of these new files from cluttering source directories, the pyc files are now collected in a "__pycache__" directory stored under the package directory.

Aside from the filenames and target directories, the new scheme has a few aspects that are visible to the programmer:

• Imported modules now have a __cached__ attribute which stores the name of the actual file that was imported:

  >>> import collections >>> collections.__cached__ # doctest: +SKIP 'c:/py32/lib/__pycache__/collections.cpython-32.pyc'

• The tag that is unique to each interpreter is accessible from the !imp{.interpreted-text role="mod"} module:

  >>> import imp # doctest: +SKIP >>> imp.get_tag() # doctest: +SKIP 'cpython-32'

• Scripts that try to deduce source filename from the imported file now need to be smarter. It is no longer sufficient to simply strip the "c" from a ".pyc" filename. Instead, use the new functions in the !imp{.interpreted-text role="mod"} module:

  >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc') # doctest: +SKIP 'c:/py32/lib/collections.py' >>> imp.cache_from_source('c:/py32/lib/collections.py') # doctest: +SKIP 'c:/py32/lib/__pycache__/collections.cpython-32.pyc'

• The py_compile{.interpreted-text role="mod"} and compileall{.interpreted-text role="mod"} modules have been updated to reflect the new naming convention and target directory. The command-line invocation of compileall has new options: -i for specifying a list of files and directories to compile and -b which causes bytecode files to be written to their legacy location rather than __pycache__.

• The importlib.abc{.interpreted-text role="mod"} module has been updated with new abstract base classes <abstract base class>{.interpreted-text role="term"} for loading bytecode files. The obsolete ABCs, !PyLoader{.interpreted-text role="class"} and !PyPycLoader{.interpreted-text role="class"}, have been deprecated (instructions on how to stay Python 3.1 compatible are included with the documentation).

::: seealso

3147{.interpreted-text role="pep"} - PYC Repository Directories

: PEP written by Barry Warsaw. :::

PEP 3149: ABI Version Tagged .so Files

The PYC repository directory allows multiple bytecode cache files to be co-located. This PEP implements a similar mechanism for shared object files by giving them a common directory and distinct names for each version.

The common directory is "pyshared" and the file names are made distinct by identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the major and minor version numbers, and optional build flags (such as "d" for debug, "m" for pymalloc, "u" for wide-unicode). For an arbitrary package "foo", you may see these files when the distribution package is installed:

/usr/share/pyshared/foo.cpython-32m.so
/usr/share/pyshared/foo.cpython-33md.so

In Python itself, the tags are accessible from functions in the sysconfig{.interpreted-text role="mod"} module:

>>> import sysconfig
>>> sysconfig.get_config_var('SOABI')       # find the version tag
'cpython-32mu'
>>> sysconfig.get_config_var('EXT_SUFFIX')  # find the full filename extension
'.cpython-32mu.so'

::: seealso

3149{.interpreted-text role="pep"} - ABI Version Tagged .so Files

: PEP written by Barry Warsaw. :::

PEP 3333: Python Web Server Gateway Interface v1.0.1

This informational PEP clarifies how bytes/text issues are to be handled by the WSGI protocol. The challenge is that string handling in Python 3 is most conveniently handled with the str{.interpreted-text role="class"} type even though the HTTP protocol is itself bytes oriented.

The PEP differentiates so-called native strings that are used for request/response headers and metadata versus byte strings which are used for the bodies of requests and responses.

The native strings are always of type str{.interpreted-text role="class"} but are restricted to code points between U+0000 through U+00FF which are translatable to bytes using Latin-1 encoding. These strings are used for the keys and values in the environment dictionary and for response headers and statuses in the !start_response{.interpreted-text role="func"} function. They must follow 2616{.interpreted-text role="rfc"} with respect to encoding. That is, they must either be ISO-8859-1 characters or use 2047{.interpreted-text role="rfc"} MIME encoding.

For developers porting WSGI applications from Python 2, here are the salient points:

• If the app already used strings for headers in Python 2, no change is needed.
• If instead, the app encoded output headers or decoded input headers, then the headers will need to be re-encoded to Latin-1. For example, an output header encoded in utf-8 was using h.encode('utf-8') now needs to convert from bytes to native strings using h.encode('utf-8').decode('latin-1').
• Values yielded by an application or sent using the !write{.interpreted-text role="meth"} method must be byte strings. The !start_response{.interpreted-text role="func"} function and environ must use native strings. The two cannot be mixed.

For server implementers writing CGI-to-WSGI pathways or other CGI-style protocols, the users must to be able access the environment using native strings even though the underlying platform may have a different convention. To bridge this gap, the wsgiref{.interpreted-text role="mod"} module has a new function, wsgiref.handlers.read_environ{.interpreted-text role="func"} for transcoding CGI variables from os.environ{.interpreted-text role="data"} into native strings and returning a new dictionary.

::: seealso

3333{.interpreted-text role="pep"} - Python Web Server Gateway Interface v1.0.1

: PEP written by Phillip Eby. :::

Other Language Changes

Some smaller changes made to the core Python language are:

• String formatting for format{.interpreted-text role="func"} and str.format{.interpreted-text role="meth"} gained new capabilities for the format character #. Previously, for integers in binary, octal, or hexadecimal, it caused the output to be prefixed with '0b', '0o', or '0x' respectively. Now it can also handle floats, complex, and Decimal, causing the output to always have a decimal point even when no digits follow it.

  >>> format(20, '#o') '0o24' >>> format(12.34, '#5.0f') ' 12.'

  (Suggested by Mark Dickinson and implemented by Eric Smith in 7094{.interpreted-text role="issue"}.)

• There is also a new str.format_map{.interpreted-text role="meth"} method that extends the capabilities of the existing str.format{.interpreted-text role="meth"} method by accepting arbitrary mapping{.interpreted-text role="term"} objects. This new method makes it possible to use string formatting with any of Python's many dictionary-like objects such as ~collections.defaultdict{.interpreted-text role="class"}, ~shelve.Shelf{.interpreted-text role="class"}, ~configparser.ConfigParser{.interpreted-text role="class"}, or dbm{.interpreted-text role="mod"}. It is also useful with custom dict{.interpreted-text role="class"} subclasses that normalize keys before look-up or that supply a ~object.__missing__{.interpreted-text role="meth"} method for unknown keys:

  >>> import shelve
  >>> d = shelve.open('tmp.shl')
  >>> 'The {project_name} status is {status} as of {date}'.format_map(d)
  'The testing project status is green as of February 15, 2011'
  
  >>> class LowerCasedDict(dict):
  ...     def __getitem__(self, key):
  ...         return dict.__getitem__(self, key.lower())
  ...
  >>> lcd = LowerCasedDict(part='widgets', quantity=10)
  >>> 'There are {QUANTITY} {Part} in stock'.format_map(lcd)
  'There are 10 widgets in stock'
  
  >>> class PlaceholderDict(dict):
  ...     def __missing__(self, key):
  ...         return '<{}>'.format(key)
  ...
  >>> 'Hello {name}, welcome to {location}'.format_map(PlaceholderDict())
  'Hello <name>, welcome to <location>'
  

(Suggested by Raymond Hettinger and implemented by Eric Smith in 6081{.interpreted-text role="issue"}.)

• The interpreter can now be started with a quiet option, -q, to prevent the copyright and version information from being displayed in the interactive mode. The option can be introspected using the sys.flags{.interpreted-text role="data"} attribute:

  $ python -q
  >>> sys.flags
  sys.flags(debug=0, division_warning=0, inspect=0, interactive=0,
  optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0,
  ignore_environment=0, verbose=0, bytes_warning=0, quiet=1)
  

  (Contributed by Marcin Wojdyr in 1772833{.interpreted-text role="issue"}).

• The hasattr{.interpreted-text role="func"} function works by calling getattr{.interpreted-text role="func"} and detecting whether an exception is raised. This technique allows it to detect methods created dynamically by ~object.__getattr__{.interpreted-text role="meth"} or ~object.__getattribute__{.interpreted-text role="meth"} which would otherwise be absent from the class dictionary. Formerly, hasattr would catch any exception, possibly masking genuine errors. Now, hasattr has been tightened to only catch AttributeError{.interpreted-text role="exc"} and let other exceptions pass through:

  >>> class A:
  ...     @property
  ...     def f(self):
  ...         return 1 // 0
  ...
  >>> a = A()
  >>> hasattr(a, 'f')
  Traceback (most recent call last):
    ...
  ZeroDivisionError: integer division or modulo by zero
  

  (Discovered by Yury Selivanov and fixed by Benjamin Peterson; 9666{.interpreted-text role="issue"}.)

• The str{.interpreted-text role="func"} of a float or complex number is now the same as its repr{.interpreted-text role="func"}. Previously, the str{.interpreted-text role="func"} form was shorter but that just caused confusion and is no longer needed now that the shortest possible repr{.interpreted-text role="func"} is displayed by default:

  >>> import math >>> repr(math.pi) '3.141592653589793' >>> str(math.pi) '3.141592653589793'

  (Proposed and implemented by Mark Dickinson; 9337{.interpreted-text role="issue"}.)

• memoryview{.interpreted-text role="class"} objects now have a ~memoryview.release{.interpreted-text role="meth"} method and they also now support the context management protocol. This allows timely release of any resources that were acquired when requesting a buffer from the original object.

  >>> with memoryview(b'abcdefgh') as v: ... print(v.tolist()) [97, 98, 99, 100, 101, 102, 103, 104]

  (Added by Antoine Pitrou; 9757{.interpreted-text role="issue"}.)

• Previously it was illegal to delete a name from the local namespace if it occurs as a free variable in a nested block:

  def outer(x):
      def inner():
          return x
      inner()
      del x
  

  This is now allowed. Remember that the target of an except{.interpreted-text role="keyword"} clause is cleared, so this code which used to work with Python 2.6, raised a SyntaxError{.interpreted-text role="exc"} with Python 3.1 and now works again:

  def f():
      def print_error():
          print(e)
      try:
          something
      except Exception as e:
          print_error()
          # implicit "del e" here
  

  (See 4617{.interpreted-text role="issue"}.)

• Struct sequence types <struct-sequence-objects>{.interpreted-text role="ref"} are now subclasses of tuple. This means that C structures like those returned by os.stat{.interpreted-text role="func"}, time.gmtime{.interpreted-text role="func"}, and sys.version_info{.interpreted-text role="data"} now work like a named tuple{.interpreted-text role="term"} and now work with functions and methods that expect a tuple as an argument. This is a big step forward in making the C structures as flexible as their pure Python counterparts:

  >>> import sys >>> isinstance(sys.version_info, tuple) True >>> 'Version %d.%d.%d %s(%d)' % sys.version_info # doctest: +SKIP 'Version 3.2.0 final(0)'

  (Suggested by Arfrever Frehtes Taifersar Arahesis and implemented by Benjamin Peterson in 8413{.interpreted-text role="issue"}.)

• Warnings are now easier to control using the PYTHONWARNINGS{.interpreted-text role="envvar"} environment variable as an alternative to using -W at the command line:

  $ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::'
  

  (Suggested by Barry Warsaw and implemented by Philip Jenvey in 7301{.interpreted-text role="issue"}.)

• A new warning category, ResourceWarning{.interpreted-text role="exc"}, has been added. It is emitted when potential issues with resource consumption or cleanup are detected. It is silenced by default in normal release builds but can be enabled through the means provided by the warnings{.interpreted-text role="mod"} module, or on the command line.

  A ResourceWarning{.interpreted-text role="exc"} is issued at interpreter shutdown if the gc.garbage{.interpreted-text role="data"} list isn't empty, and if gc.DEBUG_UNCOLLECTABLE{.interpreted-text role="const"} is set, all uncollectable objects are printed. This is meant to make the programmer aware that their code contains object finalization issues.

  A ResourceWarning{.interpreted-text role="exc"} is also issued when a file object{.interpreted-text role="term"} is destroyed without having been explicitly closed. While the deallocator for such object ensures it closes the underlying operating system resource (usually, a file descriptor), the delay in deallocating the object could produce various issues, especially under Windows. Here is an example of enabling the warning from the command line:

  $ python -q -Wdefault
  >>> f = open("foo", "wb")
  >>> del f
  __main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'>
  

  (Added by Antoine Pitrou and Georg Brandl in 10093{.interpreted-text role="issue"} and 477863{.interpreted-text role="issue"}.)

• range{.interpreted-text role="class"} objects now support index and count methods. This is part of an effort to make more objects fully implement the collections.Sequence <collections.abc.Sequence>{.interpreted-text role="class"} abstract base class{.interpreted-text role="term"}. As a result, the language will have a more uniform API. In addition, range{.interpreted-text role="class"} objects now support slicing and negative indices, even with values larger than sys.maxsize{.interpreted-text role="data"}. This makes range more interoperable with lists:

  >>> range(0, 100, 2).count(10)
  1
  >>> range(0, 100, 2).index(10)
  5
  >>> range(0, 100, 2)[5]
  10
  >>> range(0, 100, 2)[0:5]
  range(0, 10, 2)
  

  (Contributed by Daniel Stutzbach in 9213{.interpreted-text role="issue"}, by Alexander Belopolsky in 2690{.interpreted-text role="issue"}, and by Nick Coghlan in 10889{.interpreted-text role="issue"}.)

• The callable{.interpreted-text role="func"} builtin function from Py2.x was resurrected. It provides a concise, readable alternative to using an abstract base class{.interpreted-text role="term"} in an expression like isinstance(x, collections.Callable):

  >>> callable(max) True >>> callable(20) False

  (See 10518{.interpreted-text role="issue"}.)

• Python's import mechanism can now load modules installed in directories with non-ASCII characters in the path name. This solved an aggravating problem with home directories for users with non-ASCII characters in their usernames.

(Required extensive work by Victor Stinner in 9425{.interpreted-text role="issue"}.)

New, Improved, and Deprecated Modules

Python's standard library has undergone significant maintenance efforts and quality improvements.

The biggest news for Python 3.2 is that the email{.interpreted-text role="mod"} package, mailbox{.interpreted-text role="mod"} module, and !nntplib{.interpreted-text role="mod"} modules now work correctly with the bytes/text model in Python 3. For the first time, there is correct handling of messages with mixed encodings.

Throughout the standard library, there has been more careful attention to encodings and text versus bytes issues. In particular, interactions with the operating system are now better able to exchange non-ASCII data using the Windows MBCS encoding, locale-aware encodings, or UTF-8.

Another significant win is the addition of substantially better support for SSL connections and security certificates.

In addition, more classes now implement a context manager{.interpreted-text role="term"} to support convenient and reliable resource clean-up using a with{.interpreted-text role="keyword"} statement.

email

The usability of the email{.interpreted-text role="mod"} package in Python 3 has been mostly fixed by the extensive efforts of R. David Murray. The problem was that emails are typically read and stored in the form of bytes{.interpreted-text role="class"} rather than str{.interpreted-text role="class"} text, and they may contain multiple encodings within a single email. So, the email package had to be extended to parse and generate email messages in bytes format.

• New functions ~email.message_from_bytes{.interpreted-text role="func"} and ~email.message_from_binary_file{.interpreted-text role="func"}, and new classes ~email.parser.BytesFeedParser{.interpreted-text role="class"} and ~email.parser.BytesParser{.interpreted-text role="class"} allow binary message data to be parsed into model objects.

• Given bytes input to the model, ~email.message.Message.get_payload{.interpreted-text role="meth"} will by default decode a message body that has a Content-Transfer-Encoding{.interpreted-text role="mailheader"} of 8bit using the charset specified in the MIME headers and return the resulting string.

• Given bytes input to the model, ~email.generator.Generator{.interpreted-text role="class"} will convert message bodies that have a Content-Transfer-Encoding{.interpreted-text role="mailheader"} of 8bit to instead have a 7bit Content-Transfer-Encoding{.interpreted-text role="mailheader"}.

  Headers with unencoded non-ASCII bytes are deemed to be 2047{.interpreted-text role="rfc"}-encoded using the unknown-8bit character set.

• A new class ~email.generator.BytesGenerator{.interpreted-text role="class"} produces bytes as output, preserving any unchanged non-ASCII data that was present in the input used to build the model, including message bodies with a Content-Transfer-Encoding{.interpreted-text role="mailheader"} of 8bit.

• The smtplib{.interpreted-text role="mod"} ~smtplib.SMTP{.interpreted-text role="class"} class now accepts a byte string for the msg argument to the ~smtplib.SMTP.sendmail{.interpreted-text role="meth"} method, and a new method, ~smtplib.SMTP.send_message{.interpreted-text role="meth"} accepts a ~email.message.Message{.interpreted-text role="class"} object and can optionally obtain the from_addr and to_addrs addresses directly from the object.

(Proposed and implemented by R. David Murray, 4661{.interpreted-text role="issue"} and 10321{.interpreted-text role="issue"}.)

elementtree

The xml.etree.ElementTree{.interpreted-text role="mod"} package and its !xml.etree.cElementTree{.interpreted-text role="mod"} counterpart have been updated to version 1.3.

Several new and useful functions and methods have been added:

• xml.etree.ElementTree.fromstringlist{.interpreted-text role="func"} which builds an XML document from a sequence of fragments
• xml.etree.ElementTree.register_namespace{.interpreted-text role="func"} for registering a global namespace prefix
• xml.etree.ElementTree.tostringlist{.interpreted-text role="func"} for string representation including all sublists
• xml.etree.ElementTree.Element.extend{.interpreted-text role="meth"} for appending a sequence of zero or more elements
• xml.etree.ElementTree.Element.iterfind{.interpreted-text role="meth"} searches an element and subelements
• xml.etree.ElementTree.Element.itertext{.interpreted-text role="meth"} creates a text iterator over an element and its subelements
• xml.etree.ElementTree.TreeBuilder.end{.interpreted-text role="meth"} closes the current element
• xml.etree.ElementTree.TreeBuilder.doctype{.interpreted-text role="meth"} handles a doctype declaration

Two methods have been deprecated:

• !xml.etree.ElementTree.getchildren{.interpreted-text role="meth"} use list(elem) instead.
• !xml.etree.ElementTree.getiterator{.interpreted-text role="meth"} use Element.iter instead.

For details of the update, see Introducing ElementTree on Fredrik Lundh's website.

(Contributed by Florent Xicluna and Fredrik Lundh, 6472{.interpreted-text role="issue"}.)

functools

• The functools{.interpreted-text role="mod"} module includes a new decorator for caching function calls. functools.lru_cache{.interpreted-text role="func"} can save repeated queries to an external resource whenever the results are expected to be the same.

  For example, adding a caching decorator to a database query function can save database accesses for popular searches:

  >>> import functools >>> @functools.lru_cache(maxsize=300) ... def get_phone_number(name): ... c = conn.cursor() ... c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,)) ... return c.fetchone()[0]

  >>> for name in user_requests: # doctest: +SKIP ... get_phone_number(name) # cached lookup

  To help with choosing an effective cache size, the wrapped function is instrumented for tracking cache statistics:

  >>> get_phone_number.cache_info() # doctest: +SKIP CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300)

  If the phonelist table gets updated, the outdated contents of the cache can be cleared with:

  >>> get_phone_number.cache_clear()

  (Contributed by Raymond Hettinger and incorporating design ideas from Jim Baker, Miki Tebeka, and Nick Coghlan; see recipe 498245, recipe 577479, 10586{.interpreted-text role="issue"}, and 10593{.interpreted-text role="issue"}.)

• The functools.wraps{.interpreted-text role="func"} decorator now adds a __wrapped__{.interpreted-text role="attr"} attribute pointing to the original callable function. This allows wrapped functions to be introspected. It also copies ~function.__annotations__{.interpreted-text role="attr"} if defined. And now it also gracefully skips over missing attributes such as ~function.__doc__{.interpreted-text role="attr"} which might not be defined for the wrapped callable.

  In the above example, the cache can be removed by recovering the original function:

  >>> get_phone_number = get_phone_number.__wrapped__ # uncached function

  (By Nick Coghlan and Terrence Cole; 9567{.interpreted-text role="issue"}, 3445{.interpreted-text role="issue"}, and 8814{.interpreted-text role="issue"}.)

• To help write classes with rich comparison methods, a new decorator functools.total_ordering{.interpreted-text role="func"} will use existing equality and inequality methods to fill in the remaining methods.

  For example, supplying __eq__ and __lt__ will enable ~functools.total_ordering{.interpreted-text role="func"} to fill-in __le__, __gt__ and __ge__:

  @total_ordering
  class Student:
      def __eq__(self, other):
          return ((self.lastname.lower(), self.firstname.lower()) ==
                  (other.lastname.lower(), other.firstname.lower()))
  
      def __lt__(self, other):
          return ((self.lastname.lower(), self.firstname.lower()) <
                  (other.lastname.lower(), other.firstname.lower()))
  

  With the total_ordering decorator, the remaining comparison methods are filled in automatically.

  (Contributed by Raymond Hettinger.)

• To aid in porting programs from Python 2, the functools.cmp_to_key{.interpreted-text role="func"} function converts an old-style comparison function to modern key function{.interpreted-text role="term"}:

  >>> # locale-aware sort order >>> sorted(iterable, key=cmp_to_key(locale.strcoll)) # doctest: +SKIP

  For sorting examples and a brief sorting tutorial, see the Sorting HowTo tutorial.

  (Contributed by Raymond Hettinger.)

itertools

• The itertools{.interpreted-text role="mod"} module has a new ~itertools.accumulate{.interpreted-text role="func"} function modeled on APL's scan operator and Numpy's accumulate function:

  >>> from itertools import accumulate >>> list(accumulate([8, 2, 50])) [8, 10, 60]

  >>> prob_dist = [0.1, 0.4, 0.2, 0.3] >>> list(accumulate(prob_dist)) # cumulative probability distribution [0.1, 0.5, 0.7, 1.0]

  For an example using ~itertools.accumulate{.interpreted-text role="func"}, see the examples for the random module <random-examples>{.interpreted-text role="ref"}.

  (Contributed by Raymond Hettinger and incorporating design suggestions from Mark Dickinson.)

collections

• The collections.Counter{.interpreted-text role="class"} class now has two forms of in-place subtraction, the existing -= operator for saturating subtraction and the new ~collections.Counter.subtract{.interpreted-text role="meth"} method for regular subtraction. The former is suitable for multisets which only have positive counts, and the latter is more suitable for use cases that allow negative counts:

  >>> from collections import Counter >>> tally = Counter(dogs=5, cats=3) >>> tally -= Counter(dogs=2, cats=8) # saturating subtraction >>> tally Counter({'dogs': 3})

  >>> tally = Counter(dogs=5, cats=3) >>> tally.subtract(dogs=2, cats=8) # regular subtraction >>> tally Counter({'dogs': 3, 'cats': -5})

  (Contributed by Raymond Hettinger.)

• The collections.OrderedDict{.interpreted-text role="class"} class has a new method ~collections.OrderedDict.move_to_end{.interpreted-text role="meth"} which takes an existing key and moves it to either the first or last position in the ordered sequence.

  The default is to move an item to the last position. This is equivalent of renewing an entry with od[k] = od.pop(k).

  A fast move-to-end operation is useful for resequencing entries. For example, an ordered dictionary can be used to track order of access by aging entries from the oldest to the most recently accessed.

  >>> from collections import OrderedDict >>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e']) >>> list(d) ['a', 'b', 'X', 'd', 'e'] >>> d.move_to_end('X') >>> list(d) ['a', 'b', 'd', 'e', 'X']

  (Contributed by Raymond Hettinger.)

• The collections.deque{.interpreted-text role="class"} class grew two new methods ~collections.deque.count{.interpreted-text role="meth"} and ~collections.deque.reverse{.interpreted-text role="meth"} that make them more substitutable for list{.interpreted-text role="class"} objects:

  >>> from collections import deque >>> d = deque('simsalabim') >>> d.count('s') 2 >>> d.reverse() >>> d deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's'])

  (Contributed by Raymond Hettinger.)

threading

The threading{.interpreted-text role="mod"} module has a new ~threading.Barrier{.interpreted-text role="class"} synchronization class for making multiple threads wait until all of them have reached a common barrier point. Barriers are useful for making sure that a task with multiple preconditions does not run until all of the predecessor tasks are complete.

Barriers can work with an arbitrary number of threads. This is a generalization of a Rendezvous which is defined for only two threads.

Implemented as a two-phase cyclic barrier, ~threading.Barrier{.interpreted-text role="class"} objects are suitable for use in loops. The separate filling and draining phases assure that all threads get released (drained) before any one of them can loop back and re-enter the barrier. The barrier fully resets after each cycle.

Example of using barriers:

from threading import Barrier, Thread

def get_votes(site):
    ballots = conduct_election(site)
    all_polls_closed.wait()        # do not count until all polls are closed
    totals = summarize(ballots)
    publish(site, totals)

all_polls_closed = Barrier(len(sites))
for site in sites:
    Thread(target=get_votes, args=(site,)).start()

In this example, the barrier enforces a rule that votes cannot be counted at any polling site until all polls are closed. Notice how a solution with a barrier is similar to one with threading.Thread.join{.interpreted-text role="meth"}, but the threads stay alive and continue to do work (summarizing ballots) after the barrier point is crossed.

If any of the predecessor tasks can hang or be delayed, a barrier can be created with an optional timeout parameter. Then if the timeout period elapses before all the predecessor tasks reach the barrier point, all waiting threads are released and a ~threading.BrokenBarrierError{.interpreted-text role="exc"} exception is raised:

def get_votes(site):
    ballots = conduct_election(site)
    try:
        all_polls_closed.wait(timeout=midnight - time.now())
    except BrokenBarrierError:
        lockbox = seal_ballots(ballots)
        queue.put(lockbox)
    else:
        totals = summarize(ballots)
        publish(site, totals)

In this example, the barrier enforces a more robust rule. If some election sites do not finish before midnight, the barrier times-out and the ballots are sealed and deposited in a queue for later handling.

See Barrier Synchronization Patterns for more examples of how barriers can be used in parallel computing. Also, there is a simple but thorough explanation of barriers in The Little Book of Semaphores, section 3.6.

(Contributed by Kristján Valur Jónsson with an API review by Jeffrey Yasskin in 8777{.interpreted-text role="issue"}.)

datetime and time

• The datetime{.interpreted-text role="mod"} module has a new type ~datetime.timezone{.interpreted-text role="class"} that implements the ~datetime.tzinfo{.interpreted-text role="class"} interface by returning a fixed UTC offset and timezone name. This makes it easier to create timezone-aware datetime objects:

  >>> from datetime import datetime, timezone
  
  >>> datetime.now(timezone.utc)
  datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc)
  
  >>> datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z")
  datetime.datetime(2000, 1, 1, 12, 0, tzinfo=datetime.timezone.utc)
  

• Also, ~datetime.timedelta{.interpreted-text role="class"} objects can now be multiplied by float{.interpreted-text role="class"} and divided by float{.interpreted-text role="class"} and int{.interpreted-text role="class"} objects. And ~datetime.timedelta{.interpreted-text role="class"} objects can now divide one another.

• The datetime.date.strftime{.interpreted-text role="meth"} method is no longer restricted to years after 1900. The new supported year range is from 1000 to 9999 inclusive.

• Whenever a two-digit year is used in a time tuple, the interpretation has been governed by !time.accept2dyear{.interpreted-text role="data"}. The default is True which means that for a two-digit year, the century is guessed according to the POSIX rules governing the %y strptime format.

  Starting with Py3.2, use of the century guessing heuristic will emit a DeprecationWarning{.interpreted-text role="exc"}. Instead, it is recommended that !time.accept2dyear{.interpreted-text role="data"} be set to False so that large date ranges can be used without guesswork:

  >>> import time, warnings
  >>> warnings.resetwarnings()      # remove the default warning filters
  
  >>> time.accept2dyear = True      # guess whether 11 means 11 or 2011
  >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
  Warning (from warnings module):
    ...
  DeprecationWarning: Century info guessed for a 2-digit year.
  'Fri Jan  1 12:34:56 2011'
  
  >>> time.accept2dyear = False     # use the full range of allowable dates
  >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
  'Fri Jan  1 12:34:56 11'
  

  Several functions now have significantly expanded date ranges. When !time.accept2dyear{.interpreted-text role="data"} is false, the time.asctime{.interpreted-text role="func"} function will accept any year that fits in a C int, while the time.mktime{.interpreted-text role="func"} and time.strftime{.interpreted-text role="func"} functions will accept the full range supported by the corresponding operating system functions.

(Contributed by Alexander Belopolsky and Victor Stinner in 1289118{.interpreted-text role="issue"}, 5094{.interpreted-text role="issue"}, 6641{.interpreted-text role="issue"}, 2706{.interpreted-text role="issue"}, 1777412{.interpreted-text role="issue"}, 8013{.interpreted-text role="issue"}, and 10827{.interpreted-text role="issue"}.)

math

The math{.interpreted-text role="mod"} module has been updated with six new functions inspired by the C99 standard.

The ~math.isfinite{.interpreted-text role="func"} function provides a reliable and fast way to detect special values. It returns True for regular numbers and False for Nan or Infinity:

>>> from math import isfinite >>> [isfinite(x) for x in (123, 4.56, float('Nan'), float('Inf'))] [True, True, False, False]

The ~math.expm1{.interpreted-text role="func"} function computes e**x-1 for small values of x without incurring the loss of precision that usually accompanies the subtraction of nearly equal quantities:

>>> from math import expm1 >>> expm1(0.013671875) # more accurate way to compute e**x-1 for a small x 0.013765762467652909

The ~math.erf{.interpreted-text role="func"} function computes a probability integral or Gaussian error function. The complementary error function, ~math.erfc{.interpreted-text role="func"}, is 1 - erf(x):

::: {.doctest options="+SKIP"} >>> from math import erf, erfc, sqrt >>> erf(1.0/sqrt(2.0)) # portion of normal distribution within 1 standard deviation 0.682689492137086 >>> erfc(1.0/sqrt(2.0)) # portion of normal distribution outside 1 standard deviation 0.31731050786291404 >>> erf(1.0/sqrt(2.0)) + erfc(1.0/sqrt(2.0)) 1.0 :::

The ~math.gamma{.interpreted-text role="func"} function is a continuous extension of the factorial function. See https://en.wikipedia.org/wiki/Gamma_function for details. Because the function is related to factorials, it grows large even for small values of x, so there is also a ~math.lgamma{.interpreted-text role="func"} function for computing the natural logarithm of the gamma function:

>>> from math import gamma, lgamma >>> gamma(7.0) # six factorial 720.0 >>> lgamma(801.0) # log(800 factorial) 4551.950730698041

(Contributed by Mark Dickinson.)

abc

The abc{.interpreted-text role="mod"} module now supports ~abc.abstractclassmethod{.interpreted-text role="func"} and ~abc.abstractstaticmethod{.interpreted-text role="func"}.

These tools make it possible to define an abstract base class{.interpreted-text role="term"} that requires a particular classmethod{.interpreted-text role="func"} or staticmethod{.interpreted-text role="func"} to be implemented:

class Temperature(metaclass=abc.ABCMeta):
    @abc.abstractclassmethod
    def from_fahrenheit(cls, t):
        ...
    @abc.abstractclassmethod
    def from_celsius(cls, t):
        ...

(Patch submitted by Daniel Urban; 5867{.interpreted-text role="issue"}.)

io

The io.BytesIO{.interpreted-text role="class"} has a new method, ~io.BytesIO.getbuffer{.interpreted-text role="meth"}, which provides functionality similar to memoryview{.interpreted-text role="func"}. It creates an editable view of the data without making a copy. The buffer's random access and support for slice notation are well-suited to in-place editing:

>>> REC_LEN, LOC_START, LOC_LEN = 34, 7, 11

>>> def change_location(buffer, record_number, location):
...     start = record_number * REC_LEN + LOC_START
...     buffer[start: start+LOC_LEN] = location

>>> import io

>>> byte_stream = io.BytesIO(
...     b'G3805  storeroom  Main chassis    '
...     b'X7899  shipping   Reserve cog     '
...     b'L6988  receiving  Primary sprocket'
... )
>>> buffer = byte_stream.getbuffer()
>>> change_location(buffer, 1, b'warehouse  ')
>>> change_location(buffer, 0, b'showroom   ')
>>> print(byte_stream.getvalue())
b'G3805  showroom   Main chassis    '
b'X7899  warehouse  Reserve cog     '
b'L6988  receiving  Primary sprocket'

(Contributed by Antoine Pitrou in 5506{.interpreted-text role="issue"}.)

reprlib

When writing a ~object.__repr__{.interpreted-text role="meth"} method for a custom container, it is easy to forget to handle the case where a member refers back to the container itself. Python's builtin objects such as list{.interpreted-text role="class"} and set{.interpreted-text role="class"} handle self-reference by displaying "..." in the recursive part of the representation string.

To help write such ~object.__repr__{.interpreted-text role="meth"} methods, the reprlib{.interpreted-text role="mod"} module has a new decorator, ~reprlib.recursive_repr{.interpreted-text role="func"}, for detecting recursive calls to !__repr__{.interpreted-text role="meth"} and substituting a placeholder string instead:

>>> class MyList(list):
...     @recursive_repr()
...     def __repr__(self):
...         return '<' + '|'.join(map(repr, self)) + '>'
...
>>> m = MyList('abc')
>>> m.append(m)
>>> m.append('x')
>>> print(m)
<'a'|'b'|'c'|...|'x'>

(Contributed by Raymond Hettinger in 9826{.interpreted-text role="issue"} and 9840{.interpreted-text role="issue"}.)

logging

In addition to dictionary-based configuration described above, the logging{.interpreted-text role="mod"} package has many other improvements.

The logging documentation has been augmented by a basic tutorial <logging-basic-tutorial>{.interpreted-text role="ref"}, an advanced tutorial <logging-advanced-tutorial>{.interpreted-text role="ref"}, and a cookbook <logging-cookbook>{.interpreted-text role="ref"} of logging recipes. These documents are the fastest way to learn about logging.

The logging.basicConfig{.interpreted-text role="func"} set-up function gained a style argument to support three different types of string formatting. It defaults to "%" for traditional %-formatting, can be set to "{" for the new str.format{.interpreted-text role="meth"} style, or can be set to "$" for the shell-style formatting provided by string.Template{.interpreted-text role="class"}. The following three configurations are equivalent:

>>> from logging import basicConfig
>>> basicConfig(style='%', format="%(name)s -> %(levelname)s: %(message)s")
>>> basicConfig(style='{', format="{name} -> {levelname} {message}")
>>> basicConfig(style='$', format="$name -> $levelname: $message")

If no configuration is set-up before a logging event occurs, there is now a default configuration using a ~logging.StreamHandler{.interpreted-text role="class"} directed to sys.stderr{.interpreted-text role="data"} for events of WARNING level or higher. Formerly, an event occurring before a configuration was set-up would either raise an exception or silently drop the event depending on the value of logging.raiseExceptions{.interpreted-text role="data"}. The new default handler is stored in logging.lastResort{.interpreted-text role="data"}.

The use of filters has been simplified. Instead of creating a ~logging.Filter{.interpreted-text role="class"} object, the predicate can be any Python callable that returns True or False.

There were a number of other improvements that add flexibility and simplify configuration. See the module documentation for a full listing of changes in Python 3.2.

csv

The csv{.interpreted-text role="mod"} module now supports a new dialect, ~csv.unix_dialect{.interpreted-text role="class"}, which applies quoting for all fields and a traditional Unix style with '\n' as the line terminator. The registered dialect name is unix.

The csv.DictWriter{.interpreted-text role="class"} has a new method, ~csv.DictWriter.writeheader{.interpreted-text role="meth"} for writing-out an initial row to document the field names:

>>> import csv, sys
>>> w = csv.DictWriter(sys.stdout, ['name', 'dept'], dialect='unix')
>>> w.writeheader()
"name","dept"
>>> w.writerows([
...     {'name': 'tom', 'dept': 'accounting'},
...     {'name': 'susan', 'dept': 'Salesl'}])
"tom","accounting"
"susan","sales"

(New dialect suggested by Jay Talbot in 5975{.interpreted-text role="issue"}, and the new method suggested by Ed Abraham in 1537721{.interpreted-text role="issue"}.)

contextlib

There is a new and slightly mind-blowing tool ~contextlib.ContextDecorator{.interpreted-text role="class"} that is helpful for creating a context manager{.interpreted-text role="term"} that does double duty as a function decorator.

As a convenience, this new functionality is used by ~contextlib.contextmanager{.interpreted-text role="func"} so that no extra effort is needed to support both roles.

The basic idea is that both context managers and function decorators can be used for pre-action and post-action wrappers. Context managers wrap a group of statements using a with{.interpreted-text role="keyword"} statement, and function decorators wrap a group of statements enclosed in a function. So, occasionally there is a need to write a pre-action or post-action wrapper that can be used in either role.

For example, it is sometimes useful to wrap functions or groups of statements with a logger that can track the time of entry and time of exit. Rather than writing both a function decorator and a context manager for the task, the ~contextlib.contextmanager{.interpreted-text role="func"} provides both capabilities in a single definition:

from contextlib import contextmanager
import logging

logging.basicConfig(level=logging.INFO)

@contextmanager
def track_entry_and_exit(name):
    logging.info('Entering: %s', name)
    yield
    logging.info('Exiting: %s', name)

Formerly, this would have only been usable as a context manager:

with track_entry_and_exit('widget loader'):
    print('Some time consuming activity goes here')
    load_widget()

Now, it can be used as a decorator as well:

@track_entry_and_exit('widget loader')
def activity():
    print('Some time consuming activity goes here')
    load_widget()

Trying to fulfill two roles at once places some limitations on the technique. Context managers normally have the flexibility to return an argument usable by a with{.interpreted-text role="keyword"} statement, but there is no parallel for function decorators.

In the above example, there is not a clean way for the track_entry_and_exit context manager to return a logging instance for use in the body of enclosed statements.

(Contributed by Michael Foord in 9110{.interpreted-text role="issue"}.)

decimal and fractions

Mark Dickinson crafted an elegant and efficient scheme for assuring that different numeric datatypes will have the same hash value whenever their actual values are equal (8188{.interpreted-text role="issue"}):

assert hash(Fraction(3, 2)) == hash(1.5) == \
       hash(Decimal("1.5")) == hash(complex(1.5, 0))

Some of the hashing details are exposed through a new attribute, sys.hash_info{.interpreted-text role="data"}, which describes the bit width of the hash value, the prime modulus, the hash values for infinity and nan, and the multiplier used for the imaginary part of a number:

>>> sys.hash_info # doctest: +SKIP sys.hash_info(width=64, modulus=2305843009213693951, inf=314159, nan=0, imag=1000003)

An early decision to limit the interoperability of various numeric types has been relaxed. It is still unsupported (and ill-advised) to have implicit mixing in arithmetic expressions such as Decimal('1.1') + float('1.1') because the latter loses information in the process of constructing the binary float. However, since existing floating-point value can be converted losslessly to either a decimal or rational representation, it makes sense to add them to the constructor and to support mixed-type comparisons.

• The decimal.Decimal{.interpreted-text role="class"} constructor now accepts float{.interpreted-text role="class"} objects directly so there in no longer a need to use the ~decimal.Decimal.from_float{.interpreted-text role="meth"} method (8257{.interpreted-text role="issue"}).
• Mixed type comparisons are now fully supported so that ~decimal.Decimal{.interpreted-text role="class"} objects can be directly compared with float{.interpreted-text role="class"} and fractions.Fraction{.interpreted-text role="class"} (2531{.interpreted-text role="issue"} and 8188{.interpreted-text role="issue"}).

Similar changes were made to fractions.Fraction{.interpreted-text role="class"} so that the ~fractions.Fraction.from_float{.interpreted-text role="meth"} and ~fractions.Fraction.from_decimal{.interpreted-text role="meth"} methods are no longer needed (8294{.interpreted-text role="issue"}):

>>> from decimal import Decimal >>> from fractions import Fraction >>> Decimal(1.1) Decimal('1.100000000000000088817841970012523233890533447265625') >>> Fraction(1.1) Fraction(2476979795053773, 2251799813685248)

Another useful change for the decimal{.interpreted-text role="mod"} module is that the Context.clamp <decimal.Context.clamp>{.interpreted-text role="attr"} attribute is now public. This is useful in creating contexts that correspond to the decimal interchange formats specified in IEEE 754 (see 8540{.interpreted-text role="issue"}).

(Contributed by Mark Dickinson and Raymond Hettinger.)

ftp

The ftplib.FTP{.interpreted-text role="class"} class now supports the context management protocol to unconditionally consume socket.error{.interpreted-text role="exc"} exceptions and to close the FTP connection when done:

>>> from ftplib import FTP
>>> with FTP("ftp1.at.proftpd.org") as ftp:
        ftp.login()
        ftp.dir()

'230 Anonymous login ok, restrictions apply.'
dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 .
dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 ..
dr-xr-xr-x   5 ftp      ftp          4096 May  6 10:43 CentOS
dr-xr-xr-x   3 ftp      ftp            18 Jul 10  2008 Fedora

Other file-like objects such as mmap.mmap{.interpreted-text role="class"} and fileinput.input{.interpreted-text role="func"} also grew auto-closing context managers:

with fileinput.input(files=('log1.txt', 'log2.txt')) as f:
    for line in f:
        process(line)

(Contributed by Tarek Ziadé and Giampaolo Rodolà in 4972{.interpreted-text role="issue"}, and by Georg Brandl in 8046{.interpreted-text role="issue"} and 1286{.interpreted-text role="issue"}.)

The ~ftplib.FTP_TLS{.interpreted-text role="class"} class now accepts a context parameter, which is a ssl.SSLContext{.interpreted-text role="class"} object allowing bundling SSL configuration options, certificates and private keys into a single (potentially long-lived) structure.

(Contributed by Giampaolo Rodolà; 8806{.interpreted-text role="issue"}.)

popen

The os.popen{.interpreted-text role="func"} and subprocess.Popen{.interpreted-text role="func"} functions now support with{.interpreted-text role="keyword"} statements for auto-closing of the file descriptors.

(Contributed by Antoine Pitrou and Brian Curtin in 7461{.interpreted-text role="issue"} and 10554{.interpreted-text role="issue"}.)

select

The select{.interpreted-text role="mod"} module now exposes a new, constant attribute, ~select.PIPE_BUF{.interpreted-text role="const"}, which gives the minimum number of bytes which are guaranteed not to block when select.select{.interpreted-text role="func"} says a pipe is ready for writing.

>>> import select >>> select.PIPE_BUF # doctest: +SKIP 512

(Available on Unix systems. Patch by Sébastien Sablé in 9862{.interpreted-text role="issue"})

gzip and zipfile

gzip.GzipFile{.interpreted-text role="class"} now implements the io.BufferedIOBase{.interpreted-text role="class"} abstract base class{.interpreted-text role="term"} (except for truncate()). It also has a ~gzip.GzipFile.peek{.interpreted-text role="meth"} method and supports unseekable as well as zero-padded file objects.

The gzip{.interpreted-text role="mod"} module also gains the ~gzip.compress{.interpreted-text role="func"} and ~gzip.decompress{.interpreted-text role="func"} functions for easier in-memory compression and decompression. Keep in mind that text needs to be encoded as bytes{.interpreted-text role="class"} before compressing and decompressing:

>>> import gzip >>> s = 'Three shall be the number thou shalt count, ' >>> s += 'and the number of the counting shall be three' >>> b = s.encode() # convert to utf-8 >>> len(b) 89 >>> c = gzip.compress(b) >>> len(c) 77 >>> gzip.decompress(c).decode()[:42] # decompress and convert to text 'Three shall be the number thou shalt count'

(Contributed by Anand B. Pillai in 3488{.interpreted-text role="issue"}; and by Antoine Pitrou, Nir Aides and Brian Curtin in 9962{.interpreted-text role="issue"}, 1675951{.interpreted-text role="issue"}, 7471{.interpreted-text role="issue"} and 2846{.interpreted-text role="issue"}.)

Also, the zipfile.ZipExtFile <zipfile.ZipFile.open>{.interpreted-text role="class"} class was reworked internally to represent files stored inside an archive. The new implementation is significantly faster and can be wrapped in an io.BufferedReader{.interpreted-text role="class"} object for more speedups. It also solves an issue where interleaved calls to read and readline gave the wrong results.

(Patch submitted by Nir Aides in 7610{.interpreted-text role="issue"}.)

tarfile

The ~tarfile.TarFile{.interpreted-text role="class"} class can now be used as a context manager. In addition, its ~tarfile.TarFile.add{.interpreted-text role="meth"} method has a new option, filter, that controls which files are added to the archive and allows the file metadata to be edited.

The new filter option replaces the older, less flexible exclude parameter which is now deprecated. If specified, the optional filter parameter needs to be a keyword argument{.interpreted-text role="term"}. The user-supplied filter function accepts a ~tarfile.TarInfo{.interpreted-text role="class"} object and returns an updated ~tarfile.TarInfo{.interpreted-text role="class"} object, or if it wants the file to be excluded, the function can return None:

>>> import tarfile, glob

>>> def myfilter(tarinfo):
...     if tarinfo.isfile():             # only save real files
...         tarinfo.uname = 'monty'      # redact the user name
...         return tarinfo

>>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:
...     for filename in glob.glob('*.txt'):
...         tf.add(filename, filter=myfilter)
...     tf.list()
-rw-r--r-- monty/501        902 2011-01-26 17:59:11 annotations.txt
-rw-r--r-- monty/501        123 2011-01-26 17:59:11 general_questions.txt
-rw-r--r-- monty/501       3514 2011-01-26 17:59:11 prion.txt
-rw-r--r-- monty/501        124 2011-01-26 17:59:11 py_todo.txt
-rw-r--r-- monty/501       1399 2011-01-26 17:59:11 semaphore_notes.txt

(Proposed by Tarek Ziadé and implemented by Lars Gustäbel in 6856{.interpreted-text role="issue"}.)

hashlib

The hashlib{.interpreted-text role="mod"} module has two new constant attributes listing the hashing algorithms guaranteed to be present in all implementations and those available on the current implementation:

>>> import hashlib

>>> hashlib.algorithms_guaranteed
{'sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'}

>>> hashlib.algorithms_available
{'md2', 'SHA256', 'SHA512', 'dsaWithSHA', 'mdc2', 'SHA224', 'MD4', 'sha256',
'sha512', 'ripemd160', 'SHA1', 'MDC2', 'SHA', 'SHA384', 'MD2',
'ecdsa-with-SHA1','md4', 'md5', 'sha1', 'DSA-SHA', 'sha224',
'dsaEncryption', 'DSA', 'RIPEMD160', 'sha', 'MD5', 'sha384'}

(Suggested by Carl Chenet in 7418{.interpreted-text role="issue"}.)

ast

The ast{.interpreted-text role="mod"} module has a wonderful a general-purpose tool for safely evaluating expression strings using the Python literal syntax. The ast.literal_eval{.interpreted-text role="func"} function serves as a secure alternative to the builtin eval{.interpreted-text role="func"} function which is easily abused. Python 3.2 adds bytes{.interpreted-text role="class"} and set{.interpreted-text role="class"} literals to the list of supported types: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None.

>>> from ast import literal_eval

>>> request = "{'req': 3, 'func': 'pow', 'args': (2, 0.5)}"
>>> literal_eval(request)
{'args': (2, 0.5), 'req': 3, 'func': 'pow'}

>>> request = "os.system('do something harmful')"
>>> literal_eval(request)
Traceback (most recent call last):
  ...
ValueError: malformed node or string: <_ast.Call object at 0x101739a10>

(Implemented by Benjamin Peterson and Georg Brandl.)

os

Different operating systems use various encodings for filenames and environment variables. The os{.interpreted-text role="mod"} module provides two new functions, ~os.fsencode{.interpreted-text role="func"} and ~os.fsdecode{.interpreted-text role="func"}, for encoding and decoding filenames:

>>> import os >>> filename = 'Sehenswürdigkeiten' >>> os.fsencode(filename) b'Sehenswxc3xbcrdigkeiten'

Some operating systems allow direct access to encoded bytes in the environment. If so, the os.supports_bytes_environ{.interpreted-text role="const"} constant will be true.

For direct access to encoded environment variables (if available), use the new os.getenvb{.interpreted-text role="func"} function or use os.environb{.interpreted-text role="data"} which is a bytes version of os.environ{.interpreted-text role="data"}.

(Contributed by Victor Stinner.)

shutil

The shutil.copytree{.interpreted-text role="func"} function has two new options:

• ignore_dangling_symlinks: when symlinks=False so that the function copies a file pointed to by a symlink, not the symlink itself. This option will silence the error raised if the file doesn't exist.
• copy_function: is a callable that will be used to copy files. shutil.copy2{.interpreted-text role="func"} is used by default.

(Contributed by Tarek Ziadé.)

In addition, the shutil{.interpreted-text role="mod"} module now supports archiving operations <archiving-operations>{.interpreted-text role="ref"} for zipfiles, uncompressed tarfiles, gzipped tarfiles, and bzipped tarfiles. And there are functions for registering additional archiving file formats (such as xz compressed tarfiles or custom formats).

The principal functions are ~shutil.make_archive{.interpreted-text role="func"} and ~shutil.unpack_archive{.interpreted-text role="func"}. By default, both operate on the current directory (which can be set by os.chdir{.interpreted-text role="func"}) and on any sub-directories. The archive filename needs to be specified with a full pathname. The archiving step is non-destructive (the original files are left unchanged).

>>> import shutil, pprint

>>> os.chdir('mydata')  # change to the source directory
>>> f = shutil.make_archive('/var/backup/mydata',
...                         'zip')      # archive the current directory
>>> f                                   # show the name of archive
'/var/backup/mydata.zip'
>>> os.chdir('tmp')                     # change to an unpacking
>>> shutil.unpack_archive('/var/backup/mydata.zip')  # recover the data

>>> pprint.pprint(shutil.get_archive_formats())  # display known formats
[('bztar', "bzip2'ed tar-file"),
 ('gztar', "gzip'ed tar-file"),
 ('tar', 'uncompressed tar file'),
 ('zip', 'ZIP file')]

>>> shutil.register_archive_format(     # register a new archive format
...     name='xz',
...     function=xz.compress,           # callable archiving function
...     extra_args=[('level', 8)],      # arguments to the function
...     description='xz compression'
... )

(Contributed by Tarek Ziadé.)

sqlite3

The sqlite3{.interpreted-text role="mod"} module was updated to pysqlite version 2.6.0. It has two new capabilities.

• The !sqlite3.Connection.in_transit{.interpreted-text role="attr"} attribute is true if there is an active transaction for uncommitted changes.
• The sqlite3.Connection.enable_load_extension{.interpreted-text role="meth"} and sqlite3.Connection.load_extension{.interpreted-text role="meth"} methods allows you to load SQLite extensions from ".so" files. One well-known extension is the fulltext-search extension distributed with SQLite.

(Contributed by R. David Murray and Shashwat Anand; 8845{.interpreted-text role="issue"}.)

html

A new html{.interpreted-text role="mod"} module was introduced with only a single function, ~html.escape{.interpreted-text role="func"}, which is used for escaping reserved characters from HTML markup:

>>> import html >>> html.escape('x > 2 && x < 7') 'x > 2 && x < 7'

socket

The socket{.interpreted-text role="mod"} module has two new improvements.

• Socket objects now have a ~socket.socket.detach{.interpreted-text role="meth"} method which puts the socket into closed state without actually closing the underlying file descriptor. The latter can then be reused for other purposes. (Added by Antoine Pitrou; 8524{.interpreted-text role="issue"}.)
• socket.create_connection{.interpreted-text role="func"} now supports the context management protocol to unconditionally consume socket.error{.interpreted-text role="exc"} exceptions and to close the socket when done. (Contributed by Giampaolo Rodolà; 9794{.interpreted-text role="issue"}.)

ssl

The ssl{.interpreted-text role="mod"} module added a number of features to satisfy common requirements for secure (encrypted, authenticated) internet connections:

• A new class, ~ssl.SSLContext{.interpreted-text role="class"}, serves as a container for persistent SSL data, such as protocol settings, certificates, private keys, and various other options. It includes a ~ssl.SSLContext.wrap_socket{.interpreted-text role="meth"} for creating an SSL socket from an SSL context.
• A new function, !ssl.match_hostname{.interpreted-text role="func"}, supports server identity verification for higher-level protocols by implementing the rules of HTTPS (from 2818{.interpreted-text role="rfc"}) which are also suitable for other protocols.
• The ssl.wrap_socket() <ssl.SSLContext.wrap_socket>{.interpreted-text role="func"} constructor function now takes a ciphers argument. The ciphers string lists the allowed encryption algorithms using the format described in the OpenSSL documentation.
• When linked against recent versions of OpenSSL, the ssl{.interpreted-text role="mod"} module now supports the Server Name Indication extension to the TLS protocol, allowing multiple "virtual hosts" using different certificates on a single IP port. This extension is only supported in client mode, and is activated by passing the server_hostname argument to ssl.SSLContext.wrap_socket{.interpreted-text role="meth"}.
• Various options have been added to the ssl{.interpreted-text role="mod"} module, such as ~ssl.OP_NO_SSLv2{.interpreted-text role="data"} which disables the insecure and obsolete SSLv2 protocol.
• The extension now loads all the OpenSSL ciphers and digest algorithms. If some SSL certificates cannot be verified, they are reported as an "unknown algorithm" error.
• The version of OpenSSL being used is now accessible using the module attributes ssl.OPENSSL_VERSION{.interpreted-text role="const"} (a string), ssl.OPENSSL_VERSION_INFO{.interpreted-text role="const"} (a 5-tuple), and ssl.OPENSSL_VERSION_NUMBER{.interpreted-text role="const"} (an integer).

(Contributed by Antoine Pitrou in 8850{.interpreted-text role="issue"}, 1589{.interpreted-text role="issue"}, 8322{.interpreted-text role="issue"}, 5639{.interpreted-text role="issue"}, 4870{.interpreted-text role="issue"}, 8484{.interpreted-text role="issue"}, and 8321{.interpreted-text role="issue"}.)

nntp

The !nntplib{.interpreted-text role="mod"} module has a revamped implementation with better bytes and text semantics as well as more practical APIs. These improvements break compatibility with the nntplib version in Python 3.1, which was partly dysfunctional in itself.

Support for secure connections through both implicit (using !nntplib.NNTP_SSL{.interpreted-text role="class"}) and explicit (using !nntplib.NNTP.starttls{.interpreted-text role="meth"}) TLS has also been added.

(Contributed by Antoine Pitrou in 9360{.interpreted-text role="issue"} and Andrew Vant in 1926{.interpreted-text role="issue"}.)

certificates

http.client.HTTPSConnection{.interpreted-text role="class"}, urllib.request.HTTPSHandler{.interpreted-text role="class"} and urllib.request.urlopen{.interpreted-text role="func"} now take optional arguments to allow for server certificate checking against a set of Certificate Authorities, as recommended in public uses of HTTPS.

(Added by Antoine Pitrou, 9003{.interpreted-text role="issue"}.)

imaplib

Support for explicit TLS on standard IMAP4 connections has been added through the new imaplib.IMAP4.starttls{.interpreted-text role="mod"} method.

(Contributed by Lorenzo M. Catucci and Antoine Pitrou, 4471{.interpreted-text role="issue"}.)

http.client

There were a number of small API improvements in the http.client{.interpreted-text role="mod"} module. The old-style HTTP 0.9 simple responses are no longer supported and the strict parameter is deprecated in all classes.

The ~http.client.HTTPConnection{.interpreted-text role="class"} and ~http.client.HTTPSConnection{.interpreted-text role="class"} classes now have a source_address parameter for a (host, port) tuple indicating where the HTTP connection is made from.

Support for certificate checking and HTTPS virtual hosts were added to ~http.client.HTTPSConnection{.interpreted-text role="class"}.

The ~http.client.HTTPConnection.request{.interpreted-text role="meth"} method on connection objects allowed an optional body argument so that a file object{.interpreted-text role="term"} could be used to supply the content of the request. Conveniently, the body argument now also accepts an iterable{.interpreted-text role="term"} object so long as it includes an explicit Content-Length header. This extended interface is much more flexible than before.

To establish an HTTPS connection through a proxy server, there is a new ~http.client.HTTPConnection.set_tunnel{.interpreted-text role="meth"} method that sets the host and port for HTTP Connect tunneling.

To match the behavior of http.server{.interpreted-text role="mod"}, the HTTP client library now also encodes headers with ISO-8859-1 (Latin-1) encoding. It was already doing that for incoming headers, so now the behavior is consistent for both incoming and outgoing traffic. (See work by Armin Ronacher in 10980{.interpreted-text role="issue"}.)

unittest

The unittest module has a number of improvements supporting test discovery for packages, easier experimentation at the interactive prompt, new testcase methods, improved diagnostic messages for test failures, and better method names.

• The command-line call python -m unittest can now accept file paths instead of module names for running specific tests (10620{.interpreted-text role="issue"}). The new test discovery can find tests within packages, locating any test importable from the top-level directory. The top-level directory can be specified with the -t option, a pattern for matching files with -p, and a directory to start discovery with -s:

  $ python -m unittest discover -s my_proj_dir -p _test.py
  

  (Contributed by Michael Foord.)

• Experimentation at the interactive prompt is now easier because the unittest.TestCase{.interpreted-text role="class"} class can now be instantiated without arguments:

  >>> from unittest import TestCase >>> TestCase().assertEqual(pow(2, 3), 8)

  (Contributed by Michael Foord.)

• The unittest{.interpreted-text role="mod"} module has two new methods, ~unittest.TestCase.assertWarns{.interpreted-text role="meth"} and ~unittest.TestCase.assertWarnsRegex{.interpreted-text role="meth"} to verify that a given warning type is triggered by the code under test:

  with self.assertWarns(DeprecationWarning):
      legacy_function('XYZ')
  

  (Contributed by Antoine Pitrou, 9754{.interpreted-text role="issue"}.)

  Another new method, ~unittest.TestCase.assertCountEqual{.interpreted-text role="meth"} is used to compare two iterables to determine if their element counts are equal (whether the same elements are present with the same number of occurrences regardless of order):

  def test_anagram(self):
      self.assertCountEqual('algorithm', 'logarithm')
  

  (Contributed by Raymond Hettinger.)

• A principal feature of the unittest module is an effort to produce meaningful diagnostics when a test fails. When possible, the failure is recorded along with a diff of the output. This is especially helpful for analyzing log files of failed test runs. However, since diffs can sometime be voluminous, there is a new ~unittest.TestCase.maxDiff{.interpreted-text role="attr"} attribute that sets maximum length of diffs displayed.

• In addition, the method names in the module have undergone a number of clean-ups.

  For example, ~unittest.TestCase.assertRegex{.interpreted-text role="meth"} is the new name for !assertRegexpMatches{.interpreted-text role="meth"} which was misnamed because the test uses re.search{.interpreted-text role="func"}, not re.match{.interpreted-text role="func"}. Other methods using regular expressions are now named using short form "Regex" in preference to "Regexp" -- this matches the names used in other unittest implementations, matches Python's old name for the re{.interpreted-text role="mod"} module, and it has unambiguous camel-casing.

  (Contributed by Raymond Hettinger and implemented by Ezio Melotti.)

• To improve consistency, some long-standing method aliases are being deprecated in favor of the preferred names:

  Old Name Preferred Name

  ---

  !assert_{.interpreted-text role="meth"} .assertTrue{.interpreted-text role="meth"} !assertEquals{.interpreted-text role="meth"} .assertEqual{.interpreted-text role="meth"} !assertNotEquals{.interpreted-text role="meth"} .assertNotEqual{.interpreted-text role="meth"} !assertAlmostEquals{.interpreted-text role="meth"} .assertAlmostEqual{.interpreted-text role="meth"} !assertNotAlmostEquals{.interpreted-text role="meth"} .assertNotAlmostEqual{.interpreted-text role="meth"}

  Likewise, the TestCase.fail* methods deprecated in Python 3.1 are expected to be removed in Python 3.3.

  (Contributed by Ezio Melotti; 9424{.interpreted-text role="issue"}.)

• The !assertDictContainsSubset{.interpreted-text role="meth"} method was deprecated because it was misimplemented with the arguments in the wrong order. This created hard-to-debug optical illusions where tests like TestCase().assertDictContainsSubset({'a':1, 'b':2}, {'a':1}) would fail.

  (Contributed by Raymond Hettinger.)

random

The integer methods in the random{.interpreted-text role="mod"} module now do a better job of producing uniform distributions. Previously, they computed selections with int(n*random()) which had a slight bias whenever n was not a power of two. Now, multiple selections are made from a range up to the next power of two and a selection is kept only when it falls within the range 0 <= x < n. The functions and methods affected are ~random.randrange{.interpreted-text role="func"}, ~random.randint{.interpreted-text role="func"}, ~random.choice{.interpreted-text role="func"}, ~random.shuffle{.interpreted-text role="func"} and ~random.sample{.interpreted-text role="func"}.

(Contributed by Raymond Hettinger; 9025{.interpreted-text role="issue"}.)

poplib

~poplib.POP3_SSL{.interpreted-text role="class"} class now accepts a context parameter, which is a ssl.SSLContext{.interpreted-text role="class"} object allowing bundling SSL configuration options, certificates and private keys into a single (potentially long-lived) structure.

(Contributed by Giampaolo Rodolà; 8807{.interpreted-text role="issue"}.)

asyncore

!asyncore.dispatcher{.interpreted-text role="class"} now provides a !handle_accepted{.interpreted-text role="meth"} method returning a (sock, addr) pair which is called when a connection has actually been established with a new remote endpoint. This is supposed to be used as a replacement for old !handle_accept{.interpreted-text role="meth"} and avoids the user to call !accept{.interpreted-text role="meth"} directly.

(Contributed by Giampaolo Rodolà; 6706{.interpreted-text role="issue"}.)

tempfile

The tempfile{.interpreted-text role="mod"} module has a new context manager, ~tempfile.TemporaryDirectory{.interpreted-text role="class"} which provides easy deterministic cleanup of temporary directories:

with tempfile.TemporaryDirectory() as tmpdirname:
    print('created temporary dir:', tmpdirname)

(Contributed by Neil Schemenauer and Nick Coghlan; 5178{.interpreted-text role="issue"}.)

inspect

• The inspect{.interpreted-text role="mod"} module has a new function ~inspect.getgeneratorstate{.interpreted-text role="func"} to easily identify the current state of a generator-iterator:

  >>> from inspect import getgeneratorstate
  >>> def gen():
  ...     yield 'demo'
  ...
  >>> g = gen()
  >>> getgeneratorstate(g)
  'GEN_CREATED'
  >>> next(g)
  'demo'
  >>> getgeneratorstate(g)
  'GEN_SUSPENDED'
  >>> next(g, None)
  >>> getgeneratorstate(g)
  'GEN_CLOSED'
  

  (Contributed by Rodolpho Eckhardt and Nick Coghlan, 10220{.interpreted-text role="issue"}.)

• To support lookups without the possibility of activating a dynamic attribute, the inspect{.interpreted-text role="mod"} module has a new function, ~inspect.getattr_static{.interpreted-text role="func"}. Unlike hasattr{.interpreted-text role="func"}, this is a true read-only search, guaranteed not to change state while it is searching:

  >>> class A:
  ...     @property
  ...     def f(self):
  ...         print('Running')
  ...         return 10
  ...
  >>> a = A()
  >>> getattr(a, 'f')
  Running
  10
  >>> inspect.getattr_static(a, 'f')
  <property object at 0x1022bd788>
  

(Contributed by Michael Foord.)

pydoc

The pydoc{.interpreted-text role="mod"} module now provides a much-improved web server interface, as well as a new command-line option -b to automatically open a browser window to display that server:

$ pydoc3.2 -b

(Contributed by Ron Adam; 2001{.interpreted-text role="issue"}.)

dis

The dis{.interpreted-text role="mod"} module gained two new functions for inspecting code, ~dis.code_info{.interpreted-text role="func"} and ~dis.show_code{.interpreted-text role="func"}. Both provide detailed code object information for the supplied function, method, source code string or code object. The former returns a string and the latter prints it:

>>> import dis, random
>>> dis.show_code(random.choice)
Name:              choice
Filename:          /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/random.py
Argument count:    2
Kw-only arguments: 0
Number of locals:  3
Stack size:        11
Flags:             OPTIMIZED, NEWLOCALS, NOFREE
Constants:
   0: 'Choose a random element from a non-empty sequence.'
   1: 'Cannot choose from an empty sequence'
Names:
   0: _randbelow
   1: len
   2: ValueError
   3: IndexError
Variable names:
   0: self
   1: seq
   2: i

In addition, the ~dis.dis{.interpreted-text role="func"} function now accepts string arguments so that the common idiom dis(compile(s, '', 'eval')) can be shortened to dis(s):

>>> dis('3*x+1 if x%2==1 else x//2')
  1           0 LOAD_NAME                0 (x)
              3 LOAD_CONST               0 (2)
              6 BINARY_MODULO
              7 LOAD_CONST               1 (1)
             10 COMPARE_OP               2 (==)
             13 POP_JUMP_IF_FALSE       28
             16 LOAD_CONST               2 (3)
             19 LOAD_NAME                0 (x)
             22 BINARY_MULTIPLY
             23 LOAD_CONST               1 (1)
             26 BINARY_ADD
             27 RETURN_VALUE
        >>   28 LOAD_NAME                0 (x)
             31 LOAD_CONST               0 (2)
             34 BINARY_FLOOR_DIVIDE
             35 RETURN_VALUE

Taken together, these improvements make it easier to explore how CPython is implemented and to see for yourself what the language syntax does under-the-hood.

(Contributed by Nick Coghlan in 9147{.interpreted-text role="issue"}.)

dbm

All database modules now support the !get{.interpreted-text role="meth"} and !setdefault{.interpreted-text role="meth"} methods.

(Suggested by Ray Allen in 9523{.interpreted-text role="issue"}.)

ctypes

A new type, ctypes.c_ssize_t{.interpreted-text role="class"} represents the C ssize_t{.interpreted-text role="c:type"} datatype.

site

The site{.interpreted-text role="mod"} module has three new functions useful for reporting on the details of a given Python installation.

• ~site.getsitepackages{.interpreted-text role="func"} lists all global site-packages directories.
• ~site.getuserbase{.interpreted-text role="func"} reports on the user's base directory where data can be stored.
• ~site.getusersitepackages{.interpreted-text role="func"} reveals the user-specific site-packages directory path.

>>> import site
>>> site.getsitepackages()
['/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages',
 '/Library/Frameworks/Python.framework/Versions/3.2/lib/site-python',
 '/Library/Python/3.2/site-packages']
>>> site.getuserbase()
'/Users/raymondhettinger/Library/Python/3.2'
>>> site.getusersitepackages()
'/Users/raymondhettinger/Library/Python/3.2/lib/python/site-packages'

Conveniently, some of site's functionality is accessible directly from the command-line:

$ python -m site --user-base
/Users/raymondhettinger/.local
$ python -m site --user-site
/Users/raymondhettinger/.local/lib/python3.2/site-packages

(Contributed by Tarek Ziadé in 6693{.interpreted-text role="issue"}.)

sysconfig

The new sysconfig{.interpreted-text role="mod"} module makes it straightforward to discover installation paths and configuration variables that vary across platforms and installations.

The module offers access simple access functions for platform and version information:

• ~sysconfig.get_platform{.interpreted-text role="func"} returning values like linux-i586 or macosx-10.6-ppc.
• ~sysconfig.get_python_version{.interpreted-text role="func"} returns a Python version string such as "3.2".

It also provides access to the paths and variables corresponding to one of seven named schemes used by distutils. Those include posix_prefix, posix_home, posix_user, nt, nt_user, os2, os2_home:

• ~sysconfig.get_paths{.interpreted-text role="func"} makes a dictionary containing installation paths for the current installation scheme.
• ~sysconfig.get_config_vars{.interpreted-text role="func"} returns a dictionary of platform specific variables.

There is also a convenient command-line interface:

C:\Python32>python -m sysconfig
Platform: "win32"
Python version: "3.2"
Current installation scheme: "nt"

Paths:
        data = "C:\Python32"
        include = "C:\Python32\Include"
        platinclude = "C:\Python32\Include"
        platlib = "C:\Python32\Lib\site-packages"
        platstdlib = "C:\Python32\Lib"
        purelib = "C:\Python32\Lib\site-packages"
        scripts = "C:\Python32\Scripts"
        stdlib = "C:\Python32\Lib"

Variables:
        BINDIR = "C:\Python32"
        BINLIBDEST = "C:\Python32\Lib"
        EXE = ".exe"
        INCLUDEPY = "C:\Python32\Include"
        LIBDEST = "C:\Python32\Lib"
        SO = ".pyd"
        VERSION = "32"
        abiflags = ""
        base = "C:\Python32"
        exec_prefix = "C:\Python32"
        platbase = "C:\Python32"
        prefix = "C:\Python32"
        projectbase = "C:\Python32"
        py_version = "3.2"
        py_version_nodot = "32"
        py_version_short = "3.2"
        srcdir = "C:\Python32"
        userbase = "C:\Documents and Settings\Raymond\Application Data\Python"

(Moved out of Distutils by Tarek Ziadé.)

pdb

The pdb{.interpreted-text role="mod"} debugger module gained a number of usability improvements:

• pdb.py{.interpreted-text role="file"} now has a -c option that executes commands as given in a .pdbrc{.interpreted-text role="file"} script file.
• A .pdbrc{.interpreted-text role="file"} script file can contain continue and next commands that continue debugging.
• The ~pdb.Pdb{.interpreted-text role="class"} class constructor now accepts a nosigint argument.
• New commands: l(list), ll(long list) and source for listing source code.
• New commands: display and undisplay for showing or hiding the value of an expression if it has changed.
• New command: interact for starting an interactive interpreter containing the global and local names found in the current scope.
• Breakpoints can be cleared by breakpoint number.

(Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.)

configparser

The configparser{.interpreted-text role="mod"} module was modified to improve usability and predictability of the default parser and its supported INI syntax. The old !ConfigParser{.interpreted-text role="class"} class was removed in favor of !SafeConfigParser{.interpreted-text role="class"} which has in turn been renamed to ~configparser.ConfigParser{.interpreted-text role="class"}. Support for inline comments is now turned off by default and section or option duplicates are not allowed in a single configuration source.

Config parsers gained a new API based on the mapping protocol:

>>> parser = ConfigParser()
>>> parser.read_string("""
... [DEFAULT]
... location = upper left
... visible = yes
... editable = no
... color = blue
...
... [main]
... title = Main Menu
... color = green
...
... [options]
... title = Options
... """)
>>> parser['main']['color']
'green'
>>> parser['main']['editable']
'no'
>>> section = parser['options']
>>> section['title']
'Options'
>>> section['title'] = 'Options (editable: %(editable)s)'
>>> section['title']
'Options (editable: no)'

The new API is implemented on top of the classical API, so custom parser subclasses should be able to use it without modifications.

The INI file structure accepted by config parsers can now be customized. Users can specify alternative option/value delimiters and comment prefixes, change the name of the DEFAULT section or switch the interpolation syntax.

There is support for pluggable interpolation including an additional interpolation handler ~configparser.ExtendedInterpolation{.interpreted-text role="class"}:

>>> parser = ConfigParser(interpolation=ExtendedInterpolation())
>>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'},
...                   'custom': {'prefix': '/usr/local'}})
>>> parser.read_string("""
... [buildout]
... parts =
...   zope9
...   instance
... find-links =
...   ${buildout:directory}/downloads/dist
...
... [zope9]
... recipe = plone.recipe.zope9install
... location = /opt/zope
...
... [instance]
... recipe = plone.recipe.zope9instance
... zope9-location = ${zope9:location}
... zope-conf = ${custom:prefix}/etc/zope.conf
... """)
>>> parser['buildout']['find-links']
'\n/home/ambv/zope9/downloads/dist'
>>> parser['instance']['zope-conf']
'/usr/local/etc/zope.conf'
>>> instance = parser['instance']
>>> instance['zope-conf']
'/usr/local/etc/zope.conf'
>>> instance['zope9-location']
'/opt/zope'

A number of smaller features were also introduced, like support for specifying encoding in read operations, specifying fallback values for get-functions, or reading directly from dictionaries and strings.

(All changes contributed by Łukasz Langa.)

urllib.parse

A number of usability improvements were made for the urllib.parse{.interpreted-text role="mod"} module.

The ~urllib.parse.urlparse{.interpreted-text role="func"} function now supports IPv6 addresses as described in 2732{.interpreted-text role="rfc"}:

>>> import urllib.parse >>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') # doctest: +NORMALIZE_WHITESPACE ParseResult(scheme='http', netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]', path='/foo/', params='', query='', fragment='')

The ~urllib.parse.urldefrag{.interpreted-text role="func"} function now returns a named tuple{.interpreted-text role="term"}:

>>> r = urllib.parse.urldefrag('http://python.org/about/#target')
>>> r
DefragResult(url='http://python.org/about/', fragment='target')
>>> r[0]
'http://python.org/about/'
>>> r.fragment
'target'

And, the ~urllib.parse.urlencode{.interpreted-text role="func"} function is now much more flexible, accepting either a string or bytes type for the query argument. If it is a string, then the safe, encoding, and error parameters are sent to ~urllib.parse.quote_plus{.interpreted-text role="func"} for encoding:

>>> urllib.parse.urlencode([
...      ('type', 'telenovela'),
...      ('name', '¿Dónde Está Elisa?')],
...      encoding='latin-1')
'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F'

As detailed in parsing-ascii-encoded-bytes{.interpreted-text role="ref"}, all the urllib.parse{.interpreted-text role="mod"} functions now accept ASCII-encoded byte strings as input, so long as they are not mixed with regular strings. If ASCII-encoded byte strings are given as parameters, the return types will also be an ASCII-encoded byte strings:

>>> urllib.parse.urlparse(b'http://www.python.org:80/about/\') # doctest: +NORMALIZE_WHITESPACE ParseResultBytes(scheme=b'http', netloc=b'www.python.org:80\', path=b'/about/', params=b'', query=b'', fragment=b'')

(Work by Nick Coghlan, Dan Mahn, and Senthil Kumaran in 2987{.interpreted-text role="issue"}, 5468{.interpreted-text role="issue"}, and 9873{.interpreted-text role="issue"}.)

mailbox

Thanks to a concerted effort by R. David Murray, the mailbox{.interpreted-text role="mod"} module has been fixed for Python 3.2. The challenge was that mailbox had been originally designed with a text interface, but email messages are best represented with bytes{.interpreted-text role="class"} because various parts of a message may have different encodings.

The solution harnessed the email{.interpreted-text role="mod"} package's binary support for parsing arbitrary email messages. In addition, the solution required a number of API changes.

As expected, the ~mailbox.Mailbox.add{.interpreted-text role="meth"} method for mailbox.Mailbox{.interpreted-text role="class"} objects now accepts binary input.

~io.StringIO{.interpreted-text role="class"} and text file input are deprecated. Also, string input will fail early if non-ASCII characters are used. Previously it would fail when the email was processed in a later step.

There is also support for binary output. The ~mailbox.Mailbox.get_file{.interpreted-text role="meth"} method now returns a file in the binary mode (where it used to incorrectly set the file to text-mode). There is also a new ~mailbox.Mailbox.get_bytes{.interpreted-text role="meth"} method that returns a bytes{.interpreted-text role="class"} representation of a message corresponding to a given key.

It is still possible to get non-binary output using the old API's ~mailbox.Mailbox.get_string{.interpreted-text role="meth"} method, but that approach is not very useful. Instead, it is best to extract messages from a ~mailbox.Message{.interpreted-text role="class"} object or to load them from binary input.

(Contributed by R. David Murray, with efforts from Steffen Daode Nurpmeso and an initial patch by Victor Stinner in 9124{.interpreted-text role="issue"}.)

turtledemo

The demonstration code for the turtle{.interpreted-text role="mod"} module was moved from the Demo directory to main library. It includes over a dozen sample scripts with lively displays. Being on sys.path{.interpreted-text role="data"}, it can now be run directly from the command-line:

$ python -m turtledemo

(Moved from the Demo directory by Alexander Belopolsky in 10199{.interpreted-text role="issue"}.)

Multi-threading

• The mechanism for serializing execution of concurrently running Python threads (generally known as the GIL{.interpreted-text role="term"} or Global Interpreter Lock) has been rewritten. Among the objectives were more predictable switching intervals and reduced overhead due to lock contention and the number of ensuing system calls. The notion of a "check interval" to allow thread switches has been abandoned and replaced by an absolute duration expressed in seconds. This parameter is tunable through sys.setswitchinterval{.interpreted-text role="func"}. It currently defaults to 5 milliseconds.

  Additional details about the implementation can be read from a python-dev mailing-list message (however, "priority requests" as exposed in this message have not been kept for inclusion).

  (Contributed by Antoine Pitrou.)

• Regular and recursive locks now accept an optional timeout argument to their ~threading.Lock.acquire{.interpreted-text role="meth"} method. (Contributed by Antoine Pitrou; 7316{.interpreted-text role="issue"}.)

• Similarly, threading.Semaphore.acquire{.interpreted-text role="meth"} also gained a timeout argument. (Contributed by Torsten Landschoff; 850728{.interpreted-text role="issue"}.)

• Regular and recursive lock acquisitions can now be interrupted by signals on platforms using Pthreads. This means that Python programs that deadlock while acquiring locks can be successfully killed by repeatedly sending SIGINT to the process (by pressing Ctrl+C{.interpreted-text role="kbd"} in most shells). (Contributed by Reid Kleckner; 8844{.interpreted-text role="issue"}.)

Optimizations

A number of small performance enhancements have been added:

• Python's peephole optimizer now recognizes patterns such x in {1, 2, 3} as being a test for membership in a set of constants. The optimizer recasts the set{.interpreted-text role="class"} as a frozenset{.interpreted-text role="class"} and stores the pre-built constant.

  Now that the speed penalty is gone, it is practical to start writing membership tests using set-notation. This style is both semantically clear and operationally fast:

  extension = name.rpartition('.')[2]
  if extension in {'xml', 'html', 'xhtml', 'css'}:
      handle(name)
  

  (Patch and additional tests contributed by Dave Malcolm; 6690{.interpreted-text role="issue"}).

• Serializing and unserializing data using the pickle{.interpreted-text role="mod"} module is now several times faster.

  (Contributed by Alexandre Vassalotti, Antoine Pitrou and the Unladen Swallow team in 9410{.interpreted-text role="issue"} and 3873{.interpreted-text role="issue"}.)

• The Timsort algorithm used in list.sort{.interpreted-text role="meth"} and sorted{.interpreted-text role="func"} now runs faster and uses less memory when called with a key function{.interpreted-text role="term"}. Previously, every element of a list was wrapped with a temporary object that remembered the key value associated with each element. Now, two arrays of keys and values are sorted in parallel. This saves the memory consumed by the sort wrappers, and it saves time lost to delegating comparisons.

  (Patch by Daniel Stutzbach in 9915{.interpreted-text role="issue"}.)

• JSON decoding performance is improved and memory consumption is reduced whenever the same string is repeated for multiple keys. Also, JSON encoding now uses the C speedups when the sort_keys argument is true.

  (Contributed by Antoine Pitrou in 7451{.interpreted-text role="issue"} and by Raymond Hettinger and Antoine Pitrou in 10314{.interpreted-text role="issue"}.)

• Recursive locks (created with the threading.RLock{.interpreted-text role="func"} API) now benefit from a C implementation which makes them as fast as regular locks, and between 10x and 15x faster than their previous pure Python implementation.

  (Contributed by Antoine Pitrou; 3001{.interpreted-text role="issue"}.)

• The fast-search algorithm in stringlib is now used by the ~str.split{.interpreted-text role="meth"}, ~str.rsplit{.interpreted-text role="meth"}, ~str.splitlines{.interpreted-text role="meth"} and ~str.replace{.interpreted-text role="meth"} methods on bytes{.interpreted-text role="class"}, bytearray{.interpreted-text role="class"} and str{.interpreted-text role="class"} objects. Likewise, the algorithm is also used by ~str.rfind{.interpreted-text role="meth"}, ~str.rindex{.interpreted-text role="meth"}, ~str.rsplit{.interpreted-text role="meth"} and ~str.rpartition{.interpreted-text role="meth"}.

  (Patch by Florent Xicluna in 7622{.interpreted-text role="issue"} and 7462{.interpreted-text role="issue"}.)

• Integer to string conversions now work two "digits" at a time, reducing the number of division and modulo operations.

  (6713{.interpreted-text role="issue"} by Gawain Bolton, Mark Dickinson, and Victor Stinner.)

There were several other minor optimizations. Set differencing now runs faster when one operand is much larger than the other (patch by Andress Bennetts in 8685{.interpreted-text role="issue"}). The !array.repeat{.interpreted-text role="meth"} method has a faster implementation (1569291{.interpreted-text role="issue"} by Alexander Belopolsky). The ~http.server.BaseHTTPRequestHandler{.interpreted-text role="class"} has more efficient buffering (3709{.interpreted-text role="issue"} by Andrew Schaaf). The operator.attrgetter{.interpreted-text role="func"} function has been sped-up (10160{.interpreted-text role="issue"} by Christos Georgiou). And ~configparser.ConfigParser{.interpreted-text role="class"} loads multi-line arguments a bit faster (7113{.interpreted-text role="issue"} by Łukasz Langa).

Unicode

Python has been updated to Unicode 6.0.0. The update to the standard adds over 2,000 new characters including emoji symbols which are important for mobile phones.

In addition, the updated standard has altered the character properties for two Kannada characters (U+0CF1, U+0CF2) and one New Tai Lue numeric character (U+19DA), making the former eligible for use in identifiers while disqualifying the latter. For more information, see Unicode Character Database Changes.

Codecs

Support was added for cp720 Arabic DOS encoding (1616979{.interpreted-text role="issue"}).

MBCS encoding no longer ignores the error handler argument. In the default strict mode, it raises an UnicodeDecodeError{.interpreted-text role="exc"} when it encounters an undecodable byte sequence and an UnicodeEncodeError{.interpreted-text role="exc"} for an unencodable character.

The MBCS codec supports 'strict' and 'ignore' error handlers for decoding, and 'strict' and 'replace' for encoding.

To emulate Python3.1 MBCS encoding, select the 'ignore' handler for decoding and the 'replace' handler for encoding.

On Mac OS X, Python decodes command line arguments with 'utf-8' rather than the locale encoding.

By default, tarfile{.interpreted-text role="mod"} uses 'utf-8' encoding on Windows (instead of 'mbcs') and the 'surrogateescape' error handler on all operating systems.

Documentation

The documentation continues to be improved.

• A table of quick links has been added to the top of lengthy sections such as built-in-funcs{.interpreted-text role="ref"}. In the case of itertools{.interpreted-text role="mod"}, the links are accompanied by tables of cheatsheet-style summaries to provide an overview and memory jog without having to read all of the docs.

• In some cases, the pure Python source code can be a helpful adjunct to the documentation, so now many modules now feature quick links to the latest version of the source code. For example, the functools{.interpreted-text role="mod"} module documentation has a quick link at the top labeled:

  Source code Lib/functools.py{.interpreted-text role="source"}.

  (Contributed by Raymond Hettinger; see rationale.)

• The docs now contain more examples and recipes. In particular, re{.interpreted-text role="mod"} module has an extensive section, re-examples{.interpreted-text role="ref"}. Likewise, the itertools{.interpreted-text role="mod"} module continues to be updated with new itertools-recipes{.interpreted-text role="ref"}.

• The datetime{.interpreted-text role="mod"} module now has an auxiliary implementation in pure Python. No functionality was changed. This just provides an easier-to-read alternate implementation.

  (Contributed by Alexander Belopolsky in 9528{.interpreted-text role="issue"}.)

• The unmaintained Demo{.interpreted-text role="file"} directory has been removed. Some demos were integrated into the documentation, some were moved to the Tools/demo{.interpreted-text role="file"} directory, and others were removed altogether.

  (Contributed by Georg Brandl in 7962{.interpreted-text role="issue"}.)

IDLE

• The format menu now has an option to clean source files by stripping trailing whitespace.

  (Contributed by Raymond Hettinger; 5150{.interpreted-text role="issue"}.)

• IDLE on Mac OS X now works with both Carbon AquaTk and Cocoa AquaTk.

  (Contributed by Kevin Walzer, Ned Deily, and Ronald Oussoren; 6075{.interpreted-text role="issue"}.)

Code Repository

In addition to the existing Subversion code repository at https://svn.python.org there is now a Mercurial repository at https://hg.python.org/\ .

After the 3.2 release, there are plans to switch to Mercurial as the primary repository. This distributed version control system should make it easier for members of the community to create and share external changesets. See 385{.interpreted-text role="pep"} for details.

To learn to use the new version control system, see the Quick Start or the Guide to Mercurial Workflows.

Build and C API Changes

Changes to Python's build process and to the C API include:

• The idle, pydoc and 2to3 scripts are now installed with a version-specific suffix on make altinstall (10679{.interpreted-text role="issue"}).

• The C functions that access the Unicode Database now accept and return characters from the full Unicode range, even on narrow unicode builds (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others). A visible difference in Python is that unicodedata.numeric{.interpreted-text role="func"} now returns the correct value for large code points, and repr{.interpreted-text role="func"} may consider more characters as printable.

  (Reported by Bupjoe Lee and fixed by Amaury Forgeot D'Arc; 5127{.interpreted-text role="issue"}.)

• Computed gotos are now enabled by default on supported compilers (which are detected by the configure script). They can still be disabled selectively by specifying --without-computed-gotos.

  (Contributed by Antoine Pitrou; 9203{.interpreted-text role="issue"}.)

• The option --with-wctype-functions was removed. The built-in unicode database is now used for all functions.

  (Contributed by Amaury Forgeot D'Arc; 9210{.interpreted-text role="issue"}.)

• Hash values are now values of a new type, Py_hash_t{.interpreted-text role="c:type"}, which is defined to be the same size as a pointer. Previously they were of type long, which on some 64-bit operating systems is still only 32 bits long. As a result of this fix, set{.interpreted-text role="class"} and dict{.interpreted-text role="class"} can now hold more than 2**32 entries on builds with 64-bit pointers (previously, they could grow to that size but their performance degraded catastrophically).

  (Suggested by Raymond Hettinger and implemented by Benjamin Peterson; 9778{.interpreted-text role="issue"}.)

• A new macro !Py_VA_COPY{.interpreted-text role="c:macro"} copies the state of the variable argument list. It is equivalent to C99 va_copy but available on all Python platforms (2443{.interpreted-text role="issue"}).

• A new C API function !PySys_SetArgvEx{.interpreted-text role="c:func"} allows an embedded interpreter to set sys.argv{.interpreted-text role="data"} without also modifying sys.path{.interpreted-text role="data"} (5753{.interpreted-text role="issue"}).

• !PyEval_CallObject{.interpreted-text role="c:func"} is now only available in macro form. The function declaration, which was kept for backwards compatibility reasons, is now removed -- the macro was introduced in 1997 (8276{.interpreted-text role="issue"}).

• There is a new function PyLong_AsLongLongAndOverflow{.interpreted-text role="c:func"} which is analogous to PyLong_AsLongAndOverflow{.interpreted-text role="c:func"}. They both serve to convert Python int{.interpreted-text role="class"} into a native fixed-width type while providing detection of cases where the conversion won't fit (7767{.interpreted-text role="issue"}).

• The PyUnicode_CompareWithASCIIString{.interpreted-text role="c:func"} function now returns not equal if the Python string is NUL terminated.

• There is a new function PyErr_NewExceptionWithDoc{.interpreted-text role="c:func"} that is like PyErr_NewException{.interpreted-text role="c:func"} but allows a docstring to be specified. This lets C exceptions have the same self-documenting capabilities as their pure Python counterparts (7033{.interpreted-text role="issue"}).

• When compiled with the --with-valgrind option, the pymalloc allocator will be automatically disabled when running under Valgrind. This gives improved memory leak detection when running under Valgrind, while taking advantage of pymalloc at other times (2422{.interpreted-text role="issue"}).

• Removed the O? format from the PyArg_Parse functions. The format is no longer used and it had never been documented (8837{.interpreted-text role="issue"}).

There were a number of other small changes to the C-API. See the Misc/NEWS file for a complete list.

Also, there were a number of updates to the Mac OS X build, see Mac/BuildScript/README.txt for details. For users running a 32/64-bit build, there is a known problem with the default Tcl/Tk on Mac OS X 10.6. Accordingly, we recommend installing an updated alternative such as ActiveState Tcl/Tk 8.5.9. See https://www.python.org/download/mac/tcltk/ for additional details.

Porting to Python 3.2

This section lists previously described changes and other bugfixes that may require changes to your code:

• The configparser{.interpreted-text role="mod"} module has a number of clean-ups. The major change is to replace the old !ConfigParser{.interpreted-text role="class"} class with long-standing preferred alternative !SafeConfigParser{.interpreted-text role="class"}. In addition there are a number of smaller incompatibilities:

  • The interpolation syntax is now validated on ~configparser.ConfigParser.get{.interpreted-text role="meth"} and ~configparser.ConfigParser.set{.interpreted-text role="meth"} operations. In the default interpolation scheme, only two tokens with percent signs are valid: %(name)s and %%, the latter being an escaped percent sign.
  • The ~configparser.ConfigParser.set{.interpreted-text role="meth"} and ~configparser.ConfigParser.add_section{.interpreted-text role="meth"} methods now verify that values are actual strings. Formerly, unsupported types could be introduced unintentionally.
  • Duplicate sections or options from a single source now raise either ~configparser.DuplicateSectionError{.interpreted-text role="exc"} or ~configparser.DuplicateOptionError{.interpreted-text role="exc"}. Formerly, duplicates would silently overwrite a previous entry.
  • Inline comments are now disabled by default so now the ; character can be safely used in values.
  • Comments now can be indented. Consequently, for ; or # to appear at the start of a line in multiline values, it has to be interpolated. This keeps comment prefix characters in values from being mistaken as comments.
  • "" is now a valid value and is no longer automatically converted to an empty string. For empty strings, use "option =" in a line.

• The !nntplib{.interpreted-text role="mod"} module was reworked extensively, meaning that its APIs are often incompatible with the 3.1 APIs.

• bytearray{.interpreted-text role="class"} objects can no longer be used as filenames; instead, they should be converted to bytes{.interpreted-text role="class"}.

• The !array.tostring{.interpreted-text role="meth"} and !array.fromstring{.interpreted-text role="meth"} have been renamed to array.tobytes() <array.array.tobytes>{.interpreted-text role="meth"} and array.frombytes() <array.array.frombytes>{.interpreted-text role="meth"} for clarity. The old names have been deprecated. (See 8990{.interpreted-text role="issue"}.)

• PyArg_Parse*() functions:

  • "t#" format has been removed: use "s#" or "s*" instead
  • "w" and "w#" formats has been removed: use "w*" instead

• The !PyCObject{.interpreted-text role="c:type"} type, deprecated in 3.1, has been removed. To wrap opaque C pointers in Python objects, the PyCapsule{.interpreted-text role="c:type"} API should be used instead; the new type has a well-defined interface for passing typing safety information and a less complicated signature for calling a destructor.

• The !sys.setfilesystemencoding{.interpreted-text role="func"} function was removed because it had a flawed design.

• The random.seed{.interpreted-text role="func"} function and method now salt string seeds with an sha512 hash function. To access the previous version of seed in order to reproduce Python 3.1 sequences, set the version argument to 1, random.seed(s, version=1).

• The previously deprecated !string.maketrans{.interpreted-text role="func"} function has been removed in favor of the static methods bytes.maketrans{.interpreted-text role="meth"} and bytearray.maketrans{.interpreted-text role="meth"}. This change solves the confusion around which types were supported by the string{.interpreted-text role="mod"} module. Now, str{.interpreted-text role="class"}, bytes{.interpreted-text role="class"}, and bytearray{.interpreted-text role="class"} each have their own maketrans and translate methods with intermediate translation tables of the appropriate type.

  (Contributed by Georg Brandl; 5675{.interpreted-text role="issue"}.)

• The previously deprecated !contextlib.nested{.interpreted-text role="func"} function has been removed in favor of a plain with{.interpreted-text role="keyword"} statement which can accept multiple context managers. The latter technique is faster (because it is built-in), and it does a better job finalizing multiple context managers when one of them raises an exception:

  with open('mylog.txt') as infile, open('a.out', 'w') as outfile:
      for line in infile:
          if '<critical>' in line:
              outfile.write(line)
  

  (Contributed by Georg Brandl and Mattias Brändström; appspot issue 53094.)

• struct.pack{.interpreted-text role="func"} now only allows bytes for the s string pack code. Formerly, it would accept text arguments and implicitly encode them to bytes using UTF-8. This was problematic because it made assumptions about the correct encoding and because a variable-length encoding can fail when writing to fixed length segment of a structure.

  Code such as struct.pack('<6sHHBBB', 'GIF87a', x, y) should be rewritten with to use bytes instead of text, struct.pack('<6sHHBBB', b'GIF87a', x, y).

  (Discovered by David Beazley and fixed by Victor Stinner; 10783{.interpreted-text role="issue"}.)

• The xml.etree.ElementTree{.interpreted-text role="class"} class now raises an xml.etree.ElementTree.ParseError{.interpreted-text role="exc"} when a parse fails. Previously it raised an xml.parsers.expat.ExpatError{.interpreted-text role="exc"}.

• The new, longer str{.interpreted-text role="func"} value on floats may break doctests which rely on the old output format.

• In subprocess.Popen{.interpreted-text role="class"}, the default value for close_fds is now True under Unix; under Windows, it is True if the three standard streams are set to None, False otherwise. Previously, close_fds was always False by default, which produced difficult to solve bugs or race conditions when open file descriptors would leak into the child process.

• Support for legacy HTTP 0.9 has been removed from urllib.request{.interpreted-text role="mod"} and http.client{.interpreted-text role="mod"}. Such support is still present on the server side (in http.server{.interpreted-text role="mod"}).

  (Contributed by Antoine Pitrou, 10711{.interpreted-text role="issue"}.)

• SSL sockets in timeout mode now raise socket.timeout{.interpreted-text role="exc"} when a timeout occurs, rather than a generic ~ssl.SSLError{.interpreted-text role="exc"}.

  (Contributed by Antoine Pitrou, 10272{.interpreted-text role="issue"}.)

• The misleading functions !PyEval_AcquireLock{.interpreted-text role="c:func"} and !PyEval_ReleaseLock{.interpreted-text role="c:func"} have been officially deprecated. The thread-state aware APIs (such as PyEval_SaveThread{.interpreted-text role="c:func"} and PyEval_RestoreThread{.interpreted-text role="c:func"}) should be used instead.

• Due to security risks, !asyncore.handle_accept{.interpreted-text role="func"} has been deprecated, and a new function, !asyncore.handle_accepted{.interpreted-text role="func"}, was added to replace it.

  (Contributed by Giampaolo Rodola in 6706{.interpreted-text role="issue"}.)

• Due to the new GIL{.interpreted-text role="term"} implementation, !PyEval_InitThreads{.interpreted-text role="c:func"} cannot be called before Py_Initialize{.interpreted-text role="c:func"} anymore.