Copyright © 2006 Manoj Srivastava
| Revision History | ||
|---|---|---|
| Revision 1.0.4 | 25 Jul 2006 | |
Obstacles to conformance with the new python policy. While the new Python policy, specifically the "Packaged Modules" chapter, contains the elements that must be present in the debian/control filename, it is not very explicit about how the values are to be substituted. The Debian Wiki falls back on calling dh_python, which is not helpful in understanding the actual logic to be followed. This article is an attempt to correct this gap in documentation.
While trying to update SELinux packages, I ran across problems in trying to determine if my packages were complying with the new python policy: any practical tips for packaging generally devolved to the statement "Oh, just run dh_python". This is my attempt to offer more concrete tips for packaging. While this document started by reverse engineering dh_python, it has, thanks to help from various people more knowledgeable about Python than I, moved beyond that, and is closer to being a specification unfettered by the idiosyncrasies of current tools and implementations.
At any given time, the package python will represent the current default Debian Python version.
Apart from the default version, legacy versions of Python or beta versions of future releases may be included as well in the distribution, as long as they are needed by other packages, or as long as it seems reasonable to provide them. (Note: For the scope of this document, Python versions are synonymous to feature releases, i.e. Python 2.0 and 2.0.1 are subminor versions of the same Python version 2.0, but Python 2.1 and 2.2 are indeed different versions.)
For any version, the main package must be called pythonX.Y.
The set of currently supported python versions can be found in /usr/share/python/debian_defaults.
As long as you don't install other versions of Python in your path, Debian's Python versions won't be affected by a new version.
If you install a different subrelease of the version of python you've got installed, you'll need to be careful to install all the modules you use for that version of python too.
Python scripts depending on the default Python version or not depending on a specific Python version should use python (unversioned) as the interpreter name.
Python scripts that only work with a specific Python version must explicitly use the versioned interpreter name (pythonX.Y).
The preferred specification for the Python interpreter is /usr/bin/python or /usr/bin/pythonX.Y. This ensures that a Debian installation of python is used and all dependencies on additional python modules are met.
If a maintainer would like to provide the user with the possibility to override the Debian Python interpreter, he may want to use /usr/bin/env python or /usr/bin/env pythonX.Y. However this is not advisable as it bypasses Debian's dependency checking and makes the package vulnerable to incomplete local installations of python.
This consists of software directly called by an end user of external program, and is independently interpreted by the Python interpreter. Usually starts with the magic bytes #!, with the interpreter being /usr/bin/python* or /usr/bin/env python*.
This is code included in python "programs/scripts", and not invoked directly (serving as library modules in compiled languages).
Modules can be categorized under two orthogonal criteria: firstly, based on the whether or not they are implemented purely in python, like so:
These are python source code, to be interpreted by the Python interpreter just like program/script code is, and may work across many versions of Python.
Extensions are C code compiled and linked against a specific version of the libpython library, and so can only be used by one version of Python.
If a package provides any binary-independent modules (foo.py files), the corresponding bytecompiled modules (foo.pyc files) and optimized modules (foo.pyo files) must not ship in the package. Instead, they should be generated in the package's postinst, and removed in the package's prerm. The package's prerm has to make sure that both foo.pyc and foo.pyo are removed.
A package should only byte-compile the files which belong to the package.
The file /etc/python/debian_config allows configuration how modules should be byte-compiled. The postinst scripts should respect these settings.
Modules in private installation directories and in /usr/lib/site-python should be byte-compiled, when the default python version changes.
Another way of categorizing modules is based on whether or not they are available for use by third party scripts/modules.
Public modules are available for use in other Python scripts or modules using the import directive. They are installed in one of the directories:
This directory is reserved for official python modules. No other package apart from upstream official Python modules should install modules in this directory.
This is where most add-on modules live. Often, packages do not directly install modules here, but instead use utility packages like python-central and python-support to byte compile and install modules as needed.
Packages using python-support actually have their packages linked in from this directory, but no package should directly install modules there directly. See the documentation for python-support for details.
Packages may install public Python modules in directories specific to Python packaging utilities -- which specify the directories under which such modules should be droppped, and the the structure of these directories is defined by the utilities themselves. Please note that these directories are not in the path for Python, and are not available for modules to be imported from. At the time of writing, such uility specific directories include:
| /usr/share/pycentral |
| /usr/share/python-support |
The module search path for Debian has been amended to include a directory tree in /usr/local at the beginning of the path. By default, sys.path is searched in the following order:
| /usr/lib/pythonXY.zip |
| /usr/lib/pythonX.Y |
| /usr/lib/pythonX.Y/plat-linux2 |
| /usr/lib/pythonX.Y/lib-tk |
| /usr/lib/pythonX.Y/lib-dynload |
| /usr/local/lib/pythonX.Y/site-packages |
| /usr/lib/pythonX.Y/site-packages |
| /var/lib/python-support/pythonX.Y |
| /usr/lib/pythonX.Y/site-packages/module-dir |
| /usr/lib/site-python |
The use of the /usr/lib/site-python directory is deprecated. The directory may be dropped from the path in a future version. The /usr/lib/pythonXY.zip archive appeared in python2.3; it is not currently used in Debian.
Private modules are modules that are used internally by a program, and are generally only accessible to that specific program or suite of programs included in the same package. They are not intended to be shared with third party programs. They are installed in special directories, for example:
| /usr/lib/<package> |
| /usr/share/<package> |
| /usr/lib/games/<package> |
| /usr/share/games/<package> |
Python documentation is split out in separate packages pythonX.Y-doc. The package python-doc will always provide the documentation for the default Debian Python version.
TODO: Policy for documentation of third party packages.
The new policy is designed to reduce the load on people packaging python modules when one of the following events occur, and, by the dint of doing so, ease the transition that occur as new Python versions are introduced, old ones removed, and as the default version of Python changes, with minimal impact on the target system. In case of the following events:
Most pure Python modules with no restrictions on the version of Python supported, and those pure Python modules that only have a lower bound on the versions of python supported (for example, "2.3-", or "all"), would require no upload; they are merely recompiled using the rtupdate hooks, or utilities like python-central and python-support that hook themselves into rtupdate.
A number of public extension modules that do not have restrictions on the version of Python supported can just be recompiled. These include packages like:
Packages using $(shell pyversions -s) to determine which python versions to build for at build time, and which build depend upon python-all-dev would work seamlessly.
Some packages using CDBS will also work out of the box.
Packages using the distutils build system should also work.
Private modules are only built for one Python version, usually the default (pyversion -d). Private modules that do not have restrictions on the version of Python required, or which intelligently use $(shell pyversions -s) and their internal restrictions to discover which version of Python to build for would also merely need to be recompiled.
The only packages that will need manual upgrades, are packages that needed a range of python versions that did not contained the (at the upload time) current python version, and a couple of other cases, which should reduce the packages that need a non-automatized action from the maintainer (like a new source upload). This reduction of manual intervention should reduce the effort and time taken for changing the Python version or adding or dropping versions of Python much more painless than the old policy.
The new policy aims to reduce the pressure on packagers when th default Python version (what /usr/bin/python points to) changes. In that case, it tries to:
Any packages that do not need to change need not be rebuilt or uploaded. Pure Python modules, either public or private, should be byte compiled automatically on th target machine using the rtupdate mechanisms.
Packages that have already an extension installed for the new version of Python do not need to change either, since they would continue to keep working.
The new policy also aims to reduce the pressure when an old version of Python is dropped, since only packages directly dependent on that version will need a manual upgrade (this includes, for example, any package with scripts that use /usr/bin/pythonX.Y).
The new policy also reduces the numbers of packages in the archive, by supporting multiple versions of Python in the same binary package (at the cost of increased size of that one package, but it should still result in space saving.)
This presupposes that the package build mechanism utilizes the utility pyversions to get information about the supported Python versions, the currently installed Python executable versions, and the default Python runtime, instead of hard coding values in the debian/rules script. This script can also be used to parse the value of the XS-Python-Version field in the debian/control or debian/pyversions.
| Next | ||
| General packaging concepts |