Skip to content

Latest commit

 

History

History
344 lines (239 loc) · 14.3 KB

README.md

File metadata and controls

344 lines (239 loc) · 14.3 KB
Raw

python-build

python-build is a pyenv plugin that provides a pyenv install command to compile and install different versions of Python on UNIX-like systems.

You can also use python-build without pyenv in environments where you need precise control over Python version installation.

See the list of releases for changes in each version.

Installation

Installing as a pyenv plugin (recommended)

Since python-build is bundled with pyenv by default, you do not need to do anything.

Installing as a standalone program (advanced)

Installing python-build as a standalone program will give you access to the python-build command for precise control over Python version installation. If you have pyenv installed, you will also be able to use the pyenv install command.

git clone https://github.com/pyenv/pyenv.git
cd pyenv/plugins/python-build
./install.sh

This will install python-build into /usr/local. If you do not have write permission to /usr/local, you will need to run sudo ./install.sh instead. You can install to a different prefix by setting the PREFIX environment variable.

To update python-build after it has been installed, run git pull in your cloned copy of the repository, then re-run the install script.

Installing with Homebrew (for OS X users)

Mac OS X users can install python-build with the Homebrew package manager. This will give you access to the python-build command. If you have pyenv installed, you will also be able to use the pyenv install command.

This is the recommended method of installation if you installed pyenv with Homebrew.

brew install pyenv

Or, if you would like to install the latest development release:

brew install --HEAD pyenv

Usage

Before you begin, you should ensure that your build environment has the proper system dependencies for compiling the wanted Python Version (see our recommendations).

Using pyenv install with pyenv

To install a Python version for use with pyenv, run pyenv install with exact name of the version you want to install. For example,

pyenv install 2.7.4

Python versions will be installed into a directory of the same name under ~/.pyenv/versions.

To see a list of all available Python versions, run pyenv install --list. You may also tab-complete available Python versions if your pyenv installation is properly configured.

Using python-build standalone

If you have installed python-build as a standalone program, you can use the python-build command to compile and install Python versions into specific locations.

Run the python-build command with the exact name of the version you want to install and the full path where you want to install it. For example,

python-build 2.7.4 ~/local/python-2.7.4

To see a list of all available Python versions, run python-build --definitions.

Pass the -v or --verbose flag to python-build as the first argument to see what's happening under the hood.

Custom definitions

Both pyenv install and python-build accept a path to a custom definition file in place of a version name. Custom definitions let you develop and install versions of Python that are not yet supported by python-build.

See the python-build built-in definitions as a starting point for custom definition files.

Adding definitions with a Pyenv plugin

You can add your own definitions with a Pyenv plugin by placing them under $PYENV_ROOT/plugins/your_plugin_name/share/python-build.

Default build configuration

Without the user customizing the build with environment variables (see below), python-build builds Python with mostly default Configure options to maintain the principle of the least surprise.

The exceptions -- non-default options that are set by default -- are listed below:

Option/Behavior Rationale
--enable-shared is on by default. Pass --disable-shared to Configure options to override The official CPython Docker image uses it. It's required to embed CPython.
argument to --enable-universalsdk is ignored and set to /
--with-universal-archs defaults to universal2 on ARM64 architecture the only dual-architecture Macs in use today are Apple Silicon which can only build that one
argument to --enable-framework is ignored and set to a specific value CPython's build logic requires a very specific argument to avoid installing the Applications part globally
argument to --enable-unicode in non-MacOS is overridden to ucs4 for 2.x-3.3
MACOSX_DEPLOYMENT_TARGET defaults to the running MacOS version

Integration with 3rd-party package ecosystems

Homebrew

Homebrew is used to find dependency packages if brew is found on PATH:

  • In MacOS, or
  • If the running Pyenv itself is installed with Homebrew

Set PYTHON_BUILD_USE_HOMEBREW or PYTHON_BUILD_SKIP_HOMEBREW to override this default.

When Homebrew is used, its include and lib paths are added to compiler search path (the latter is also set as rpath), and also Python dependencies that are typically keg-only are searched for in the Homebrew installation and added individually.

NOTE: Homebrew is not used in Linux by default because it's rolling-release which causes a problem. Upgrading a Python dependency in Homebrew to a new major version (that brew does without warning) would break all Pyenv-managed installations that depend on it. You can use a community plugin fix-version to fix installations in such a case.

Portage

In FreeBSD, if pkg is on PATH, Ports are searched for some dependencies that Configure is known to not search for via pkg-config. (Later versions of CPython search for more packages via pkg-config so this may eventually become redundant.)

Special environment variables

You can set certain environment variables to control the build process.

  • TMPDIR sets the location where python-build stores temporary files.
  • PYTHON_BUILD_BUILD_PATH sets the location in which sources are downloaded and built. By default, this is a subdirectory of TMPDIR.
  • PYTHON_BUILD_CACHE_PATH, if set, specifies a directory to use for caching downloaded package files.
  • PYTHON_BUILD_MIRROR_URL overrides the default mirror URL root to one of your choosing.
  • PYTHON_BUILD_MIRROR_URL_SKIP_CHECKSUM, if set, does not append the SHA2 checksum of the file to the mirror URL.
  • PYTHON_BUILD_SKIP_MIRROR, if set, forces python-build to download packages from their original source URLs instead of using a mirror.
  • PYTHON_BUILD_HTTP_CLIENT, explicitly specify the HTTP client type to use. aria2, curl and wget are the supported values and by default, are searched in that order.
  • PYTHON_BUILD_CURL_OPTS, PYTHON_BUILD_WGET_OPTS, PYTHON_BUILD_ARIA2_OPTS pass additional parameters to the corresponding HTTP client.
  • PYTHON_BUILD_SKIP_HOMEBREW, if set, will not search for libraries installed by Homebrew when it would normally will.
  • PYTHON_BUILD_USE_HOMEBREW, if set, will search for libraries installed by Homebrew when it would normally not.
  • PYTHON_BUILD_HOMEBREW_OPENSSL_FORMULA, override the Homebrew OpenSSL formula to use.
  • PYTHON_BUILD_ROOT overrides the default location from where build definitions in share/python-build/ are looked up.
  • PYTHON_BUILD_DEFINITIONS can be a list of colon-separated paths that get additionally searched when looking up build definitions.
  • CC sets the path to the C compiler.
  • PYTHON_CFLAGS lets you pass additional options to the default CFLAGS. Use this to override, for instance, the -O3 option.
  • CONFIGURE_OPTS lets you pass additional options to ./configure.
  • MAKE lets you override the command to use for make. Useful for specifying GNU make (gmake) on some systems.
  • MAKE_OPTS (or MAKEOPTS) lets you pass additional options to make.
  • MAKE_INSTALL_OPTS lets you pass additional options to make install.
  • PYTHON_CONFIGURE_OPTS and PYTHON_MAKE_OPTS and PYTHON_MAKE_INSTALL_OPTS allow you to specify configure and make options for building CPython. These variables will be passed to Python only, not any dependent packages (e.g. libyaml).

Applying patches to Python before compiling

Both pyenv install and python-build support the --patch (-p) flag that signals that a patch from stdin should be applied to Python, Jython or PyPy source code before the ./configure and compilation steps.

Example usage:

# applying a single patch
$ pyenv install --patch 2.7.10 < /path/to/python.patch

# applying a patch from HTTP
$ pyenv install --patch 2.7.10 < <(curl -sSL http://git.io/python.patch)

# applying multiple patches
$ cat fix1.patch fix2.patch | pyenv install --patch 2.7.10

Building for maximum performance

Building CPython with --enable-optimizations will result in a faster interpreter at the cost of significantly longer build times. Most notably, this enables PGO (profile guided optimization). While your mileage may vary, it is common for performance improvement from this to be in the ballpark of 30%.

env PYTHON_CONFIGURE_OPTS='--enable-optimizations --with-lto' PYTHON_CFLAGS='-march=native -mtune=native' pyenv install 3.6.0

You can also customize the task used for profile guided optimization by setting the PROFILE_TASK environment variable, for instance, PROFILE_TASK='-m test.regrtest --pgo -j0' will run much faster than the default task.

Checksum verification

If you have the shasum, openssl, or sha256sum tool installed, python-build will automatically verify the SHA2 checksum of each downloaded package before installing it.

Checksums are optional and specified as anchors on the package URL in each definition. (All bundled definitions include checksums.)

Package download mirrors

python-build will first attempt to download package files from a mirror hosted on GitHub Pages. If this fails, it will fall back to the official URL specified in the definition file.

You can point python-build to another mirror by specifying the PYTHON_BUILD_MIRROR_URL environment variable.

Package mirror URLs are constructed by joining $PYTHON_BUILD_MIRROR_URL with the SHA2 checksum of the package file as specified in the URL in the installation script (the part after the hash sign). E.g.:

https://mycache.example.com/0419e9085bf51b7a672009b3f50dbf1859acdf18ba725d0ec19aa5c8503f0ea3

If you have replicated the directory structure of an official site, the easiest way to adapt would be to make symlinks at the mirror's root:

0419e9085bf51b7a672009b3f50dbf1859acdf18ba725d0ec19aa5c8503f0ea3 -> 3.10.10/Python-3.10.10.tar.xz

The rationale is to abstract away difference between directory structures of sites of various Python flavors and their occasional changes as well as to accomodate people who only wish to cache some select downloads. This also allows to mirror multiple sites at once.

If the mirror being used does not have the same checksum (e.g. with a pull-through cache like Artifactory), you can set the PYTHON_BUILD_MIRROR_URL_SKIP_CHECKSUM environment variable.

If you don't have an SHA2 program installed, python-build will skip the download mirror and use official URLs instead. You can force python-build to bypass the mirror by setting the PYTHON_BUILD_SKIP_MIRROR environment variable.

The official python-build download mirror is provided by GitHub Pages.

Package download cache

Python-build will keep a cache of downloaded package files at the location specified by the PYTHON_BUILD_CACHE_PATH environment variable if it exists. The default is ~/.pyenv/cache, so you can enable caching by just creating that directory.

The name of the would-be cached file is reported by Pyenv in the "Downloading <filename>..." message. It's possible to warm up the cache by manually putting the file there under an appropriate name.

Keeping the build directory after installation

Both python-build and pyenv install accept the -k or --keep flag, which tells python-build to keep the downloaded source after installation. This can be useful if you need to use gdb and memprof with Python.

Source code will be kept in a parallel directory tree ~/.pyenv/sources when using --keep with the pyenv install command. You should specify the location of the source code with the PYTHON_BUILD_BUILD_PATH environment variable when using --keep with python-build.

Getting Help

Please see the pyenv wiki for solutions to common problems.

If you can't find an answer on the wiki, open an issue on the issue tracker. Be sure to include the full build log for build failures.

Contributing

Testing new python versions

If you are contributing a new python version for python-build, you can test the build in a docker container based on Ubuntu 18.04.

With docker installed:

docker build -t my_container .
docker run my_container pyenv install <my_version>

To enter a shell which will allow you to build and then test a python version, replace the second line with

docker run -it my_container

The container will need to be rebuilt whenever you change the repo, but after the first build, this will be very fast, as the layer including the build dependencies will be cached.

Changes made inside the container will not be persisted.

To test all new versions since a particular revision (e.g. master), cd to the root of your pyenv repo, and run this script:

set -e
set -x

docker build -t pyenv-test-container .

git diff --name-only master \
  | grep '^plugins/python-build/share/python-build/' \
  | awk -F '/' '{print $NF}' \
  | xargs -I _ docker run pyenv-test-container pyenv install _
  • Build the docker image with the tag pyenv-test-container
  • Look for the names files changed since revision master
  • Filter out any which don't live where python-build keeps its build scripts
  • Look only at the file name (i.e. the python version name)
  • Run a new docker container for each, building that version