############################################################ Utilities for building wheels that use third-party libraries ############################################################

Update: Uploads, Rackspace, Anaconda

The original Multibuild default was to upload wheels to a Rackspace container, where Rackspace kindly donated the hosting to the Scikit-learn team. Rackspace finally stopped subsidizing this container. Some projects using Multibuild have moved to using https://anaconda.org/scientific-python-nightly-wheels, see SPEC04_ for more info.

.. _SPEC04: https://scientific-python.org/specs/spec-0004/

Introduction

A set of scripts to automate builds of macOS and manylinux wheels. It will work with windows, but that is not the prime target.

The CI scripts are designed to build and test:

64-bit macOS x86_64 wheels built for macOS 10.9+
64-bit macOS arm64 wheels built for macOS 10.6+
manylinux wheels in all the variations

You can build and test against all released versions of CPython and PyPy.

How does it work?

Multibuild is a series of bash scripts that define bash functions to build and test wheels.

Configuration is by overriding the default build function, and defining a test function.

The bash scripts are layered, in the sense that they are composed of a number of scripts which are sourced in sequence, each one potentially overriding previous ones.

macOS

The following bash scripts are sourced in this order::

multibuild/common_utils.sh
multibuild/osx_utils.sh
env_vars.sh
multibuild/configure_build.sh
multibuild/library_builders.sh
config.sh

See multibuild/travis_osx_steps.sh

The macOS build / test phases run on the VM started by CI. Therefore any environment variable defined in the CI script or the bash shell scripts listed above are available for your build and test.

Build options are controlled mainly by the following environment variables:

MB_PYTHON_VERSION sets the Python version targeted: major.minor.patch for CPython, or pypy-major.minor for PyPy.
MB_PYTHON_OSX_VER sets the minimum macOS SDK version for any C extensions. For CPython targets it may be set to 10.6, 10.9 or 11.0, provided a corresponding Python build is available at python.org <https://www.python.org/downloads/mac-osx/>_. It defaults to the highest version available. It's ignored for PyPy targets.
PLAT sets the architectures built for any C extensions: x86_64 or arm64. It defaults to the same arches as the target Python version: arm64 for macOS 11.0; x86_64 for CPython macOS 10.9 or PyPy; and 64/32-bit for CPython 10.6.

In most cases it's best to rely on the defaults for MB_PYTHON_OSX_VER and PLAT, rather than setting them explicitly. Examples of exceptions to this guideline include:

setting MB_PYTHON_OSX_VER=10.6 to build a 10.6 CPython wheel for Python 2.7 (default for 2.7 is 10.9 64-bit)
setting MB_PYTHON_OSX_VER=10.6 and PLAT=x86_64 to build a 10.6 64-bit wheel.

The build_wheel function builds the wheel, and install_run function installs and tests it. Look in multibuild/common_utils.sh for default definitions of these functions. See below for more details, many of which are common to macOS and Linux.

Manylinux

The build phase is in a Manylinux Docker container, but the test phase is in a clean container.

Build phase

Specify the Manylinux version to build for with the MB_ML_VER environment variable. The default version is 2014. Versions that are currently valid are:

1 corresponding to manylinux1 (see PEP 513 <https://www.python.org/dev/peps/pep-0513>_).
2010 corresponding to manylinux2010 (see PEP 571 <https://www.python.org/dev/peps/pep-0571>_).
2014 corresponding to manylinux2014 and adds more architectures to PLAT (see PEP 599 <https://www.python.org/dev/peps/pep-0599>_).
_2_24 corresponding to manylinux_2_24 (see PEP 600 <https://www.python.org/dev/peps/pep-0600>_).
_2_28 corresponding to manylinux_2_28 (see PEP 600 <https://www.python.org/dev/peps/pep-0600>_).

The environment variable specified which Manylinux docker container you are building in.

The PLAT environment variable can be one of

x86_64, for 64-bit x86
i686, for 32-bit x86
s390x, for 64-bit s390x
ppc64le, for PowerPC
aarch64, for ARM
arm64, for Apple silicon
universal2, for both Apple silicon and 64-bit x86

The default is x86_64. Only x86_64 and i686 are valid on manylinux1 and manylinux2010.

multibuild/travis_linux_steps.sh defines the build_wheel function, which starts up the Manylinux1 Docker container to run a wrapper script multibuild/docker_build_wrap.sh, that (within the container) sources the following bash scripts::

multibuild/common_utils.sh
multibuild/manylinux_utils.sh
env_vars.sh
multibuild/configure_build.sh
multibuild/library_builders.sh
config.sh

See docker_build_wrap.sh to review the order of script sourcing.

See the definition of build_multilinux in multibuild/travis_linux_steps.sh for the environment variables passed from Travis CI to the Manylinux1 container.

Once in the container, after sourcing the scripts above, the wrapper runs the real build_wheel function, which now comes (by default) from multibuild/common_utils.sh.

Test phase

Specify the version to test with the DOCKER_TEST_IMAGE environment variable. The default version is dependent on MB_ML_LIBC and PLAT.

When MB_ML_LIBC is musllinux:

multibuild/alpine3.22_x86_64, when PLAT is x86_64
multibuild/alpine3.22_arm64v8, when PLAT is aarch64

Otherwise:

multibuild/noble_x86_64, when PLAT is x86_64
matthewbrett/trusty:32 when PLAT is i686 (Yes, an older image for 32-bit)
multibuild/noble_arm64v8 when PLAT is aarch64
multibuild/noble_ppc64le when PLAT is ppc64le
multibuild/noble_s390x when PLAT is s390x

Other valid values are any in https://quay.io/organization/pypa, using the correct platform code. Alternatively, you can use the substitution pattern multibuild/noble_{PLAT} in the .travis.yml file.

See multibuild/docker_test_wrap.sh.

multibuild/travis_linux_steps.sh defines the install_run function, which starts up the testing Docker container with the wrapper script multibuild/docker_test_wrap.sh. The wrapper script sources the following bash scripts::

multibuild/common_utils.sh
config.sh

See docker_test_wrap.sh for script source order.

See install_run in multibuild/travis_linux_steps.sh for the environment variables passed into the container.

It then (in the container) runs the real install_run command, which comes (by default) from multibuild/common_utils.sh.

Standard build and test functions

The standard build command is build_wheel. This is a bash function. By default the function that is run on macOS, and in the Manylinux container for the build phase, is defined in multibuild/common_utils.sh. You can override the default function in the project config.sh file (see below).

If you are building a wheel from PyPI, rather than from a source repository, you can use the build_index_wheel command, again defined in multibuild/common_utils.sh.

Typically, you can get away with leaving the default build_wheel / build_index_wheel functions to do their thing, but you may need to define a pre_build function in config.sh. The default build_wheel and build_index_wheel functions will call the pre_build function, if defined, before building the wheel, so pre_build is a good place to build any required libraries.

The standard test command is the bash function install_run. The version run on macOS and in the Linux testing container is also defined in multibuild/common_utils.sh. Typically, you do not override this function, but you in that case you will need to define a run_tests function, to run your tests, returning a non-zero error code for failure. The default install_run implementation calls the run_tests function, which you will likely define in config.sh. See the examples below for examples of less and more complicated builds, where the complicated builds override more of the default implementations.

To use these scripts

Make a repository for building wheels - e.g. https://github.com/MacPython/astropy-wheels - or in your case maybe https://github.com/your-org/your-project-wheels;
Add this (here) repository as a submodule::

git submodule add https://github.com/multi-build/multibuild.git
Add your own project repository as another submodule::

git submodule add https://github.com/your-org/your-project.git
For Travis CI, create a .travis.yml file, something like this::

env: global: - REPO_DIR=your-project # Commit from your-project that you want to build - BUILD_COMMIT=v0.1.0 # pip dependencies to build your project - BUILD_DEPENDS="cython numpy" # pip dependencies to test your project. Include any dependencies # that you need, that are also specified in BUILD_DEPENDS, this will be # a separate install. # Now see the Uploads section for the stuff you need to # upload your wheels after CI has built them.

You will likely prefer "language: generic" for travis configuration,

rather than, say "language: python". Multibuild doesn't use

Travis-prov

Multibuild

Install / Use

README

macOS

Manylinux

Build phase

Test phase

You will likely prefer "language: generic" for travis configuration,

rather than, say "language: python". Multibuild doesn't use

Travis-prov