Wagon

A wagon (also spelt waggon in British and Commonwealth English) is a heavy four-wheeled vehicle pulled by draught animals, used for transporting goods, commodities, agricultural materials, supplies, and sometimes people. Wagons are distinguished from carts, which have two wheels, and from lighter four-wheeled vehicles primarily for carrying people, such as carriages.

or.. it is just a set of (Python) Wheels.

NOTE: To accommodate for the inconsistencies between wagon and pip, and to allow for additional required functionality, we will have to perform breaking changes until we can release v1.0.0. Please make sure you hardcode your wagon versions up until then.

Incentive

Cloudify Plugins are packaged as sets of Python Wheels in tar.gz/zip archives and so we needed a tool to create such entities; hence, Wagon.

Requirements

• Wagon requires pip 1.4+ to work as this is the first version of pip to support Wheels.
• Wagon supports Linux, Windows and OSX on Python 2.7 and 3.4+. Python 2.5 will not be supported as it is not supported by pip. Python 2.6.x is not longer supported as wheel itself doesn't support it.
• Wagon is currently tested on both Linux and Windows (via Travis and AppVeyor).
• To be able to create Wagons of Wheels which include C extensions on Windows, you must have the C++ Compiler for Python installed.
• To be able to create Wagons of Wheels which include C extensions on Linux or OSX, you must have the required compiler installed depending on your base distro. Usually:
  • RHEL based required gcc and python-devel.
  • Debian based require gcc and python-dev.
  • Other linux distributions will usually require gcc but might require additional packages.
  • OSX requires gcc.

Installation

pip install wagon

# latest development version
pip install http://github.com/cloudify-cosmo/wagon/archive/master.tar.gz

Backward Compatilibity

NOTE: pip 10.x breaks wagon<=0.7.0 due to the removal of the --use-wheel. If you're using pip>=10.x, please make sure you use wagon>=0.8.0. Also, if you're using pip<=7.x, use wagon<=0.7.0.

NOTE: wagon>=0.7.0 drops support for Python 2.6, 3.2 and 3.3 since Wheel itself no longer supports these versions. Please use wagon<=0.6.1 if you still need to support those versions.

Usage

NOTE: Currently, Wagon allows to pass arbitrary args to pip wheel and pip install. The way in which this is implemented is inconsistent with pip's implementation (wagon just allows passing a -a flag for all args.) This will be changed in the future to correspond to pip's implementation. See https://github.com/cloudify-cosmo/wagon/issues/70 for more information.

$ wagon
usage: wagon [-h] [-v] {create,install,validate,show,repair} ...

Create and install wheel based packages with their dependencies

positional arguments:
  {create,install,validate,show,repair}
    create              Create a Wagon archive
    install             Install a Wagon archive
    validate            Validate a wagon archive
    show                Print out the metadata of a wagon
    repair              Repair a Wagon archive

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Set verbose logging level (default: False)

...

Create Packages

$ wagon create flask
...

Creating archive for flask...
Retrieving source...
Source is: Flask
Downloading Wheels for Flask...
Collecting Flask
Using cached Flask-0.12-py2.py3-none-any.whl
Saved /tmp/tmpcYHwh0/Flask/wheels/Flask-0.12-py2.py3-none-any.whl
Collecting itsdangerous>=0.21 (from Flask)
Saved /tmp/tmpcYHwh0/Flask/wheels/itsdangerous-0.24-cp27-none-any.whl
Collecting click>=2.0 (from Flask)
Using cached click-6.7-py2.py3-none-any.whl
Saved /tmp/tmpcYHwh0/Flask/wheels/click-6.7-py2.py3-none-any.whl
...
Skipping MarkupSafe, due to already being wheel.
Platform is: linux_x86_64
Generating Metadata...
Writing metadata to file: /tmp/tmpcYHwh0/Flask/package.json
Creating tgz archive: ./Flask-0.12-py27-none-linux_x86_64.wgn...
Removing work directory...
Wagon created successfully at: ./Flask-0.12-py27-none-linux_x86_64.wgn

...

Requirement Files

NOTE: Beginning with Wagon 0.5.0, Wagon no longer looks up requirement files within archives or in the local directory when creating wagons. You must expclitly specify requirement files.

You can provide multiple requirement files to be resolved by using the -r flag (multiple times).

Editable Mode

Wagon doesn't currently provide a way for packaging packages that are in editable mode. So, for instance, providing a dev-requirements file which contains a -e DEPENDENCY requirement will not be taken into consideration. This is not related to wagon but rather to the default pip wheel implementation stating that it will be "Skipping bdist_wheel for #PACKAGE#, due to being editable".

Install Packages

$ wagon install Flask-0.12-py27-none-linux_x86_64.wgn
...

Installing Flask-0.12-py27-none-linux_x86_64.wgn
Retrieving source...
Extracting tgz Flask-0.12-py27-none-linux_x86_64.wgn to /tmp/tmplXv6Fi...
Source is: /tmp/tmplXv6Fi/Flask
Validating Platform linux_x86_64 is supported...
Installing Flask...
Installing within current virtualenv
Collecting Flask
Collecting itsdangerous>=0.21 (from Flask)
...
Installing collected packages: itsdangerous, Werkzeug, Flask
Successfully installed Flask-0.12 Werkzeug-0.11.15 itsdangerous-0.24

...

NOTE: --pre is appended to the installation command to enable installation of prerelease versions.

Installing Manually

While wagon provides a generic way of installing wagon created archives, you might not want to use the installer as you might not wish to install wagon on your application servers. Installing the package manually via pip is as easy as running (for example):

# For Linux (Windows wagon archives are zip files)
tar -xzvf ./Flask-0.12-py27-none-linux_x86_64.wgn
pip install --no-index --find-links Flask/wheels flask

Validate Packages

The validate function provides shallow validation of a Wagon archive. Basically, that all required wheels for a package are present and that the package is installable.

This shallow validation should, at the very least, verify that a Wagon archive is not corrupted. Note that the --validate flag provided with the create function uses this same validation method. Also note that validation must take place only on an OS distribution which supports the wagon archive if it contains C extensions. For instance, a win32 specific wagon archive will fail to validate on a Linux machine.

venv Python's stdlib module must be installed for Wagon to be able to validate an archive to not pollute the current environment.

$ wagon validate Flask-0.12-py27-none-linux_x86_64.wgn
...

Validating Flask-0.12-py27-none-linux_x86_64.wgn
Retrieving source...
Extracting tgz Flask-0.12-py27-none-linux_x86_64.wgn to /tmp/tmp2gqpy1...
Source is: /tmp/tmp2gqpy1/Flask
Verifying that all required files exist...
Testing package installation...
Creating Virtualenv /tmp/tmpdPNDIi...
Using real prefix '/usr'
New python executable in /tmp/tmpdPNDIi/bin/python2
Also creating executable in /tmp/tmpdPNDIi/bin/python
Installing setuptools, pip, wheel...done.
Installing /tmp/tmp2gqpy1/Flask
Retrieving source...
Source is: /tmp/tmp2gqpy1/Flask
Validating Platform linux_x86_64 is supported...
Installing Flask...
Collecting Flask
...
Installing collected packages: itsdangerous, click, MarkupSafe, Jinja2, Werkzeug, Flask
Successfully installed Flask-0.12 Jinja2-2.9.2 MarkupSafe-0.23 Werkzeug-0.11.15 click-6.7 itsdangerous-0.24
Package Flask is installed in /tmp/tmpdPNDIi
Validation Passed!

...

Show Metadata

Given a Wagon archive, this will print its metadata.

$ wagon show Flask-0.12-py27-none-linux_x86_64.wgn
...

{
    "archive_name": "Flask-0.12-py27-none-linux_x86_64.wgn",
    "build_server_os_properties": {
        "distribution": "antergos",
        "distribution_release": "archcode",
        "distribution_version": ""
    },
    "created_by_wagon_version": "0.6.0",
    "package_name": "Flask",
    "package_source": "flask",
    "package_version": "0.12",
    "supported_platform": "linux_x86_64",
    "supported_python_versions": [
        "py27"
    ],
    "wheels": [
        "MarkupSafe-0.23-cp27-cp27mu-linux_x86_64.whl",
        "Werkzeug-0.11.15-py2.py3-none-any.whl",
        "Jinja2-2.9.2-py2.py3-none-any.whl",
        "click-6.7-py2.py3-none-any.whl",
        "itsdangerous-0.24-cp27-none-any.whl",
        "Flask-0.12-py2.py3-none-any.whl"
    ]
}

...

Repair Wagon

auditwheel is a tool (currently under development) provided by pypa t