lbuild: generic, modular code generation in Python 3

The Library Builder (pronounced lbuild) is a BSD licensed [Python 3 tool][python] for describing repositories containing modules which can copy or generate a set of files based on the user provided data and options.

lbuild allows splitting up complex code generation projects into smaller modules with configurable options, and provides for their transparent discovery, documentation and dependency management. Each module is written in Python 3 and declares its options and how to generate its content via the [Jinja2 templating engine][jinja2] or a file/folder copy.

You can [install lbuild via PyPi][pypi]: pip install lbuild

Projects using lbuild:

• [modm generates a HAL for thousands of embedded devices][modm] using lbuild and a data-driven code generation pipeline.
• [Taproot: a friendly control library and framework for RoboMaster robots][taproot] uses lbuild.
• [OUTPOST - Open modUlar sofTware PlatfOrm for SpacecrafT][outpost] uses lbuild to assemble an execution platform targeted at embedded systems running mission critical software.

The dedicated maintainer of lbuild is [@salkinium][salkinium].

Overview

Consider this repository:

 $ lbuild discover
Parser(lbuild)
╰── Repository(repo @ ../repo)
    ├── Option(option) = value in [value, special]
    ├── Module(repo:module)
    │   ├── Option(option) = yes in [yes, no]
    │   ├── Module(repo:module:submodule)
    │   │   ╰── Option(option) = REQUIRED in [1, 2, 3, 4, 5]
    │   ╰── Module(repo:module:submodule2)
    ╰── Module(modm:module2)

lbuild is called by the user with a configuration file which contains the repositories to scan, the modules to include and the options to configure them with:

<library>
  <repositories>
    <repository><path>../repo/repo.lb</path></repository>
  </repositories>
  <options>
    <option name="repo:option">special</option>
    <option name="repo:module:option">3</option>
  </options>
  <modules>
    <module>repo:module</module>
  </modules>
</library>

The repo.lb file is compiled by lbuild and the two functions init, prepare are called:

def init(repo):
    repo.name = "repo"
    repo.add_option(EnumerationOption(name="option",
                                      enumeration=["value", "special"],
                                      default="value"))

def prepare(repo, options):
    repo.find_modules_recursive("src")

This gives the repository a name and declares a string option. The prepare step adds all module files in the src/ folder.

Each module.lb file is then compiled by lbuild, and the three functions init, prepare and build are called:

def init(module):
    module.name = ":module"

def prepare(module, options):
    if options["repo:option"] == "special":
        module.add_option(EnumerationOption(name="option", enumeration=[1, 2, 3, 4, 5]))
        return True
    return False

def build(env):
    env.outbasepath = "repo/module"
    env.copy("static.hpp")
    for number in range(env["repo:module:option"]):
        env.template("template.cpp.in", "template_{}.cpp".format(number + 1))

The init step sets the module's name and its parent name. The prepare step then adds a EnumerationOption and makes the module available, if the repository option is set to "special". Finally in the build step, a number of files are generated based on the option's content.

The files are generated at the call-site of lbuild build which would then look something like this:

 $ ls
main.cpp        project.xml
 $ lbuild build
 $ tree
.
├── main.cpp
├── repo
│   ├── module
│   │   ├── static.hpp
│   │   ├── template_1.cpp
│   │   ├── template_2.cpp
│   │   └── template_3.cpp

Documentation

The above example shows a minimal feature set, but lbuild has a few more tricks up its sleeves. Let's have a look at the API in more detail with examples from [the modm repository][modm].

Command Line Interface

Before you can build a project you need to provide a configuration. lbuild aims to make discovery easy from the command line:

 $ lbuild --repository ../modm/repo.lb discover
Parser(lbuild)
╰── Repository(modm @ ../modm)   modm: a barebone embedded library generator
    ╰── Option(target) = REQUIRED in [at90can128, at90can32, at90can64, ...

This gives you an overview of the repositories and their options. In this case the modm:target repository option is required, so let's check that out:

 $ lbuild -r ../modm/repo.lb discover-options
modm:target = REQUIRED in [at90can128, at90can32, at90can64, at90pwm1, at90pwm161, at90pwm2,
                           ... a really long list ...
                           stm32l4s9vit, stm32l4s9zij, stm32l4s9zit, stm32l4s9ziy]

  Meta-HAL target device

You can then choose this repository option and discover the available modules for this specific repository option:

 $ lbuild -r ../modm/repo.lb --option modm:target=stm32f407vgt discover
Parser(lbuild)
╰── Repository(modm @ ../modm)   modm: a barebone embedded library generator
    ├── Option(target) = stm32f407vgt in [at90can128, at90can32, at90can64, ...]
    ├── Configuration(modm:disco-f407vg)
    ├── Module(modm:board)   Board Support Packages
    │   ╰── Module(modm:board:disco-f469ni)   STM32F469IDISCOVERY
    ├── Module(modm:build)   Build System Generators
    │   ├── PathOption(build.path) = build/parent-folder in [String]
    │   ├── Option(project.name) = parent-folder in [String]
    │   ╰── Module(modm:build:scons)  SCons Build Script Generator
    │       ├── Option(info.build) = no in [yes, no]
    │       ╰── Option(info.git) = Disabled in [Disabled, Info, Info+Status]
    ├── Module(modm:platform)   Platform HAL
    │   ├── Module(modm:platform:can)   Controller Area Network (CAN)
    │   │   ╰── Module(modm:platform:can:1)   Instance 1
    │   │       ├── Option(buffer.rx) = 32 in [1 .. 32 .. 65534]
    │   │       ╰── Option(buffer.tx) = 32 in [1 .. 32 .. 65534]
    │   ├── Module(modm:platform:core)   ARM Cortex-M Core
    │   │   ├── Option(allocator) = newlib in [block, newlib, tlsf]
    │   │   ├── Option(main_stack_size) = 3072 in [256 .. 3072 .. 65536]
    │   │   ╰── Option(vector_table_location) = rom in [ram, rom]

You can now discover all module options in more detail:

 $ lbuild -r ../modm/repo.lb -D modm:target=stm32f407vgt discover-options
modm:target = stm32f407vgt in [at90can128, at90can32, at90can64, ...]

  Meta-HAL target device

modm:build:build.path = build/parent-folder in [String]

  Path to the build folder

modm:build:project.name = parent-folder in [String]

  Project name for executable

Or check out specific module and option descriptions:

 $ lbuild -r ../modm/repo.lb -D modm:target=stm32f407vgt discover -n :build
>> modm:build

# Build System Generators

This parent module defines a common set of functionality that is independent of
the specific build system generator implementation.

>>>> modm:build:project.name  [StringOption]

# Project Name

The project name defaults to the folder name you're calling lbuild from.

Value: parent-folder
Inputs: [String]

>>>> modm:build:build.path  [StringOption]

# Build Path

The build path is defaulted to `build/{modm:build:project.name}`.

Value: build/parent-folder
Inputs: [String]

The complete lbuild command line interface is available with lbuild -h.

Configuration

Even though lbuild can be configured sorely via the command line, it is strongly recommended to create a configuration file (default is project.xml) which lbuild will search for in the current working directory.

<library>
  <repositories>
    <!-- Declare all your repository locations relative to this file here -->
    <repository><path>path/to/repo.lb</path></repository>
    <!-- You can also use environment variables in all nodes -->
    <repository><path>${PROJECTHOME}/repo2.lb</path></repository>
    <!-- You can also search for repository files -->
    <glob>ext/**/repo.lb</glob>
  </repositories>
  <!-- You can also inherit from another configfile. The options you specify
       here will be overwritten. -->
  <extends>path/to/config.xml</extends>
  <!-- A repository may provide aliases for configurations, so that you can
       use a string as well, instead of a path. This saves you from knowing
       exactly where the configuration file is stored in the repo.
       See also `repo.add_configuration(...)`. -->
  <extends>repo:name_of_config</extends>
  <!-- A configuration alias may also be versioned -->
  <extends>repo:name_of_config:specific_version</extends>
  <!-- You can declare the *where* the output should be generated, default is cwd -->
  <outpath>generated/folder</outpath>
  <options>
    <!-- Options are treated as key-value pairs -->
    <option name="repo:repo_option_name">value</option>
    <!-- An option set is the only one allowing multiple values -->
    <option name="repo:module:module_option_name">set, options, may, contain, commas</option>
  </options>
  <modules>
    <!-- You only need to declare the modules you are actively using.
         The dependencies are automatically resolved by lbuild. -->
    <module>repo:module</module>
    <module>repo:other_module:submodule</module>
  </modules>
</library>

On startup, lbuild will search the current working directory upwards for one or more lbuild.xml files, which if found, are used as the base configuration, inherited by all other configurations. This is very useful when several projects all require the same repositories, and you don't want to specify each repository path for each project.

<library>
  <repositories>
    <repository><path>path/to/common/repo.lb</path></repository>
  </repositories>
  <modules>
    <module>repo:module-required-by-all</module>
  </modules>
</library>

In the simplest case your project just <extends> this base config.

<lib