Setup.shl: A simple Bash library to replace Makefiles

make (and similar dependency-chasing tools, such as SCons, Rake, Waf, Ant, Maven, Gradle et al) offer very useful primitives for building complex file hierarchies from a small set of source files, while minimizing the amount of work needed to rebuild after a small change.

Setup.shl tries to offer the same basic set of features, as a (mostly pure) Bash library. It supports parallel compilation with buffered outputs, continuous builds, nested builds, as well as a new kind of dependency, all in 20kB of Bash code. Other than that, it tries to be as easy to use as possible.

Installing and using Setup.shl

Since it is just a Bash script at its core, you can start using Setup.shl from your shell by simply setting the SETUP_INSTALL_DIR variable to the root of this project, and sourcing the lib/setup.shl file (if you are using Bash, that is).

Even if it is mostly Bash, the core Setup.shl still needs a few utilities to function, which are listed below :

• mkfifo to create the pipes used to communicate with worker subshells
• mktemp to create a temporary directory used to hold setup-specific state information
• date to query files for their timestamps and get the system date
• GNU getopt, only if you are using bin/setup, to parse command-line arguments
• optionally, if cc is installed on your system, it is used to speed up some parts of the build process.

This file defines two functions, prepare and setup (and a third, teardown, if you want to perform continuous builds), whose jobs are to respectively prepare computations and run them.

This repository also provides a bin/setup executable that does something similar to the make tool : it searches a file named Setup in the current directory or its parents, and runs that file in an environment where the Setup.shl library was already sourced. It can also optionally, using the --watch option, keep its eye on all source files and trigger a new build every time they are written to.

This project provides a Setup file to illustrate its own usage. Installing Setup.shl can be done by running bin/setup and installing the resulting archive (called .pkg.tar.gz because I like my generated files to be hidden) to the root of your filesystem :

bin/setup package
sudo tar -xvzf .pkg.tar.gz -C /

The prepare function

Usage: prepare FILE... = COMMAND (-OPT|@FILE|FILE)...

This function declares that the FILEs on the left of the = are the result of the application of the COMMAND to its arguments. It does not itself run the command.

An argument can take three shapes :

• starting with a - indicates a flag argument, which doesn't need to describe a file
• starting with a @ indicates a splice dependency, which is described in more detail here
• otherwise, it is a simple dependency that triggers the command when it becomes newer than any of the FILEs

The setup function

Usage: setup TARGET...

This function computes all the files that have been prepared before it, in addition to the TARGETs passed as arguments.

Like make, setup uses timestamps to avoid wastefully recompiling when a file is already more recent than its dependencies.

The [bin/setup](bin/setup) interpreter automatically calls this function after loading a Setup script, so you won't usually need to call it yourself. It may be useful if you decide to write your own interpreter.

The teardown function

Usage: teardown FILE...

This function makes the build system treat the FILEs as though they had just been modified. Subsequent calls to setup will rebuild every intermediate target that depends on those FILEs (although the FILEs themselves are considered up-to-date and won't be rebuilt).

Other useful functions and variables

Alongside those two main functions, Setup.shl also allows minor configurations to take place, by using the following functions and variables :

• the SETUP_JOBS variable can be set at any point in the script to the desired number of parallel jobs to run (default: 8) on the next call to setup

• the Setup.params PARAM... function prints the value of the first PARAM that was specified on the command-line, if it exists. If that PARAM starts with -, it is not printed out. The function exits non-zero if none of the PARAMs were specified on the command-line. As such it can be used to define build flags, like so :

  if Setup.params -install; then
     ...
  fi
  

  Coupled with an assignment, it can also retrieve a parameter's value in a variable :

  if target="$(Setup.params target)"; then
      # The 'target' param was specified, and is now in the 'target' variable
      ...
  fi
  

  These flag can then be triggered by running the commands setup install or setup target=TARGET (or setup install target=TARGET for both).

• the Setup.use MODULE... function loads modules from the $SETUP_INSTALL_DIR/lib/setup.d/MODULE.shl files. It is an almost trivial wrapper around source.

  After loading a module, it also tries to sources a local module file from .setup/lib/MODULE.shl if it exists, allowing you to override some module-specific functions on a per-project basis.

• the Setup.load SETUP_FILE PARAM... function loads another Setup file in the current context, in order to use its preparations as dependencies further down the line.

  Each PARAM can be what you would specify on the command-line for a standalone invocation, such as install or target=X (see the Setup.params function for more information), and is made available alongside the PARAMs of the current context, overriding them when they already exist.

• the Setup.hook FUNCTION... can be used to define new automatic dependency generators.

  A generator is a function which take a single file name as argument, and should prepare this file if it knows how. Otherwise, it should return non-zero to signal to try another generator.

• the Setup.state-file NAME function specifies an optional state file, which is used by Setup.shl to remember information from one build to the next and speed up dependency resolution upon successive invocations of setup.

  This file is created automatically by Setup.shl and can safely be deleted if it causes problems (it shouldn't but there are always exceptions).

  Warning: the prepare function doesn't do anything if a file is already known to the build system. When using a state file, that means that once a file is specified, you can no longer change the command it is associated with, or its arguments, even in subsequent builds (splice arguments are still recomputed correctly, though). If that happens, simply delete the state file, and restart the build to acknowledge the new dependency graph.

Automatic targets via dependency hooks

Other build tools usually provide a sort of "wildcard target" to avoid repetition, as in :

%.o: %.c
     ...

Setup.shl is no exception. You can declare your own callbacks to prepare files whenever they are needed, by using the Setup.hook function. For example, the following code would be equivalent to the above rule :

Setup.hook C.auto_o
function C.auto_o() {
    case "$1" in
        *.o) prepare "$1" = CC "${1/%.o/.c}";;
        *) return 1;;
    esac
}

It doesn't look as pretty, but it is a much more powerful way to describe automatic dependencies, as it allows the full power of Bash to be brought forth to take advantage of contextual information.

Although, for simple cases like the above, there is a prepare-match function that can be used like so :

prepare-match '(.*)\.o' = CC '$1.c' @'$1.includes'

The first parameter is a regex, which is used to match the file name, and every parameter after the equal sign can use the regex matches as positional parameters (quoted, because they are not in scope when we define the rule).

For more example of automatic dependencies, visit the lib/setup.d directory.

What Setup can do

<a name="splice-dependencies"></a>

Splice dependencies for a more expressive build process

The complexities of some build processes are not accurately captured by the dependency model of Make-like tools, specifically automatically-generated dependencies. As an example, consider the following example :

file: test.c

#include "test.h"

int main() { printf("%d\n",f()); }

file: test.h

#include "A.h"

A_type f(void);

file: A.h

typedef int A_type;

file: Makefile

%.o: %.c ???
    gcc -c $< -o $@

With generic targets, Make offers no simple way for test.o to know that it depends on A.h, or even test.h. We would like to be able to express the dependency as "%.o depends on %.c and all the includes of %.c', but our tools don't allow us to express that last part because the includes of %.c are generated, and not known when the build script is read.

This inability to use the content of generated files within the dependency graph leads to a lot of silliness down the line. For instance, many compilers/int