Luabuild, A Custom Lua 5.2 Builder

Rationale

Deploying Lua applications as standalone applications can be tricky, as it is in all dynamic languages. The first thing that needs to happen is that all the external Lua dependencies of the application need to be found and a standalone single Lua file generated. This is what Mathew Wild's Squish and Jay Carlson's Soar do for us. The result can be made executable (by shebang or batch file) and depends on the currently installed Lua distribution. This file can be glued to a stub executable using Luiz Henrique de Figueiredo's srlua to provide a standalone executable. This is particularly useful on platforms where there is no system package manager that can easily deliver Lua.

However, most non-trivial Lua applications will depend on external binary modules, since the Lua core is deliberately kept compact and conforming to what is provided by the C89 standard. So (for instance) iterating over files and directories requires LuaFileSystem. So we need to statically link these external C modules with Lua to get a stub executable with no external dependencies apart from the C runtime.

This turns out to be particularly straightforward with Lua 5.2; linit.c provides a list of built-in Lua modules to be included, and a separate list of external modules which will be pre-loaded. The core of Luabuild is a flexible way to build Lua 5.2 with a modified linit.c that will optionally exclude standard modules and include external modules.

The build system uses Lake which allows us to write the build logic in a higher-level, platform-independent way. (I've included a very fresh version in this distribution for convenience). lake does need Lua and LuaFileSystem, which gives us a classic bootstrapping situation where you do need an existing Lua (5.1 or 5.2) to get things going. lake allows a single build script to cope with both Microsoft and Gnu compilers on Windows, and abstracts away some of the more irritating platform preculiarities (for instance, how to create a shared C extension on OS X, etc)

If you don't have a suitable lua.exe available, then a standalone lake.exe is available for Windows.

Included Modules

Lua C extensions are usually shipped with makefiles, which are designed to build shared libries or DLLs. Makefiles are usually a mess, despite the best intentions of the authors, and have a tendency to be either hard to read or specific to the author's machine. Luabuild contains a number of common/convenient C extensions that are available for static linking:

• LuaFileSystem File System Support
• lcomplex Complex number Support (requires C99)
• LPeg
• LuaSocket Networking
• luaposix POSIX API
• winapi Minimal Windows API
• ltcltk Binding to Tcl/Tk
• LuaSQLite3
• lua-linenoise

These are all all Lua 5.2 compatible, which required some extra (but necessary) work. Some of these (luaposix and winapi) are very platform-dependent; ltcltk could be in principle built on Windows, but Tcl/Tk is an awkward dependency on that platform. I don't claim that these modules represent some kind of ideal extended core, simply that they are (a) widely used and (b) small enough to link in statically. 'Small enough' is a tough requirement when contemplating GUI toolkits in particular, because even the bindings to common cross-platform kits like wxWidgets and Qt get rather large.

The available modules are listed in modules/manifest and there may be a corresponding .lake file. For instance, complex.lake

if CC == 'cl' then
    quit "MSVC does not do C99 complex"
end

luabuild.test 'test.lua'

return c99.library{LIBDIR..'complex',src='lcomplex',args=ARGS}

They are plain Lua files invoked from the main lakefile, and have the usual lake globals available - here we can complain about Microsoft's C99 support. Luabuild defines some globals like LIBDIR (which is $LB/lib/) and ARGS which provides the common default settings, like the include directory and the base directory for finding the sources.

Expressing the build in this higher-level way makes it very flexible; we don't have to worry about the platform details of making a C99 library, whether static or dynamic.

A more complete example is for the socket.core module of luasocket:

----- building socket/core -----
COMMON='timeout buffer auxiliar options io'
COMMON = COMMON..' '..choose(WINDOWS,'wsocket','usocket')
SCORE=COMMON..' luasocket inet tcp udp except select'

luabuild.lua('ftp.lua http.lua smtp.lua tp.lua url.lua','socket')
luabuild.lua('socket.lua mime.lua ltn12.lua')
luabuild.test 'test-driver.lua'

defines='LUASOCKET_DEBUG'
if not luabuild.config.no_lua51_compat then
    defines = defines..' LUA_COMPAT_ALL'
end

return c.library {LIBDIR..'socket/core',src=SCORE,needs='sockets',defines=defines,
    args = ARGS
}

Note how any pure Lua components can be copied as well; these go into the lua directory, and will be on the module path for the default lua52 executable.

The global luabuild provides the means to add a test file. It became clear that Luabuild needed to do more than statically link an appropriate executable; without the means to easily test the result. And it was going to be necessary to also provide the traditional shared library versions of these modules and put them in a convenient place. Since some of these modules have a Lua part as well, these needed to be copied onto the module path as well. In this way, Luabuild has moved from a quick hack to being a small, flexible, source-based Lua 5.2 distribution. This was not my original intention, since there is already two big source-based distributions. It seems that the Luadist project understands that provision for static linking provides the means to customize the Lua executable, see this comment by David Manura. It's possible because that project uses another high-level build platform, CMake.

Using luabuild

The default build is controlled by default.config:

----- Luabuild default configuration
OPTIMIZE='O2'
-- or you can get a debug build
--DEBUG = true

-- this is the default; build with as much 5.1 compatibility as possible
no_lua51_compat = false

if  not STATIC then
    build_shared = true
    --the excutable/lib will find its modules in $LB/libs and $LIB/lua
    custom_lua_path = true
    -- default to linking against linenoise - set READLINE if you really
    -- want the old dog back
    readline = READLINE and true or 'linenoise'
else
    --it will be a statically-linked executable that can't link dynamically
    no_dlink = true
    -- can switch off readline (useful for self-contained executables)
    readline = false
end

-- set this if you want MSVC builds to link against runtime
-- (they will be smaller but less portable)
dynamic = DYNAMIC

if not STATIC then
    name = 'lua52'
else
    name = 'lua52s'  -- for 'static'
end

if PLAT == 'Windows' then
    dll = name
end

if PLAT == 'Windows' then
    include = 'lfs winapi socket.core mime.core lpeg struct'
    if CC ~= 'cl' then -- sorry, MSVC does not do C99 complex...
        include = include .. ' complex'
    end
else
    include = 'lfs socket.core mime.core lpeg complex posix_c struct '
    if not READLINE then
        include = include .. ' linenoise '
    end
    -- either satisfy external requirements, or just leave these out
    include = include .. ' curses_c ltcl lxp lsqlite3'
end

Again, it's a Lua file, and any global variables (defined as uppercase names) are available. Setting any uppercase name makes this available as a lake global, so you can trivially get a debug build of Lua 5.2 and the modules by using DEBUG, or a cross-platform build using PREFIX. For instance, to build for ARM Linux, I needed to add just one line:

PREFIX = 'arm-linux'

And thereafter the compiler, etc would become arm-linux-gcc and so forth.

Note custom_lua_path: the default build will put shared libraries in libs/ and Lua files in lua/ and will modify the Lua module path to look in these directories - this is the only major patch to the 5.2 sources. This is particularly useful on non-Windows platforms where the default module path is only superuser-writable, and you wish to have a 'sandboxed' Lua build.

The default build makes a fairly conventional Lua 5.2 executable (or DLL on Windows) with the external modules as shared libraries. (On POSIX systems there is an option link against readline, but you can choose to statically-link in linenoise instead.)

$ lua lake

It may complain about missing development headers, such as ncurses and tcl; for a clean Debian install I needed the libncurses-dev, libreadline-dev and libtcl8.4-dev (plus Tk8.4 for the tk part; this appears to be available out of the box for OS X machines). Or you can leave out the problematic modules and concentrate on the core modules.

To build everything, including the 'fat' Lua 5.2 executable with the modules linked in statically:

$ ./build-all