LuaMacro

LuaMacro - a macro preprocessor for Lua

This is a library and driver script for preprocessing and evaluating Lua code. Lexical macros can be defined, which may be simple C-preprocessor style macros or macros that change their expansion depending on the context.

It is a new, rewritten version of the Luaforge project of the same name, which required the token filter patch by Luiz Henrique de Figueiredo. This patch allowed Lua scripts to filter the raw token stream before the compiler stage. Within the limits imposed by the lexical filter approach this worked pretty well. However, the token filter patch is unlikely to ever become part of mainline Lua, either in its original or revised form. So the most portable option becomes precompilation, but Lua bytecode is not designed to be platform-independent and in any case changes faster than the surface syntax of the language. So using LuaMacro with LuaJIT would have required re-applying the patch, and would remain within the ghetto of specialized, experimental use.

This implementation uses a LPeg lexical analyser originally by Peter Odding to tokenize Lua source, and builds up a preprocessed string explicitly, which then can be loaded in the usual way. This is not as efficient as the original, but it can be used by anyone with a Lua interpreter, whether it is Lua 5.1, 5.2 or LuaJIT 2. An advantage of fully building the output is that it becomes much easier to debug macros when you can actually see the generated results. (Another example of a LPeg-based Lua macro preprocessor is Luma)

It is not possible to discuss macros in Lua without mentioning Fabien Fleutot's Metalua which is an alternative Lua compiler which supports syntactical macros that can work on the AST (Abstract Syntax Tree) itself of Lua. This is clearly a technically superior way to extend Lua syntax, but again has the disadvantage of being a direct-to-bytecode compiler. (Perhaps it's also a matter of taste, since I find it easier to think about extending Lua on the lexical level.)

My renewed interest in Lua lexical macros came from some discussions on the Lua mailing list about numerically optimal Lua code using LuaJIT. We have been spoiled by modern optimizing C/C++ compilers, where hand-optimization is often discouraged, but LuaJIT is new and requires some assistance. For instance, unrolling short loops can make a dramatic difference, but Lua does not provide the key concept of constant value to assist the compiler. So a very straightforward use of a macro preprocessor is to provide named constants in the old-fashioned C way. Very efficient code can be generated by generalizing the idea of 'varargs' into a statically-compiled 'tuple' type.

tuple(3) A,B

The assigment A = B is expanded as:

A_1,A_2,A_3 = B_1,B_2,B_3

I will show how the expansion can be made context-sensitive, so that the loop-unrolling macro do_ changes this behaviour:

do_(i,1,3,
    A = 0.5*B
)

expands to:

A_1 = 0.5*B_1
A_2 = 0.5*B_2
A_3 = 0.5*B_3

Another use is crafting DSLs, particularly for end-user scripting. For instance, people may be more comfortable with forall x in t do rather than for _,x in ipairs(t) do; there is less to explain in the first form and it translates directly to the second form. Another example comes from this common pattern:

some_action(function()
  ...
end)

Using the following macro:

def_ block (function() _END_CLOSE_

we can write:

some_action block
   ...
end

A criticism of traditional lexical macros is that they don't respect the scoping rules of the language itself. Bad experiences with the C preprocessor lead many to regard them as part of the prehistory of computing. The macros described here can be lexically scoped, and can be as 'hygenic' as necessary, since their expansion can be finely controlled with Lua itself.

For me, a more serious charge against 'macro magic' is that it can lead to a private dialect of the language (the original Bourne shell was written in C 'skinned' to look like Algol 68.) This often indicates a programmer uncomfortable with a language, who wants it to look like something more familiar. Relying on a preprocessor may mean that programmers need to immerse themselves more in the idioms of the new language.

That being said, macros can extend a language so that it can be more expressive for a particular task, particularly if the users are not professional programmers.

Basic Macro Substitution

To install LuaMacro, expand the archive and make a script or batch file that points to luam.lua, for instance:

lua /home/frodo/luamacro/luam.lua $*

(Or '%*' if on Windows.) Then put this file on your executable path.

Any Lua code loaded with luam goes through four distinct steps:

• loading and defining macros
• preprocessing
• compilation
• execution

The last two steps happen within Lua itself, but always occur, even though the Lua compiler is fast enough that we mostly do not bother to save the generated bytecode.

For example, consider this hello.lua:

print(HELLO)

and hello-def.lua:

local macro = require 'macro'
macro.define 'HELLO "Hello, World!"'

To run the program:

$> luam -lhello-def hello.lua
Hello, World!

So the module hello-def.lua is first loaded (compiled and executed, but not preprocessed) and only then hello.lua can be preprocessed and then loaded.

Naturaly, there are easier ways to use LuaMacro, but I want to emphasize the sequence of macro loading, preprocessing and script loading. luam has a -d flag, meaning 'dump', which is very useful when debugging the output of the preprocessing step:

$> luam -d -lhello-def hello.lua
print("Hello, World!")

hello2.lua is a more sensible first program:

require_ 'hello-def'
print(HELLO)

You cannot use the Lua require function at this point, since require is only executed when the program starts executing and we want the macro definitions to be available during the current compilation. require_ is the macro version, which loads the file at compile-time.

New with 2.5 is the default @ shortcut available when using luam, so require_ can be written @require. (@ is itself a macro, so you can redefine it if needed.)

There is also include_/@include, which is analogous to #include in cpp. It takes a file path in quotes, and directly inserts the contents of the file into the current compilation. Although tempting to use, it will not work here because again the macro definitions will not be available at compile-time.

hello3.lua fits much more into the C preprocessor paradigm, which uses the def_ macro:

@def HELLO "Hello, World!"
print(HELLO)

(Like cpp, such macro definitions end with the line; however, there is no equivalent of \ to extend the definition over multiple lines.)

With 2.1, an alternative syntax def_ (name body) is also available, which can be embedded inside a macro expression:

def_ OF_ def_ (of elseif _value ==)

Or even extend over several lines:

def_ (complain(msg,n)
  for i = 1,n do
    print msg
  end
)

def_ works pretty much like #define, for instance, def_ SQR(x) ((x)*(x)). A number of C-style favourites can be defined, like assert_ using _STR_, which is a predefined macro that 'stringifies' its argument.

def_ assert_(condn) assert(condn,_STR_(condn))

def_ macros are lexically scoped:

local X = 1
if something then
    def_ X 42
    assert(X == 42)
end
assert(X == 1)

LuaMacro keeps track of Lua block structure - in particular it knows when a particular lexical scope has just been closed. This is how the _END_CLOSE_ built-in macro works

def_ block (function() _END_CLOSE_

my_fun block
  do_something_later()
end

When the current scope closes with end, LuaMacro appends the necessary ')' to make this syntax valid.

A common use of macros in both C and Lua is to inline optimized code for a case. The Lua function assert() always evaluates its second argument, which is not always optimal:

def_ ASSERT(condn,expr) if condn then else error(expr) end

ASSERT(2 == 1,"damn! ".. 2 .." is not equal to ".. 1)

If the message expression is expensive to execute, then this can give better performance at the price of some extra code. ASSERT is now a statement, not a function, however.

Conditional Compilation

For this to work consistently, you need to use the @ shortcut:

@include 'test.inc'
@def A 10
...

This makes macro 'preprocessor' statements stand out more. Conditional compilation works as you would expect from C:

-- test-cond.lua
@if A
print 'A defined'
@else
print 'A not defined'
@end
@if os.getenv 'P'
print 'Env P is defined'
@end

Now, what is A? It is a Lua expression which is evaluated at preprocessor time, and if it returns any value except nil or false it is true, using the usual Lua rule. Assuming A is just a global variable, how can it be set?

$ luam test-cond.lua
A not defined
$ luam -VA test-cond.lua
A defined
$ export P=1
$ luam test-cond.lua
A not defined
Env P is defined

Although this looks very much like the standard C preprocessor, the implementation is rather different - @if is a special macro which evaluates its argument (everything on the rest of the line) as a Lua expression and skips upto @end (or @else or @elseif) if that condition is false.

Using macro.define

macro.define is less convenient than def_ but much more powerful. The extended form allows the substitution to be a _fu