A Small but Useful Lua library

The Lua standard library is deliberately kept small, based on the abstract platform defined by the C89 standard. It is intended as a base for further development, so Lua programmers tend to collect small useful functions for their projects.

Microlight is an attempt at 'library golf', by analogy to the popular nerd sport 'code golf'. The idea here is to try capture some of these functions in one place and document them well enough so that it is easier to use them than to write them yourself.

This library is intended to be a 'extra light' version of Penlight, which has nearly two dozen modules and hundreds of functions.

In Lua, anything beyond the core involves 'personal' choice, and this list of functions does not claim to aspire to 'canonical' status. It emerged from discussion on the Lua Mailing list started by Jay Carlson, and was implemented by myself and Dirk Laurie.

Strings

THere is no built-in way to show a text representation of a Lua table, which can be frustrating for people first using the interactive prompt. Microlight provides tstring. Please note that globally redefining tostring is not a good idea for Lua application development! This trick is intended to make experimation more satisfying:

> require 'ml'.import()
> tostring = tstring
> = {10,20,name='joe'}
{10,20,name="joe"}

The Lua string functions are particularly powerful but there are some common functions missing that tend to come up in projects frequently. There is table.concat for building a string out of a table, but no table.split to break a string into a table.

>  = split('hello dolly')
{"hello","dolly"}
> = split('one,two',',')
{"one","two"}

The second argument is a string pattern that defaults to spaces.

Although it's not difficult to do string interpolation in Lua, there's no little function to do it directly. So Microlight provides ml.expand.

> = expand("hello $you, from $me",{you='dolly',me='joe'})
hello dolly, from joe

expand also understands the alternative ${var} and may also be given a function, just like string.gsub. (But pick one or the other consistently.)

Lua string functions match using string patterns, which are a powerful subset of proper regular expressions: they contain 'magic' characters like '.','$' etc which you need to escape before using. escape is used when you wish to match a string literally:

> = ('woo%'):gsub(escape('%'),'hoo')
"woohoo"   1
> = split("1.2.3",escape("."))
{"1","2","3"}

Files and Paths

Although access is available on most platforms, it's not part of the standard, (which is why it's spelt _access on Windows). So to test for the existance of a file, you need to attempt to open it. So the exist function is easy to write:

function ml.exists (filename)
    local f = io.open(filename)
    if not f then
        return nil
    else
        f:close()
        return filename
    end
end

The return value is not a simple true or false; it returns the filename if it exists so we can easily find an existing file out of a group of candidates:

> = exists 'README' or exists 'readme.txt' or exists 'readme.md'
"readme.md"

Lua is good at slicing and dicing text, so a common strategy is to read all of a not-so-big file and process the string. This is the job of readfile. For instance, this returns the first 128 bytes of the file opened in binary mode:

> txt = readfile('readme.md',true):sub(1,128)

Note I said bytes, not characters, since strings can contain any byte sequence.

If readfile can't open a file, or can't read from it, it will return nil and an error message. This is the pattern followed by io.open and many other Lua functions; it is considered bad form to raise an error for a routine problem.

Breaking up paths into their components is done with splitpath and splitext:

> = splitpath(path)
"/path/to/dogs" "bonzo.txt"
> = splitext(path)
"/path/to/dogs/bonzo"   ".txt"
> = splitpath 'frodo.txt'
""      "frodo.txt"
> = splitpath '/usr/'
"/usr"  ""
> = splitext '/usr/bin/lua'
"/usr/bin/lua"  ""
>

These functions return two strings, one of which may be the empty string (rather than nil). On Windows, they use both forward- and back-slashes, on Unix only forward slashes.

Inserting and Extending

Most of the Microlight functions work on Lua tables. Although these may be both arrays and hashmaps, generally we tend to use them as one or the other. From now on, we'll use array and map as shorthand terms for tables

update adds key/value pairs to a map, and extend appends an array to an array; they are two complementary ways to add multiple items to a table in a single operation.

> a = {one=1,two=2}
> update(a,{three=3,four=4})
> = a
{one=1,four=4,three=3,two=2}
> t = {10,20,30}
> extend(t,{40,50})
> = t
{10,20,30,40,50}

As from version 1.1, both of these functions take an arbitrary number of tables.

To 'flatten' a table, just unpack it and use extend:

> pair = {{1,2},{3,4}}
> = extend({},unpack(pair))
{1,2,3,4}

extend({},t) would just be a shallow copy of a table.

More precisely, extend takes an indexable and writeable object, where the index runs from 1 to #O with no holes, and starts adding new elements at O[#O+1]. Simularly, the other arguments are indexable but need not be writeable. These objects are typically tables, but don't need to be. You can exploit the guarantee that extend always goes sequentially from 1 to #T, and make the first argument an object:

> obj = setmetatable({},{ __newindex = function(t,k,v) print(v) end })
> extend(obj,{1,2,3})
1
2
3

To insert multiple values into a position within an array, use insertvalues. It works like table.insert, except that the third argument is an array of values. If you do want to overwrite values, then use true for the fourth argument:

> t = {10,20,30,40,50}
> insertvalues(t,2,{11,12})
> = t
{10,11,12,20,30,40,50}
> insertvalues(t,3,{2,3},true)
> = t
{10,11,2,3,30,40,50}

(Please note that the original table is modified by these functions.)

update' works like extend`. except that all the key value pairs from the input tables are copied into the first argument. Keys may be overwritten by subsequent tables.

> t = {}
> update(t,{one=1},{ein=1},{one='ONE'})
> = t
{one="ONE",ein=1}

import is a specialized version of update; if the first argument is nil then it's assumed to be the global table. If no tables are provided, it brings in the ml table itself (hence the lazy require "ml".import() idiom).

If the arguments are strings, then we try to require them. So this brings in LuaFileSystem and imports lfs into the global table. So it's a lazy way to do a whole bunch of requires. A module 'package.mod' will be brought in as mod. Note that the second form actually does bring all of lpeg's functions in.

> import(nil,'lfs')
> import(nil,require 'lpeg')

Extracting and Mapping

The opposite operation to extending is extracting a number of items from a table.

There's sub, which works just like string.sub and is the equivalent of list slicing in Python:

> numbers = {10,20,30,40,50}
> = sub(numbers,1,1)
{10}
> = sub(numbers,2)
{20,30,40,50}
> = sub(numbers,1,-2)
{10,20,30,40}

indexby indexes a table by an array of keys:

> = indexby(numbers,{1,4})
{10,40}
> = indexby({one=1,two=2,three=3},{'three','two'})
{[3,2}

Here is the old standby imap, which makes a new array by applying a function to the original elements:

> words = {'one','two','three'}
> = imap(string.upper,words)
{"ONE","TWO","THREE"}
> s = {'10','x','20'}
> ns = imap(tonumber,s)
> = ns
{10,false,20}

imap must always return an array of the same size - if the function returns nil, then we avoid leaving a hole in the array by using false as a placeholder.

Another popular function indexof does a linear search for a value and returns the 1-based index, or nil if not successful:

> = indexof(numbers,20)
2
> = indexof(numbers,234)
nil

This function takes an optional third argument, which is a custom equality function.

In general, you want to match something more than just equality. ifind will return the first value that satisfies the given function.

> s = {'x','10','20','y'}
> = ifind(s,tonumber)
"10"

The standard function tonumber returns a non-nil value, so the corresponding value is returned - that is, the string. To get all the values that match, use ifilter:

> = ifilter(numbers,tonumber)
{"10","20"}

There is a useful hybrid between imap and ifilter called imapfilter which is particularly suited to Lua use, where a function commonly returns either something useful, or nothing. (Phillip Janda originally suggested calling this transmogrify, since no-one has preconceptions about it, except that it's a cool toy for imaginative boys).

> = imapfilter(tonumber,{'one',1,'f',23,2})
{1,23,2}

collect makes a array out of an iterator. 'collectuntilcan be given a custom predicate andcollectntakes up to a maximum number of values, which is useful for iterators that never terminate. (Note that we need to pass it either a proper iterator, likepairs, or a function or exactly one function - which isn't the case with math.random`)

> s = 'my dog ate your homework'
> words = collect(s:gmatch '%a+')
> = words
{"my","dog","ate","your","homework"}
> R = function() return math.random() end
> = collectn(3,R)
{0.0012512588885159,0.56358531449324,0.19330423902097}
> lines = collectuntil(4,io.lines())
one
two
three
four
> = lines