Overview
Warnings
1. Numeric Precision
2. Known Incompatibilities
Setup
1. Installation
2. Testing
3. Loading
4. Backwards Compatibility
API
1. Variables
2. Functions
Format string syntax
1. Repeat markers
2. Tables
3. Names
4. Bitpacks
5. Backreferences
Data items
1. Controlling endianness
2. Seeking
3. Reading and writing
Adding new formats
Credits

1. Overview

VStruct is a library for Lua 5.1-5.4 and LuaJIT 2. It provides functions for manipulating binary data, in particular for reading binary files or byte buffers into Lua values and for writing Lua values back into files or buffers. Supported data types include:

signed and unsigned integers of arbitrary size
fixed and floating point numbers
fixed-size, length-prefixed, and null-terminated strings
booleans and bitmasks
bit-packed integers, booleans and bitmasks

In addition, the library supports seeking, alignment, and byte order controls, repetition, grouping of data into tables, and named fields.

2. Warnings

2.1 Numeric Precision

When reading and writing numeric formats, vstruct is inherently limited by lua's number format, which is by default the IEEE 754 double. What this means in practice is that formats i, u, c, and p (see Data Items) may be subject to data loss, if they contain more than 52 significant bits. (The same is true of numeric constants declared in Lua itself, of course, and other libraries which store values in lua numbers). In other words - be careful when manipulating data, especially 64-bit ints, that may not fit in Lua's native types.

Formats not listed above are not subject to this limitation, as they either do not use Lua numbers at all, or do so only in ways that are guaranteed to be lossless.

2.2 Known Incompatibilities

Lua 5.0 is not supported, as vstruct makes heavy use of features introduced in 5.1.

LuaJIT 2.0b7 has a bug in the code generator that affects vstruct. This is fixed in 2.0b9. If you need to run vstruct in 2.0b7, you will need to disable JIT compilation for the m.read function:

jit.off(require("vstruct.io.m").read, true)

The strict module raises an error when using vstruct outside of LuaJIT, as it checks for the presence of LuaJIT by checking if the global variable jit is defined (resulting in a "variable not declared" error). This can be worked around by setting jit = false before loading vstruct.

3. Setup

3.1 Installation

From LuaRocks

vstruct can be installed from the LuaRocks package management system as follows.

luarocks install vstruct

From Source

vstruct is a pure-lua module, and as such requires no seperate build step; it can be installed as-is. It automatically detects which version of Lua you're using it and enables compatibility shims as needed, so one installation can be shared by multiple versions of Lua/LuaJIT.

The initializer is vstruct/init.lua, so it should be installed in a way that means require "vstruct" will load that file, and that require "vstruct.foo" will load vstruct/foo.lua. In a default install of Lua 5.1 on Linux, you can do this simply by copying the vstruct/ directory from the vstruct distribution into any of these directories:

/usr/local/share/lua/5.1/
/usr/share/lib/5.1/
/usr/local/lib/lua/5.1/
/usr/lib/lua/5.1/

This also works for Lua 5.2, replacing the 5.1 in each path with 5.2.

If you install it elsewhere, you can accomplish this by modifying package.path to include the following entries (assuming that $LIBDIR is the directory containing vstruct/):

$LIBDIR/?.lua
$LIBDIR/?/init.lua

Note that installing it to ./ will not work unless you also add an entry for ./?/init.lua to package.path; by default, lua will look for ./?.lua but not ./?/init.lua.

3.2 Testing

vstruct comes with a number of builtin tests. To run them, simply invoke vstruct/test.lua:

lua vstruct/test.lua

If any of the tests fail, it will report them on standard output and then raise an error; if all of the tests pass, it will exit cleanly.

If vstruct is properly installed, you can also load the tests as a library:

lua -lvstruct.test
require "vstruct.test"

With the same behaviour - it will load cleanly if the tests succeeded and raise an error if any of them failed.

3.3 Loading

vstruct makes itself available under the name vstruct. Note that, in accordance with current module conventions, it does not assign the module to a global; you must assign the return value of require yourself:

local vstruct = require "vstruct"
print(vstruct._VERSION)

vstruct does make one global modification when loaded: if the function math.trunc is not already defined, it will install its own definition.

3.4 Backwards Compatibility

vstruct 2.x is not backwards compatible with 1.x versions, and code using 1.x APIs will not work with it. Converting 1.x code to use 2.x is very easy - just replace calls to unpack, unpackvals, and pack with read, readvals, and write, and if you are using custom formats, rename their width, pack*, and unpack* functions to size, write, and read.

If you do need to run legacy code and can't, for whatever reason, update it to use the 2.x API or install an older version of vstruct, there is a compatibility module available. require "vstruct.compat1x" and it will be loaded, making the 1.x API available through a translation layer. By default, it will emit a warning on stderr every time a legacy function is called; to disable this, set vstruct.WARN = false after loading it.

4. API

vstruct, once loaded, exports a number of variables and functions, all of them available in the table returned by require "vstruct".

In this section, a format string means the string used to describe a binary format, controlling how data is read or written. The syntax for format strings is described in section 5; the semantics in section 6. They are not be confused with the format strings used by string.format.

4.1 Error Handling

In vstruct, all errors are hard errors: they are raised with error and must be caught with pcall. The vstruct API functions will always either return a valid result or raise an error. Barring bugs in vstruct itself, the only errors one is likely to encounter involve invalid arguments to the API - wrong types, a syntactically invalid format string, or a data table that doesn't match up with the format string. The one potential surprise is seeking past the start or end of a file; in the standard io library, this is a soft error (returning nil,error), while in vstruct this will immediately raise.

There are also some conditions which are not currently treated as errors:

Seeking past the end of the file. This will extend the file when writing; it is an error when reading only if you subsequently attempt to read data without seeking back into the bounds of the file.
Lossy numeric conversion. In particular, passing a float where an int is expected will truncate the float, and trying to read a number with more bits of precision than Lua supports will give you as many bits as it can manage.
Passing numbers where strings are expected, or numeric strings where numbers are expected, is permitted in some cases due to Lua's automatic coercion between strings and numbers. This may not always be the case, so I recommend not relying on this.
One can often substitute any boolean true value when true is expected, or nil when false is expected. As above, this is not guaranteed to work and is not to be relied on.

4.2 Variables

These variables are used to control various settings of the library at runtime. At present there is only one public setting, but others may be added in the future.

vstruct.cache = (true|false|nil)

Enables or disables caching of compiled format strings. This can improve performance in the common case where you are re-using the same format strings many times; if, on the other hand, you are generating lots of different format strings, this will increase memory usage - perhaps significantly - for no performance benefit.

If true, enables caching. If false, existing cache entries will be used, but new ones will not be created. If nil, the cache is entirely disabled.

The default is nil.

4.3 Functions

vstruct.cursor(string)

Wraps a string so that it can be used as a file. The returned object ('cur') supports cur:seek, cur:read(num_bytes) and cur:write(string), with the same behaviours as the file methods of the same names. In general, vstruct will attempt to automatically wrap strings if they are passed to it where a file is expected (and unwrap them before returning them); this function is primarily useful when more control over the process is required.

To access the wrapped string, use cur.str; to determine where the read/write pointer is, use cur.pos.

vstruct.sizeof(fmt)

Takes a format string and returns the on-disk size of the data it describes, in bytes. If the size cannot be determined solely from the string (for example, it contains backreferences or variable-width fields), or if it performs seeks (even if the seek distance is known in advance), returns nil.

vstruct.read(fmt, <fd or string>, [data])

read takes a format string and a buffer or file to read from, and returns a table of unpacked data items. The items will appear in the table with the same order and/or names that they had in the format string.

The data argument is an optional table. If pre

Vstruct

Install / Use

README

Contents