Hexi is a lightweight, header-only C++23 library for safely handling binary data from arbitrary sources (but primarily network data). It sits somewhere between manually memcpying bytes from network buffers and full-blown serialisation libraries.

Some of the protocols Hexi is used to handle include DNS, STUN, NAT Port Mapping Protocol, Port Control Protocol, World of Warcraft (emulators) and GameSpy (emulators). Of course, nothing stops you from defining your own!

The design goals are ease of use, safety when dealing with untrusted data and in the face of programmer mistakes, a reasonable level of flexibility, and keeping overhead to a minimum.

What Hexi doesn't offer: versioning, conversion between different formats, handling of text-based formats, unloading the dishwasher.

Hexi is dual-licensed under MIT and Apache License, Version 2.0. This means you can use Hexi under the license you prefer.

Incorporating Hexi into your project is simple! The easiest way is to simply copy hexi.h from single_include into your own project. If you'd rather only include what you use, you can add include to your include paths or incorporate it into your own CMake project with target_link_library. To build the unit tests, run CMake with ENABLE_TESTING.

Here's what some libraries might call a very simple motivating example:

#include <hexi.h>
#include <array>
#include <vector>
#include <cstddef>

struct LoginPacket {
    uint64_t user_id;
    uint64_t timestamp;
    std::array<uint8_t, 16> ipv6;
};

auto deserialise(std::span<const char> network_buffer) {
    hexi::buffer_adaptor adaptor(network_buffer); // wrap the buffer
    hexi::binary_stream stream(adaptor);          // create a binary stream
    
    // deserialise!
    LoginPacket packet;
    stream >> packet;
    return packet;
}

auto serialise(const LoginPacket& packet) {
    std::vector<uint8_t> buffer;
    hexi::buffer_adaptor adaptor(buffer); // wrap the buffer
    hexi::binary_stream stream(adaptor);  // create a binary stream
    
    // serialise!
    stream << packet;
    return buffer;
}

By default, Hexi will try to serialise basic structures such as our LoginPacket if they meet requirements for being safe to directly copy the bytes. Now, for reasons of portability, it's not recommended that you do things this way unless you're positive that the data layout is identical on the system that wrote the data. Not to worry, this is easily solved. Plus, we didn't do any error or endianness handling. All in good time.

The two classes you'll primarily deal with are buffer_adaptor and binary_stream.

binary_stream takes a container as its argument and is used to do the reading and writing. It doesn't know much about the details of the underlying container.

To support containers that weren't written to be used with Hexi, buffer_adaptor is used as a wrapper that binary_stream can interface with. As with binary_stream, it also provides read and write operations but at a lower level.

buffer_adaptor can wrap any contiguous container or view that provides data and size member functions. From the standard library, that means the following can be used out of the box:

[x] std::array
[x] std::span
[x] std::string_view
[x] std::string
[x] std::vector

Plenty of non-standard library containers will work out of the box, too, as long as they provide a vaguely similar API.

The container's value type must be a byte type (e.g. char, std::byte, uint8_t). std::as_bytes can be used as a workaround if this poses a problem.

Hexi supports custom containers, including non-contiguous containers. In fact, there's a non-contiguous container included in the library. You simply need to provide a few functions such as read and size to allow the binary_stream class to be able to use it.

static_buffer.h provides a simple example of a custom container that can be used directly with binary_stream.

As mentioned, Hexi is intended to be safe to use even when dealing with untrusted data. An example might be network messages that have been manipulated to try to trick your code into reading out of bounds.

binary_stream performs bounds checking to ensure that it will never read more data than the buffer has available and optionally allows you to specify an upper bound on the amount of data to read. This can be useful when you have multiple messages in a buffer and want to limit the deserialisation from potentially eating into the next.

buffer_t buffer;
// ... read data
hexi::binary_stream stream(buffer, 32); // will never read more than 32 bytes

The default error handling mechanism is exceptions. Upon encountering a problem with reading data, an exception derived from hexi::exception will be thrown. These are:

hexi::buffer_underrun - attempt to read out of bounds
hexi::stream_read_limit - attempt to read more than the imposed limit

Exceptions from binary_stream can be disabled by specifying no_throw as an argument. This argument eliminates exception branches at compile-time, so there's zero run-time overhead.

hexi::binary_stream stream(buffer, hexi::no_throw);

Regardless of the error handling mechanism you use, the state of a binary_stream can be checked as follows:

hexi::binary_stream stream(buffer, hexi::no_throw);
// ... assume an error happens

// simplest way to check whether any errors have occurred
if (!stream) {
    // handle error
}

// or we can get the state
if (auto state = stream.state(); state != hexi::stream_state::ok) {
    // handle error
}

In the first example, reading our LoginPacket would only work as expected if the program that wrote the data laid everything out in the same way as our own program. This might not be the case for reasons of architecture differences, compiler flags, etc.

Here's the same example but doing it portably.

#include <hexi.h>
#include <span>
#include <string>
#include <vector>
#include <cstddef>
#include <cstdint>

// an example structure that has separate serialise and deserialise functions
struct LoginPacket {
    uint64_t user_id;
    std::string username;
    uint64_t timestamp;
    uint8_t has_optional_field;
    uint32_t optional_field;  // pretend this is big-endian in the protocol

    // deserialise
    auto& operator>>(auto& stream) {
        stream >> user_id >> username >> timestamp >> has_optional_field;

        if (has_optional_field) {
            // fetch explicitly as big-endian ('be') value
            stream >> hexi::endian::be(optional_field);
        }

        // we can manually trigger an error if something went wrong
        // stream.set_error_state();
        return stream;
    }

    // serialise
    auto& operator<<(auto& stream) const {
        stream << user_id << username << timestamp << has_optional_field;

        if (has_optional_field) {
            // write explicitly as big-endian ('be') value
            stream << hexi::endian::be(optional_field);
        }

        return stream;
    }
};

// an example of a packet that can serialise and deserialise with the same function
struct LogoutPacket {
    std::string username;
    std::uint32_t user_id;
    std::uint8_t has_timestamp;
    std::uint64_t timestamp;    // pretend this is optional & big endian in the protocol

    void serialise(auto& stream) const {
        stream(username, user_id);

        if (has_timestamp) {
            stream(hexi::endian::be(timestamp));

            // can also do this to write a single field:
            // stream & hexi::endian::be(timestamp);
        }
    }
};


// pretend we're reading network data
void read() {
    std::vector<char> buffer;
    const auto bytes_read = socket.read(buffer);

    // ... logic for determining packet type, etc

    bool result {};

    switch (packet_type) {
        case login_packet:
            result = handle_login_packet(buffer);
            break;
        case logout_packet:
            result = handle_logout_packet(buffer);
            break;
    }

    // ... handle result
}

auto handle_login_packet(std::span<const char> buffer) {
    hexi::buffer_adaptor adaptor(buffer);

    /**
     * hexi::endian::little tells the stream to convert to/from
     * little-endian unless told otherwise by using the endian
     * adaptors. If no argument is provided, it does not perform
     * any conversions by default.
     */
    hexi::binary_stream stream(adaptor, hexi::endian::little);

    LoginPacket packet;
    stream >> packet;

    if (stream) {
        // ... do something with the packet
        return true;
    } else {
        return false;
    }
}

auto handle_logout_packet(std::span<const char> buffer) {
    hexi::buff

Hexi

Install / Use

README