musli

Excellent performance, no compromises[^1]!

Müsli is a flexible, fast, and generic binary serialization framework for Rust, in the same vein as [serde].

It provides a set of formats, each with its own well-documented set of features and tradeoffs. Every byte-oriented serialization method including escaped formats like [musli::json] has full #[no_std] support with or without alloc. And a particularly neat component providing low-level refreshingly simple [zero-copy serialization][zerocopy].

[^1]: As in Müsli should be able to do everything you need and more.

Overview

See [derives] to learn how to implement [Encode] and [Decode].
See [data_model] to learn about the abstract data model of Müsli.
See [benchmarks] and [size comparisons] to learn about the performance of this framework.
See [tests] to learn how this library is tested.
See [musli::serde] for seamless compatibility with [serde]. You might also be interested to learn how Müsli is different.

Usage

Add the following to your Cargo.toml using the format you want to use:

[dependencies]
musli = { version = "0.0.149", features = ["storage"] }

Design

The heavy lifting is done by the [Encode] and [Decode] derives which are documented in the [derives] module.

Müsli operates based on the schema represented by the types which implement these traits.

use musli::{Encode, Decode};

#[derive(Encode, Decode)]
struct Person {
    /* .. fields .. */
}

Note by default a field is identified by its numerical index which would change if they are re-ordered. Renaming fields and setting a default naming policy can be done by configuring the [derives].

The binary serialization formats provided aim to efficiently and accurately encode every type and data structure available in Rust. Each format comes with well-documented tradeoffs and aims to be fully memory safe to use.

Internally we use the terms "encoding", "encode", and "decode" because it's distinct from [serde]'s use of "serialization", "serialize", and "deserialize" allowing for the clearer interoperability between the two libraries. Encoding and decoding also has more of a "binary serialization" vibe, which more closely reflects the focus of this framework.

Müsli is designed on similar principles as [serde]. Relying on Rust's powerful trait system to generate code which can largely be optimized away. The end result should be very similar to handwritten, highly optimized code.

As an example of this, these two functions both produce the same assembly (built with --release):

const OPTIONS: Options = options::new().fixed().native_byte_order().build();
const ENCODING: Encoding<OPTIONS> = Encoding::new().with_options();

#[derive(Encode, Decode)]
#[musli(packed)]
pub struct Storage {
    left: u32,
    right: u32,
}

fn with_musli(storage: &Storage) -> Result<[u8; 8]> {
    let mut array = [0; 8];
    ENCODING.encode(&mut array[..], storage)?;
    Ok(array)
}

fn without_musli(storage: &Storage) -> Result<[u8; 8]> {
    let mut array = [0; 8];
    array[..4].copy_from_slice(&storage.left.to_ne_bytes());
    array[4..].copy_from_slice(&storage.right.to_ne_bytes());
    Ok(array)
}

Müsli is different from [`serde`]

Müsli's data model does not speak Rust. There are no serialize_struct_variant methods which provides metadata about the type being serialized. The [Encoder] and [Decoder] traits are agnostic on this. Compatibility with Rust types is entirely handled using the [Encode] and [Decode] derives in combination with modes.

We use GATs to provide easier to use abstractions. GATs were not available when serde was designed.

Everything is a [Decoder] or [Encoder]. Field names are therefore not limited to be strings or indexes, but can be named to [arbitrary types][musli-name-type] if needed.

Visitor are only used when needed. serde [completely uses visitors] when deserializing and the corresponding method is treated as a "hint" to the underlying format. The deserializer is then free to call any method on the visitor depending on what the underlying format actually contains. In Müsli, we swap this around. If the caller wants to decode an arbitrary type it calls [decode_any]. The format can then either signal the appropriate underlying type or call [Visitor::visit_unknown] telling the implementer that it does not have access to type information.

We've invented moded encoding allowing the same Rust types to be encoded in many different ways with much greater control over how things encoded. By default we include the [Binary] and [Text] modes providing sensible defaults for binary and text-based formats.

Müsli fully supports [no-std and no-alloc] from the ground up without compromising on features using safe and efficient [scoped allocations].

We support [detailed tracing] when decoding for much improved diagnostics of where something went wrong.

Formats

Formats are currently distinguished by supporting various degrees of upgrade stability. A fully upgrade stable encoding format must tolerate that one model can add fields that an older version of the model should be capable of ignoring.

Partial upgrade stability can still be useful as is the case of the [musli::storage] format below, because reading from storage only requires decoding to be upgrade stable. So if correctly managed with #[musli(default)] this will never result in any readers seeing unknown fields.

The available formats and their capabilities are:

| | reorder | missing | unknown | self | |-|-|-|-|-| | [musli::packed] (with #[musli(packed)]) | ✗ | ✗ | ✗ | ✗ | | [musli::storage] | ✔ | ✔ | ✗ | ✗ | | [musli::wire] | ✔ | ✔ | ✔ | ✗ | | [musli::descriptive] | ✔ | ✔ | ✔ | ✔ | | [musli::json] [^json] | ✔ | ✔ | ✔ | ✔ |

reorder determines whether fields must occur in exactly the order in which they are specified in their type. Reordering fields in such a type would cause unknown but safe behavior of some kind. This is only suitable for communication where the data models of each client are strictly synchronized.

missing determines if reading can handle missing fields through something like Option<T>. This is suitable for on-disk storage, because it means that new optional fields can be added as the schema evolves.

unknown determines if the format can skip over unknown fields. This is suitable for network communication. At this point you've reached upgrade stability. Some level of introspection is possible here, because the serialized format must contain enough information about fields to know what to skip which usually allows for reasoning about basic types.

self determines if the format is self-descriptive. Allowing the structure of the data to be fully reconstructed from its serialized state. These formats do not require models to decode and can be converted to and from dynamic containers such as [musli::value] for introspection. Such formats also allows for type-coercions to be performed, so that a signed number can be correctly read as an unsigned number if it fits in the destination type.

For every feature you drop, the format becomes more compact and efficient. [musli::storage] using #[musli(packed)] for example is roughly as compact as [bincode] while [musli::wire] is comparable in size to something like [protobuf]. All formats are primarily byte-oriented, but some might perform [bit packing] if the benefits are obvious.

[^json]: This is strictly not a binary serialization, but it was implemented as a litmus test to ensure that Müsli has the necessary framework features to support it. Luckily, the implementation is also quite good!

Upgrade stability

The following is an example of full upgrade stability using [musli::wire]. Version1 can be decoded from an instance of Version2 because it understands how to skip fields which are part of Version2. We're also explicitly adding `#[musli(name = ..)]

Musli

Install / Use

README

musli

Overview

Usage

Design

Müsli is different from [`serde`]

Formats

Upgrade stability

Musli

Install / Use

README

musli

Overview

Usage

Design

Müsli is different from [serde]

Formats

Upgrade stability

Müsli is different from [`serde`]