SeqProto

This library provides a simple way to serialize and deserialize objects in binary format.

Why another serialization library?

While I have been writing this library, I have in mind the following main goals:

• Runtime independent - I want to have a library that runs in every javascript runtime, from Node.JS, through browsers, to CloudFlare.
• Performance - I want to have a library that is fast and easy to use.
• Small size - I want to have a library that is small and easy to use.
• Customizable - Due to the JavaScript nature, the data structures are limited.
• TypeScript support - I want to have a library that is easy to use in TypeScript.

Installation

Seqproto works in any JavaScript environment. You can install it via npm:

npm install seqproto

Or via CDN:

import { createSer, createDes } from 'https://unpkg.com/seqproto@latest/dist/esm/index.js'

Usage

Examples

For more examples, see the examples directory.

import { createSer, createDes } from 'seqproto'

// Create a serializer
const ser = createSer()

// Serialize some data
ser.serializeBoolean(true)
ser.serializeUInt32(42)
ser.serializeFloat32(-0.5)
ser.serializeString('hello world')
ser.serializeArray([1, 2, 3], (ser, n) => ser.serializeUInt32(n))

// Get ArrayBuffer with serialized data
const buffer = ser.getBuffer()

// Create a deserializer
const des = createDes(buffer)

// Deserialize data
const b = des.deserializeBoolean()
const i = des.deserializeUInt32()
const f = des.deserializeFloat32()
const s = des.deserializeString()
const a = des.deserializeArray((des) => des.deserializeUInt32())

console.log({ b, i, f, s, a })

Object

import type { Ser, Des } from 'seqproto'
import { createSer, createDes } from 'seqproto'

interface Todo {
  id: number
  userId: number
  title: string
  completed: boolean
}

function serializeTodo (ser: Ser, todo: Todo) {
  ser.serializeUInt32(todo.id)
  ser.serializeUInt32(todo.userId)
  ser.serializeString(todo.title)
  ser.serializeBoolean(todo.completed)
}

function deserializeTodo (des: Des): Todo {
  const id = des.deserializeUInt32()
  const userId = des.deserializeUInt32()
  const title = des.deserializeString()
  const completed = des.deserializeBoolean()
  return { id, userId, title, completed }
}

const ser: Ser = createSer()

serializeTodo(ser, {
  id: 1,
  userId: 1,
  title: 'hello',
  completed: false,
})

const buffer = ser.getBuffer()

const des: Des = createDes(buffer)
const todo = deserializeTodo(des)

console.log(JSON.stringify(todo, null, 2))

Array of object

import type { Ser, Des } from 'seqproto'
import { createSer, createDes } from 'seqproto'

let buffer

// Serialize
const todos = [
  { userId: 1, id: 1, completed: false, title: "delectus aut autem" },
  { userId: 1, id: 2, completed: true, title: "quis ut nam facilis et officia qui" }
]

const ser: Ser = createSer()

ser.serializeArray(todos, (ser, todo) => {
  ser.serializeUInt32(todo.id)
  ser.serializeUInt32(todo.userId)
  ser.serializeString(todo.title)
  ser.serializeBoolean(todo.completed)
})

buffer = ser.getBuffer()

// Deserialize
const des: Des = createDes(buffer)

const deserializedTodos = des.deserializeArray((des) => {
  const id = des.deserializeUInt32()
  const userId = des.deserializeUInt32()
  const title = des.deserializeString()
  const completed = des.deserializeBoolean()
  return { id, userId, title, completed }
})

console.log(deserializedTodos)

API

This library exports the following functions:

• createSer()/createSer({ bufferSize: number }): creates a new serializer.
• createDes(buffer): creates a new deserializer.
• ser.reset(): reset the serializer.
• ser.serializeBoolean(b): serializes a boolean value.
• des.deserializeBoolean(): deserializes a boolean value.
• ser.serializeUInt32(uint32): serializes a 32-bit unsigned integer.
• des.deserializeUInt32(uint32): deserializes a 32-bit unsigned integer.
• ser.serializeNumber(n): serializes a 32-bit unsigned integer or signed integer or float.
• des.deserializeNumber(n): deserializes a 32-bit unsigned integer or signed integer or float.
• ser.serializeString(string): serializes a string.
• des.deserializeString(): deserializes a string.
• ser.serializeArray(array, (ser, item) => { ... }): serializes an array.
• des.deserializeArray((des) => { ... }): deserializes an array.
• ser.serializeIterable(iterable, (ser, item) => { ... }): serializes an iterable.
• des.deserializeIterable((des) => { ... }): deserializes an iterable.
• ser.serializeFloat32(float32): serializes float 32bit.
• des.deserializeFloat32(): deserializes float 32bit.
• ser.getBuffer(): returns the serialized buffer.

Benchmarks

We created 3 different benchmarks to compare the performance of SeqProto with JSON and Avro.

Isolated benchmark

You can run the benchmarks with the following command:

npm run benchmark:serdes

Serialization / Deserialization: | name | ops | margin | percentSlower | | -------- | ------- | -------- | ------- | | seqproto | 29764 | 0.72 | 0 | | protobuf | 13698 | 0.19 | 53.98 | | avro | 24204 | 0.14 | 18.68 | | cbor | 803 | 0.22 | 97.3 | | cborx | 9707 | 0.32 | 67.39 | | msgpack | 6857 | 0.06 | 76.96 | | msgpackr | 10449 | 0.27 | 64.89 | | JSON | 14434 | 0.07 | 51.51 |

Http benchmark

You can run the benchmarks using 2 shells.

cd bechmarks/e2e
pnpm install
pnpm start

cd bechmarks/e2e
pnpm run autocannon:json
pnpm run autocannon:seqproto
pnpm run autocannon:avro

| type | req (in 10s) | Avg req/sec | Avg Bytes/Sec | Avg Latency (ms) | | -------- | ---- | --------- | ---- | ---- | | JSON | 164k | 14892 | 275 | 0.11 | | SeqProto | 269k | 26865.6 | 321 | 0.01 | | Avro | 197k | 17926.55 | 169 | 0.04 |

e2e benchmark

You can run the benchmarks with the following command:

cd bechmarks/e2e
pnpm install
pnpm start

And go to http://localhost:3000/public/index.html.

| iteration | parallelism | type | ms | | -------- | ------- | -------- | ------- | | 10 | 1 | JSON | 30.69999998807907 | | 10 | 1 | SeqProto | 25.600000023841858 | | 10 | 1 | Avro | 30.399999976158142 | | 100 | 1 | JSON | 108.80000001192093 | | 100 | 1 | SeqProto | 96.80000001192093 | | 100 | 1 | Avro | 96 | | 100 | 3 | JSON | 162.10000002384186 | | 100 | 3 | SeqProto | 152.4000000357628 | | 100 | 3 | Avro | 167.5 | | 100 | 6 | JSON | 277.19999998807907 | | 100 | 6 | SeqProto | 263.30000001192093 | | 100 | 6 | Avro | 308.19999998807907 |

Contributing

Contributions are welcome! Please open an issue if you have any ideas for improvement or found a bug.

License

Apache-2.0