MWire - Magenta Wire Format

High-performance, schema-agnostic binary data serialization format and CLI tool.

Overview

MWire is a compact, efficient binary format designed for:

IoT & Telemetry - Sensor data, device metrics
Time-Series Data - Logs, events, monitoring
Any Structured Data - General-purpose serialization

Key Features

Fast: 50,000-100,000+ records/sec throughput
Compact: 50-70% smaller than JSON (up to 84% with compression)
Verified: Zero-loss round-trip validation
Reliable: Built-in CRC32 checksums
Research-Grade: Benchmarking and statistics tools
Schema-Agnostic: Not limited to telemetry use cases

Installation

Build from Source

git clone https://github.com/ndourc/mwire.git
cd mwire
cargo build --release --bin mwire

Binary location: target/release/mwire

Usage

1. Encode JSON to MTF

mwire encode -i data.json -o data.mtf --compress

2. Decode MTF to JSON

mwire decode -i data.mtf -o data.json --pretty

3. Inspect MTF Files

mwire inspect dataset.mtf --first 1 --last 10

4. Verify Integrity

mwire verify data.json

5. Analyze Performance

mwire benchmark data.json --iterations 1000

6. View Statistics

mwire stats dataset.mtf

Documentation

| Document | Description | |----------|-------------| | Quick Start | 5-minute tutorial to get started | | Full Documentation | Complete user guide | | API Reference | Detailed command reference | | Format Specification | MTF binary format spec (MTS-1) | | Implementation Plan | Roadmap and planned features |

Use Cases

IoT Data Pipeline

# Compress sensor data for transmission
mwire encode -i sensors.json -o sensors.mtf --compress

# 70% size reduction - save bandwidth and storage costs

Research & Benchmarking

# Generate publishable performance data
mwire benchmark dataset.json --iterations 10000 --json > results.json

# Prove zero-loss serialization
mwire verify dataset.json

CI/CD Integration

# Validate data quality in pipelines
mwire verify export.json || exit 1
mwire stats export.mtf --json | jq '.data_completeness_percent'

Performance

Throughput

Encode: ~65,000 records/second
Decode: ~112,000 records/second
With LZ4: ~45,000 encode / 95,000 decode per second

Size Comparison (vs JSON)

| Format | Size (bytes) | Ratio | Reduction | |--------|--------------|-------|-----------| | JSON (pretty) | 582 | 1.00x | - | | JSON (compact) | 387 | 0.66x | 34% | | MessagePack | 245 | 0.42x | 58% | | CBOR | 238 | 0.41x | 59% | | MTF | 137 | 0.24x | 76% | | MTF + LZ4 | 94 | 0.16x | 84% |

Available Commands

| Command | Description | |---------|-------------| | encode | Convert JSON to MTF binary | | decode | Convert MTF binary to JSON | | inspect | View MTF file contents | | verify | Round-trip integrity check | | benchmark | Performance analysis | | stats | Dataset statistics | | stress-test | High-volume performance test |

See API Reference for detailed documentation.

Project Structure

mwire/
├── cli/                  # MWire CLI tool
│   └── src/
│       ├── main.rs      # CLI entry point
│       ├── encode.rs    # JSON → MTF
│       ├── decode.rs    # MTF → JSON
│       ├── inspect.rs   # File inspection
│       ├── verify.rs    # Round-trip verification
│       ├── benchmark.rs # Performance testing
│       └── stats.rs     # Statistics analysis
├── core/                # Core MTF format implementation
│   └── src/
│       ├── encoder.rs   # Binary encoder
│       ├── decoder.rs   # Binary decoder
│       ├── types.rs     # Data structures
│       └── compression.rs # LZ4 compression
├── python/              # Python bindings
├── docs/                # Documentation
│   ├── README.md        # Full documentation
│   ├── API.md          # API reference
│   ├── FORMAT.md       # Format specification
│   └── QUICKSTART.md   # Quick start guide
└── examples/            # Example files

Format Specification

MTS-1 (Magenta Telemetry Specification v1.0)

Packet Structure

┌─────────────────────────────────┐
│ HEADER (15 bytes)               │
│  - Version, flags, timestamp    │
│  - Sequence number              │
├─────────────────────────────────┤
│ VARIABLE DATA                   │
│  - Device ID (UTF-8)            │
│  - UUID (16 bytes)              │
│  - Field-encoded payload        │
├─────────────────────────────────┤
│ CHECKSUM (CRC32, 4 bytes)       │
└─────────────────────────────────┘

Features:

Little-endian encoding
Sparse field encoding (only present fields)
Optional LZ4 compression
Delta encoding support
Built-in checksums for integrity

See FORMAT.md for complete specification.

Language Bindings

| Language | Status | Package | |----------|--------|---------| | Rust | Native | magenta-telemetry-core | | Python | Available | magenta-telemetry (PyPI) | | JavaScript | Planned | TBD | | Go | Planned | TBD |

Roadmap

Phase 1: Foundation (Complete)

Core encode/decode
CLI tool (MWire)
Inspection and verification
Benchmarking and statistics
Python bindings

Phase 2: Flexibility (In Progress)

Schema validation framework
STDIN/STDOUT pipeline support
Custom schema definitions
CSV/Parquet export

Phase 3: Advanced Features

Encryption support
Digital signatures
Delta encoding visualization
Synthetic data generation
Memory profiling

Phase 4: Ecosystem

Schema registry
Migration tools (v1→v2)
Plugin system
Additional language bindings

Contributing

Contributions are welcome! Please:

Fork the repository
Create a feature branch
Make your changes
Add tests
Submit a pull request

See CONTRIBUTING.md for details.

📄 License

MIT License - see LICENSE for details.

🙋 Support

Issues: GitHub Issues
Documentation: docs/
Examples: examples/

🏆 Acknowledgments

Built with:

Rust - Systems programming language
LZ4 - Ultra-fast compression
CRC32Fast - Hardware-accelerated checksums

📈 Metrics

Binary Size: ~2MB (release build)
Dependencies: Minimal (Rust only)
Test Coverage: >90%
Documentation: Comprehensive

MWire: Fast. Compact. Verified.

Efficient data serialization for the modern world.

Mwire

Install / Use

README