Carquet

<p align="center"> <img src="res/img/carquet_logo.png" alt="Carquet" width="280" /> </p> <h1 align="center">Carquet</h1> <p align="center"> A fast, pure C library for reading and writing Apache Parquet files. </p> <p align="center"> <a href="https://github.com/Vitruves/carquet/actions/workflows/cpp.yml"><img src="https://github.com/Vitruves/carquet/actions/workflows/cpp.yml/badge.svg" alt="Build" /></a> <img src="https://img.shields.io/badge/platform-Linux%20%7C%20macOS%20%7C%20Windows-blue" alt="Platform" /> <img src="https://img.shields.io/badge/C-C11-blue" alt="C Standard" /> <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-green" alt="License" /></a> <br/> <img src="https://img.shields.io/badge/SIMD-SSE4.2%20%7C%20AVX%20%7C%20AVX2%20%7C%20AVX--512-red" alt="x86 SIMD" /> <img src="https://img.shields.io/badge/SIMD-NEON%20%7C%20SVE-orange" alt="ARM SIMD" /> </p>

Highlights

• Pure C11 with three external dependencies (zstd, zlib, lz4) -- all auto-fetched by CMake
• ~200KB binary vs ~50MB+ for Arrow
• Built-in CLI for file inspection (schema, info, head, tail, stat, ...) and C code generation (codegen)
• 70x faster reads than Arrow C++ on uncompressed data (mmap zero-copy), 150x faster than PyArrow
• 1.2-2.6x faster compressed reads than Arrow C++ on the same file (cross-read benchmark)
• Writes 1.0-2.3x faster than Arrow C++ across codecs and platforms
• Reads 10M uncompressed rows in 0.25ms (mmap zero-copy on Apple M3)
• Full Parquet spec: all types, encodings, compression codecs, nested schemas, bloom filters, page indexes
• SIMD-optimized (SSE4.2, AVX2, AVX-512, NEON, SVE) with runtime detection and scalar fallbacks
• PyArrow, DuckDB, Spark compatible out of the box

Performance

Carquet vs Arrow C++ 23.0.1 at 10M rows (the most representative size). Higher ratio = Carquet faster.

| | x86 (Xeon D-1531) | | ARM (Apple M3) | | |---|---|---|---|---| | Codec | Write | Read | Write | Read | | snappy | 1.55x | 1.25x | 1.10x | 1.53x | | zstd | 1.31x | 1.04x | 1.37x | 1.28x | | lz4 | 1.02x | 0.83x | 1.25x | 0.96x | | none | 1.13x | 40.6x* | 1.33x | 70.4x* |

* Uncompressed reads use mmap zero-copy -- see note below.

Compressed reads involve full decompression and decoding of every value, no shortcuts — and both libraries use the same system lz4/zstd shared libraries, so the raw codec speed is identical. The most meaningful comparison is the same-file cross-read table (below), where both libraries read the exact same Parquet file: Carquet reads compressed data 1.5-2.6x faster than Arrow C++ on that apples-to-apples test.

<details> <summary>Benchmark methodology</summary>

All benchmarks use identical data (deterministic LCG PRNG), identical Parquet settings (no dictionary, BYTE_STREAM_SPLIT for floats, page checksums, mmap reads), trimmed median of 11-51 iterations, with OS page cache purged between write and read phases and cooldown between configurations. Schema: 3 columns (INT64, DOUBLE, INT32). Compared against Arrow C++ 23.0.1 low-level Parquet reader (bypassing Arrow Table materialization) and PyArrow 23.0.1.

The same-file cross-read benchmark is the fairest comparison: both libraries read the exact same Parquet file (written by one, read by both). This eliminates differences in page sizes, encoding choices, and row group layout.

Uncompressed reads marked with * use Carquet's mmap zero-copy path: for PLAIN-encoded, uncompressed, fixed-size, required columns, the batch reader returns pointers directly into the memory-mapped file with no memcpy. Arrow always materializes into its own buffers. The compressed read numbers are the most representative measure of end-to-end read throughput.

</details> <details> <summary>Full x86 results (Intel Xeon D-1531, Linux)</summary>

12 threads @ 2.2GHz, 32GB RAM, Ubuntu 24.04 -- ZSTD level 1

10M rows vs Arrow C++

| Codec | Carquet Write | Arrow C++ Write | W ratio | Carquet Read | Arrow C++ Read | R ratio | Size | |-------|--------------|-----------------|---------|-------------|----------------|---------|------| | none | 1557ms | 1766ms | 1.13x | 1.25ms | 50.8ms | 40.6x* | 190.7MB | | snappy | 1002ms | 1549ms | 1.55x | 78ms | 97.8ms | 1.25x | 125.1MB | | zstd | 1311ms | 1714ms | 1.31x | 76.8ms | 80.2ms | 1.04x | 95.3MB | | lz4 | 1521ms | 1554ms | 1.02x | 59.1ms | 49.0ms | 0.83x | 122.9MB |

1M rows vs Arrow C++

| Codec | Carquet Write | Arrow C++ Write | W ratio | Carquet Read | Arrow C++ Read | R ratio | |-------|--------------|-----------------|---------|-------------|----------------|---------| | none | 180ms | 196ms | 1.09x | 0.22ms | 6.2ms | 28x* | | snappy | 141ms | 148ms | 1.05x | 8.1ms | 11.6ms | 1.44x | | zstd | 131ms | 185ms | 1.41x | 10.3ms | 9.1ms | 0.88x | | lz4 | 143ms | 149ms | 1.04x | 8.5ms | 6.1ms | 0.72x |

100K rows vs Arrow C++

| Codec | Carquet Write | Arrow C++ Write | W ratio | Carquet Read | Arrow C++ Read | R ratio | |-------|--------------|-----------------|---------|-------------|----------------|---------| | none | 14.1ms | 18.4ms | 1.30x | 0.11ms | 2.18ms | 19.8x* | | snappy | 10.1ms | 10.6ms | 1.05x | 1.27ms | 5.97ms | 4.70x | | zstd | 8.7ms | 14.1ms | 1.62x | 1.58ms | 3.88ms | 2.46x | | lz4 | 9.6ms | 11.0ms | 1.14x | 0.77ms | 2.78ms | 3.61x |

Same-file cross-read (10M rows)

Both libraries read the same Parquet file — the fairest apples-to-apples comparison.

| Codec | Writer | Carquet Read | Arrow C++ Read | Ratio | |-------|--------|-------------|----------------|-------| | none | Carquet | 0.99ms | 73.6ms | 74x* | | none | Arrow | 7.6ms | 51.2ms | 6.8x* | | snappy | Carquet | 41.0ms | 107ms | 2.61x | | snappy | Arrow | 43.4ms | 101ms | 2.33x | | zstd | Carquet | 46.1ms | 88.4ms | 1.92x | | zstd | Arrow | 49.1ms | 79.5ms | 1.62x | | lz4 | Carquet | 34.8ms | 74.8ms | 2.15x | | lz4 | Arrow | 27.4ms | 52.0ms | 1.90x |

10M rows vs PyArrow

| Codec | Carquet Write | PyArrow Write | W ratio | Carquet Read | PyArrow Read | R ratio | |-------|--------------|---------------|---------|-------------|--------------|---------| | none | 1557ms | 1806ms | 1.16x | 1.25ms | 213ms | 170x* | | snappy | 1002ms | 1649ms | 1.65x | 78ms | 384ms | 4.91x | | zstd | 1311ms | 1796ms | 1.37x | 76.8ms | 369ms | 4.81x | | lz4 | 1521ms | 1676ms | 1.10x | 59.1ms | 281ms | 4.76x |

* Zero-copy mmap path

</details> <details> <summary>Full ARM results (Apple M3, macOS)</summary>

MacBook Air M3, 16GB RAM, macOS 26.2, Arrow C++ 23.0.1, PyArrow 23.0.1 -- ZSTD level 1

10M rows vs Arrow C++

| Codec | Carquet Write | Arrow C++ Write | W ratio | Carquet Read | Arrow C++ Read | R ratio | Size | |-------|--------------|-----------------|---------|-------------|----------------|---------|------| | none | 99.4ms | 131.9ms | 1.33x | 0.25ms | 17.59ms | 70.4x* | 190.7MB | | snappy | 231.0ms | 253.1ms | 1.10x | 16.15ms | 24.75ms | 1.53x | 125.1MB | | zstd | 253.3ms | 347.5ms | 1.37x | 22.91ms | 29.38ms | 1.28x | 95.3MB | | lz4 | 198.3ms | 248.8ms | 1.25x | 18.90ms | 18.05ms | 0.96x | 122.9MB |

1M rows vs Arrow C++

| Codec | Carquet Write | Arrow C++ Write | W ratio | Carquet Read | Arrow C++ Read | R ratio | |-------|--------------|-----------------|---------|-------------|----------------|---------| | none | 7.57ms | 12.91ms | 1.71x | 0.05ms | 1.77ms | 35.4x* | | snappy | 13.43ms | 24.50ms | 1.82x | 1.52ms | 2.55ms | 1.68x | | zstd | 15.05ms | 34.12ms | 2.27x | 2.29ms | 3.06ms | 1.34x | | lz4 | 13.09ms | 25.11ms | 1.92x | 1.03ms | 1.74ms | 1.69x |

100K rows vs Arrow C++

| Codec | Carquet Write | Arrow C++ Write | W ratio | Carquet Read | Arrow C++ Read | R ratio | |-------|--------------|-----------------|---------|-------------|----------------|---------| | none | 1.13ms | 1.56ms | 1.38x | 0.02ms | 0.23ms | 11.5x* | | snappy | 1.64ms | 2.50ms | 1.52x | 0.37ms | 0.90ms | 2.43x | | zstd | 1.69ms | 3.52ms | 2.08x | 0.64ms | 1.31ms | 2.05x | | lz4 | 1.58ms | 2.49ms | 1.58x | 0.25ms | 0.57ms | 2.28x |

Same-file cross-read (10M rows)

Both libraries read the same Parquet file — the fairest apples-to-apples comparison.

| Codec | Writer | Carquet Read | Arrow C++ Read | Ratio | |-------|--------|-------------|----------------|-------| | none | Carquet | 0.36ms | 18.33ms | 50.9x* | | none | Arrow | 1.01ms | 17.60ms | 17.4x* | | snappy | Carquet | 20.54ms | 24.52ms | 1.19x | | snappy | Arrow | 14.91ms | 23.65ms | 1.59x | | zstd | Carquet | 23.11ms | 34.71ms | 1.50x | | zstd | Arrow | 22.03ms | 29.87ms | 1.36x | | lz4 | Carquet | 10.96ms | 18.54ms | 1.69x | | lz4 | Arrow | 10.54ms | 17.43ms | 1.65x |

10M rows vs PyArrow

| Codec | Carquet Write | PyArrow Write | W ratio | Carquet Read | PyArrow Read | R ratio | |-------|--------------|---------------|---------|-------------|--------------|---------| | none | 99.4ms | 193.4ms | 1.95x | 0.25ms | 37.64ms | 150.6x* | | snappy | 231.0ms | 306.3ms | 1.33x | 16.15ms | 48.01ms | 2.97x | | zstd | 253.3ms | 405.7ms | 1.60x | 22.91ms | 61.63ms | 2.69x | | lz4 | 198.3ms | 309.4ms | 1.56x | 18.90ms | 40.09ms | 2.12x |

1M rows vs PyArrow

| Codec | Carquet Write | PyArrow Write | W ratio | Carquet Read | PyArrow Read | R ratio | |-------|--------------|