MemCP — Persistent Main Memory Database with MySQL Compatibility

MemCP is a MySQL-compatible database that keeps your data fully in main memory for maximum speed. Your data is safe — it's persisted to disk automatically and survives restarts — but because there's no disk I/O on the query path, reads and aggregations run 10–100× faster than MySQL.

Use your existing MySQL connector, ORMs, and SQL queries. No migration, no rewrite.

Status: Beta. Core SQL (SELECT, INSERT, UPDATE, DELETE, JOIN, GROUP BY, subqueries, triggers) works well in production. Some advanced SQL edge cases are still being improved — check open issues if a specific query doesn't behave as expected.

Why Switch from MySQL? 💡

⚡ 10-100x Faster on Aggregations and Reports

No disk I/O on the query path — data lives in main memory, always ready
Fast for writes and reads — handles inserts and complex statistics in the same database
Built-in REST API — query your database directly over HTTP, no middleware needed
Sub-millisecond response times even on large tables with GROUP BY and aggregations

Why is it so much faster for statistics? MySQL reads entire rows to compute a SUM() or COUNT(). MemCP stores each column separately, so SELECT region, SUM(revenue) FROM orders GROUP BY region only reads the two columns it actually needs — not every field of every row.

🔌 Drop-in MySQL Compatibility

-- Your existing MySQL queries work immediately
CREATE TABLE users (id INT, name VARCHAR(100), email VARCHAR(255));
INSERT INTO users VALUES (1, 'Alice', 'alice@example.com');
SELECT * FROM users WHERE id = 1;

🌐 Built-in REST API Server

# Start MemCP with REST API
./memcp --api-port=4321 lib/main.scm

# Query via HTTP instantly
curl -X POST http://localhost:4321/sql/mydb \
  -d "SELECT * FROM users" \
  -H "Authorization: Basic cm9vdDphZG1pbg=="

API Endpoints:

/sql/<database> — MySQL-dialect SQL
/psql/<database> — PostgreSQL-dialect SQL
/rdf/<database> — SPARQL queries
/rdf/<database>/load_ttl — load RDF/Turtle data
/dashboard — admin dashboard with live CPU/memory/connection gauges, database browser, shard and compression statistics, and user management

📊 Perfect for Modern Workloads

Microservices - Embedded database per service
APIs and Web Apps - Ultra-low latency responses
Real-time Analytics - Process data as fast as it arrives
Development & Testing - Instant setup, no configuration

Architecture & Languages 🏗️

MemCP combines the best of multiple worlds with a carefully chosen tech stack:

Go (Storage Engine & Core)

High-performance storage engine built in Go
Concurrent request handling with goroutines
Memory-efficient data structures
Cross-platform compatibility

Scheme (SQL Parser & Compiler)

Advanced SQL parser written in Scheme
Query optimization and compilation
Extensible language for complex transformations
Functional programming advantages for parsing

Flexible Scripting Support

Command-line argument support for automation
Dynamic query generation and processing
Easy integration with existing workflows

Key Advantages 🎯

🔥 Ultra-Fast REST APIs

Traditional setup: Client → HTTP Server → Database Connection → Disk I/O MemCP: Client → HTTP Server → Main Memory ✨

// Response times you'll see
MySQL (with network + disk):  10-50ms
MemCP (main memory):          0.1-1ms  // 50x faster!

⚡ Docker

docker pull carli2/memcp
docker run -it -p 4321:4321 -p 3307:3307 carli2/memcp

🧠 Persistent and Safe

Data is written to disk — restarts and crashes don't lose your data
S3, MinIO, and Ceph backends for cloud and distributed deployments
Automatic compression reduces storage footprint significantly vs. MySQL
Configurable data directory — point it at any local or remote path

Memory Management 🧩

MemCP is designed to run alongside other services on the same machine without blowing up your RAM.

Automatic compression — MemCP stores each column in the most compact format that fits the data: small integers get bit-packed, repeated strings become dictionary-encoded, sequential IDs are stored as ranges. A table that takes 10 GB in MySQL often fits in 1–3 GB in MemCP.

Configurable memory budget — by default MemCP uses at most 50% of your server's RAM. You can set an exact limit via the dashboard or the settings API:

# Limit to 4 GB total
curl -u root:admin -X POST http://localhost:4321/dashboard/api/settings \
  -d '{"key":"MaxRamBytes","value":4294967296}'

# Or as a percentage of total RAM (default: 50)
curl -u root:admin -X POST http://localhost:4321/dashboard/api/settings \
  -d '{"key":"MaxRamPercent","value":40}'

Automatic eviction — when MemCP approaches its memory limit, it automatically unloads the least recently used data from RAM. That data stays safe on disk and is transparently reloaded the next time a query needs it. Frequently accessed hot data stays in memory; cold data steps aside.

System-wide pressure awareness — if the whole server runs low on free RAM (below 10%), MemCP detects this and proactively releases its own cache — even if its own budget isn't exhausted yet. This keeps your application, web server, and OS responsive regardless of load spikes.

Separate budget for persistent data — a second budget (default: 30% of RAM) controls how much space the on-disk data loaded into RAM may occupy, independently of temporary query working memory. Both limits are tunable at runtime without restart.

Storage Engines

MemCP supports several storage engines, selectable per table via CREATE TABLE ... ENGINE=<engine> or ALTER TABLE ... ENGINE=<engine>:

| Engine | Data durability | Evictable | Description | |--------|----------------|-----------|-------------| | safe | Persisted to disk (default) | Yes | All writes are immediately durable; data is evicted from RAM when memory is tight and reloaded on demand. | | logged | Persisted to disk | Yes | Like safe but uses a write-ahead log for faster writes with the same durability guarantee. | | sloppy | Persisted to disk | Yes | Data is written to disk asynchronously; faster writes but brief data loss on crash. | | memory | RAM only | No | Data lives entirely in RAM and is never written to disk. Not registered with the memory manager — data is never evicted. Lost on restart. | | cache | Schema only (RAM data) | Yes | Like memory, data lives in RAM and is never written to disk. Unlike memory, the data is registered with the memory manager and will be automatically cleared when MemCP approaches its memory limit. The table schema is always persisted; after eviction the table is accessible but empty. Useful for derived or precomputed data that can be regenerated on demand. |

🔧 Developer-Friendly

Comprehensive test suite with 2470+ SQL tests across 100+ test suites
YAML-based testing framework
Extensive error handling and validation
Built-in performance monitoring

Quick Start 🚀

# 1. Build MemCP
go mod download
make

# 2. Start with REST API
./memcp --api-port=4321 --mysql-port=3307 lib/main.scm

# Run as a background daemon (use --no-repl to avoid exiting when stdin closes)
./memcp --no-repl --api-port=4321 --mysql-port=3307 lib/main.scm &

# 3. Create your first database
curl -X POST http://localhost:4321/sql/system \
  -d "CREATE DATABASE myapp" \
  -u root:admin

# 4. Start building lightning-fast apps!
curl -X POST http://localhost:4321/sql/myapp \
  -d "CREATE TABLE products (id INT, name VARCHAR(100), price DECIMAL(10,2))" \
  -u root:admin

CLI Flags

| Flag | Default | Description | |------|---------|-------------| | --api-port=PORT | 4321 | HTTP API listen port | | --mysql-port=PORT | 3307 | MySQL protocol listen port | | --mysql-socket=PATH | /tmp/memcp.sock | MySQL Unix socket path | | --root-password=PASSWORD | admin | Initial root password (first run only) | | --disable-api | — | Disable HTTP API server | | --disable-mysql | — | Disable MySQL protocol server | | --no-repl | — | Disable interactive REPL (required for daemon/background use) | | -data DIR | ./data | Data directory |

Authentication

Security note: Never expose MemCP directly to the internet with default credentials. Always set a strong --root-password before any network-accessible deployment.

Default credentials: root / admin.
Set the initial root password via CLI: --root-password=supersecret at the first run (on a fresh -data folder), or via Docker env ROOT_PASSWORD.
Docker Compose example:

services:
  memcp:
    image: carli2/memcp:latest
    environment:
      - ROOT_PASSWORD=supersecret
      - PARAMS=--api-port=4321
    ports:
      - "4321:4321"  # HTTP API
      - "3307:3307"  # MySQL protocol
    volumes:
      - memcp_data:/data
volumes:
  memcp_data: {}

Change the credentials with:

curl -X POST http://localhost:4321/sql/system \
  -d "ALTER USER root IDENTIFIED BY 'supersecret'" \
  -u root:admin

Importing Existing Data 📥

MemCP can bulk-import schema and data from MySQL or PostgreSQL with a single Scheme call. The import drops and recreates the target tables on every run, so it is safe to re-run after schema changes.

Import from MySQL

; import all databases (skip system dbs)
(mysql_import nil nil "root" "secret")

; import one specific database
(mysql_import nil nil "root" "secret" "myapp")

; import into a differently-named MemCP database
(mysql_import nil nil "root" "secret" "myapp" "myapp_memcp")

Parameters: host (nil → 127.0.0.1), port (nil → 3306), username, password, sourcedb (nil → all), targetdb (nil → sourced

Memcp

Install / Use

README