Elastic Event Store
A fully serverless event store
Install / Use
npx skills add vladikk/elastic-event-storeInstalls into whichever agent you are using.
README
Elastic Event Store
A fully serverless storage for event sourcing-based systems.
Table of Contents
- What is Event Sourcing?
- What is Event Store?
- Getting Started
- Push Subscriptions
- Pull Subscriptions
- Architecture
- Data Model
- Ordering Guarantees
- Testing
- Limitations
What is Event Sourcing?
Traditionally, software systems operate on state-based data. In other words, business entities and concepts are represented as a snapshot of their current state. E.g.:
| Id | Name | Team | | --- | ---------- | ----------------- | | 1 | Gillian | Administration | | 2 | Krzysztof | Accounting | | 3 | Robyn | Frontend |
In the above example, all we know about the data is its current state. But how did it get to the current state? — We don't know. The Event Sourcing pattern answers this and many other questions.
Event Sourcing introduces the dimension of time into the modeling of business entities and their lifecycles. Instead of capturing an entity's current state, an event-sourced system keeps a transactional record of all events that have occurred during an entity's lifecycle. For example:
{ "id": 3, "type": "initialized", "name": "Robyn", "timestamp": "2021-01-05T13:15:30Z" }
{ "id": 3, "type": "assigned", "team": "Frontend", "timestamp": "2021-01-05T16:15:30Z" }
{ "id": 3, "type": "promoted", "position": "team-leader", "timestamp": "2021-01-22T16:15:30Z" }
By modeling and persisting events, we capture exactly what happened during an entity's lifecycle. Events become the system's source of truth. Hence the name: event sourcing.
Not only can we derive the current state by sequentially applying events, but the flexible event-based model also allows projecting different state models optimized for different tasks.
Finally, Event Sourcing is not Event-Driven Architecture (EDA):
<a name="WhatIsEventStore"/>EventSourcing is not Event driven architecture. The former is about events inside the app. The latter is about events between (sub)systems
~ @ylorph
What is Event Store?
An event store is a storage mechanism optimized for event-sourcing-based systems. It should provide the following functionality:
- Append events to a stream (stream = events of a distinct entity).
- Read events from a stream.
- Concurrency management to detect collisions when multiple processes write to the same stream.
- Enumerate events across all streams (e.g., for CQRS projections).
- Push newly committed events to interested subscribers.
All of the above functions are supported by the Elastic Event Store.
<a name="GettingStarted"/>Getting Started
<a name="Installing"/>Installing
-
Install AWS SAM CLI and configure your AWS credentials.
-
Clone the repository:
git clone https://github.com/doitintl/elastic-event-store.git
cd elastic-event-store
- Build and deploy a new instance:
sam build
# ... Build Succeeded
sam deploy --guided
# ...
# ApiEndpoint: https://XXXXXXXXXXXX.execute-api.XXXXXXXX.amazonaws.com/Prod/
Verify installation:
curl https://XXXXXXXXXXXX.execute-api.XXXXXXXX.amazonaws.com/Prod/version
# { "version": "0.0.1" }
Using
1. Submit a few changesets
EES_URL=https://XXXXXXXXXXXX.execute-api.XXXXXXXX.amazonaws.com/Prod
curl $EES_URL/streams/stream-aaa-111 \
-H 'Content-Type: application/json' \
-X POST \
--data @- <<BODY
{
"metadata": {
"command": "do_something",
"issuedBy": "me"
},
"events": [
{ "type": "init", "data": 1 },
{ "type": "sell", "data": 20 },
{ "type": "buy", "data": 5 }
]
}
BODY
The Elastic Event Store is opinionated about concurrency control: it is mandatory. When committing to an existing stream, you must specify the expected last changeset:
curl "$EES_URL/streams/stream-aaa-111?expected_last_changeset=1" \
-H 'Content-Type: application/json' \
-X POST \
--data @- <<BODY
{
"metadata": {
"command": "do_something_else",
"issuedBy": "me"
},
"events": [
{ "type": "buy", "data": 100 },
{ "type": "buy", "data": 220 },
{ "type": "sell", "data": 15 }
]
}
BODY
2. Fetch changesets:
curl $EES_URL/streams/stream-aaa-111/changesets
3. Fetch events:
curl $EES_URL/streams/stream-aaa-111/events
4. Fetch statistics:
curl $EES_URL/streams
<a name="PushSubscriptions"/>Note: Statistics are updated asynchronously every minute.
Push Subscriptions
The CloudFormation stack includes two SNS FIFO topics:
ees_changesets_XXX_XXX_.fifo— for new changesetsees_events_XXX_XXX_.fifo— for individual events
Pull (Catchup) Subscriptions
To enumerate global changesets:
curl "$EES_URL/changesets?checkpoint=0"
Use the next_checkpoint value to fetch the next batch. This endpoint is critical for CQRS projections and state rebuilds.
Architecture
- REST API exposed via API Gateway
- System logic in AWS Lambda
- Events stored in DynamoDB
- DynamoDB Streams trigger Lambdas for global indexing and publishing
- SNS FIFO topics for push subscriptions
- SQS DLQs for failed stream processing
Data Model
Each partition in the events table represents a stream — i.e., a business entity's event history.
Main DynamoDB schema:
| Column | Type | Description | | ----------------- | --------------------- | ----------- | | stream_id | Partition Key (String) | Stream ID | | changeset_id | Sort Key (Number) | Commit ID in stream | | events | JSON (String) | Committed events | | metadata | JSON (String) | Changeset metadata | | timestamp | String | Commit timestamp | | first_event_id | LSI (Number) | First event ID in stream | | last_event_id | LSI (Number) | Last event ID in stream | | page | GSI Partition (Number) | For global ordering | | page_item | GSI Sort (Number) | Index within global page |
<a name="OrderingGuarantees"/>Ordering Guarantees
- Intra-stream order is preserved and strongly consistent.
- Inter-stream order is not guaranteed but is repeatable — global enumeration always yields the same result.
Testing
- Set the
SAM_ARTIFACTS_BUCKETenvironment variable:
export SAM_ARTIFACTS_BUCKET=your-bucket-name
- Deploy the test environment:
./deploy-test-env.sh
- Run unit tests:
./run-unit-tests.sh
- Run unit and integration tests:
./run-all-tests.sh
Limitations
Because DynamoDB is used:
- Maximum item (changeset) size: 400 KB
- Maximum item collection (stream) size: 10 GB
As with all serverless solutions, at high scale, a self-managed deployment may be more cost-effective.
Related Skills
obsidian
385.5kWork with Obsidian vaults using the official obsidian CLI: read/search/create/edit notes, tasks, links, properties, plugins.
html-anything
8.1k✨ The agentic HTML editor — your local AI agent writes the HTML, you ship it. 🚀 75 Skills × 9 Surfaces (magazine · deck · poster · XHS / tweet · prototype · data report · Hyperframes) 🛡️ Sandboxed preview · 📤 1-click to WeChat / X / Zhihu / HTML / PNG 🔑 Zero API key — Claude Code / Cursor / Code…
openpencil
4.7kThe world's first open-source AI-native vector design tool and the first to feature concurrent Agent Teams. Design-as-Code. Turn prompts into UI directly on the live canvas. A modern alternative to Pencil.
openpencil
4.7kThe world's first open-source AI-native vector design tool and the first to feature concurrent Agent Teams. Design-as-Code. Turn prompts into UI directly on the live canvas. A modern alternative to Pencil.
