SkillAgentSearch skills...

ddia-systems

Design data systems by understanding storage engines, replication, partitioning, transactions, and consistency models

Install / Use

npx skills add wondelai/skills --skill ddia-systems

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

84/100

Category

Automation

Supported Platforms

Universal

Our assessment of ddia-systems

ddia-systems scores 84/100 on our quality scale, 2061st of 2,790 Automation skills we index.

Its SKILL.md is 16 KB long, well organised into 15 sections and no code examples: a thorough specification that gives an agent plenty to work with.

With 2,248 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
30/30
Structure
13/20
Description
12/15
Adoption
14/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated about 2 months ago, so ddia-systems is actively maintained.
  • Our last check on 2026-11-05 found the source still online.
  • It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-10-01. Automated pattern scan on 2026-10-01. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

ddia-systems compared with similar skills

All 4 of these similar skills score higher than ddia-systems; compare them before choosing.

SkillScoreStarsUpdatedFormat
ddia-systems (this skill)by wondelai842.2k2mo agoSKILL.md
claude-memby thedotmack100117.6ktodayCLAUDE.md
taste-skillby Leonxlnx100116.8k8d agoCLAUDE.md
Agent-Reachby Panniantong100112.3k8d agoCLAUDE.md
rufloby ruvnet10081.9ktodayCLAUDE.md

Frequently asked questions

How do I install ddia-systems?
Run npx skills add wondelai/skills --skill ddia-systems. The install tabs above show the steps for each supported agent.
Which AI agents does ddia-systems work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is ddia-systems safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is ddia-systems still maintained?
The repository was last updated about 2 months ago, so ddia-systems is actively maintained.

name: ddia-systems description: 'Design data systems by understanding storage engines, replication, partitioning, transactions, and consistency models. Use when the user mentions "database choice", "which database should I use", "SQL or NoSQL", "replication lag", "partitioning strategy", "consistency vs availability", "stream processing", "ACID transactions", "eventual consistency", "my queries are slow at scale", or "data is inconsistent across replicas". Also trigger when choosing a datastore, designing data pipelines, or debugging distributed-system consistency issues. Covers data models, batch/stream processing, and distributed consensus. For system design, see system-design. For resilience, see release-it.' license: MIT metadata: author: wondelai version: "1.4.0"

Designing Data-Intensive Applications Framework

A principled approach to building reliable, scalable, and maintainable data systems. Apply these principles when choosing databases, designing schemas, architecting distributed systems, or reasoning about consistency and fault tolerance.

Core Principle

Data outlives code. Applications are rewritten and frameworks come and go, but data persists for decades -- prioritize the long-term correctness, durability, and evolvability of the data layer. Most applications are data-intensive, not compute-intensive: the hard problems are data volume, complexity, and rate of change, and explicit consistency/availability/latency trade-offs separate robust systems from fragile ones.

Scoring

Goal: 10/10. Score a data architecture by the seven Quick Diagnostic rows below: award ~1.4 points per row answered "yes" with evidence (deliberate, documented trade-off), 0 where the answer is "no" or unknown.

  • 9-10: every domain choice -- data model, storage engine, replication, partitioning, isolation, derived-data, fault handling -- is deliberate, documented, and matched to actual read/write/consistency requirements; failover tested.
  • 5-6: core choices made but two or three diagnostic rows fail -- e.g. default isolation level unknown, hot-key risk unhandled, or failover untested.
  • <=3: choices driven by familiarity, not requirements; ignored failure modes (replication lag, write skew, hot partitions) and accidental complexity dominate.

Report the current score, which diagnostic rows failed, and the improvements needed to reach 10/10.

The DDIA Framework

Seven domains for reasoning about data-intensive systems:

1. Data Models and Query Languages

Core concept: The data model shapes how you think about the problem. Relational, document, and graph models each impose different constraints and enable different query patterns.

Why it works: Choosing the wrong data model forces application code to compensate for representational mismatch, adding accidental complexity that compounds over time.

Key insights:

  • Relational models excel at many-to-many relationships and ad-hoc queries; document models at one-to-many relationships and locality; graph models at recursive traversals over interconnected data
  • Schema-on-write (relational) catches errors early; schema-on-read (document) offers flexibility
  • Polyglot persistence -- different stores for different access patterns -- is often the right answer
  • Object-relational impedance mismatch is a real cost; document models reduce it for self-contained aggregates

Code applications:

| Context | Pattern | Example | |---------|---------|---------| | User profiles with nested data | Document model for self-contained aggregates | Profile, addresses, and preferences in one MongoDB document | | Social network connections | Graph model for relationship traversal | Neo4j Cypher: MATCH (a)-[:FOLLOWS*2]->(b) for friend-of-friend | | Financial ledger with joins | Relational model for referential integrity | PostgreSQL foreign keys between accounts, transactions, entries |

See references/data-models.md when picking relational vs document vs graph or evaluating schema-on-read -- adds the full trade-off matrix and query-language comparisons.

2. Storage Engines

Core concept: Storage engines trade off read performance against write performance. Log-structured engines (LSM trees) optimize writes; page-oriented engines (B-trees) balance reads and writes.

Key insights:

  • LSM trees: append-only writes, periodic compaction, excellent write throughput, higher read amplification
  • B-trees: in-place updates, predictable read latency, write amplification from page splits
  • Write amplification (one logical write causing multiple physical writes) matters for SSDs with limited write cycles
  • Column-oriented storage dramatically improves analytical queries through compression and vectorized processing
  • In-memory databases are fast because they avoid encoding overhead, not because they avoid disk

Code applications:

| Context | Pattern | Example | |---------|---------|---------| | High write throughput | LSM-tree engine | Cassandra or RocksDB for time-series ingestion at 100K+ writes/sec | | Mixed read/write OLTP | B-tree engine | PostgreSQL B-tree indexes for transactional point lookups | | Analytical queries | Column-oriented storage | ClickHouse or Parquet for scanning billions of rows, few columns |

See references/storage-engines.md when a workload is read/write-bound or you must choose indexes -- adds write/read-path diagrams, compaction strategies, column storage, and a benchmark-driven decision procedure.

3. Replication

Core concept: Replication keeps copies of data on multiple machines for fault tolerance, scalability, and latency reduction. The core challenge is handling changes consistently.

Why it works: Every replication strategy trades off consistency, availability, and latency. Making the trade-off explicit prevents subtle anomalies that surface only under load or failure.

Key insights:

  • Single-leader: simple, strong consistency possible, but the leader is a bottleneck and single point of failure
  • Multi-leader: better write availability across data centers, but complex conflict resolution
  • Leaderless: highest availability via quorum reads/writes, but needs careful conflict handling
  • Replication lag causes read-your-writes, monotonic-read, and causality violations
  • Synchronous replication guarantees durability but adds latency; asynchronous risks data loss on failover
  • CRDTs and last-writer-wins resolve conflicts with very different correctness guarantees

Code applications:

| Context | Pattern | Example | |---------|---------|---------| | Read-heavy web app | Single-leader with read replicas | PostgreSQL primary + read replicas behind pgBouncer | | Multi-region writes | Multi-leader replication | CockroachDB or Spanner with bounded staleness | | Shopping cart availability | Leaderless with merge | DynamoDB with last-writer-wins or application-level cart merge |

See references/replication.md when choosing single/multi/leaderless or debugging stale reads -- adds lag anomalies, quorum math, conflict resolution, and CRDTs.

4. Partitioning

Core concept: Partitioning (sharding) distributes data across nodes so each handles a subset, enabling horizontal scaling beyond a single machine.

Key insights:

  • Key-range partitioning supports efficient range scans but risks hotspots on sequential keys
  • Hash partitioning distributes load evenly but destroys sort order, making range queries expensive
  • Local secondary indexes require scatter-gather queries; global secondary indexes require cross-partition updates
  • Hotspots occur even with hashing when a single key is extremely popular (celebrity problem)
  • Rebalancing strategies: fixed partition count, dynamic splitting, or proportional to nodes

Code applications:

| Context | Pattern | Example | |---------|---------|---------| | Time-series data | Key-range partitioning by time + source | Partition by (sensor_id, date) to avoid current-day write hotspot | | User data at scale | Hash partitioning on user ID | Cassandra consistent hashing on user_id for even distribution | | Celebrity/hot-key problem | Key splitting with random suffix | Append random digit to hot key, fan out reads across 10 sub-partitions |

See references/partitioning.md when sharding or fighting a hot key -- adds rebalancing strategies, request routing, and local-vs-global secondary index trade-offs.

5. Transactions and Consistency

Core concept: Transactions provide safety guarantees (ACID) that simplify application code by letting you pretend failures and concurrency don't exist -- within the transaction's scope.

Why it works: Without transactions, every piece of application code must handle partial failures, races, and concurrent modification. Transactions move that complexity into the database, handled correctly once.

Key insights:

  • Isolation levels are a spectrum: read uncommitted, read committed, snapshot isolation, serializable
  • Most databases default to read committed or snapshot isolation -- NOT serializable -- so you must understand the anomalies this permits
  • Write skew: two transactions read the same data, decide, and write different records -- no row lock prevents it
  • Serializable snapshot isolation (SSI) gives full serializability optimistically: no blocking, but aborts on conflict; two-phase locking blocks and deadlocks under contention
  • Distributed transactions (two-phase commit) are expensive and fragile; design around single-partition operations instead

Code applications:

| Context | Pattern | Example | |---------|---------|---------| | Account balance transfer | Serializable transaction | BEGIN; UPDATE accounts ... -100 WHERE id=1; UPDATE accounts ... +100 WHERE id=2; COMMIT; | | Inventory reservation | SELECT FOR UPDATE to prevent write skew | SELECT stock FROM items WHERE id = X FOR UPDATE before decrementing | | Cross-service operations | Saga instead of distributed transaction | Charge card, reserve inventory; on failure, run compensating refund |

See references/transactions.md when setting isolation levels or chasing a concurrency bug -- adds per-isolation anomaly tables, write-skew examples, 2PL vs SSI, and distributed-transaction pitfalls.

6. Batch and Stream Processing

Core concept: Batch processing transforms bounded datasets in bulk; stream processing transforms unbounded event streams continuously. Both compute derived data.

Why it works: Separating the system of record from derived data (caches, indexes, materialized views) lets each be optimized independently and rebuilt from source when requirements change.

Key insights:

  • MapReduce is conceptually simple but operationally awkward; dataflow engines (Spark, Flink) generalize it with arbitrary DAGs
  • Change data capture (CDC) turns database writes into a stream downstream systems can consume
  • Stream-table duality: a stream is the changelog of a table; a table is the materialized state of a stream
  • Exactly-once semantics require idempotent operations or transactional output
  • Time windowing (tumbling, hopping, session) is essential for aggregating unbounded streams

Code applications:

| Context | Pattern | Example | |---------|---------|---------| | Daily analytics pipeline | Batch processing with Spark | Read day's events from S3, aggregate, write to warehouse | | Real-time fraud detection | Stream processing with Flink | Kafka payment events, rules over 5-second tumbling windows | | Syncing search index | Change data capture | Debezium captures PostgreSQL WAL, Kafka feeds Elasticsearch | | Audit trail / event replay | Event sourcing | Store OrderPlaced, OrderShipped events; rebuild state by replaying |

See [references/batch-stream.md](ref

Truncated for display — read the full file on GitHub.

Related Skills

View on GitHub
GitHub Stars2.2k
CategoryAutomation
Updated2mo ago
Forks228

Languages

Shell

Trust signals

100/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

No cautions