content-core

Platform kernel for a content-automation system. Provides provider-neutral model access, workflow execution, retrieval, persistence, evaluation, and a REST service layer. Four pipelines and one operations console are built on it.

Status: actively maintained · 53 tests · 11 architecture decision records

---

Context

This is a side project built by an engineer whose primary background is enterprise Java — Spring Boot services, Kafka event architectures, and JVM operations in financial services. It exists to study LLM system design with the same engineering discipline applied to production backend systems.

The implementation language is Python for reasons documented in ADR-0011 — vendor SDK maturity in this specific domain, not language preference. The architectural patterns are deliberately conventional: layered dependencies, provider abstraction behind a stable interface, constructor injection, repository-style persistence, typed retry at integration boundaries, and instrumentation at a single choke point. That mapping is set out in docs/ENGINEERING_PRINCIPLES.md.

---

Documentation

| Document | Contents | |---|---| | ARCHITECTURE.md | Component boundaries, request flow, execution model, data model, deployment topology | | ENGINEERING_PRINCIPLES.md | Design patterns applied and their enterprise equivalents | | SCALABILITY.md | Scaling sequence with explicit triggers per stage | | PERFORMANCE.md | Where time and cost are spent; optimisations and deliberate non-optimisations | | TESTING.md | Test strategy, substitution seams, coverage and gaps | | deploy/DEPLOY.md | Deployment for Compose and managed platforms | | docs/adr/ | Eleven decision records, including declined technologies |

---

Modules

| Module | Responsibility | |---|---| | content_core.llm | Provider-neutral generation over Anthropic, OpenAI, and Google; retry, exception translation, telemetry emission | | content_core.workflow | Ordered step execution with per-stage retry and failure policy; run reporting | | content_core.retry | Typed retry decorator with exponential backoff | | content_core.rag | Embedding, indexing, and similarity retrieval with content-addressed cache | | content_core.agents | Role-specialised orchestration with a bounded revision loop | | content_core.eval | Rule-based and model-based scoring, prompt registry, benchmarking, experiments | | content_core.feedback | Performance capture and feature correlation producing scoring weights | | content_core.approvals | Threshold-based approval gate with a review queue | | content_core.telemetry | Per-call token, latency, and cost recording; structured logging | | content_core.db | SQLAlchemy models, session lifetime, query surface | | content_core.api | FastAPI service layer with background job execution | | mcp_server.py | Model Context Protocol server exposing platform capabilities as tools |

---

Installation

Optional dependency groups keep consumers from installing what they do not use.

pip install -e .                 # core: provider abstraction, workflow, retry
pip install -e ".[rag]"          # + embedding and vector search
pip install -e ".[db]"           # + SQLAlchemy, Alembic
pip install -e ".[api]"          # + FastAPI, Uvicorn
pip install -e ".[mcp]"          # + MCP server
pip install -e ".[dev]"          # + pytest

Configuration is environment-based. Copy .env.example to .env; no credential is read from source.

---

Usage

Generation through the provider abstraction:

from content_core import LLMProvider

llm = LLMProvider(provider="claude")          # or "openai" / "gemini"
text = llm.generate("...", model="claude-sonnet-4-6", max_tokens=800)

Workflow execution with per-stage policy:

from content_core.workflow import Workflow, Step

workflow = Workflow("daily-run", steps=[
    Step("fetch_topics",  fetch_topics,  retries=2),
    Step("generate",      generate,      retries=2),
    Step("thumbnail",     thumbnail,     on_error="skip"),   # optional stage
    Step("publish",       publish,       retries=3),
])

report = workflow.run(initial_context={"channel": "..."})
print(report.summary())        # per-step status, attempts, duration

Evaluation against golden cases:

from content_core.eval import evaluate

report = evaluate(prompt_template, cases, criteria=[
    {"scorer": "length_bounds", "min_words": 30, "max_words": 90},
    {"scorer": "excludes_all", "forbidden": ["as an AI"]},
    {"scorer": "llm_judge", "rubric": "Rate factual tone and hook strength."},
])

---

Service layer

pip install -e ".[api]"
uvicorn content_core.api.app:app --port 8000
# OpenAPI documentation at /docs

| Endpoint | Purpose | |---|---| | POST /jobs/script, POST /jobs/episode | Submit generation work; returns a job identifier | | GET /jobs/{id}, GET /jobs | Job status and result | | GET /runs | Workflow run history with per-step timing | | GET /metrics/usage | Token and cost aggregates by provider and model | | GET /eval/runs, GET /benchmarks | Evaluation and benchmark history | | GET /approvals, POST /approvals/{id}/decide | Review queue | | GET /feedback/{channel} | Learned scoring weights | | GET /health | Liveness and version |

Authentication is a static API key via X-API-Key, disabled when unset for local development. Rationale in ADR-0004.

---

Persistence

SQLAlchemy models with Alembic migrations. SQLite by default; PostgreSQL by setting DATABASE_URL. The schema uses portable column types only, so engine selection is configuration rather than a code change (ADR-0001).

DATABASE_URL=sqlite:///content_core.db alembic upgrade head

---

Testing

pip install -e ".[dev]"
pytest -q          # 53 tests, no credentials, no network, under six seconds

Vendor SDKs, embedding models, and judge responses are substituted at their seams; persistence runs against a real engine on a temporary file. See TESTING.md.

---

Deployment

cd deploy && cp ../.env.example .env
docker compose up                      # API + console, SQLite
docker compose --profile postgres up   # + PostgreSQL

Managed platform guidance in deploy/DEPLOY.md. Container orchestration was evaluated and not adopted at single-node scale (ADR-0003, and see SCALABILITY.md Stage 6).

---

Decision records

docs/adr/ documents eleven decisions. The declines are the more informative set — each records what the technology solves, why that problem was absent here, and the threshold at which the decision reverses.

Declined: distributed task queue, graph-based agent orchestration framework, multi-provider routing library, distributed cache, container orchestration, token-based identity, distributed tracing stack, hosted vector store, general plugin framework.

---

Consumers

Four content pipelines (three Python, one Node.js) and platform-dashboard, a React and TypeScript operations console.