Cleanframe

<p align="center"> <img src="https://raw.githubusercontent.com/inboxpraveen/Cleanframe/main/docs/assets/Logo.svg" alt="CleanFrame logo" width="120" /> </p> <h1 align="center">CleanFrame</h1> <p align="center"> <b>The reproducible data-cleaning engine for Python.</b><br/> AI decides once. The recipe runs forever — deterministic, diffable, reviewable. </p> <p align="center"> <a href="https://inboxpraveen.github.io/Cleanframe/"><b>▶ Try it in your browser</b></a> (nothing to install, your file never leaves the tab) &nbsp;·&nbsp; <code>pip install cleanframe-engine && cleanframe demo</code> </p> <p align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/inboxpraveen/Cleanframe/main/docs/assets/Hero-dark.png"> <img src="https://raw.githubusercontent.com/inboxpraveen/Cleanframe/main/docs/assets/Hero.png" width="100%" alt="Real output of cleanframe demo: a messy customers CSV becomes clean data through a reviewable YAML recipe. Changed cells are highlighted, a row with an invalid email is held in quarantine, and next month's renamed column stops the replay."/> </picture> </p> <p align="center"> <a href="https://pypi.org/project/cleanframe-engine/"><img src="https://img.shields.io/pypi/v/cleanframe-engine" alt="PyPI"/></a> <a href="https://github.com/inboxpraveen/Cleanframe/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-blue" alt="License"/></a> <a href="https://github.com/inboxpraveen/Cleanframe/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/inboxpraveen/Cleanframe/ci.yml?branch=main" alt="CI"/></a> <img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python"/> <img src="https://img.shields.io/badge/LLM-optional-success" alt="LLM optional"/> <a href="https://github.com/inboxpraveen/Cleanframe/wiki"><img src="https://img.shields.io/badge/docs-wiki-informational" alt="Wiki"/></a> <a href="https://inboxpraveen.github.io/Cleanframe/"><img src="https://img.shields.io/badge/playground-in_browser-ff6f00" alt="Playground"/></a> <a href="https://github.com/inboxpraveen/Cleanframe/blob/main/docs/decisions.md"><img src="https://img.shields.io/badge/decision_models-Jev--style-8a2be2" alt="Decision models"/></a> </p>

---

Every team has that file. The vendor spreadsheet that arrives monthly with dates in three formats. The CRM export where Bengaluru, Bangalore, and BLR are three different cities. The finance sheet with ₹1,20,000 in a column typed as text.

You clean it by hand. Next month, it arrives broken in a new way, and you clean it again.

CleanFrame ends that loop. It profiles your data, detects issues, proposes a cleanup plan (with an LLM's help — or without one), executes it with pure pandas, and saves the whole thing as a recipe: a versionable YAML file that replays on every future file with zero AI calls, and alerts you when the incoming schema drifts.

AI suggests.  Pandas executes.  Rules validate.  You approve.  Everything is reproducible.

Documentation

| | | |--|--| | Wiki (full docs) | Getting started, API, recipe/schema specs, production guide | | docs/ | Same guides in-repo | | CONTRIBUTING.md | Invariants + how to add detectors/ops | | SECURITY.md | Vulnerability reporting | | CHANGELOG.md | Release notes |

Why not just use an LLM agent on my dataframe?

Because you can't ship "the model probably fixed it" to production. Chat-with-your-data tools are great for exploration and say so themselves — they are not built for pipelines. CleanFrame is built on one rule:

The LLM never touches your data. It only writes the plan.

The plan compiles to deterministic pandas operations. Same input → same output, every time. Every changed cell is tracked. Every step is reviewable, reversible, and exportable as plain Python you can read.

30-second demo

No API key, no sample files to download. The demo data ships inside the package:

pip install cleanframe-engine
cleanframe demo

It profiles a messy customer file (HTML report + quality score), cleans it into a recipe, then replays that recipe on next month's file. That file has a renamed column and a new date format, so the replay stops and tells you why. Finally the demo drafts the patch that makes it replay cleanly. Every step prints the command to run on your own data.

The same thing in Python:

import pandas as pd
import cleanframe as cf

df = pd.read_csv("cleanframe-demo/messy_customers.csv")      # written by `cleanframe demo`

result = cf.clean(
    df,
    target_schema="cleanframe-demo/customer.schema.yaml",    # or: cf.infer_schema(df)
    mode="review",                                           # review | auto | strict
    # decide="typesafe/jev-latest",   # optional decision model for judgment calls (TYPESAFE_API_KEY)
)

result.diff.show()                # cell-level before/after, git-diff style
result.recipe.save("customer.recipe.yaml")   # ← the durable artifact
result.code.save("clean_customers.py")       # plain pandas, no cleanframe dependency
clean_df = result.dataframe
result.quarantine                 # rows validation held back, with reasons — never deleted

The sample file has one genuinely invalid email, so the demo quarantines that row and cleans the other five.

Next month, the file comes back. No LLM, no tokens, no variance:

cleanframe apply new_customers.csv --recipe customer.recipe.yaml --out clean.csv

If the new file's schema drifted (a renamed column, a new currency format), CleanFrame stops and tells you instead of silently corrupting data:

⚠ Schema drift detected in new_customers.csv
  • Column "Amt (INR)" is new — 94% match to recipe column "amount_inr"
  • 312 value(s) in "signup_date" match no allowed date format (new: 'Jan 5, 26')
  • 40 of 312 numeric value(s) in "Amount" do not fit the recipe's number format
    (decimal '.', thousands ','; new: '€1.200,50')
  Run `cleanframe suggest new_customers.csv --recipe <recipe.yaml> --update` to review a patch.

Decision models: AI makes the judgment calls, under a 0.9 bar

Some cleaning steps are judgment calls, not parsing. Is BLR another name for Bangalore? Is Banglore a typo, or a different place? Does none in an allergies column mean "missing" or "no allergies"? Which column is the schema's amount_inr? Rules can't settle these safely. With decide=, a decision model answers each one as a typed question (pick an option, or yes/no) with a probability. That can be Jev, an open "System One" model, or any LLM through an adapter.

cleanframe clean vendor.csv --decide typesafe/jev-latest --llm-exposure sample

• An answer changes data only at p ≥ 0.9 (decision_threshold). Below that it is recorded as a suggestion in the recipe and never executed. A confident "no, these differ" vetoes a typo merge the rules would have made.
• Asked once, at planning time. Accepted answers become ordinary recipe ops (normalize_values, to_na, rename_to), and every answer is logged in recipe.meta.decisions. Replay makes zero AI calls.
• Privacy follows llm_exposure. Category values are sent only under sample, and only for columns whose values repeat. Dates and numbers are never asked: they stay with the deterministic parsers.
• Backends: typesafe/…, openrouter/typesafe/…, self-hosted systemone/…, offline rules, or any --llm spec. No SDK is needed.

How good are decision models at this? CleanBench measures it on hand-labelled messy columns full of traps, scoring safe-apply precision: of the answers that would change your data, how many are right. See Decision models.

Bring your own API key (or no key at all)

CleanFrame is LLM-optional and provider-agnostic.

| Mode | What runs | Data leaves your machine? | |---|---|---| | Rules-only (default) | Deterministic detectors + heuristics | Never | | Metadata | LLM sees column names, dtypes, and value patterns (regex sketches) — never raw values | Only metadata | | Sample | LLM sees a small shuffled sample of values. Emails, phones and long strings are redacted; other short values (names, cities) are sent verbatim. Passing llm_exposure="sample" is the opt-in — there is no separate approval step | Sample values, per the redaction above | | Replay | Saved recipes | Never — recipes need no LLM | | Decisions (decide=) | Typed questions to a decision model while planning; same exposure rules | Metadata only by default; distinct category values under sample |

• Keys come from environment variables (ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, GROQ_API_KEY, …). CleanFrame never stores or logs them, and each provider reads only its own variable — OPENAI_API_KEY is never sent to another provider.
• Any provider that speaks OpenAI Chat Completions works out of the box — Anthropic (native), OpenAI, OpenRouter, Groq, Together, Fireworks, DeepSeek, Mistral, Google Gemini, xAI, Perplexity, Cohere, plus local Ollama / LM Studio or any custom OPENAI_BASE_URL. Spec format: provider/model (e.g. openrouter/anthropic/claude-sonnet-4, groq/llama-3.3-70b-versatile).
• Cost cap: cf.clean(..., max_tokens_budget=50_000) is checked before each call (against an estimate) and again against the usage the provider reports — leave headroom, since a call that overshoots has already been billed.
• Fully offline / air-gapped operation is a supported first-class mode, not a degraded one.

What it cleans

| Problem | Example | |---|---| | Invisible characters | zero-width spaces, BOMs, non-breaking spaces and decomposed accents (café typed two ways) → one normalised form | | Column name chaos | Cust Name, customer_name, CustomerName → customer_name | | Date formats | 12/01/24, 1 Jan 2024, 2024-01-01 → ISO dates | | Currency & numbers | ₹1,20,000, $1,200, -$5, (1,200), 1.234,56, 1200 INR → typed floats (+ a currency column); a value whose grouping is invalid or ambiguous becomes a counted, warned null, never a guess | | Category variants | Bengaluru / bengaluru / BENGALURU and typos (Banglore, one edit away) → one canonical value; look-alikes such as Austria/Australia stay apart; abbreviations such as BLR need a seed map or a decision model | | Emails & phones | validation, normalization, 00/+ international prefixes, optional default country code | | Duplicates | exact duplicates are dropped (tracked in the diff); fuzzy near-duplicates and key collisions are reported with keep/drop proposals for you to act on | | Missing values | detected and explained; strategies proposed, never silently applied | | Units | 5kg, 5000 g, 5 KG → normalized | | Schema mapping | messy file → your target schema, with confidence scores | | Outliers | flagged with evidence — detected, never auto-"fixed" |

Production-ready defaults

CleanFrame is built for pipelines, not just demos:

• Plan once, replay forever — commit recipes; fail on schema drift
• Multi-sheet workbooks — clean every Excel tab into one reviewable recipe; write-back refuses to overwrite the source in place
• Out-of-core streaming — replay row-independent recipes over larger-than-RAM CSVs at chunk-bounded memory
• **Format