<div align="center"> <img src="assets/logo.png" alt="" width="104" height="104">

Toadstone

A toadstone was a gem people wore to detect poison. It never detected anything.<br> This one is named after that, on purpose.

<table> <tr> <td width="50%" align="center">

▶  Open the live demo

The real interface, frozen over a real crawl.<br> Click the tabs. Open a finding. No sign-up.

</td> <td width="50%" align="center">

🎬  Watch the tour

Two minutes. Why it exists, what it found,<br> and every view in the app.

</td> </tr> </table>

Why this exists · Quick start · What it finds · Screenshots · How it compares · For agents · Architecture · Security · FAQ

</div> <picture> <source media="(prefers-color-scheme: dark)" srcset="assets/screens/hero-dark.png"> <img alt="Toadstone's issues view: findings grouped by check, each with its machine-readable slug and a count" src="assets/screens/hero-light.png"> </picture> <div align="center">_{A real crawl of <a href="https://freshlycited.com/">freshlycited.com</a>, a site we own and maintain: 141 URLs, 8,374 links, <b>zero errors</b> — and nine things worth fixing anyway, each with its machine-readable slug. <a href="#reproduce-the-screenshots">Reproduce it</a> against your own site, or against the broken fixture that ships in this repo.}</div>

---

Why this exists

I run an SEO team of 22 people. Screaming Frog is licensed per machine.

That is not a complaint about the price — it is a good tool and worth its licence. It is that the shape is wrong for how we work. We are in Ahrefs every day. We need a full technical crawl maybe five or six times a month. Paying a seat for every executive so each of them can run a crawl a handful of times a month is paying for the wrong thing.

What I wanted was one instance I host myself, on my own Mac, that the whole team can reach — and that Claude Code can talk to directly over MCP, so a crawl comes back as a list of issues to fix rather than a CSV someone has to open, read and turn into tickets.

Nothing available had that shape. So I forked an open-source crawler and started building.

claude mcp add toadstone -- python -m src.mcp

That is the whole integration. Ten tools, read-only by default, unmetered — because they are your crawls, on your machine. "Crawl freshlycited.com and tell me what to fix" comes back as findings grouped by check, each with its evidence and a one-line fix, ready to become tickets. No credits, no units, no per-seat anything. See the tools →

Then I found something I was not looking for

Most SEO crawlers advertise how many checks they have. That number is almost meaningless, and here is why.

I counted how many of the ones I had inherited had ever been observed to fire on a real page. There were about 225. Eighty-four of them had ever fired. The rest passed their unit tests and sat inert:

• eight AMP checks that could not fire on any site, because the markup they read was discarded before they ran;
• every PDF check, dead from the moment JavaScript rendering was switched on;
• a raw-versus-rendered comparison that had been comparing a document against itself;
• a failed PageSpeed run that reported as a clean bill of performance — the API returned a quota error, the integration swallowed it, and the report came back with no performance issues at all;
• three "checks" that were not checks, only sample data left in a test file.

None of that looked broken. Every report came back clean and confident, and clean was the wrong answer.

So the project built a site that is wrong in as many ways as possible, crawls it with the real crawler, and records which check slugs actually appear.

The last sweep, run while this README was being written, fired 180 distinct checks across the fixture's six modes — 162 in the default run alone, on 77 crawled pages. Here is that sweep, in full:

python tools/coverage_sweep.py

--- default                 77 pages crawled, 162 checks fired
--- sitemap-limits-1         3 pages crawled,  76 checks fired   50,001 URLs, 51 children
--- sitemap-limits-2         3 pages crawled,  70 checks fired   a sitemap listing nothing
--- sitemap-limits-3         3 pages crawled,  75 checks fired   52MB of sitemap
--- silent-image            85 pages crawled, 166 checks fired   a handler that answers nothing
--- https                    4 pages crawled,  30 checks fired   the checks that need TLS

180 distinct checks fired across 6 mode(s)

Each mode needs a crawl of its own — a sitemap of 50,001 URLs cannot share a run with one that works — and the sweep keeps a separate slug log per mode, because the harness truncates its log at the start of every run. (Six runs pointed at one file gives you the sixth mode's slugs and a confidently wrong total. That mistake was made while writing this, and it reported 30.)

BUILD-LOG.md carries the rest of the accounting day by day — including the thirteen checks that rest on a stubbed transport with real parsing either side, the one that rests on a model's verdict being supplied rather than asked for, the two that cannot honestly be reached at all, and every wrong turn taken on the way. It is written in the first person and it does not flatter the work.

Those commands are the whole argument. Run them before you trust anything above.

---

What it is

A self-hosted SEO site crawler with a browser UI, an HTTP API, and an MCP server so an AI agent can query crawls directly. It renders JavaScript, keeps crawls so it can tell you what changed since the last one, and reports findings with evidence a developer can act on without re-fetching the page.

It is a fork of LibreCrawl by Phiality, with roughly a hundred commits of checks, integrations, API, agent support and UI work on top. See FORK.md for exactly what the fork adds.

No per-URL pricing, no seats, no cloud. Point it at a site you have permission to crawl and it runs on your machine.

What a finding looks like

Every finding, wherever it surfaces — UI, API, export, MCP — carries the same shape. This one is real, from the crawl in the screenshot above:

{
  "id": "69230cd05f9f7ca9ec7a66ee0f91a85ccf9f0d90",
  "check": "links_to_broken_urls",
  "type": "error",
  "category": "Technical",
  "issue": "Page Links to Broken URLs",
  "url": "http://toadstone-demo.localtest.me:8977/",
  "details": "1 link(s) on this page are broken: http://toadstone-demo.localtest.me:8977/missing-entirely.png (404)",
  "evidence": {
    "found": "1 broken links",
    "related_urls": ["http://toadstone-demo.localtest.me:8977/missing-entirely.png"]
  },
  "how_to_fix": "Update or remove the link; a visitor following it hits an error.",
  "docs_url": "/checks#links_to_broken_urls",
  "status": "open",
  "first_seen_crawl": 2
}

Two things in there do more work than they look like they do:

• id is sha1(url + check + offending value). The same defect keeps the same identity between crawls. A fixed issue never comes back as "new", an accepted one stays accepted, and first_seen_crawl tells you when it appeared. Most tools re-number their findings every run, which makes "what changed?" unanswerable.
• check is a slug, never free text. You can filter on it, alert on it, and diff on it. docs_url points at the generated reference page for it.

---

Watch it work

<div align="center">_{A real crawl of <a href="https://freshlycited.com/">freshlycited.com</a>, time-lapsed. Nothing staged: the URL is typed, the counters climb, and the findings at the end are the ones it actually found.}</div>

Open this crawl in the live demo → — the real interface, not a report. Click anything; starting a new crawl is the one thing it cannot do. It is the actual interface, frozen over a real crawl — the same templates, stylesheet and JavaScript, with every API response captured and replayed. Tabs switch, filters filter, findings expand, the command palette opens. Starting a new crawl is the one thing it cannot do. Built by tools/build_demo_app.py.

---

What it finds

222 checks, 180 of them observed firing in the last sweep. Every one has a slug, a severity, a category, an evidence block and a one-line fix. The running install generates its own reference page at /checks, built from what it has actually reported rather than from a hand-kept list that drifts.

| Area | What it looks at | |---|---| | Technical | Canonicals (relative, cross-domain, chained, pointing at a redirect or a noindex page), redirect chains and loops, noindex and nofollow, hreflang, headings, images, response times, page weight, mixed host and protocol links, tracking parameters on internal links | | Sitemaps | robots.txt → index → children; URLs that 404, redirect, are noindex, canonicalise elsewhere or are blocked; crawled pages missing from the sitemap; lastmod missing, malformed, in the future or never bumped; the 50,000-URL and 50MB limits; duplicates across children; http/https and www drift | | Duplication | Scored on the page body with header, nav, footer and sidebar stripped, by exact hash or 64-bit simhash, reporting the shared passage rather than a percentage. Pages that share a layout no longer read as duplicates because of their chrome, and URLs that canonicalise to each other never do | | Dates and freshness | Missing, unparseable or timezone-less dates; JSON-LD, article:modified_time, HTTP Last-Modified, sitemap lastmod and the visible "Updated on…" disagreeing; stale pages; future dates; modified before published; dates that are two reads of one clock rather than two events | | Accessibility | axe-core 4.10.2, around 200 rules, each becoming its own check with its own slug (a11y_color-contrast, a11y_image-alt, …) so you can filter and diff on them | | Rendered vs raw | What the browser produced against what the server sent: content only JavaScript creates, links only JavaScript creates, text hidden after render, console errors, layout shift sources | | Security headers | HSTS and its max-age, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, cookie flags, insecure form actions, active and passive mixed content | | Server logs | Your own access log against the crawl: Googlebot user agents whose reverse DNS does not confirm, crawl budget spent on pages that only 404, pages nothing links to that are requested anyway, pages search engines have never requested, unreadable log lines | | Structured data | JSON-LD parse errors, microdata and RDFa, FAQ markup, breadcrumbs, reviews | | AMP | Thirteen checks, including the ones that only mean anything when you compare the