unlimited-search

English | 한국어 | 中文 | 日本語 | Español

<p align="center"><strong>From public pages to usable signal.</strong></p> <p align="center"> <img src="assets/hero.png" width="860" alt="unlimited-search hero showing public URLs routed through routes, HTTP, RSS, archives, and media into clean structured text for MCP and CLI."> </p>

unlimited-search is a Python CLI and MCP server for reading public web content when a normal direct fetch is not enough. It combines platform public routes, browser-like HTTP identities, content fallbacks, public archive fallbacks, and media metadata extraction behind one local tool.

It is built for agents and automation that need usable text from public URLs. It is not intended to bypass logins, paywalls, CAPTCHA, private networks, account restrictions, IP bans, or access controls.

Quickstart

Prerequisite: Python 3.12 or newer.

python -m pip install unlimited-search
unlimited-search read https://en.wikipedia.org/wiki/OpenAI --max-content-chars 800

The command returns JSON with page content, a verdict, request metadata, and an attempt trace.

For MCP clients, install the package and register the stdio server:

{
  "mcpServers": {
    "unlimited-search": {
      "command": "unlimited-search",
      "args": ["serve"]
    }
  }
}

What It Provides

| Capability | What it does | | --- | --- | | Public platform routes | Uses unauthenticated public APIs, feeds, or metadata routes before generic fetching. | | Browser-like fetching | Tries multiple TLS/browser identities, URL variants, referer strategies, and response validation. | | Content fallbacks | Recovers public text through Jina Reader, RSS/Atom discovery, and embedded page metadata. | | Archive fallbacks | Falls back to Wayback and archive.today/archive.ph public snapshots when live reads fail. | | Media metadata | Uses yt-dlp --dump-json for public media URLs without downloading media. | | MCP tools | Exposes single-URL reads, batch reads, diagnostics, and media extraction to MCP clients. |

See Platform coverage for the full support matrix and known gaps.

Install

Install from PyPI:

python -m pip install unlimited-search

Update:

python -m pip install --upgrade unlimited-search

Remove:

python -m pip uninstall unlimited-search

CLI Usage

unlimited-search help
unlimited-search <command> [arguments]

Commands:

serve           Start the MCP stdio server
read <url>      Read a public URL
diagnose <url>  Diagnose access without full content
media <url>     Extract public media metadata
help            Show this help

Common examples:

unlimited-search read https://example.com --max-content-chars 1000
unlimited-search diagnose https://example.com
unlimited-search media https://www.youtube.com/watch?v=dQw4w9WgXcQ

Fallback-focused smoke tests:

unlimited-search read https://example.com --no-public-routes --max-attempts 0 --max-content-chars 500
unlimited-search read http://www.whitehouse.gov/1600/presidents/barackobama --no-public-routes --max-attempts 1 --max-content-chars 500

Useful flags:

| Flag | Applies to | Purpose | | --- | --- | --- | | --timeout SECONDS | read, diagnose, media | Set request timeout. | | --max-attempts N | read, diagnose | Limit generic HTTP-grid attempts. | | --max-content-chars N | read | Limit returned content size. | | --no-public-routes | read, diagnose | Skip platform public routes. | | --preferred-identity NAME | read, diagnose | Try a specific browser-like identity first. | | --success-selector CSS | read | Treat matching CSS selectors as success signals. Can be repeated. |

MCP Tools

The MCP server runs over stdio:

unlimited-search serve

Available tools:

| Tool | Purpose | | --- | --- | | read_public_url | Read one public URL and return content, verdict, metadata, and trace. | | read_public_urls | Read multiple public URLs in one tool call. | | diagnose_access | Return a compact diagnosis and attempt trace without full content. | | extract_media | Extract public media metadata through yt-dlp without downloading media. |

More configuration examples are in MCP configuration.

How Reads Work

<p align="center"> <img src="assets/pipeline.png" width="860" alt="Pipeline diagram: Public URL to platform routes, HTTP grid, content fallbacks, archive fallbacks, media metadata, and clean public text, with login, paywall, or CAPTCHA stopping safely."> </p>

read_public_url tries the least invasive public routes first, then progressively broader recovery paths:

1. Platform-specific public routes for sites such as Reddit, X/Twitter, Bluesky, Hacker News, Google News, Stack Overflow, Wikipedia, GitHub, npm, PyPI, Wayback, Naver, Amazon, and Google Scholar.
2. A generic HTTP grid with browser-like TLS identities, URL variants, referer strategies, response validation, and selected HTTP/1.1 transport fallback.
3. Non-browser content fallbacks:
   • Jina Reader JSON content
   • RSS/Atom discovery through Jina external.alternate
   • common origin feed paths such as /feed, /rss, and /atom.xml
   • OGP, JSON-LD, Schema.org, and Next.js payload metadata salvage
4. Public archive fallbacks:
   • Wayback Available API
   • Wayback latest/direct snapshot
   • Wayback CDX latest 200 snapshot
   • archive.today/archive.ph best-effort snapshots
5. yt-dlp metadata extraction for known public media hosts.

Fallback successes are reported as weak_ok or suspect_ok, not strong_ok, because recovered content can be incomplete, stale, or metadata-only.

Safety Boundaries

unlimited-search is a public-content reader. It should stop or report failure when a target requires authentication, payment, CAPTCHA, private network access, or hard anti-abuse bypassing.

The reader rejects private, loopback, link-local, multicast, reserved, and metadata-service targets by default. Redirects are checked before fallback routes are allowed to continue.

Returned HTML, JSON, RSS, archive text, and metadata should be treated as untrusted content.

Development

uv sync --extra dev
uv run unlimited-search read https://example.com --max-content-chars 300
uv run unlimited-search serve
uv run pytest
uv build

Run the live eval set when changing routes or fallback behavior:

uv run python scripts/run_eval.py --list
uv run python scripts/run_eval.py
uv run python scripts/run_eval.py --markdown eval-results/report.md --csv eval-results/report.csv
uv run python scripts/run_eval.py --baseline eval-results/eval-20260707T000000Z.jsonl --fail-on-regression

The default eval cases live in scripts/eval_urls.yaml. Difficult sites such as NamuWiki, TikTok, Naver Search, Amazon, and Google Scholar are optional so remote blocking or rate limits do not fail the whole run.

Project Docs

• Platform coverage
• MCP configuration
• Troubleshooting
• Security
• Contributing
• Code of conduct
• Disclaimer
• Privacy
• Changelog
• License