Deepcrawl

WARNING: DO NOT USE DEEPCRAWL IN PRODUCTION RIGHT NOW AS IT IS SUBJECT TO CHANGE AND STILL UNDER RAPID DEVELOPMENT. USE AT YOUR OWN RISK!

100% free and open-source Firecrawl alternative with better performance and flexibility.

Ask DeepWiki about this repo

NOTE: DeepCrawl doesn’t target anti-scraping or anti-bot purposes. It’s optimized for high‑frequency agent workloads that scrape public pages to extract cleaned Markdown and a hierarchical links tree.

Deepcrawl is an agent-oriented website data context extraction platform. It extracts cleaned markdown of page content, agent-favoured hierarchical links tree, and metadata that LLMs can digest with minimal token cost to reduce context switching and hallucination.

Full Platform (Nextjs Dashboard, API Workers, Auth Workers, and Database) is open and transparent.

Documentation

Visit https://deepcrawl.dev/docs to view the documentation.

Contributing

Please read the contributing guide.

License

Open Source. Open Code - built with ❤️ by @felixLu.

Star History

<a href="https://www.star-history.com/#lumpinif/deepcrawl&type=date&legend=top-left"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=lumpinif/deepcrawl&type=date&theme=dark&legend=top-left" /> <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=lumpinif/deepcrawl&type=date&legend=top-left" /> <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=lumpinif/deepcrawl&type=date&legend=top-left" /> </picture> </a>