DevDocs by CyberAGI 🚀

[!WARNING] 📌 DevDocs Status: Not publicly maintained. Enhanced internal version at CyberAGI — public release coming soon. If you have any questions please reach out to info@cyberagi.ai

<div align="center"> <img src="assets/image.png" alt="DevDocs Interface" width="800"> <p align="center"> <strong>Turn Weeks of Documentation Research into Hours of Productive Development</strong> </p> <p align="center"> <a href="#-perfect-for">Perfect For</a> • <a href="#-features">Features</a> • <a href="#-why-devdocs">Why DevDocs</a> • <a href="#-getting-started">Getting Started</a> • <a href="#-scripts-and-their-purpose">Scripts</a> • <a href="#-pricing-comparison">Compare to FireCrawl</a> • <a href="#-join-our-community">Discord</a> • <a href="#-devdocs-roadmap">DevDocs Roadmap</a> </p> </div>

🚀 Technology Partners

<div align="center" style="display: flex; justify-content: center; align-items: center; gap: 30px; flex-wrap: wrap; padding: 20px 0;"> <img src="https://cdn.brandfetch.io/idmJWF3N06/theme/light/symbol.svg?c=1dxbfHSJFAPEGdCLU4o5B" alt="Anthropic" style="height: 60px; max-width: 150px; object-fit: contain;" /> <img src="https://cdn.brandfetch.io/idR3duQxYl/w/800/h/810/theme/light/symbol.png?c=1dxbfHSJFAPEGdCLU4o5B" alt="OpenAI" style="height: 60px; max-width: 150px; object-fit: contain;" /> <img src="assets/image-8.png" alt="CyberAGI" style="height: 60px; max-width: 150px; object-fit: contain;" /> <img src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/powered-by-disco.svg" alt="Crawl4AI" style="height: 60px; max-width: 150px; object-fit: contain;" /> </div>

🎯 Perfect For

🏢 Enterprise Software Developers

Skip weeks of reading documentation and dealing with technical debt. Implement ANY technology faster by letting DevDocs handle the heavy lifting of documentation understanding.

🕸️ Web Scrapers

Pull entire contents of websites with Smart Discovery of Child URLs up to level 5. Perfect for both internal and external website documentation with intelligent crawling.

👥 Development Teams

Leverage internal documentation with built-in MCP servers and Claude integration for intelligent data querying. Transform your team's knowledge base into an actionable resource.

🚀 Indie Hackers

DevDocs + VSCode(cline) + Your Idea = Ship products fast with ANY technology. No more getting stuck in documentation hell when building your next big thing.

✨ Features

🧠 Intelligent Crawling

• Smart Depth Control: Choose crawl depth from 1-5 levels
• Automatic Link Discovery: Finds and categorizes all related content
• Selective Crawling: Pick exactly what you want to extract
• Child URL Detection: Automatically discovers and maps website structure

⚡ Performance & Speed

• Parallel Processing: Crawl multiple pages simultaneously
• Smart Caching: Never waste time on duplicate content
• Lazy Loading Support: Handles modern web apps effortlessly
• Rate Limiting: Respectful crawling that won't overload servers

🎯 Content Processing

• Clean Extraction: Get content without the fluff
• Multiple Formats: Export to MD or JSON for LLM fine-tuning
• Structured Output: Logically organized content
• MCP Server Integration: Ready for AI processing

🛡️ Enterprise Features

• Error Recovery: Auto-retry on failures
• Full Logging: Track every operation
• API Access: Integrate with your tools
• Team Management: Multiple seats and roles

🤔 Why DevDocs?

The Problem

Documentation is everywhere and LLMs are OUTDATED in their knowledge. Reading it, understanding it, and implementing it takes weeks of research and development even for senior engineers. We cut down that time to hours.

Our Solution

DevDocs brings documentation to you. Point it at any tech documentation URL, and watch as it:

1. Discovers all related pages to that technology
2. Extracts meaningful content without the fluff
3. Organizes information logically inside an MCP server ready for your LLM to query
4. Presents it in a clean, searchable format in MD or JSON for finetuning LLM purpose

🔥 We want anyone in the world to have the ability to build amazing products quickly using the most cutting edge LLM technology.

💰 Pricing Comparison

| Feature | DevDocs | Firecrawl | |---------|---------|-----------| | Free Tier | Unlimited pages | None | | Starting Price | Free Forever | $16/month | | Enterprise Plan | Custom | $333/month | | Crawl Speed | 1000/min | 20/min | | Depth Levels | Up to 5 | Limited | | Team Seats | Unlimited | 1-5 seats | | Export Formats | MD, JSON, LLM-ready MCP servers | Limited formats | | API Access | Coming Soon | Limited | | Model Context Protocol Integration | ✅ | ❌ | | Support | Priority Available via Discord | Standard only | | Self-hosted (free use) | ✅ | ❌ |

🚀 Getting Started

DevDocs is designed to be easy to use with Docker, requiring minimal setup for new users.

Prerequisites

• Docker installed on your system
• Git for cloning the repository

Quick Start with Docker (Recommended)

For Mac/Linux users:

# Clone the repository
git clone https://github.com/cyberagiinc/DevDocs.git

# Navigate to the project directory
cd DevDocs

# Configure environment variables
# Copy the template file to .env
cp .env.template .env

# Ensure NEXT_PUBLIC_BACKEND_URL in .env is set correctly (e.g., http://localhost:24125)
# This allows the frontend (running in your browser) to communicate with the backend service.


# Start all services using Docker
./docker-start.sh

For Windows users: Experimental Only (Not Tested Yet)

# Clone the repository
git clone https://github.com/cyberagiinc/DevDocs.git

# Navigate to the project directory

cd DevDocs

# Configure environment variables
# Copy the template file to .env

copy .env.template .env

# Ensure NEXT_PUBLIC_BACKEND_URL in .env is set correctly (e.g., http://localhost:24125)

# This allows the frontend (running in your browser) to communicate with the backend service.

# Prerequisites: Install WSL 2 and Docker Desktop
# Docker Desktop for Windows requires WSL 2. Please ensure you have WSL 2 installed and running first.
# 1. Install WSL 2: Follow the official Microsoft guide: https://learn.microsoft.com/en-us/windows/wsl/install
# 2. Install Docker Desktop for Windows: Download and install from the official Docker website. Docker Desktop includes Docker Compose.



# Start all services using Docker
docker-start.bat

<details> <summary>Note for Windows Users</summary>

If you encounter permission issues, you may need to run the script as administrator or manually set permissions on the logs, storage, and crawl_results directories. The script uses the icacls command to set permissions, which might require elevated privileges on some Windows systems.

Manually Setting Permissions on Windows:

If you need to manually set permissions, you can do so using either the Windows GUI or command line:

Using Windows Explorer:

1. Right-click on each directory (logs, storage, crawl_results)
2. Select "Properties"
3. Go to the "Security" tab
4. Click "Edit" to change permissions
5. Click "Add" to add users/groups
6. Type "Everyone" and click "Check Names"
7. Click "OK"
8. Select "Everyone" in the list
9. Check "Full control" under "Allow"
10. Click "Apply" and "OK"

Using Command Prompt (as Administrator):

icacls logs /grant Everyone:F /T
icacls storage /grant Everyone:F /T
icacls crawl_results /grant Everyone:F /T

</details> <details> <summary>Note about docker-compose.yml on Windows</summary>

If you encounter issues with the docker-compose.yml file (such as "Top-level object must be a mapping" error), the docker-start.bat script automatically fixes this by ensuring the file has the correct format and encoding. This fix is applied every time you run the script, so you don't need to manually modify the file.

</details>

This single command will:

1. Create all necessary directories
2. Set appropriate permissions
3. Build and start all Docker containers
4. Monitor the services to ensure they're running properly

Accessing DevDocs

Once the services are running:

• Frontend UI: http://localhost:3001
• Backend API: http://localhost:24125
• Crawl4AI Service: http://localhost:11235

Logs and Monitoring

When using Docker, logs can be accessed :

1. Container Logs (recommended for debugging):

   # View logs from a specific container
   docker logs devdocs-frontend
   docker logs devdocs-backend
   docker logs devdocs-mcp
   docker logs devdocs-crawl4ai
   
   # Follow logs in real-time
   docker logs -f devdocs-backend
   

To stop all services, press Ctrl+C in the terminal where docker-start is running.

📜 Scripts and Their Purpose

DevDocs includes various utility scripts to help with development, testing, and maintenance. Here's a quick reference:

Startup Scripts

• start.sh / start.bat / start.ps1 - Start all services (frontend, backend, MCP) for local development.
• docker-start.sh / docker-start.bat - Start all services using Docker containers.

MCP Server Scripts

• check_mcp_health.sh - Verify the MCP server's health and configuration status.
• restart_and_test_mcp.sh - Restart Docker containers with updated MCP configuration and test connectivity.

Crawl4AI Scripts

• check_crawl4ai.sh - Check the status and health of the Crawl4AI service.
• debug_crawl4ai.sh - Run Crawl4AI in debug mode with verbose logging for troubleshooting.
• test_crawl4ai.py - Run tests against the Crawl4AI service to verify functionality.
• test_from_container.sh - Test the Crawl4AI service from within a Docker container.

Utility Scripts

• view_result.sh - Display crawl results in a formatted view.
• `fi