AnyTool

<div align="center"> <picture> <img src="assets/AnyTool_logo.png" width="800px" style="border: none; box-shadow: none;" alt="AnyTool Logo"> </picture>

AnyTool: Universal Tool-Use Layer for AI Agents

✨ One Line of Code to Supercharge any Agent with <br>Fast, Scalable and Powerful Tool Use ✨

| ⚡ Fast - Lightning Tool Retrieval  |  📈 Self-Evolving Tool Orchestration  |  ⚡ Universal Tool Automation |

</div>

🎯 What is AnyTool?

AnyTool is a Universal Tool-Use Layer that transforms how AI agents interact with tools. It solves three fundamental challenges that prevent reliable agent automation: overwhelming tool contexts, unreliable community tools, and limited capability coverage -- delivering the first truly intelligent tool orchestration system for production AI agents.

💡 Research Highlights

⚡ Fast - Lightning Tool Retrieval

• Smart Context Management: Progressive tool filtering delivers exact tools in milliseconds through multi-stage pipeline, eliminating context pollution while maintaining speed.

• Zero-Waste Processing: Pre-computed embeddings and lazy initialization eliminate redundant processing - tools are instantly ready across all executions.

📈 Scalable - Self-Evolving Tool Orchestration

• Adaptive MCP Tool Selection: Smart caching and selective re-indexing maintain constant performance from 10 to 10,000 tools with optimal resource usage.

• Self-Evolving Tool Optimization: System continuously improves through persistent memory, becoming more efficient as your tool ecosystem expands.

🌍 Powerful - Universal Tool Automation

• Quality-Aware Selection: Built-in reliability tracking and safety controls deliver production-ready automation through persistent learning and execution safeguards.

• Universal Tool-Use Capability: Multi-backend architecture seamlessly extends beyond web APIs to system operations, GUI automation, and deep research through unified interface.

⚡ Easy-to-Use and Effortless Integration

One line to get intelligent tool orchestration. Zero-config setup transforms complex multi-tool workflows into a single API call.

from anytool import AnyTool

# One line to get intelligent tool orchestration
async with AnyTool() as tool_layer:
    result = await tool_layer.execute(
        "Research trending AI coding tools from GitHub and tech news, "
        "collect their features and user feedback, analyze adoption patterns, "
        "then create a comparison report with insights"
    )

---

📋 Table of Contents

• 🎯 Quick Start
• 🚀 Technical Innovation & Implementation
• 🔧 Configuration Guide
• 📖 Code Structure
• 🔗 Related Projects

---

🎯 Quick Start

1. Environment Setup

# Clone repository
git clone https://github.com/HKUDS/AnyTool.git
cd AnyTool

# Create and activate conda environment (includes ffmpeg for video recording)
conda create -n anytool python=3.12 ffmpeg -c conda-forge -y
conda activate anytool

# Install dependencies
pip install -r requirements.txt

[!NOTE] Create a .env file and add your API keys (refer to anytool/.env.example).

2. Execution Mode: Local vs Server

AnyTool's Shell and GUI backends support two execution modes. You can configure the mode in anytool/config/config_grounding.json:

{
  "shell": { "mode": "local", ... },  // or "server"
  "gui":   { "mode": "local", ... }   // or "server"
}

Local Mode (Default — no server needed)

In local mode, Shell and GUI operations are executed directly in-process via subprocess / asyncio. This is the simplest setup — no local server required. Just use AnyTool as normal, see Quick Integration for usage examples.

[!TIP] Use local mode when you are running AnyTool on the same machine you want to control (your own laptop / desktop). This is the recommended mode for most users.

Server Mode (for remote VMs / isolation)

In server mode, Shell and GUI operations are sent over HTTP to a running local_server Flask service. This is required when:

• Controlling a remote VM — the agent runs on your host, while the server runs inside the VM.
• Process isolation / sandboxing — you want script execution in a separate process for security or stability.
• Multi-machine deployments — the agent and the execution environment are on different machines.

To use server mode, set "mode": "server" in config_grounding.json, then install platform-specific dependencies and start the server:

[!IMPORTANT] Platform-specific setup required: Different operating systems need different dependencies for desktop control. Please install the required dependencies for your OS before starting the local server:

<details> <summary><b>macOS Setup</b></summary>

# Install macOS-specific dependencies
pip install pyobjc-core pyobjc-framework-cocoa pyobjc-framework-quartz atomacos

Permissions Required: macOS will automatically prompt for permissions when you first run the local server. Grant the following:

• Accessibility (for GUI control)
• Screen Recording (for screenshots and video capture)

If prompts don't appear, manually grant permissions in System Settings → Privacy & Security.

</details> <details> <summary><b>Linux Setup</b></summary>

# Install Linux-specific dependencies
pip install python-xlib pyatspi numpy

# Install system packages
sudo apt install at-spi2-core python3-tk scrot

[!NOTE] Optional dependencies:

• Accessibility: pyatspi + at-spi2-core
• Window management: wmctrl
• Cursor in screenshots: libx11-dev + libxfixes-dev

</details> <details> <summary><b>Windows Setup</b></summary>

# Install Windows-specific dependencies
pip install pywinauto pywin32 PyGetWindow

</details>

After installing the platform-specific dependencies, start the local server:

python -m anytool.local_server.main

[!NOTE] See anytool/local_server/README.md for complete API documentation and advanced configuration.

Mode Comparison

| | Local Mode ("local") | Server Mode ("server") | |---|---|---| | Setup | Zero — just run your agent | Start local_server first | | Use case | Same-machine development | Remote VMs, sandboxing, multi-machine | | Shell execution | asyncio.subprocess in-process | HTTP → Flask → subprocess | | GUI execution | Direct pyautogui / ScreenshotHelper | HTTP → Flask → pyautogui | | Dependencies | Only core AnyTool | Core + Flask + platform deps | | Network | None required | HTTP between agent ↔ server |

3. Quick Integration

AnyTool is a plug-and-play Universal Tool-Use Layer for any AI agent. The task passed to execute() can come from your agent's planning module, user input, or any workflow system.

import asyncio
from anytool import AnyTool
from anytool.tool_layer import AnyToolConfig

async def main():
    config = AnyToolConfig(
        enable_recording=True,
        recording_backends=["gui", "shell", "mcp", "web"],
        enable_screenshot=True,
        enable_video=True,
    )
    
    async with AnyTool(config=config) as tool_layer:
        result = await tool_layer.execute(
            "Research trending AI coding tools from GitHub and tech news, "
            "collect their features and user feedback, analyze adoption patterns, "
            "then create a comparison report with insights"
        )
        print(result["response"])

asyncio.run(main())

[!TIP] MCP Server Configuration: For tasks requiring specific tools, add relevant MCP servers to anytool/config/config_mcp.json. Unsure which servers to add? Simply add all potentially useful ones, AnyTool's Smart Tool RAG will automatically select the appropriate tools for your task. See MCP Configuration for details.

---

Technical Innovation & Implementation

🧩 Challenge 1: MCP Tool Context Overload

The Problem. Current MCP agents suffer from a fundamental design flaw: they load ALL configured servers and tools at every execution step, creating an overwhelming action space, creates three critical issues:

• ⚡ Slow Performance with Massive Context Loading<br> Complete tool set from all pre-configured servers loaded simultaneously at every step, degrading execution speed

• 🎯 Poor Accuracy from Blind Tool Setup<br> Users cannot preview tools before connecting, leading to over-setup "just in case" and confusing tool selection

• 💸 Resource Waste with No Memory<br> Same tools reloaded at every execution step with no caching, causing redundant loading

✅ AnyTool's Solution: Tool Context Management Framework

Motivation: "Load Everything" → "Retrieve What's Needed"<br> Improvement: Faster tool selection, cleaner context, and efficient resource usage through smart retrieval and memory.

Technical Innovation:<br>

🎯 Multi-Stage Tool Retrieval Pipeline

• Progressive MCP Tool Filtering: server selection → tool name matching → tool semantic search → LLM ranking
• Reduces MCP Tool Search Space: Each stage narrows down candidate tools for optimizing precision and speed

**