Blind Clicks

AI Agent Framework for the Real World
Build smarter automations — even for legacy software with no APIs.

🚧 Note: This project is currently under active development and will be released as an open-source project soon. You can already star this repository to stay updated, and follow the build-in-public journey on YouTube. For more details, visit blindclicks.dev.

🖱️ What Is Blind Clicks?

Blind Clicks is a self-hosted AI agent development framework that gives you full control over your automation workflows. Inspired by OpenAI's Operator and Anthropic's Computer Use, it goes one step further:
It works blindly — by simulating actual user clicks and keystrokes at the OS level.

It's like macros on steroids, but with an LLM-powered fallback vision layer — for when things don't go as planned.

⚙️ How It Works

Blind Clicks uses:

• RobotJS to simulate OS-level mouse and keyboard actions
• Dockerized Linux (via Webtop) to run applications in a browser-accessible container
• LLMs (your choice — bring your own) for prompt understanding
• Vision fallback (via OmniParser and others) to analyze screen metadata and adapt intelligently when clicks fail

Automate boring, complex workflows — even in ancient software that still runs on Windows XP.

💡 Why It Matters

This isn't another browser automation tool.
No Puppeteer. No Playwright. No DOM required.
Just raw, deterministic automation — designed for:

• Legacy systems with no APIs
• On-prem software where you can't use cloud agents
• Privacy-sensitive use cases (100% self-hosted)
• Developers & integrators building custom AI agents for real-world, messy UIs

🧠 Smart Recovery with LLMs

When clicks fail or a UI changes unexpectedly, Blind Clicks captures the screen, analyzes it with tools like Microsoft's OmniParser, and adapts the workflow intelligently.
Fallback logic is promptable, explainable, and customizable.

Human-in-the-loop? Absolutely.
Even your AI agent can call you if it gets stuck.

🧾 Real-World Demo: Dativ Accounting System

Here's a real demo of an AI agent built using Blind Clicks — automating the accounting system Dativ (widely used in Germany).
This agent logs in, uploads invoices, parses and reconciles OCR data, and notifies the human when done. It even handles two-factor authentication with a human-in-the-loop fallback.

"If you've ever had to upload hundreds of invoices manually, you'll understand why this matters."
– Aemal

📺 Building in Public — Follow the Journey

Blind Clicks is being built live, every Sunday, in public.
Every bug. Every breakthrough. Every idea. All documented and shared.

▶️ Watch the full YouTube Playlist
Episodes include live coding, real debugging sessions, design choices, and honest reflections.

💬 "It's messy, fun, and very real. I lock myself in a room, hit record, and build this agent layer by layer."

Subscribe, follow the journey, and learn how AI agents are made — for real-world tasks.

🛠 Built for Builders

Blind Clicks is a developer-first tool.
It's open-source, hackable, and still early in its journey. Every Sunday, I build this live on my YouTube channel.

💬 "This project started as a way to help my wife automate a boring task. Now it's my playground for rethinking automation from the ground up."
– Aemal Sayer, Creator of Blind Clicks

🔐 100% GDPR & AI Act Compliant

Run it offline. Use your own LLM. Lock it behind a firewall.
Blind Clicks is privacy-first by design — perfect for European governments, hospitals, or any company that says "no" to OpenAI.

👨‍💻 About the Author

Aemal Sayer is a software engineer and AI enthusiast passionate about building practical solutions to real-world problems. With a background in automation and AI, he created Blind Clicks to solve a personal pain point that turned into a broader mission to make automation accessible for legacy systems.

• 🌐 Personal Website
• 💼 LinkedIn

👋 Try It. Break It. Improve It.

This is not a product pitch. It's a project — a creative challenge — an open invitation.

If you're excited by deterministic AI agents, OS-level automation, or voice-based workflows that just work…

📬 Reach out or
📺 Subscribe on YouTube — and let's build this together.