llm-crawler-access-check
Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot
Install / Use
npx skills add davepoon/buildwithclaude --skill llm-crawler-access-checkInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
AI & Machine LearningSupported Platforms
Our assessment of llm-crawler-access-check
llm-crawler-access-check scores 86/100 on our quality scale, 397th of 794 AI & Machine Learning skills we index (top 50%).
We have not analysed the SKILL.md file itself yet, so the content part of this score is an estimate until our crawler reaches it.
With 3,537 GitHub stars, it is one of the more widely adopted skills in the catalogue.
Maintenance, license and trust
- The repository was last updated 5 days ago, so llm-crawler-access-check is actively maintained.
- It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
llm-crawler-access-check compared with similar skills
All 4 of these similar skills score higher than llm-crawler-access-check; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| llm-crawler-access-check (this skill)by davepoon | 86 | 3.5k | 5d ago | SKILL.md |
| claude-memby thedotmack | 100 | 94.8k | 1d ago | CLAUDE.md |
| Understand-Anythingby Egonex-AI | 100 | 84.3k | 15d ago | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 73.9k | today | CLAUDE.md |
| CowAgentby zhayujie | 100 | 47.1k | today | CLAUDE.md |
Frequently asked questions
- How do I install llm-crawler-access-check?
- Run
npx skills add davepoon/buildwithclaude --skill llm-crawler-access-check. The install tabs above show the steps for each supported agent. - Which AI agents does llm-crawler-access-check work with?
- It is written for Claude Code, Gemini CLI and GitHub Copilot, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is llm-crawler-access-check safe to use?
- It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is llm-crawler-access-check still maintained?
- The repository was last updated 5 days ago, so llm-crawler-access-check is actively maintained.
Related Skills
claude-mem
94.8kPersistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
Understand-Anything
84.3kGraphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
headroom
73.9kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
CowAgent
47.1kOpen-source super AI assistant & Agent Harness. Plans tasks, runs tools and skills, self-evolves with memory and knowledge. Multi-agent, multi-model, multi-channel. Lightweight, extensible, one-line install.
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.