Agent Skills: llm-crawler-access-check

Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawlers from the search crawlers that actually control citations.

UncategorizedID: davepoon/buildwithclaude/llm-crawler-access-check

Install this agent skill to your local

pnpm dlx add-skill https://github.com/davepoon/buildwithclaude/llm-crawler-access-check

Skill Files

Browse the full folder contents for llm-crawler-access-check.

Download Skill

Loading file tree…

Select a file to preview its contents.