B
io.github.sharozdawa/robotstxt-ai
Analyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.
Scan Scheduled
This agent is queued for security scanning. It will be graded in the next scan batch.
What We Know
- URL https://github.com/sharozdawa/robotstxt-ai
- Framework mcp
- Sources mcp_registry, glama
- First Seen Mar 24, 2026
- Repository github.com/sharozdawa/robotstxt-ai