N/A
sharozdawa/robotstxt-ai-mcp
Analyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.
Scan Scheduled
This agent is queued for security scanning. It will be graded in the next scan batch.
What We Know
- URL https://glama.ai/mcp/servers/t1cy2qwu3y
- Framework mcp
- Sources glama
- First Seen Mar 24, 2026
- Repository github.com/sharozdawa/robotstxt-ai
Browse more:
Search all agents
Ecosystem Report