Which AI crawlers can read your website?
Enter your website to see whether ChatGPT, Claude, Perplexity, Gemini and other AI crawlers are allowed by your robots.txt, then copy a ready-made robots.txt block and a starting llms.txt.
Frequently asked questions
Straight answers before you paste your URL.
Search and answer bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) fetch your pages to answer someone's question, and usually link back, so they can send you visitors. Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) collect pages to train AI models and send no visitors. Many sites allow the first group and block the second.
Put your website to work tonight
Paste your URL and chat with your agent in minutes.
Free for 100 messages a month · No credit card required