Where do ChatGPT, Claude, Perplexity and Gemini get their sources?
Short answer
Each assistant searches the web with its own crawlers or index: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude, PerplexityBot for Perplexity, and Google's search index for Gemini and Google's AI answers. To get cited everywhere, let these crawlers in and make sure you are well indexed in Google and Bing.
How does an AI assistant find its sources?
An AI assistant answers either from its training knowledge or by searching the web when the question is asked. In the second case, it queries a search index, selects pages and cites those that answer best. Each company uses different, publicly declared crawlers, which a website can allow or block in its robots.txt.
Which crawlers do ChatGPT, Claude and Perplexity use?
| Assistant | Search crawler | Training crawler | On-demand crawler |
|---|---|---|---|
| ChatGPT (OpenAI) | OAI-SearchBot | GPTBot | ChatGPT-User |
| Claude (Anthropic) | Claude-SearchBot | ClaudeBot | Claude-User |
| Perplexity | PerplexityBot | no training crawler declared | Perplexity-User |
- OpenAI: according to its documentation, OAI-SearchBot is used to surface websites in ChatGPT search, and blocking GPTBot only signals that content should not be used for training.
- Anthropic: according to its help center, Claude-SearchBot indexes content to improve search answers, Claude-User visits pages when a user asks a question, and ClaudeBot collects content that may be used for training.
- Perplexity: according to its documentation, PerplexityBot is used to surface and link websites in its results and is not used to train models; Perplexity-User acts on behalf of users.
Where do Gemini and Google’s AI answers get their sources?
Google relies on its own search index. For AI Overviews and AI Mode, Google states that there are no additional requirements beyond SEO best practices, and that these features may issue multiple related searches across subtopics before answering (the “query fan-out”).
For Gemini, Google offers a separate robots.txt token, Google-Extended. According to Google, it lets you choose whether crawled content may be used to train future Gemini models and to ground answers in the Gemini apps, but it has no effect on inclusion or ranking in Google Search.
Which sources do assistants prefer?
Published studies point to two useful trends:
- Lists and comparisons: according to Ahrefs, “best X” lists make up 43.8% of the pages ChatGPT cites for recommendation questions.
- Recent content: according to Ahrefs, AI assistants cite content that is 25.7% fresher than Google’s classic results, with the exception of AI Overviews.
What should you do to be cited by all of them?
- Allow each assistant’s search crawler in your robots.txt: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot.
- Check your firewall: blocking “AI bots” can override your robots.txt.
- Get your website indexed in Google Search Console and Bing Webmaster Tools.
- Decide separately about training: you can block GPTBot or ClaudeBot without disappearing from search. Think harder before blocking Google-Extended, which also covers grounding Gemini’s answers.
- Measure each assistant separately, with the same questions, because each cites different sources.
Frequently asked questions
Does blocking training crawlers stop you from being cited?
Not at OpenAI, Anthropic and Perplexity, which separate training from search: GPTBot and ClaudeBot are for training, OAI-SearchBot, Claude-SearchBot and PerplexityBot for search. At Google, the Google-Extended token does not affect Google Search, but it also covers grounding Gemini's answers in content from the index.
Do the assistants cite the same websites?
No. Each has its own index and criteria, and answers vary from one try to the next. You need to measure your presence in each assistant separately.