> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fullreach.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AI crawlers

> Every AI crawler that the crawler audit and the crawler logs know, and what each one feeds.

The [Crawlability](/reference/crawlability) audit reads robots.txt for each crawler below, and [Crawler Visits](/reference/crawler-visits) counts the requests of those that send any. A crawler that feeds no tracked platform feeds an assistant or a model that FullReach AI does not measure.

| Crawler               | Token                  | Operator     | Purpose           | Feeds                               | Sends requests      | Addresses published |
| --------------------- | ---------------------- | ------------ | ----------------- | ----------------------------------- | ------------------- | ------------------- |
| Bingbot               | `bingbot`              | Microsoft    | Search            | Microsoft Copilot                   | Yes                 | Yes                 |
| Google Extended       | `Google-Extended`      | Google       | Search            | Google Gemini                       | No, robots.txt only | No                  |
| Googlebot             | `Googlebot`            | Google       | Search            | Google AI Overviews, Google AI Mode | Yes                 | Yes                 |
| MSNBot (legacy)       | `msnbot`               | Microsoft    | Search            | Microsoft Copilot                   | Yes                 | Yes                 |
| OpenAI Search         | `OAI-SearchBot`        | OpenAI       | Search            | ChatGPT                             | Yes                 | Yes                 |
| PerplexityBot         | `PerplexityBot`        | Perplexity   | Search            | Perplexity                          | Yes                 | Yes                 |
| Amazon Search         | `Amzn-SearchBot`       | Amazon       | Search            | -                                   | Yes                 | No                  |
| Applebot              | `Applebot`             | Apple        | Search            | -                                   | Yes                 | Yes                 |
| Claude Search         | `Claude-SearchBot`     | Anthropic    | Search            | -                                   | Yes                 | Yes                 |
| Meta Web Indexer      | `meta-webindexer`      | Meta         | Search            | -                                   | Yes                 | No                  |
| YouBot                | `YouBot`               | You.com      | Search            | -                                   | Yes                 | No                  |
| ChatGPT User          | `ChatGPT-User`         | OpenAI       | Asked by a person | ChatGPT                             | Yes                 | Yes                 |
| Perplexity User       | `Perplexity-User`      | Perplexity   | Asked by a person | Perplexity                          | Yes                 | Yes                 |
| Amazon User           | `Amzn-User`            | Amazon       | Asked by a person | -                                   | Yes                 | No                  |
| Claude User           | `Claude-User`          | Anthropic    | Asked by a person | -                                   | Yes                 | Yes                 |
| DuckAssistBot         | `DuckAssistBot`        | DuckDuckGo   | Asked by a person | -                                   | Yes                 | Yes                 |
| Meta External Fetcher | `meta-externalfetcher` | Meta         | Asked by a person | -                                   | Yes                 | No                  |
| Mistral User          | `MistralAI-User`       | Mistral      | Asked by a person | -                                   | Yes                 | Yes                 |
| cohere-ai             | `cohere-ai`            | Cohere       | Asked by a person | -                                   | Yes                 | No                  |
| GPTBot                | `GPTBot`               | OpenAI       | Training          | ChatGPT                             | Yes                 | Yes                 |
| AI2Bot                | `AI2Bot`               | Ai2          | Training          | -                                   | Yes                 | No                  |
| Applebot Extended     | `Applebot-Extended`    | Apple        | Training          | -                                   | No, robots.txt only | No                  |
| Bytespider            | `Bytespider`           | ByteDance    | Training          | -                                   | Yes                 | No                  |
| ClaudeBot             | `ClaudeBot`            | Anthropic    | Training          | -                                   | Yes                 | Yes                 |
| Common Crawl          | `CCBot`                | Common Crawl | Training          | -                                   | Yes                 | Yes                 |
| FacebookBot           | `FacebookBot`          | Meta         | Training          | -                                   | Yes                 | No                  |
| Meta External Agent   | `meta-externalagent`   | Meta         | Training          | -                                   | Yes                 | No                  |
| anthropic-ai (legacy) | `anthropic-ai`         | Anthropic    | Training          | -                                   | Yes                 | Yes                 |
| Amazonbot             | `Amazonbot`            | Amazon       | Other             | -                                   | Yes                 | No                  |
| Diffbot               | `Diffbot`              | Diffbot      | Other             | -                                   | Yes                 | No                  |
| GoogleOther           | `GoogleOther`          | Google       | Other             | -                                   | Yes                 | Yes                 |
| OpenAI Ads            | `OAI-AdsBot`           | OpenAI       | Other             | -                                   | Yes                 | Yes                 |

**Purpose** says what a crawler's request is for. A search crawler builds the index an answer searches. A crawler asked by a person fetches a page while an assistant answers somebody. A training crawler collects pages for future models.

**Addresses published** means the operator publishes the addresses its crawler sends from, so a logged request can be verified. For the others, a request can only claim the name.