--- title: "AI crawler configuration: search, training and user-initiated access" lang: "en" canonical: "https://winin.ai/en/guides/ai-crawlers/" alternate: "https://winin.ai/zh/guides/ai-crawlers/" datePublished: "2026-09-12" dateModified: "2026-09-12" section: "guides" --- # AI crawler configuration: search, training and user-initiated access **Direct answer: AI-related crawlers are not a binary allow/disallow switch. At a minimum, separate three categories: training crawlers (such as GPTBot and ClaudeBot, potentially used for model training), search crawlers (such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot, used to fetch content for retrieval results), and user-triggered access (such as ChatGPT-User, Claude-User, and Perplexity-User, triggered by user actions and generally not automatic crawlers). Mixing them together makes it easy to be led by vague claims during procurement. A more important premise: robots.txt is a convention, not identity verification, not access control, and it cannot guarantee that the requester complies. Blocking a crawler does not mean visibility drops to zero, and allowing a crawler does not mean your content will be cited.** --- ## 1. Three access categories based on official vendor documentation Only identifications documented officially are listed below. Identifiers without official documentation are omitted. Platform documentation changes; check the latest official documentation before use. | Vendor/Platform | Identifier | Nature | Official documentation | |---|---|---|---| | Google | Googlebot | Search foundation crawler | https://developers.google.com/search/docs/crawling-indexing/googlebot | | Google | Google-Extended | Does not control appearance in Google Search | https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers | | OpenAI | GPTBot | Possibly used for training | https://developers.openai.com/api/docs/bots | | OpenAI | OAI-SearchBot | Used for search | https://developers.openai.com/api/docs/bots | | OpenAI | ChatGPT-User | User-triggered access, should not be treated as an automatic crawler | https://developers.openai.com/api/docs/bots | | Anthropic | ClaudeBot | Training | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler | | Anthropic | Claude-SearchBot | Search | Same as above | | Anthropic | Claude-User | User-triggered | Same as above | | Perplexity | PerplexityBot | Search, not training | https://docs.perplexity.ai/docs/resources/perplexity-crawlers | | Perplexity | Perplexity-User | User-triggered, generally not subject to robots rules | Same as above | **Two key points**: 1. Google AI search features build on existing SEO fundamentals. They do not require MD or llms files and do not guarantee indexing. Do not treat uploading a file as a prerequisite for appearing in AI search. 2. Google-Extended does not control appearance in Google Search. Confusing it with Googlebot is a common configuration error. --- ## 2. Comparing three strategies | Dimension | Allow all | Search crawlers only | Block all | |---|---|---|---| | Training crawlers | Allow | Disallow | Disallow | | Search crawlers | Allow | Allow | Disallow | | User-triggered access | Depends on platform policy | Depends on platform policy | Depends on platform policy | | Likelihood of entering training corpora | Higher | Lower (depends on vendor separation) | Lower | | Likelihood of being fetched for retrieval | Retained | Retained | Given up | | Maintenance cost | Low | Medium | Low | **Allow all**: suits scenarios where brand awareness comes first and content barriers are low. **Search crawlers only**: a common choice. Note that overseas vendors have separated training and search identifiers, while some vendors may not separate them as finely. Precision varies by vendor. **Block all**: understand the cost. It does not necessarily mean "visibility drops to zero," but it markedly reduces the routes by which AI systems can obtain information from your own site. Avoid assertions such as "welding the access door shut." --- ## 3. Example configuration (examples, not this site's current configuration) Below are three short, independent examples to show differences in syntax. They do not represent this site's current robots.txt and do not require readers to allow training crawlers. Decide based on your own copyright, legal, and business considerations. **Example A: Disallow training types only** ``` User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / ``` **Example B: Allow search types only (can be combined with Example A)** ``` User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / ``` **Example C: Sensitive paths and default rules** ``` User-agent: * Allow: / Disallow: /admin/ Disallow: /private/ Disallow: /cart/ Disallow: /api/ Sitemap: https://www.example.com/sitemap.xml ``` ### What you can do after configuration 1. **Read back robots.txt**: use curl or a browser to confirm the file itself can be read correctly. 2. **Clear CDN cache**: clear the cache after changing robots.txt so old rules do not persist. 3. **Watch access logs**: filter by UA field and observe frequency and status code distribution over a period. Use this as a troubleshooting input, not a conclusion. 4. **Do not use robots for access control**: it cannot replace authentication. robots is not identity verification and does not guarantee that the requester complies. --- ## 4. Common supplementary actions beyond robots.txt These actions relate to whether AI can obtain your content, but none guarantees citation: 1. **Static key facts**: if prices or product descriptions exist only in client-rendered output, AI crawlers may not read them. 2. **Structured data**: clear markup helps machines understand pages but does not guarantee citation. 3. **Sitemap**: helps discovery but does not guarantee indexing. 4. **Third-party sources**: independent sources help cross-verification, but citation behavior is not under your control. 5. **llms.txt**: some projects discuss or adopt this format, but Google's official documentation does not require it and does not guarantee indexing. Do not treat file placement as a prerequisite for AI search. 6. **Monitor actual access**: observe through server logs. Do not assume a UA is genuine. 7. **Quarterly re-testing**: manually check visibility on different platforms using a fixed set of queries. This is a spot check, not a coverage statistic. --- ## 5. Questions to ask before buying When a GEO vendor claims that configuring robots.txt will "make AI find you," follow up: 1. Do you mean search crawlers are allowed, content is indexed, or the brand appears in answers? 2. Do you have this client's actual access logs or citation records? If not, what supports the conclusion? 3. How do you handle vendors that separate training and search crawlers differently? 4. Are you aware that user-triggered access such as Perplexity-User is generally not governed by robots rules? 5. How do you distinguish the scope of Googlebot and Google-Extended? Be cautious if the answer is only vague claims such as "allow it and you will be indexed" or "block it and you disappear." --- ## 6. Boundaries and reminders - **robots is not access control**: it relies on the requester's compliance and cannot replace authentication. - **Blocking does not mean zero visibility**: AI systems may obtain information through third-party sources, user-triggered access, and other routes. - **Allowing does not mean citation**: crawling, indexing, retrieval, and citation are different layers. - **Do not invent UAs**: cite identifiers only when official documentation exists. Unverified domestic UA names should not enter formal content. - **Google AI search builds on SEO fundamentals**: it does not require MD or llms files and does not guarantee indexing. - **Do not promise outcomes**: any claim of "citation rate increase" or "recommendation rate increase" based on robots configuration lacks a basis. --- ## Conclusion robots.txt is a convention between you and AI systems. It is not a switch and not a contract. Before buying GEO services, instead of asking "allow or block," ask "allow whom, allow what, and how will the business benefit be verified afterward." --- Winin's material on this topic focuses on fact governance and verifiable monitoring. It does not promise crawling, indexing, citation, or recommendation outcomes on any platform. If you are organizing your own crawler policy, you can use the categorization and examples above as reference material. > When combining examples, specific User-agent groups generally do not inherit path rules from `User-agent: *`. Repeat applicable path restrictions in each specific group. Protect sensitive data with authentication, not robots.txt.