Training
Builds longer-lived model knowledge from periodically fetched public content.
Technical reference · July 2026
The AI crawler list ecommerce teams need.
Know who is crawling, why they visit, and where access should stop. This field guide turns a changing bot landscape into one audit your team can finish today.
Read the index14listed user agents
Across eight operators and three distinct crawler jobs.
Quick answer
Most ecommerce sites should permit the established crawlers that support AI training, search indexes, and user-triggered retrieval—while continuing to block admin, checkout, account, API, and other non-public paths. Treat ByteDance’s Bytespider as a business-specific decision, not a universal default.
Indexing
Creates a current retrieval layer that AI products can query for sources.
Live fetch
Retrieves a specific page when a user asks for current information.
Full index · 14 entries
Crawler by crawler.
Recommendations reflect a citation-first posture for public ecommerce content. Your legal, privacy, and commercial requirements may justify narrower rules.
| User agent | Operator | Purpose | AI surface | Default |
|---|---|---|---|---|
GPTBot | OpenAI | Training | ChatGPT · GPT models | Allow |
OAI-SearchBot | OpenAI | Search index | ChatGPT Search | Allow |
ChatGPT-User | OpenAI | Live fetch | ChatGPT retrieval | Allow |
ClaudeBot | Anthropic | Training + retrieval | Claude | Allow |
anthropic-ai | Anthropic | Legacy crawler | Claude | Allow |
PerplexityBot | Perplexity | Search index | Perplexity | Allow |
Perplexity-User | Perplexity | Live fetch | Perplexity retrieval | Allow |
Google-Extended | Training control | Gemini | Allow | |
Applebot-Extended | Apple | Training control | Apple Intelligence | Allow |
Amazonbot | Amazon | Search index | Alexa for Shopping | Allow |
Meta-ExternalAgent | Meta | Training | Meta AI | Allow |
CCBot | Common Crawl | Open web corpus | AI datasets | Allow |
Bytespider | ByteDance | Training | TikTok AI · Doubao | Conditional |
cohere-ai | Cohere | Training | Command models | Allow |
Baseline configuration
Open the public store. Close the private rooms.
Start with explicit rules for high-priority crawlers, then preserve the same private-path restrictions you already apply to search engines. Replace the example sitemap and adapt paths to your platform.
Allow product, collection, editorial, and help content.
Disallow accounts, carts, checkout, admin, and internal APIs.
# AI crawlers — review quarterly
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: CCBot
Allow: /
Disallow: /admin/
Disallow: /checkout/
Disallow: /cart/
Disallow: /account/
Sitemap: https://example.com/sitemap.xmlDecision guide
Where a default changes.
Allow the discovery layer
Keep product, category, editorial, comparison, and support pages available if AI citation visibility is part of your acquisition strategy.
Protect paid content
Restrict premium reports, member libraries, staging environments, or other material whose value depends on controlled access.
Check the CDN too
A network-level bot rule can block a crawler before it ever reads robots.txt. Review Cloudflare, WAF, and hosting controls alongside the file.
Make it a rhythm, not a rescue.
- MonthlyScan server logs for expected crawler traffic and unexplained gaps.
- QuarterlyRead robots.txt end to end and update the known user-agent list.
- After changesRecheck access after SEO plugin, CDN, firewall, or platform updates.