A website can be crawled by AI bots, indexed by search engines, and still never get cited in an AI answer, because crawled, indexed, and cited are three separate states and most sites only ever confirm the first two. The most common reason a business is missing from ChatGPT, Microsoft Copilot, Perplexity, and Google's AI answers is not weak content; it is a robots.txt line, a CDN bot-protection toggle, a stray noindex, or JavaScript-only text quietly blocking the AI search crawlers. A non-technical owner does not need to read the config to catch this. Three questions to whoever built the site surface it: are the AI search bots allowed in by name, is anything silently blocking them, and can you prove our content is reachable and not just indexed.
Most sites that are invisible in AI answers have an access problem, not a content problem. Before you rewrite a single page, you have to confirm that the AI search crawlers can reach it, read it, and are permitted to keep it. That confirmation is a three-question conversation with your developer, and every business owner can have it without touching a config file. This piece gives you the three questions, the crawlers to name, the four common accidental blocks, and a live robots.txt that does it right.
This is the technical-access companion to You Can Rank #4 on Google and Be Invisible to ChatGPT, which showed the same buyer question producing different winners on Google, Google's AI answer, and ChatGPT. That piece proved the split between ranking and citation. This one covers the step underneath both: whether the machines can even get to your page in the first place.
Crawled, indexed, and cited are three different states
Crawled, indexed, and cited are three separate states, and confirming the first two tells you nothing about the third. A bot fetching your page is one event. That page becoming eligible to appear in search is a second. A passage from it being used as evidence inside an AI answer is a third. Sites routinely pass the first two and fail the last, then blame the copy.
| State | What it means | What confirms it |
|---|---|---|
| Crawled | An AI or search bot successfully fetched the URL | A 200 response to the bot's user-agent in your server logs |
| Indexed | The page is eligible to appear in search results | The page shows in Google Search Console coverage with no noindex |
| Cited | A passage from the page is used as evidence in an AI answer | The page appears in the Bing AI Performance report or Google's AI reports, or is named inline in a live answer |
Indexing is the floor, not the finish line. Google is explicit that a page has to clear the floor before it can even be considered for an AI answer.
So "it's indexed" is the wrong finish line to settle for. Indexed means the page is allowed into the pool an AI system draws from. Whether a passage gets lifted from it is a separate outcome, decided after the page is already reachable and readable. The three questions below make sure you actually clear the floor, because a page blocked at the crawl step never even gets to compete.
Question one: does our robots.txt allow the AI search bots by name?
The first question is whether your robots.txt names the AI search crawlers and lets them in. A default site configuration was written before these bots existed, so it neither allows nor blocks them on purpose; it just leaves the outcome to chance and to whatever your CDN decides. Naming the bots removes the guesswork. Here are the ones that matter and what allowing each actually does.
| Crawler | Who runs it | What allowing it does |
|---|---|---|
| GPTBot | OpenAI | Uses your pages for model training only |
| OAI-SearchBot | OpenAI | Surfaces your site in ChatGPT search |
| ChatGPT-User | OpenAI | Fetches your page when a user's ChatGPT prompt triggers a visit |
| ClaudeBot / Claude-SearchBot | Anthropic | Training, and search-quality crawling for Claude |
| PerplexityBot | Perplexity | Surfaces and links your site in Perplexity answers |
| Googlebot | Crawls for Search, AI Overviews, and AI Mode (one crawler) | |
| bingbot | Microsoft | Crawls for Bing Search and Copilot / Bing AI summaries (one crawler) |
The trap here is treating "AI bots" as one switch. Training crawlers and search-retrieval crawlers are independent, and blocking one does not block the other. OpenAI documents this directly.
That independence is the whole point of naming bots individually. GPTBot crawls for training; OAI-SearchBot is what puts you in ChatGPT search; ChatGPT-User is triggered by a person's prompt. All three respect robots.txt. Anthropic works the same way: ClaudeBot trains, Claude-SearchBot supports search quality, Claude-User is user-triggered, and all honor robots.txt. PerplexityBot surfaces and links sites in Perplexity search, is not used for foundation-model training, and Perplexity recommends allowing it. If you want to appear in a given AI system's answers, allow that system's search bot specifically.
Google and Bing do not have separate AI crawlers. Google's AI Overviews and AI Mode run on the regular Googlebot and the same search index, with no separate AI crawler and no separate submission step, so an unblocked Googlebot already covers them. Bing is a single switch too, and that cuts the other way: bingbot crawls for Bing Search and for Copilot and Bing AI summaries at once, with no separate Bing AI opt-out user-agent.
One more distinction that trips people up: Google-Extended is not a crawler at all. It is a robots.txt control token that governs whether your already-crawled content can be used to train Gemini and to ground answers in Gemini Apps and Vertex AI. Setting it is a consent choice, and it is safe either way.
Question two: is anything silently blocking them?
The second question is whether something is blocking these crawlers without anyone deciding to. Silent blocks are the usual culprit, because they get shipped by a template or a security default and no one connects them to a missing AI citation. There are four common ones, and your developer can rule each in or out in minutes.
| Accidental block | What it looks like | Why it hides you |
|---|---|---|
| Copied "block-AI" robots.txt | A pasted rule list that disallows the training crawlers and the AI search bots together | Removes the site from AI answers while looking like a privacy win |
| CDN bot protection | Cloudflare Bot Fight Mode, a "Block AI bots" managed rule, or AI Crawl Control turned on | Flags legitimate AI search crawlers as scrapers, all-or-nothing across training and retrieval |
| Stray noindex | A noindex meta tag or X-Robots-Tag: noindex shipped from a staging template |
Pulls a money page out of the index, so it cannot be cited |
| JS-only content | Business name, service, and location render only client-side | Leaves the server HTML empty, so there is nothing to lift as evidence |
Block one is a copy-paste casualty. Someone found a "block the AI bots" snippet, pasted it to protect the brand, and did not notice it lists OAI-SearchBot and PerplexityBot right next to GPTBot. The training refusal and the search removal ride together, so the site quietly disappears from ChatGPT search and Perplexity while the owner thinks they only opted out of training.
Block two lives at the CDN, not in your files, which is why developers miss it. Bot-protection features flag automated traffic by default, and legitimate AI search crawlers look automated. The toggle is typically all-or-nothing across training and search-retrieval bots, so a security setting nobody revisited can be the entire reason you are absent from AI answers. If you are on a managed platform, this is the first place to look.
Block three is a staging leftover. A noindex tag belongs on a development build and is supposed to be stripped at launch. When a template ships it to production, the page is crawlable but ineligible for the index, which puts it below the floor from question one. It cannot be cited because it is not even in the running.
Block four is invisible to a quick eyeball check because the page looks fine in a browser. If the business name, service, city, and proof points are injected by JavaScript after load, the raw HTML a crawler reads can be close to empty. The page renders for humans and reads as blank to a machine.
Question three: can you prove the content is reachable, not just indexed?
The third question pushes past "it's indexed" to "prove it's reachable," because indexed is a claim and reachable is a check. Being in the index means a page cleared the floor at some point; it does not confirm that today an AI search bot can fetch the current page, read real text in the response, and not hit a wall. Ask your developer to demonstrate three things, not assert them.
- Real text in the server HTML. View the page source, not the rendered page, and confirm the business name, service, location, and at least one checkable proof point are present as plain text before any JavaScript runs. If they are missing from the raw HTML, an AI system has nothing to ground on.
- No noindex on money pages. Confirm your service pages, location pages, and key posts carry no
noindexmeta tag and noX-Robots-Tag: noindexheader. These are the pages you most want cited, so they are the ones a stray tag hurts most. - No 403s to AI bots in the logs. Have them grep the server or CDN logs for the crawler user-agents in question one and confirm those bots get 200 responses, not 403s or challenge pages. A page that answers a browser with a 200 and answers OAI-SearchBot with a 403 is blocked in the one way that matters, and only the logs show it.
Reachability is the thing you can actually verify, and it is the thing that has to be true before content quality even becomes the question. A passage cannot be chosen as evidence if the crawler could not retrieve the passage.
What a robots.txt that does it right looks like
Our own live robots.txt names the AI search crawlers, allows them, and blocks only the scrapers. It is generated in our site builder, so it ships identically on every deploy. Here is what a correct "yes" looks like, verified in our source on 2026-07-05:
User-agent: *
Allow: /
Disallow: /careers/apply
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Disallow: /careers/roles/
Disallow: /careers/apply
User-agent: AhrefsBot
Disallow: /
User-agent: AhrefsSiteAudit
Disallow: /
Sitemap: https://choice.marketing/sitemap.xml
Read the choices in it. The wildcard group allows every well-behaved bot by default; the only carve-out is a private job-application path, not anything a buyer would search for. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, and PerplexityBot each get an explicit allow, so the AI search crawlers are permitted by name rather than left to a CDN's guess. Google-Extended gets an allow too, which is our consent choice: we let our already-published pages be used for Gemini training and grounding. A business that would rather not can flip that one line to a disallow with zero effect on Google Search ranking or AI Overviews inclusion. The only blanket disallows are AhrefsBot and AhrefsSiteAudit, which are competitive-analysis scrapers, not answer engines. Nothing that surfaces you in an AI answer is blocked.
That is the standard to hold your own file to: AI search bots allowed by name, a deliberate training-and-grounding choice on Google-Extended, and blocks reserved for tools that only take.
The access question comes before the content question
Access comes before content, always. The ranking-versus-citation split is real, and rewriting pages into evidence is the work that wins citations once the machines can read them. But none of that fires if a robots.txt line, a CDN toggle, a stray noindex, or a JavaScript-only page is quietly turning the AI search crawlers away at the door. You can write the best evidence paragraph on the internet and still be invisible if the bot that would cite it gets a 403.
This is why we run client marketing like infrastructure, not like a content calendar. Across 33 active clients and 60,000-plus data points monitored daily, crawler access is a thing we verify, not assume, because it is the cheapest failure to fix and the most expensive to leave broken. A blocked crawler is a config change, not a strategy problem, and it costs nothing to correct once you have found it.
Put the three questions to whoever built your site: are the AI search bots allowed in by name, is anything silently blocking them, and can you prove our content is reachable and not just indexed. If the answers are shaky, that is your first fix, ahead of any rewrite. If you want a second set of eyes on it, ask us to check your crawler access, or see what our SEO practice verifies on every client. Either way, confirm the door is open before you worry about what is written on the other side of it.