How to Make a Site Visible to ChatGPT and Perplexity

A site becomes visible to ChatGPT and Perplexity when the bots behind these systems are not blocked in robots.txt, can read the content without executing JavaScript, and the page itself answers the question directly in its opening sentences rather than buried in the middle of the text. These are three separate conditions, and most sites fail at least one of them without even realizing it. Below is a breakdown of what actually controls visibility, and what is a trendy but ultimately irrelevant add-on.

According to a SE Ranking study covering almost 300,000 domains, llms.txt is installed on only 10.13% of sites, and the researchers found no correlation at all between having the file and how often a site gets cited in AI answers - the model predicted citation rates more accurately once this parameter was removed from the calculation. Earlier still, in July 2025, Google’s Gary Illyes confirmed that Google does not use, and has no plans to use, llms.txt, either for regular search or for AI Overviews.

This doesn’t mean visibility for AI systems can’t be controlled - it just means the trendy file everyone asks about first is not the lever that actually works. Below: what really controls bot access, how training crawls fundamentally differ from crawls that generate a live answer, and how to structure content so there’s something in it worth citing.

For a business, this is a real loss: customers increasingly type their research question straight into ChatGPT or Perplexity instead of a Google search box, and if a company’s site doesn’t make it into that answer, a competitor who configured access and content structure correctly gets the mention instead - without spending a single click on ads.

What visibility actually means for ChatGPT and Perplexity

“Visibility” breaks down into two independent events: a bot read your page at some point and can use it as a data source, and a bot cited your page right now while answering a specific user query. These are different bots, different purposes, and different consequences if you block them.

At OpenAI the split is explicit: GPTBot collects data that may end up training future models, while OAI-SearchBot indexes pages for live ChatGPT Search answers - and according to OpenAI’s official crawler overview, that indexing is not used to train models. Anthropic breaks it down even further: ClaudeBot collects training data, Claude-SearchBot indexes content for Claude’s search feature, and Claude-User fetches a page at the moment a specific Claude user asks a question about it. PerplexityBot at Perplexity isn’t involved in model training at all - according to Perplexity’s official documentation, it’s a crawler used only for search and processing user queries.

The practical takeaway: a site can allow PerplexityBot and OAI-SearchBot so it shows up in live answers, while blocking GPTBot and ClaudeBot if the company doesn’t want its text used as training material for future model versions. That’s not a contradiction - it’s a deliberate choice, available through separate lines in robots.txt.

Robots.txt is where access actually gets decided

robots.txt: a text file at the root of a site (example.com/robots.txt) that uses User-agent, Allow, and Disallow directives to tell crawlers which sections of the site they can visit. It’s the only one of the two files covered in this article that bots actually check before every visit, and that they honor in the vast majority of cases.

Unlike llms.txt, robots.txt is an old, de facto enforced standard: OpenAI, Anthropic, and Perplexity all explicitly confirm in their documentation that their crawlers respect Disallow directives. Compliance is still voluntary - there’s no technical block for ignoring it - but major legitimate players have a reputational and legal incentive to stick to it.

An example configuration that allows live-answer bots and blocks training bots:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Check the current bot list against OpenAI’s official crawler overview and the equivalent Anthropic and Perplexity documentation, not against articles like this one - the list changes, and what’s accurate today may gain a new bot name or a new default policy within six months.

A separate technical detail for headless sites and SPAs: if content renders through JavaScript and the bot doesn’t execute it (most AI crawlers don’t - they read the raw HTML the server returns), a correctly configured robots.txt won’t help if the pre-render page serves an empty <div id="root">. It’s the same architectural problem we covered in the article on why React sites lose rankings in Google - the same logic applies to AI crawlers too, except the consequence isn’t a ranking drop, it’s a total absence of content to cite.

What llms.txt is, and what it doesn’t guarantee

llms.txt: a markdown file proposed in 2024, placed at the root of a site (example.com/llms.txt), that lists a site’s sections and links in condensed form to make them easier for a language model to read. It’s not an official web standard, but a voluntary community convention - support by any specific AI system is not guaranteed.

As of 2026, the picture is mixed. On one hand, discussion of formalizing the format is underway: an IETF working group on AI crawlers started a draft user-agent registry, and a W3C initiative has looked at a proposal to standardize the format. On the other hand, not a single major model provider, including OpenAI, Anthropic, or Google, has publicly confirmed using llms.txt as a production signal. Google is the clearest example: Gary Illyes explained the position by pointing out that a separate file summarizing content can’t be verified against the live page, and compared the idea to the keywords meta tag, which Google dropped decades ago precisely because it was a self-declared, easily gamed signal.

That doesn’t mean adding llms.txt is pointless - the file costs nothing in performance terms and can’t do any harm. But treat it as an optional business card, not as a tool that controls visibility: the real levers are robots.txt for access and content structure for citability, both covered separately in this article.

Why training crawls and answer crawls are not the same thing

The difference between “a bot added your text to a training set” and “a bot cited your page in an answer to a user’s question right now” is the difference between two independent business effects, and confusing them means making blocking decisions blind.

Training crawls affect how future model versions describe your product or industry as a whole - a delayed effect, diluted across millions of sources, and impossible to measure at the level of a single site. Crawling for live answers (retrieval) has a concrete, measurable effect: if PerplexityBot or OAI-SearchBot managed to read the page, there’s a real chance of getting a mention with a direct link in an answer to a user who might visit the site a minute later.

That’s why some companies follow a simple logic: block GPTBot, Google-Extended, and CCBot, because training data brings no measurable business benefit, while keeping OAI-SearchBot, PerplexityBot, and Claude-SearchBot open, because those are the ones that form a direct visitor-acquisition channel. This isn’t universal advice - for content publishers who make money from traffic, the logic can run the other way - but the choice itself should be a deliberate decision, not a side effect of someone copying another site’s robots.txt without thinking it through.

The content structure AI systems actually cite

Even with fully open bot access, the odds of getting cited depend on how the text itself is structured - AI systems don’t retell an entire article, they extract the specific passage that answers the question asked.

AEO/GEO: Answer Engine Optimization and Generative Engine Optimization - practices for structuring content so that page passages can be extracted and cited in an AI system’s answer without losing meaning. The two terms are often used as synonyms, though GEO is sometimes read more broadly - as work on a brand’s overall citability, not just on individual answers.

What actually affects how extractable a passage is:

• A direct answer to the heading’s question in the section’s first sentence, not after three paragraphs of preamble
• Explicit H2/H3 headings phrased as a specific question or a specific statement, not vague labels like “Additional information”
• Short paragraphs of 2-4 sentences - models truncate long blocks of text when citing them, losing part of the meaning
• Structured data in JSON-LD format - Article, FAQPage, HowTo - that gives the model explicit, machine-readable markup instead of forcing it to guess the structure from the layout

We covered how to implement this markup on a headless site in detail, with code examples for different page types, in the article on structured data on a headless site. This is a case where the same work improves both regular SEO rankings and citability in AI answers at once - the effort doesn’t get duplicated.

What this looks like in practice

In a typical audit of a B2B marketing site (20-40 employees), the picture is usually similar: robots.txt either doesn’t mention AI bots at all, or was copied from a three-year-old template and blocks everything indiscriminately, including PerplexityBot and OAI-SearchBot, because whoever wrote the template in 2023 added them “just in case” alongside the training crawlers.

After sorting bots individually, the usual outcome is blocking only GPTBot, ClaudeBot, Google-Extended, and CCBot, leaving the search and retrieval crawlers open. A separate check looks at whether the main content renders exclusively through client-side JavaScript - if the site runs on outdated SPA architecture, bots simply don’t see the text, no matter how many rules are written into robots.txt. For sites built with static generation or server-side rendering (Astro, Next.js with SSR), this problem doesn’t come up at all - there’s no difference between the HTML the browser gets and the HTML the bot sees.

Companies that recently migrated from WordPress to a modern stack often get this visibility “for free” just from the architecture - we covered the migration process itself and what gets preserved in the article on moving a site from WordPress to Astro. Rewriting content for direct answers and question-style headings usually shows a noticeable effect within 4-6 weeks - that’s roughly how long most AI crawlers take to reindex a site’s main pages.

Who this matters most for

Visibility for AI systems matters most for businesses whose target audience is technically savvy buyers who increasingly type research queries straight into ChatGPT or Perplexity instead of a search box: SaaS products, consulting, B2B services with a long sales cycle. For local businesses or e-commerce with transactional traffic, the effect of this configuration will be noticeably smaller - classic search and paid ads remain the main channel there.

This also matters especially for companies whose site is technically outdated - built on WordPress with heavy plugins, on a builder with no control over rendering, or without any structured data at all. In those cases, editing robots.txt isn’t enough: you first need to make sure bots can physically read the content. On a site that’s already running, configuring robots.txt and llms.txt is usually handled as a small, self-contained technical addition that doesn’t require reworking the rest of the code.

Frequently asked questions

Should you add llms.txt to your site in 2026?

The file won’t hurt, but it also doesn’t guarantee appearing in AI system answers. The SE Ranking study covering almost 300,000 domains found no correlation between having llms.txt and citation frequency, and Google has explicitly confirmed it doesn’t use the file in any form. Priority should go to a correctly configured robots.txt and content structure - those are what actually control visibility.

How do you check whether your site is blocked for GPTBot or PerplexityBot?

Open your-domain.com/robots.txt in a browser and look for the User-agent: GPTBot, User-agent: PerplexityBot, User-agent: ClaudeBot, and similar blocks. If Disallow: / appears next to the bot you’re checking, access is fully blocked. A bot not mentioned in the file at all is allowed by default - you only need to explicitly disallow what you actually want to block.

Will blocking GPTBot hurt the site’s rankings in regular Google search?

No, these are independent systems. GPTBot belongs to OpenAI and has nothing to do with Googlebot, which crawls a site for regular Google search results. Blocking one crawler doesn’t affect indexing by another - every directive in robots.txt applies only to the User-agent it names.

Can you block all AI bots at once with a single directive?

Yes, but the result is crude: a User-agent: * rule with Disallow: / blocks every crawler, including Googlebot and Bingbot, which regular search visibility depends on. To block only AI crawlers selectively, you need a separate User-agent line for each bot, as shown in the example above.

How often should you review the list of allowed and blocked bots?

Check the list at least once a quarter: new bots appear, and existing ones change scope - for example, in February 2026 Anthropic updated its documentation and split ClaudeBot into three separate bots with different blocking logic. There’s still no single settled standard in this area, and what’s accurate today isn’t guaranteed to still be accurate six months from now.

Bottom line

A site’s visibility to ChatGPT and Perplexity is controlled by three levers, and all three can be checked in a single day:

• Robots.txt with explicit rules for each bot - the main, and only, mechanism that actually enforces access
• Content that’s technically accessible without executing JavaScript - without it, a correctly configured robots.txt won’t help
• Text structured with direct answers, clear headings, and structured data - what determines whether a passage from the page actually gets cited

llms.txt isn’t on that list - it’s an optional add-on with no confirmed effect on citability.

If you’re not sure your site is technically visible to AI crawlers, or content still renders in a way that leaves bots looking at an empty page, describe the situation to the Exceltic.dev team. We’ll check your robots.txt, rendering accessibility, and content structure, and show you exactly what’s keeping the site out of ChatGPT and Perplexity answers. This is part of the site development and optimization work Exceltic.dev does.