Is Your Site Readable by AI? A Technical Checklist

Your site is readable by AI search when its important pages load normally, are not set to noindex, are not blocked in robots.txt for the crawlers that power search (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and Claude-User), are listed in an XML sitemap, and keep their key information in text. None of this needs a special AI file or markup. It is ordinary technical housekeeping, and it is worth checking because a single wrong line can hide a whole site.

Why check this at all?

Good writing cannot be cited if it cannot be fetched. Every system that answers questions from the web, whether Google's AI features, ChatGPT search, Claude or Microsoft Copilot, depends on reaching and indexing your pages first. The post on how AI assistants find and cite companies explains how each one works. This post is the checklist.

The checklist

Work through it for your home page, About page, contact page and any source page a press release links to.

• The page returns a normal successful response, not an error or a long redirect chain
• The page is not marked noindex, in a meta tag or an HTTP header
• robots.txt does not block the search crawlers listed below
• The canonical tag points to the page itself, or to the version you want indexed
• Important information is in the page's text, not only in images, video or PDFs
• Other pages on your site link to it
• It is in your XML sitemap, and the sitemap is referenced in robots.txt
• Organization structured data, if used, matches what the page shows
• The site is verified in Google Search Console and Bing Webmaster Tools

Which crawlers matter for search?

Each company documents its own crawlers. For appearing in search and AI answers, these are the ones not to block.

Some crawlers are for model training rather than search: GPTBot (OpenAI), ClaudeBot (Anthropic) and the Google-Extended control (Google). OpenAI says its search and training settings are independent, and Google says Google-Extended does not affect inclusion or ranking in Google Search. Allowing or blocking training is a separate decision from search visibility, and yours to make.

What does a sensible robots.txt look like?

Here is an example file for a fictional site, one line per row. The site opts out of training crawlers, keeps an admin area out of crawling, and lets every other crawler, including all the search crawlers above, reach everything else.

You do not need to name a search crawler to allow it. Google's robots.txt specification explains that a crawler follows the group with the most specific matching user agent, falls back to the User-agent: group when none names it, and may crawl anything no rule restricts. One consequence: if you do add a group for a named crawler, it ignores the group, so repeat any Disallow lines you still want it to follow. The line to look for, and remove if it is there by accident, is Disallow: / under User-agent: * on its own, which asks every crawler to stay away from the whole site. It is sometimes left over from a staging site.

Your robots.txt lives at the root of your domain, for example www.example.com/robots.txt, so you can open it in a browser now. OpenAI notes that changes can take about 24 hours to reach its systems.

robots.txt is not how you hide a page

Google's robots.txt introduction says robots.txt mainly manages crawler traffic and is not a way to keep a page out of Google; to do that, use noindex or password protection. The reverse matters too: a page you want found must not carry noindex. Check templates and plugins, which sometimes add it site-wide.

Sitemaps and IndexNow

An XML sitemap lists the pages you want crawled. Bing's guidance on sitemaps in AI-powered search recommends referencing the sitemap in robots.txt, keeping the lastmod date accurate so changed pages are recrawled, and pairing sitemaps with IndexNow.

IndexNow lets you notify participating search engines, including Bing, when a URL is added, updated or deleted. You host a key file on your site and submit changed URLs. Bing states plainly that IndexNow makes search engines aware of changes but does not guarantee crawling or indexing.

Structured data: useful, not required

Google states that there is no special AI schema and no AI text file needed for its AI Overviews and AI Mode; a page needs to be indexed and eligible to show with a snippet. Organization structured data is still worth adding on your home page or About page. Google's Organization documentation says it can help Google understand your details and tell your organisation apart from others, and recommends properties such as name, URL, logo, address, telephone and sameAs. Keep it identical to the visible page and to the facts described in consistent company information.

What this checklist cannot do

Passing every item makes your pages reachable and eligible. It does not make them ranked, cited or recommended. That depends on whether the page is a good answer to a real question, and on decisions each search and AI company makes independently. The post on AI-ready business information covers what your pages need to say once they can be read.

Frequently asked questions

Do I need an llms.txt or other AI file?

Not for Google's AI features: Google says no new machine-readable files or AI text files are required. None of the official documentation cited here lists such a file as a requirement for OpenAI, Anthropic or Bing either.

If I block GPTBot and ClaudeBot, will ChatGPT and Claude stop citing me?

Not according to their documentation. Search uses OAI-SearchBot, Claude-SearchBot and Claude-User, which are separate from the training crawlers. Block those search crawlers and you reduce your visibility in those assistants.

How do I know if Google or Bing can index a page?

Verify your site in Google Search Console and Bing Webmaster Tools. Both report on how your pages are crawled and indexed, and Google says traffic from AI Overviews and AI Mode is included in the overall search traffic Search Console reports.