How a Website Becomes Citable by Search Engines and AI Assistants

TL;DR

• A citable website is a site whose pages a crawler can fetch without a login, read as text in the HTML it receives, attribute to a named organisation and cross-check against a structured data block.
• The first pass reads what the crawler actually receives: robots.txt rules, CDN and hosting rules, and server-rendered HTML with the text inside it.
• OpenAI documents four user agents with separate jobs, and a site can allow OAI-SearchBot for ChatGPT search while disallowing GPTBot for training.
• Google's AI features draw on the same index as ordinary search, and its published requirements are the ordinary ones: crawling allowed, internal links, content in textual form, and structured data that matches the visible text.
• Structured data is a standardised description of a page, Google recommends JSON-LD, and the block has to describe what a reader can see.
• The llms.txt file is a proposal for a markdown index at a site root or a path, with an H1, a summary blockquote and lists of links to markdown copies; it sits beside robots.txt and sitemaps, and OpenAI, Anthropic and Google publish one for their own developer documentation.
• The four-pass check runs to fetching, reading, identifying and structuring a page, and each failure it finds points at a specific piece of build work.

Citation readiness combines ordinary web publishing with attention to the published rules of systems that may quote a page. Pages need to load, carry text a machine can parse, and say who wrote them and what they describe. Teams also need to read the documentation of the systems that do the quoting, because each publishes its own rules about what it will fetch and show.

These jobs rely on ordinary content work. Both need a page that a program can read end to end, and a small amount of honesty about what each file and each tag actually does.

What a crawler receives when it opens the page

Start with the response body. An assistant's crawler and a search crawler both receive HTML, and text that only appears after a client-side render is text they may never see. Server rendering keeps the article, the price, the address and the opening hours in the response, which is the version every downstream system can read.

Then check permission. A robots.txt rule that disallows a path, a CDN rule that challenges non-browser clients, a staging password and a hosting firewalls each stop the fetch before content quality matters. Google's own list of technical requirements for appearing in its AI features leads with exactly this: crawling allowed in robots.txt and by any CDN or hosting infrastructure, content findable through internal links, and important content available in textual form.

The identity layer comes next. A page that names its author, its organisation and its update date gives both a search engine and a language model something to attribute. A page that hides those in an image or a footer script gives them a guess.

The four OpenAI user agents and the job of each

OpenAI's crawler documentation is short and worth reading in full, because it separates four user agents that companies routinely treat as one. OAI-SearchBot exists for search: it surfaces websites in ChatGPT's search features, sites opted out of it will not be shown in ChatGPT search answers, and OpenAI recommends allowing it in robots.txt along with its published IP ranges. GPTBot crawls content that may be used in training, and disallowing it signals that a site's content should not be used for training.

The other two handle two additional situations. ChatGPT-User appears when a person asks a question and ChatGPT visits a page to answer it, and OpenAI states that this agent serves certain user-initiated actions, for which robots.txt rules may not apply. OAI-AdsBot visits landing pages submitted as ads, and OpenAI excludes its data from training generative AI foundation models.

The practical consequence is a decision each company takes deliberately. A company that wants visibility inside ChatGPT search allows OAI-SearchBot; a company that wants its text kept out of training disallows GPTBot. OpenAI notes that the two settings are independent, that a site allowing both may have one crawl serve both uses, and that a robots.txt change can take about 24 hours to reach its systems.

Google's AI features draw on the same index

Google's guidance for AI features repeats its established SEO requirements. The page lists the requirements it always lists: crawling allowed in robots.txt and by the CDN or hosting layer, content findable through internal links, a decent page experience, important content available in textual form, high-quality images and video supporting that text, and structured data that matches the visible text.

It also states the limit plainly. Google may leave a page uncrawled, unindexed or unserved even when it meets every requirement and policy. For a site owner, that sentence sets the expectation for every AI visibility report sold as a guarantee.

Control exists as well. Snippet controls, whether nosnippet, data-nosnippet, max-snippet or noindex, apply to what Google shows, and the troubleshooting section explains the two reasons a control appears to be ignored: the rule was not visible to Googlebot, or Google has not recrawled yet, which can take anywhere from several days to several months. URL Inspection shows the HTML Googlebot received, which lets the site owner check whether the rule was visible.

Structured data that matches the visible text

Structured data is a standardised way of describing what a page contains, in a block that a machine reads separately from the prose. Google's introduction describes the purpose well: Search works hard to understand a page, and explicit clues about its meaning help, with structured data able to enable richer results.

Three formats are supported, and Google treats JSON-LD, microdata and RDFa as equally fine when the markup is valid. JSON-LD is the recommended one, partly because it can be generated dynamically and read from the rendered page, which suits a build that assembles pages from templates and a database.

The rule that gets broken most often sits in the guidelines around the markup: the block describes what a reader can see. A price in JSON-LD against a page that says "contact us for pricing" is a mismatch, and a review count with no reviews on the page is the same problem. For a service business the useful types are few: Organization with a name, address and contact point, LocalBusiness where a physical address matters, Service or Product for the offer, and FAQPage where the page genuinely answers questions in the body text.

The llms.txt file and the job it can do

The llms.txt proposal suggests a markdown file at a site root, or at any path within it, that gives a language model a short map of the site's content. A conforming file holds an H1 with the site name, a blockquote summary, optional prose and H2 sections of links to markdown copies of pages, each written as a name, a URL and an optional note. The proposal also recommends two link relations so a client can find the pieces: rel="alternate" pointing at the markdown version of a page, and rel="describedby" pointing at the file that covers it.

The proposal gives llms.txt a specific, limited role. Sitemaps list pages for search engines; llms.txt offers a curated overview for models. robots.txt records acceptable access rules. An agent reads an llms.txt file on demand when it needs information while helping a person. The authors expected the main use to be inference, and they published the document as a proposal.

Adoption is real where agents do the reading. OpenAI, Anthropic and Gemini publish llms.txt files for their own developer documentation, documentation platforms generate one automatically, and Chrome's Lighthouse audits sites for one as part of its agentic browsing checks. For a business site, the file offers agents a curated overview of selected pages after the first three passes succeed.

A four-pass check on your own site

This four-pass check produces a list of findings with no score attached.

Pass one, fetch. Take the URLs that matter, fetch each one with a plain HTTP client and read the status. Record cases such as a 200 with an empty body, a 403 from the CDN or a redirect to a cookie wall, then investigate the hosting and access rules involved.

Pass two, read. View the source of the fetched page, disable JavaScript in a browser, and confirm the text is there. Search for a sentence from the middle of the page. When it is missing, the fix sits in rendering, and the usual version is server rendering for the routes that carry content.

Pass three, identify. Confirm the organisation name, the page author and the update date appear in the response, and that an Organization block names the same entity. A page that credits a person is stronger for both search and assistant use, and it costs a line in the template.

Pass four, structure. Validate the JSON-LD against the page as a reader sees it, and remove every property the page does not support. Then add the llms.txt file and the sitemap entry after the first three passes are clean.

Where the build work lands

Most failed passes are template work. Rendering belongs to the framework or the static export; identity belongs to the CMS schema and the author model; structure belongs to a small set of components that emit JSON-LD from the same data the page renders. The relevant implementation category is custom web and mobile development.

The order matters for cost. Fetch and read failures are blocking issues that make the rest pointless, identity is cheap and improves every page at once, and structured data is where the fiddly review sits because each type has its own guidelines. Keep the output of each review in a file so future questions about the site's citation record have dated evidence.

The example contact form is one possible destination for a citable service page.

Sources

Source: The /llms.txt file, v2 — llmstxt.org, retrieved 2026-10-25 (the proposal for a /llms.txt markdown file at a site root or any path, covering the pages under that path; the required and optional sections, from the optional byte-order mark and the H1 through the blockquote summary and the H2 file lists; the rel="alternate" type="text/markdown" and rel="describedby" link relations; llms.txt designed to coexist with sitemaps and robots.txt, with sitemaps listing all pages for search engines and llms.txt offering a curated overview for LLMs, robots.txt stating acceptable access and llms.txt read on demand; the expectation that the file would mainly be useful for inference).

Source: The /llms.txt file — AnswerDotAI/llms-txt, github.com/AnswerDotAI/llms-txt, retrieved 2026-10-25 (v2 of the proposal; thousands of sites publish the file, documentation platforms generate one automatically, Chrome's Lighthouse audits sites for one as part of its agentic browsing checks; the AI labs publish llms.txt files for their own developer docs, including OpenAI at developers.openai.com/llms.txt, Anthropic at docs.anthropic.com/llms.txt and Gemini at ai.google.dev/gemini-api/docs/llms.txt; guidance to use concise language, informative descriptions and to test the file by asking an agent questions about the content).

Source: Overview of OpenAI Crawlers — OpenAI API documentation, developers.openai.com/api/docs/bots, retrieved 2026-10-25 (the four user agents OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User with their purposes; the two robots.txt tags are independent, so a site can allow OAI-SearchBot for search while disallowing GPTBot to keep content out of training; a site allowing both may have one crawl serve both uses; about 24 hours for systems to adjust after a robots.txt update; sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers though they can appear as navigational links; ChatGPT-User serves user-initiated actions, for which robots.txt rules may not apply; OAI-AdsBot visits only pages submitted as ads, and OpenAI excludes its data from training generative AI foundation models).

Source: AI features and your website — Google Search Central, developers.google.com/search/docs/appearance/ai-features, last updated 2025-12-10, retrieved 2026-10-25 (how AI features work in Search and how to appear in them; meeting every requirement, best practice and policy does not mean Google will crawl, index or serve the content, and indexing and serving is not guaranteed; the listed SEO best practices including crawling allowed in robots.txt and by any CDN or hosting infrastructure, internal links, page experience, important content available in textual form, high-quality images and videos and structured data matching the visible text; controls through nosnippet, data-nosnippet, max-snippet and noindex; troubleshooting preview controls with the URL Inspection tool and allowing time for a recrawl, which can take several days to several months).

Source: Introduction to structured data markup in Google Search — Google Search Central, developers.google.com/search/docs/appearance/structured-data/intro-structured-data, retrieved 2026-10-25 (structured data as a standardized format for providing information about a page and classifying its content; explicit clues about the meaning of a page helping Search; structured data enabling rich results; JSON-LD, microdata and RDFa supported and equally fine when the markup is valid, with JSON-LD the recommended format, readable when generated dynamically; the general structured data guidelines applying alongside the guidelines for each type).

Source: the llms.txt proposal and its project README, OpenAI's crawler documentation and Google's AI features and structured data documentation, all retrieved 2026-10-25. Every requirement, user agent, link relation and quotation above traces to one of those pages; the four-pass sequence and the build consequences form the article's editorial framework. Images are captures of those four documentation pages plus two Netics editorial diagrams. Netics editorial, neticslabs.com.