Agent Visibility Has a New Landlord, and It's Cloudflare

TL;DR • Cloudflare now says fewer than half of all HTML page requests come from humans, and it has shipped Agent Readiness plus an Answer Engine Optimization (AEO) tool that tells site owners whether agents can read their site and whether AI assistants recommend them. • The underlying problem is real and the diagnostics are genuinely useful — robots.txt, sitemap, Markdown, and agent-login hygiene have been invisible to most site owners until now. • But the tool is captive, opaque, and self-serving: it only measures traffic that passes through Cloudflare's own network, the benchmark it scores you against probes two US assistants (Claude and GPT) with prompts the product does not expose, and the recommended fixes conveniently route back to "Set up in Cloudflare" buttons. The referee is also a player.

What happened

Cloudflare is pitching "the age of AI agents" as a discovery problem. Its argument, in a post titled "From ranking to recommended: get your site ready to thrive in the age of AI agents", is that your next customer will ask an AI assistant instead of a search engine, and "the moment that determines whether a customer chooses you happens inside a model's response — before a human ever sees your homepage."

The headline number is striking: by Cloudflare's count, fewer than half of all HTML page requests now come from a human — not all are agents acting for a person, the post concedes, but the share is growing fast.

Cloudflare integrated its earlier Agent Readiness work into the dashboard, adding a new AEO tab to answer "Can agents actually use my site?" and "Am I getting recommended?". Diagnostics scans a hostname the way an agent reads it: robots.txt, sitemap, response headers, a machine-readable Markdown version, and published metadata. Checks are grouped by effort: quick wins (crawler-readable robots.txt, an XML sitemap, AI-crawler rules, clean Markdown), technical groundwork (Content Signals, an API catalog, link headers, agent login instructions), and advanced integration (OAuth discovery, MCP and A2A agent cards, a skills index, Web Bot Auth, WebMCP). Commerce protocols — x402, ACP, UCP, AP2 — are listed as informational and, notably, not counted in the score.

The AEO tab probes "the leading assistants (today, Anthropic's Claude and OpenAI's GPT)" with prompts in the site's inferred industry and category, and reports Citation Rate, Prominence, Mention Rate, Share of Voice, and an Industry Fit score. A benchmark is built once per industry and category; its baseline is reused across all accounts in that domain, which Cloudflare says gives zero-latency results and avoids redundant AI calls across thousands of scans.

Netics' take

The problem Cloudflare identifies is real: site owners can't tell from their access logs whether AI bots can read or recommend their site. Agents genuinely lean on robots.txt, sitemaps, response headers, and structured metadata far more than human visitors do. Most sites are not built for that user yet — "the bar is low," as the post says. The Diagnostics checklist is a legitimate operational hygiene checklist — crawlable, machine-readable, unambiguous about how it may be used. That part deserves credit — as does listing the commerce protocols as "informational for now, and not counted in your score."

First, the measurement is captive. The post says it plainly: "because the requests actually pass through Cloudflare, these tools measure rather than estimate where possible" — a sentence that ends, in the original, with ", and will improve over time." A site owner on any other CDN, or behind their own edge, does not get this view at all. That is a sales motion wearing an audit's clothes.

Second, the benchmark is opaque. Cloudflare infers your industry and category, builds a panel of prompts that the post does not disclose, queries two US assistants — Claude and GPT — and scores your site against a baseline reused across thousands of scans. You get a Citation Rate and a Share of Voice with no visibility into the prompts, the panel, or the scoring — no impression count, no missed-click report, as the post itself notes. For a metric that is supposed to tell you whether customers choose you, the absence of any way to audit the measurement is not a detail; it is the design. And for European companies, the benchmark contains exactly two voices, both American — so "what AI assistants think of your brand" is defined by models whose training, infrastructure, and commercial incentives sit entirely outside the EU.

To be fair, Cloudflare publishes two facts about its method: "we use Cloudflare AI Gateway to prompt each assistant multiple times across different models," and "We also use exact text analysis rather than a model grading its own output." Both deserve credit. But describing a method is not exposing it: we learn the plumbing — multi-model prompts via AI Gateway, exact-text comparison — not the panel, whose prompts, templates, baseline composition, and scoring weights stay undisclosed. Better documented, yes. Auditable, no.

Third, the fix path is self-serving. Every suggested improvement comes with a next step, and where Cloudflare has a feature, a "Set up in Cloudflare" link — Markdown for Agents, managed robots.txt, and the rest. The diagnostic, the benchmark, and the remedy are all owned by the same company. That does not make the advice wrong; it makes it structurally impossible to be independent. When the referee defines the rules of the game, builds the scoreboard, and sells the equipment, the scores deserve a discount.

"Discoverability used to mean ranking on a results page," the post says. "Now it means being found, read, and confidently recommended by the agents that guide your customers." Ranking on a results page was policed by search engines with criteria public enough for an industry to measure and argue with. Agent recommendation is policed by private benchmarks, private prompts, and private model behavior that changes underneath you. It is the same pattern we flagged when MCP went stateless: the vendor that writes the protocol also profits from the infrastructure it pushes (our take on stateless MCP). Cloudflare is not the only party profiting from that opacity — the model providers are the other — but it is the one selling you the scorecard.

What this means for SMBs

• Treat the fundamentals as vendor-neutral, not as Cloudflare features. Crawler-readable robots.txt, an XML sitemap, clean HTML and Markdown, real Content Signals, and honest metadata help you in any assistant, CDN, or country. Fix those first; they cost nothing and cannot be held hostage.
• Do not let a private scorecard define your market position. Citation Rate and Share of Voice from one dashboard are inputs, not truth. Verify with your own tests: run the same customer prompts in Claude, GPT, and a European model yourself. The benchmark's opacity is a feature of the product, not a law of physics — audit the models directly.
• Watch the sovereignty angle. If an AI assistant's recommendation becomes your storefront, where that assistant is built, what it is trained on, and who runs its infrastructure are business decisions, not technical trivia. A French retailer whose visibility is decided by two US models, measured by a US CDN, and optimized through US tools has swapped one dependency (search engines) for another that is harder to see and harder to challenge.
• Ask who profits before you ask who is right. The advice is not wrong because Cloudflare sells the fix — it is unverifiable because Cloudflare owns the whole chain. Use the checklist, ignore the score, and build for the open agentic web.

Bottom line

The agentic discovery problem is real, and Cloudflare's diagnostics are a useful start. But a tool that only measures traffic inside its own network, scores you against a private benchmark of two US assistants, and routes every fix back to its own buttons is not an independent audit of your agent visibility — it is a customer-acquisition funnel with good documentation. The fundamentals are free and vendor-neutral; the scorecard is a product. Book a free 30-minute audit and we will help you build for agents on your own terms — including, if you want, without Cloudflare in the middle.

Source: From ranking to recommended: get your site ready to thrive in the age of AI agents — blog.cloudflare.com, 2026-08-06.

Explore Netics for practical infrastructure and AI operations guidance.