Why most businesses are “AI-invisible”: answer presence vs machine readability

Picture this: a potential customer asks an AI assistant for “the best [service] in [city].” You expect your business to show up because your website ranks well in search results. Instead, the assistant names competitors… and your site doesn’t get mentioned even once.

That mismatch is exactly what the AI Visibility Index is trying to measure. In their latest edition (covering audits through August 2, 2026), 94.8% of audited sites were never named in an AI answer, even though only 8.9% block at least one AI crawler in robots.txt. In other words: blocking crawlers is uncommon, but being cited is rarer still.

So what’s actually going on? The answer is that “being visible to AI” isn’t one problem. It’s two different problems with two different fixes.

The two pillars: answer presence and machine readability

The Index separates “AI visibility” into:

1) Answer presence (do you get named?)

A customer asks for an answer, and the assistant responds with a short list of businesses. The study checks whether the audited business is one of those names. This is measured by querying ChatGPT, Claude, Gemini, and Perplexity and matching the assistant’s output against the audited business name.

This part is subtle for beginners: there’s no direct, guaranteed relationship between “your pages exist” and “your business gets named.” LLM assistants can choose a subset of entities based on relevance, familiarity, and retrieval quality.

2) Machine readability (can AI reliably retrieve what you are?)

Even if you want to be cited, an assistant’s crawler still has to be able to reach your site and interpret it. The Index treats machine readability as signals under your control, including:

• whether robots.txt allows the crawl,
• whether you publish a sitemap.xml,
• whether your homepage states identity and location in machine-readable form.

A crucial idea lands here: presence is an outcome, readability is an input. You can publish well-structured pages and still not be named. You can also be readable enough to be found by a crawler and still fail to win “the shortlist” inside an answer.

The headline result: why “only” 8.9% blocking still doesn’t create visibility

At first glance, the “blocking” number looks like the culprit. Eight-point-nine percent blocking AI crawlers sounds bad—until you compare it to the “never named” number.

If only 8.9% blocked and 94.8% were never named, blocking can’t be the main reason most businesses disappear from AI answers.

So the story becomes this:

• Some sites accidentally or intentionally prevent retrieval/citation.
• But even when crawlers can reach you, most businesses still don’t get selected as an entity worth naming.

That selection step is its own ecosystem: the assistant decides what it trusts, what it retrieves, and what it can confidently attribute.

The robots.txt trap: training crawlers vs retrieval/citation agents

Robots.txt is where many teams get tripped up, because the directives are easy to copy/paste and hard to reason about.

The Index breaks blocking into categories that matter operationally:

• Training crawlers: harvest content so models can learn patterns offline.
• Retrieval/citation agents: fetch pages to answer a live question, and are therefore directly tied to whether you get quoted or cited.

A key detail from the study is that many sites block broadly without distinguishing between these purposes. As the Index describes, blocking training costs you little in today’s answers, while blocking retrieval can make you uncitable.

In their dataset, the most common blocks were training-related user agents (like CCBot, GPTBot, Bytespider, and ClaudeBot), while blocking the retrieval agents (like OAI-SearchBot and Perplexity-User) was comparatively rare.

That doesn’t mean robots.txt is irrelevant. It means that the wrong mental model (“robots.txt decides visibility”) is incomplete.

Robots.txt can decide whether an assistant can even read your site for attribution. But robots.txt can’t force the assistant to choose you.

Structured data: helpful, but not sufficient

The Index also measures structured data adoption:

• 54.6% of audited sites publish any schema.org structured data.
• Only 19.3% publish schema.org LocalBusiness.

This is where many beginners get a false sense of progress. “We added schema.org!” sounds like a win. And it is a win—up to a point.

What schema.org structured data actually does

Schema.org structured data is a way to annotate your pages with machine-readable meaning (usually via JSON-LD). Instead of the assistant guessing what a piece of text refers to, it can parse explicit fields like “this is your business name,” “this is your address,” and “this is the category.”

Why LocalBusiness specifically matters

LocalBusiness is a schema type intended to communicate identity, location, and business context. If you only add generic structured data (or you put LocalBusiness markup in a place an assistant doesn’t read), you’re leaving a gap: the system can crawl your site, but the “who are you in this place?” information isn’t structured enough to support confident citation.

So the numbers fit together:

• Many sites are partially machine-readable.
• Far fewer supply the specific identity/location structure that makes entity-level retrieval and attribution easier.
• Most businesses still aren’t named—because structured data alone doesn’t guarantee the assistant’s ranking/selection.

Building a more “AI-citable” presence: practical mechanics

Below is a pragmatic way to translate the Index findings into technical actions.

Step 1: Treat robots.txt as retrieval access control

Robots.txt doesn’t “opt you into AI answers.” It controls which crawlers are allowed to request your content.

A common beginner mistake is blocking broad ranges (or older templates) without understanding the difference between training and retrieval/citation agents.

A safer posture for visibility is:

• ensure robots.txt doesn’t block the agents that are likely to be used for live retrieval and citation,
• avoid legacy blanket rules that were written for traditional search bots years ago.

Example robots.txt skeleton (focus on clarity, not perfection):

User-agent: *
Allow: /

# If you must block something, block narrowly and document why.
# Avoid copy-paste blocks that unintentionally hide your homepage.

# Example (do NOT blindly use this):
# User-agent: GPTBot
# Disallow: /

The important part isn’t the exact agent names—it’s the workflow: review what you block, and make sure you’re not preventing retrieval in practice.

Step 2: Publish sitemap.xml and point robots.txt at it

Even when crawlers can access your pages, discovery matters. The Index reports that most audited sites publish a sitemap, and many also point robots.txt at it.

Mechanically, the homepage doesn’t have to be the only page that’s crawled—but the homepage is where identity signals often live.

Step 3: Add machine-readable identity on the homepage (not only inner pages)

The Index notes a specific constraint: it reads crawl signals from a limited set of places (homepage, robots.txt, sitemap.xml). That’s a big deal.

If your LocalBusiness markup sits on a contact page, but your homepage lacks identity structure, you may be making it harder for an assistant to ground your business.

Step 4: Use JSON-LD LocalBusiness markup

Here’s a simplified JSON-LD example showing the kinds of fields LocalBusiness is meant to express.

<script type="application/ld+json">
{
 "@context": "https://schema.org",
 "@type": "LocalBusiness",
 "name": "Example Plumbing Co",
 "address": {
 "@type": "PostalAddress",
 "streetAddress": "123 Main St",
 "addressLocality": "Springfield",
 "addressRegion": "IL",
 "postalCode": "62701",
 "addressCountry": "US"
 },
 "telephone": "+1-555-0100"
}
</script>

The assistant doesn’t need perfect markup to crawl. But it does need enough structured identity to reduce ambiguity when producing an answer.

Why the selection step still fails for most sites

Could it be that adding markup will make you appear in every AI answer? No—and the Index data shows why.

Even among sites that are crawlable, the assistant must still:

• retrieve the right pages,
• interpret relevance for a specific buying-intent question,
• decide which businesses to name (often a short shortlist),
• and avoid hallucinating entities that it can’t ground.

So the real lesson is: technical readiness makes you eligible; it doesn’t guarantee nomination. That’s the difference between “can be cited” and “will be cited.”

A mental model that finally clicks

If we stitch the Index findings into one coherent narrative, it looks like this:

• Blocking AI crawlers is relatively uncommon (8.9%).
• Yet almost everyone is never named (94.8%).
• Therefore, the dominant bottleneck is not just access—it’s how assistants choose entities and how reliably they can ground identity in what they retrieve.

In practice, the best approach is layered:

1. Ensure retrieval access (robots.txt + sitemap).
2. Provide strong machine-readable identity (LocalBusiness on the homepage).
3. Expect that these steps improve odds, not certainty.

That’s a calmer, more accurate way to think about AI visibility—especially for small and mid-sized businesses that don’t have the budget to “game” LLM behavior. The goal isn’t magic ranking. The goal is reducing uncertainty so the assistant can cite you when it does decide you belong in the answer.