Why Your SPA Is Invisible in the Answer Engine Era

Disclosure: We build client sites in Astro commercially. We also build in React and Next.js where an application architecture is the right call. Our commercial interest is in the audit, not in any particular framework winning.

There is a specific failure mode I keep finding in audits, and it is invisible from inside the company. The site ranks well. Search Console looks healthy. Organic traffic is flat but not collapsing. And the company gets cited by AI answer engines approximately never, while a competitor with worse content and a fraction of the domain authority gets quoted constantly.

The cause is almost never content quality. It is that the crawler could not read the page.

The technical fact that breaks client-side rendering

AI crawlers do not execute JavaScript.

This is not a nuance or an edge case. Vercel and MERJ analysed over 500 million GPTBot fetches and found zero evidence of JavaScript execution. GPTBot downloads .js files roughly 11.5% of the time — and never runs them. It treats them as static text. The same pattern held across ClaudeBot, PerplexityBot, Meta’s ExternalAgent, and Bytespider.

Here is where each major crawler stood as of 25 July 2026. Crawler behaviour changes; verify before relying on it.

Crawler	Purpose	Executes JS?
Googlebot	Search indexing	Yes — headless Chromium (WRS)
Google Gemini	AI answers	Yes — shares Googlebot’s WRS
GPTBot	OpenAI training	No
OAI-SearchBot	ChatGPT search retrieval	No
ChatGPT-User	User-triggered fetch	No
ClaudeBot	Anthropic training	No
Claude-SearchBot	Claude search retrieval	No
PerplexityBot	Perplexity retrieval	No
Meta-ExternalAgent	Meta AI	No
Bytespider	ByteDance	No

Google is the outlier. Every other significant answer engine gets your raw HTTP response and nothing else.

There is no partial credit here. A crawler that does not execute JavaScript does not see a slightly degraded version of your page. It sees <div id="root"></div>.

Before the entity graph, the empty div

Search this topic and most of what you find is written for marketers. Entity authority. Trust architecture. Knowledge graphs. Nested schema. Advice to become “an AI-native knowledge source.”

None of it is wrong, exactly. All of it is downstream of a question nobody in that literature asks: did the crawler receive any content at all?

You can implement a flawless Organization → Product → Offer graph with consistent @id references throughout, and if it is injected into the DOM by React after page load, GPTBot never sees a byte of it. Schema in a client-rendered app is schema that does not exist. The retrieval layer everyone is optimising for runs on the raw HTTP response, and that is the layer to check first.

So check it first.

The diagnostic takes thirty seconds

Do not use the DevTools Elements panel. It shows the rendered DOM — the state after JavaScript has run — which is exactly the thing AI crawlers never see. It will lie to you.

Use view-source, or better, curl:

# This is byte-for-byte what GPTBot receives.
curl -s -A "GPTBot" https://yoursite.com/your-best-page | grep -c "a distinctive sentence from your content"

If that returns 0, the page does not exist as far as every non-Google answer engine is concerned.

A more complete check:

#!/usr/bin/env bash
# aeo-check.sh — does raw HTML contain the content that matters?
URL="$1"
HTML=$(curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.0)" "$URL")

echo "Bytes returned:      $(echo "$HTML" | wc -c)"
echo "H1 present:          $(echo "$HTML" | grep -c '<h1')"
echo "Paragraphs:          $(echo "$HTML" | grep -c '<p[ >]')"
echo "JSON-LD blocks:      $(echo "$HTML" | grep -c 'application/ld+json')"
echo "Visible text approx: $(echo "$HTML" | sed 's/<[^>]*>//g' | tr -s ' \n' ' ' | wc -w) words"

A client-rendered React app typically returns 2–8 KB with zero paragraphs and a word count under 50 — mostly the contents of <title> and some meta tags. A server-rendered equivalent returns the actual document.

Put this in CI. It is a genuine architectural regression when someone moves a section into a client-only component, and nothing else will catch it.

Why hydration is the wrong default

The standard SPA lifecycle for a content page:

1. Server sends a near-empty HTML shell.
2. Browser downloads the JavaScript bundle — framework runtime plus application code.
3. Browser parses and compiles it. Parse and compile cost scales roughly with bundle size and is dramatically worse on low-end mobile CPUs than on a developer laptop — measure it on a real mid-range Android with DevTools remote debugging rather than trusting any published figure, including one in this article.
4. Framework boots, builds a virtual DOM, renders it into the real DOM.
5. Framework walks the entire tree attaching event listeners: hydration.
6. The page becomes interactive.

Every step happens on the main thread. Steps 3 through 5 are the reason Interaction to Next Paint on SPA content sites is routinely terrible: during hydration, the main thread is blocked, so any tap the user makes is queued rather than handled.

The architectural error is the assumption underneath step 5. Hydration is all-or-nothing in a traditional SPA. A blog post with a single interactive element — a search box, say — still pays hydration cost for the entire page: the nav, the article body, the footer, the comment count, all of it. You are shipping and executing JavaScript to make static text interactive, which it never needed to be.

For an application — Figma, a trading dashboard, an email client — this is the correct trade. For a content site, it is paying a large, recurring cost for nothing.

Islands: the correct default is zero

Islands architecture inverts the default. Everything renders to static HTML at build or request time. Components that genuinely need interactivity are marked explicitly, and only those ship JavaScript. Each is an independent “island” that hydrates on its own.

In Astro:

---
// src/pages/specs/[slug].astro
// This frontmatter runs on the server only. It never reaches the browser.
import Layout from '../../layouts/Base.astro';
import TableOfContents from '../../components/TableOfContents.astro';
import SearchBox from '../../components/SearchBox.tsx';
import CodePlayground from '../../components/CodePlayground.tsx';
import { getEntry } from 'astro:content';

const { slug } = Astro.params;
const entry = await getEntry('specs', slug);
const { Content, headings } = await entry.render();
---

<Layout title={entry.data.title}>
  <!-- Zero JS. Pure HTML in the response. -->
  <TableOfContents headings={headings} />

  <article>
    <h1>{entry.data.title}</h1>
    <Content />
  </article>

  <!-- Island 1: hydrates immediately, above the fold -->
  <SearchBox client:load />

  <!-- Island 2: hydrates only when scrolled into view -->
  <CodePlayground client:visible code={entry.data.example} />

  <!-- Island 3: hydrates only on wide viewports -->
  <SidebarFilters client:media="(min-width: 60rem)" />
</Layout>

The directives are the whole point:

Directive	Behaviour	Use for
(none)	Renders to HTML, ships 0 KB JS	Text, nav, footer, tables — most of the page
client:load	Hydrates on page load	Above-the-fold interactive elements
client:idle	Hydrates during requestIdleCallback	Non-urgent widgets
client:visible	Hydrates when it enters viewport	Anything below the fold
client:media	Hydrates on media query match	Viewport-conditional UI
client:only	Skips SSR entirely	Components that touch window at module scope

client:visible deserves emphasis. A page with six interactive widgets, five of them below the fold, ships and executes one widget’s JavaScript on load. The rest cost nothing until the user actually scrolls to them — and if they never scroll, they never cost anything at all.

The output of the page above:

<article>
  <h1>Core Web Vitals: Thresholds and What Moves Them</h1>
  <p>Largest Contentful Paint measures...</p>
  <!-- Full content. In the raw response. Before any JavaScript. -->
</article>
<astro-island component-url="/_astro/SearchBox.a3f9c2.js" client="load">
  <input type="search" ... />
</astro-island>

GPTBot fetches that and gets the entire article. So does a user on a bad connection, before the network has finished with anything else.

“Just use SSR” is a partial answer

The common objection: Next.js already server-renders, so this is solved.

SSR fixes the crawler problem. It does not fix the hydration problem.

Server-rendered React sends complete HTML and then sends the full JavaScript bundle to hydrate it. The crawler is happy. The user on a mid-range phone still downloads, parses, and executes several hundred kilobytes, and still experiences a main thread blocked during hydration. You have solved visibility and left performance untouched.

React Server Components and selective hydration narrow the gap considerably, and if you are already deep in a React codebase, the RSC path is the pragmatic one. But note what it is doing: reimplementing partial hydration inside a framework whose architecture assumed full hydration. Islands frameworks start from the opposite default, which is why the ceiling is lower.

The qualitative shape of what these architectures deliver on a content page. Note that this table deliberately contains no bundle-size figures: those depend entirely on your dependencies, and any number quoted without a named repo and a build date is decoration.

Architecture	Content in raw HTML	JS shipped for static content	Main thread blocked on load
CSR SPA	No	Full framework + app	Severe
SSR + full hydration	Yes	Full framework + app	Severe
SSR + RSC / selective	Yes	Reduced — client components only	Moderate
Islands (Astro, Fresh, Qwik)	Yes	Zero, plus marked islands only	Minimal

To get real numbers for your own stack, build the same content page in each and compare gzipped transfer in the DevTools Network panel with cache disabled. Publish the method alongside the result.

Where this genuinely does not apply

I will not pretend islands are universal. If your product is an application — persistent client state across views, real-time collaboration, complex optimistic UI, a canvas — a SPA is the right architecture and I would build one.

The mistake is applying application architecture to content. Marketing sites, documentation, blogs, pricing pages, product pages, changelogs. That content has no meaningful client state, and shipping a framework runtime to render it is a decision nobody made deliberately; it happened because the team already knew React and the meta-framework made it easy.

The tell is simple: if a page would work fine with JavaScript disabled — and most content pages would — every kilobyte of framework you ship to it is waste.

Migration without a rewrite

You do not need to rebuild. The approach that works:

1. Rank pages by AEO value. Documentation, technical guides, comparison pages, pricing. These are the pages answer engines cite. Your app’s settings screen is not on this list.

2. Move those to static generation first. In Next.js, that means generateStaticParams and no client components in the content path. In Astro, a fresh content collection.

3. Audit the client boundary. In an RSC codebase, every "use client" is a hydration boundary. Most exist because someone needed one onClick. Push the directive down to the smallest component that actually needs it:

// Before: entire article is a client component for one button
"use client";
export default function Article({ post }) {
  const [copied, setCopied] = useState(false);
  return (
    <article>
      <h1>{post.title}</h1>
      <div dangerouslySetInnerHTML={{ __html: post.html }} />
      <button onClick={() => setCopied(true)}>Copy link</button>
    </article>
  );
}

// After: server component, one small client island
import CopyLinkButton from './CopyLinkButton'; // "use client" lives here
export default function Article({ post }) {
  return (
    <article>
      <h1>{post.title}</h1>
      <div dangerouslySetInnerHTML={{ __html: post.html }} />
      <CopyLinkButton />
    </article>
  );
}

4. Enforce it in CI. A Playwright test that fetches with JS disabled and asserts the H1 and first paragraph are present:

test('content is in raw HTML', async ({ browser }) => {
  const ctx = await browser.newContext({ javaScriptEnabled: false });
  const page = await ctx.newPage();
  await page.goto('/specs/core-web-vitals');
  await expect(page.locator('h1')).toContainText('Core Web Vitals');
  await expect(page.locator('article p').first()).not.toBeEmpty();
});

Frequently asked questions

Do AI crawlers execute JavaScript? No. GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent, and Bytespider all fetch raw HTML without executing JavaScript. Google Gemini is the exception, because it uses Googlebot’s rendering infrastructure.

Can a page rank #1 on Google and still be invisible to ChatGPT? Yes, and this is common. Googlebot runs headless Chromium and renders your client-side content. GPTBot does not. The two systems see genuinely different documents.

Does server-side rendering fix this? It fixes crawler visibility completely. It does not fix hydration cost — SSR React still ships the full bundle and blocks the main thread on hydration. Different problem, different fix.

How do I test what an AI crawler sees? curl -A "GPTBot" https://yoursite.com/page and read the output, or use view-source in the browser. Do not use the DevTools Elements panel; it shows the post-JavaScript DOM.

Is Astro the only islands framework? No. Astro, Fresh (Deno), Qwik, Marko, and Enhance all implement partial hydration. Astro has the largest ecosystem and lets you use React, Vue, or Svelte components inside islands.

Should I block AI crawlers instead? Understand the split first. Training crawlers (GPTBot, ClaudeBot) affect whether your content informs future models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) affect whether you appear in AI answers today. Blocking the search crawlers removes you from AI search results entirely. Block deliberately, per bot, not with a blanket rule.

What about my existing React app? Leave the application alone. Move the content — docs, blog, marketing, pricing — to a static or islands architecture and serve it from the same domain. Most sites are two products wearing one codebase.

---

Sources

• Vercel & MERJ, The Rise of the AI Crawler — the 500M-fetch analysis underpinning the JS-execution claims
• Astro Template Directives Reference — client:* behaviour
• Google Search Central: JavaScript SEO Basics — Googlebot rendering

Claims last verified 25 July 2026. Crawler behaviour changes; we date what we state and revise when the evidence does. See our editorial standards.

Related posts

• Fluid Type Systems: The Parts the Calculators Don't Generate
• Zero-Jank Scroll: Getting Animation Off the Main Thread