Debugging React crawlability on Cloudflare Pages

The failure: a canonical URL that redirected back

During our October 12 audit, Light Reach served /crawl/ and /signal/, but their canonical tags named /crawl and /signal. Those slashless URLs redirected to the slash versions. The page told Google to prefer one URL while the host sent requests to another. Search Console classified the served pages as alternate pages with a proper canonical tag.

That classification is often expected for a genuine duplicate. In this case, the preferred identity did not agree with the deployed route. We aligned the canonical tag, sitemap, structured page URL, and internal links with the final served URL. Do not change every excluded URL just to make an indexing report greener: redirects, private screens, and real duplicates have different jobs.

We also requested a made-up path and received the homepage with HTTP 200. Several product sites returned an application shell rather than readable public content. These were separate defects: fixing a canonical does not fix missing HTML, and rendering a page does not fix its status code.

1. Compare the requested URL with the final URL

Use a GET request rather than assuming a HEAD response behaves identically. The first command shows the status without following redirects. The second follows the chain and reports the final URL. The example uses our public site; substitute one of your deployed public pages.

For the preferred page URL, expect HTTP 200 without a redirect. An intentional alias may redirect once to that page. Neither a trailing slash nor a slashless URL is universally better. Match the hosting behavior and choose a consistent identity.

On our repaired site, /crawl/ returns 200 directly and names https://www.lightreach.io/crawl/ as its canonical. On October 12, 2026, Google URL Inspection also reported both Crawl and Signal as submitted and indexed, with the declared and Google-selected slash canonicals matching. The download above preserves that dated observation. A successful HTTP response and Google’s recorded index state are separate evidence; neither establishes a traffic gain.

page_url='https://www.lightreach.io/crawl/'
curl -sS -o /dev/null -w 'status=%{http_code}\n' "$page_url"
curl -sS -L -o /dev/null \
  -w 'status=%{http_code} redirects=%{num_redirects} final=%{url_effective}\n' \
  "$page_url"

2. Inspect the document before JavaScript runs

Google can render JavaScript. Its JavaScript SEO documentation also explains why prerendering or server rendering helps users and crawlers, including bots that cannot execute JavaScript. The useful question is whether your essential explanation depends on a successful browser session.

Fetch the page into a temporary directory and inspect its title, heading, canonical, robots directive, links, and text. This small standard-library parser reads the received HTML without running scripts. It does not simulate Googlebot, evaluate robots.txt, validate structured data, or prove indexing.

Look for the actual product answer, rather than only a brand name in a footer. Check that there is one canonical and that it agrees with the final URL. A public page should not carry noindex. Save the output with the release date so a later regression is easy to identify.

A heading can exist in the HTML and still be invisible. Our follow-up found primary headings on five product pages inside an entrance-animation ancestor with inline opacity:0. The small parser below reports text presence; it does not check visibility. Inspect the heading and its ancestors for hidden, aria-hidden, opacity:0, display:none, and visibility:hidden, then check computed CSS with JavaScript disabled. A visible screenshot after JavaScript runs can miss this failure.

audit_dir=$(mktemp -d)
curl -fsS 'https://www.lightreach.io/crawl/' -o "$audit_dir/page.html"
python3 - "$audit_dir/page.html" <<'PY'
import json, sys
from html.parser import HTMLParser
from pathlib import Path

class Audit(HTMLParser):
    def __init__(self):
        super().__init__()
        self.canonicals, self.robots, self.headings = [], [], []
        self.links, self.text, self.title = 0, [], []
        self.skip = 0
        self.in_title = False
    def handle_starttag(self, tag, attrs):
        attrs = dict(attrs)
        if tag in ('script', 'style'): self.skip += 1
        if tag == 'title': self.in_title = True
        if tag == 'h1': self.headings.append('h1')
        if tag == 'a' and attrs.get('href'): self.links += 1
        if tag == 'link' and 'canonical' in attrs.get('rel', '').split():
            self.canonicals.append(attrs.get('href'))
        if tag == 'meta' and attrs.get('name', '').lower() == 'robots':
            self.robots.append(attrs.get('content'))
    def handle_endtag(self, tag):
        if tag in ('script', 'style'): self.skip = max(0, self.skip - 1)
        if tag == 'title': self.in_title = False
    def handle_data(self, data):
        if self.in_title: self.title.append(data)
        if not self.skip and data.strip(): self.text.append(data.strip())

audit = Audit()
audit.feed(Path(sys.argv[1]).read_text(encoding='utf-8'))
print(json.dumps(dict(title=''.join(audit.title), h1_count=len(audit.headings),
    canonicals=audit.canonicals, robots=audit.robots, link_count=audit.links,
    text_preview=' '.join(audit.text)[:900]), indent=2))
PY

3. Request discovery files and a nonexistent page

A route returning 200 does not tell you whether it returned the correct resource. Read the Content-Type and body for robots.txt and sitemap.xml. The former should contain text directives; the latter should contain a sitemap document. An HTML application root at either address needs investigation.

Cloudflare Pages documents a default SPA fallback when a deployment has no top-level 404.html. An unmatched path can receive the root application. We added an error document and verified missing public paths return HTTP 404. A React component that says “not found” inside a 200 response is a different outcome.

The missing path below is deliberately invented. On our repaired host, the expected result is 404. If your application uses a catch-all Worker or dynamic router, inspect its route policy too: a static 404 file alone does not describe every edge path.

site_url='https://www.lightreach.io'
curl -sS -D - "$site_url/robots.txt"
curl -sS -D - "$site_url/sitemap.xml"
curl -sS -o /dev/null -w 'missing_status=%{http_code}\n' \
  "$site_url/crawlability-check-this-page-does-not-exist"

4. Preserve the application when adding a real 404

Before removing a fallback, inventory the routes it currently serves. We had public marketing pages, login screens, nested app routes, token routes, API paths, and user handles across different products. A blanket rule sending all unmatched paths to an error page would break valid application entry points.

Our solution renders public pages into HTML, keeps a separate noindex application shell for recognized app routes, and lets unknown public routes return an error. Authentication and authorization still protect private data. A noindex directive controls discovery; it is not an access-control mechanism.

Cloudflare also normalizes HTML filenames. During deployment validation, a rewrite targeting /app.html redirected and changed the browser URL. Targeting the extensionless /app resource preserved the requested private route on those deployments. Verify your own host rather than copying a rewrite blindly; Pages Functions and Workers can change route ownership.

1. Open login and a nested app URL directly in a fresh browser session, then reload them.
2. Confirm recognized app routes preserve their requested URL and carry noindex in the initial response.
3. Confirm the actual authentication flow still works; do not treat HTTP 200 as an authentication test.
4. Check API response types and handle/token routing separately from static marketing paths.
5. Request an unknown public path and confirm it is still a real 404.

5. Put the shared rule at the rendering boundary

We generate the public HTML, sitemap, canonical metadata, and structured page identity from the same page inventory. That prevents the sitemap and canonical from becoming separately maintained lists. A build check walks every published page and compares those representations before deployment.

Rendering existing React components on the server exposed browser-only reads of window and localStorage. Move those reads behind the appropriate browser lifecycle or guard them at their owning boundary. Preserve the client behavior and test the server build. Avoid patching each page with a different version of the same route rule.

For our Framer Motion pages, AnimatePresence initial={false} at the public route boundary made the initial document use the visible animation frame. We extended the existing generated-page check to reject suppressed primary headings and verified all 22 main-site sitemap pages after deployment. Choose a policy that fits your own animation library and verify browser behavior too; text extraction alone is not a visibility test.

Include the public page’s CSS in its initial document. A route stylesheet that loads only after JavaScript runs can leave the prerendered content briefly unstyled. Our guide stylesheet needed this check. Rendering readable HTML is one part of serving a usable page.

Local output is necessary evidence, but the CDN is the final serving boundary. After deployment, repeat the HTTP checks against the canonical production host, then inspect the page on desktop and mobile. We verified 33 public page responses and 10 private/error route checks across our four sites in the initial repair pass; the counts describe that pass, not every possible route.

What counts as a completed crawlability repair?

Keep a small release record: requested URL, final URL, status, canonical, initial content, robots directive, sitemap membership, and private/error route behavior. Attach the deployed revision and verification date. That gives the next engineer a reproducible contract instead of “SEO looks good.”

Then monitor separately. A sitemap fetch shows that Google received a file. A notification shows that a submission was accepted. URL Inspection shows Google’s recorded crawl and indexing state. Search impressions, clicks, and qualified inquiries describe later outcomes. None of the first three proves the last three.

For Google’s AI search features, ordinary search eligibility and useful content remain relevant; no special AI file guarantees inclusion. Our October 12 work establishes a more coherent, readable serving layer. We have not measured a traffic increase caused by these repairs.