Technical SEO: A Checklist for Google, Bing and AI Bots in 2026

What Is Technical SEO?

Technical SEO is the work that decides whether search engines and AI search can crawl a page, render it (run any JavaScript it needs) and index it, and how fast and reliably that happens. It covers what Google, Bing, ChatGPT search, Claude and Perplexity need from a site before they can use a page at all.

Technical SEO makes a page usable: reachable, working, readable and stored in an engine's index. It doesn't make the page win. A technically sound page can still lose on relevance or trust, but a page that fails the technical basics never gets to compete.

Technical SEO vs. on-page SEO comes down to one test. Technical SEO is whether an engine can reach and read the page; on-page SEO is what the page says and how it's formatted, which the On-Page SEO & AI Answers Optimization guide goes deeper on. What a page should target in the first place is the subject of SEO Content Fundamentals.

Whether an engine trusts the page once it can read it is the subject of E-E-A-T in SEO.

Structured data, site architecture, international setup and product data for AI shopping agents are technical SEO too, and each has its own guide. This one covers the base they all rest on: crawling, rendering, indexing, mobile and speed.

"Technical SEO for AI" is also used to mean using AI tools or agents to do technical SEO. That is a question of who does the work, which the Agentic SEO guide covers. This guide is about what the work has to achieve.

One Floor, Five Search Crawlers: What Google, Bing and AI Search Require

Google and Bing document the same minimum for classic and AI search: a page crawlers can reach, that works, and that has content they can index. The AI search engines' crawler pages ask for the first part of it, that their crawler isn't blocked.

On Google's side, its technical requirements set that floor in three lines: Googlebot isn't blocked, the page works (an HTTP 200, or success, status code), and the page has indexable content.

Meeting them makes a page eligible, not guaranteed: Google "doesn't guarantee that it will crawl, index, or serve your page", per its guide to how Search works.

Its AI features add nothing technical beyond a Google Search Console setting that is on by default. A page must be "indexed and eligible to be shown in Google Search with a snippet", says Google's guide to generative AI search.

Bing makes the same point about its own AI answers. Its Webmaster Guidelines open with "SEO Fundamentals Still Apply to Grounding and AI Experiences": Bing and Copilot "rely on the same core crawling, indexing, and ranking foundation as traditional search."

So when vendors sell "technical GEO" (generative engine optimization) as a separate checklist, nothing in either engine's documentation calls for one. Whether AI SEO as a whole is still SEO is a broader question, argued in the AI SEO guide.

A search crawler fetches pages so they can appear in an engine's results or answers. Five of them serve the classic and AI search products named above. Checked against each vendor's documentation in September 2026:

Search crawler	What it feeds	Runs JavaScript?	Source
Googlebot (Smartphone and Desktop)	Google Search, including AI Overviews and AI Mode	Yes, with a recent version of Chrome, after a rendering queue	Googlebot documentation; How Search works; JavaScript SEO basics; Google's guide to generative AI search
Bingbot	Bing search, Copilot and Bing's grounding results	Yes, with Microsoft Edge	Bing's evergreen Bingbot announcement (2019); Bing Webmaster Guidelines
OAI-SearchBot	Search results in ChatGPT search	Not documented	OpenAI crawler overview
Claude-SearchBot	Search results in Claude	Not documented	Anthropic's crawler article
PerplexityBot	Search results in Perplexity	Not documented	Perplexity crawler documentation

Other AI crawlers answer a different question. Training crawlers (GPTBot, ClaudeBot), user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) and Google-Extended, a robots.txt token rather than a crawler, are about what a site lets AI systems use.

The AI SEO Crawlers & Agent Accessibility guide goes deeper on those, and on the robots.txt rules for each.

Where the engines differ, in this guide's reading, is capability rather than requirements, above all whether a crawler runs JavaScript, which the rendering section below takes up.

Classic and AI search share one technical floor: a page that crawlers can reach, that works, and that has content they can index. Google and Bing document it for their AI answers too. AI search added more crawlers that must clear that floor, not new rules.

Crawlability vs. Indexability: Getting Crawled, Then Getting Indexed

Crawlability is whether a crawler can fetch a page; indexability is whether the engine is allowed and willing to store it. A page can be crawlable and still not indexed.

Google's guide to how Search works describes three stages: crawling, indexing and serving results. For pages built with JavaScript, rendering sits between crawling and indexing, which the next section covers.

Six controls act on these two stages. Each does one job, and the table shows what each one doesn't do:

Control	What it controls	What it doesn't do	Source
robots.txt	Which URLs a crawler may fetch	Keep a page out of the index, or lock anyone out: its rules "are not a form of access authorization"	Google's robots.txt introduction; Request for Comments (RFC) 9309, the robots.txt standard
noindex (robots meta tag or HTTP header)	Whether the engine may store the page	Work on a page the crawler can't fetch. The rule has to be read to count	Google's noindex documentation
Canonical tag	Which of several duplicate URLs should be the main one: a "strong signal" in Google's words, weighed with others, so Google can still pick another URL	Replace a redirect. When content moves, Bing says to use a redirect, not a canonical	Google's canonicalization documentation; Bing Webmaster Guidelines
HTTP status codes	200: the page works. 301 or 308: moved for good. 302: a short-term move, "less than 2 days" per Bing. 404 or 410: removed. 5xx: a server problem	Hide an error. An error page that returns 200 is a "soft 404"	Google's HTTP status codes documentation; Bing Webmaster Guidelines
XML sitemap	Which URLs an engine should know about; an accurate lastmod tells it which changed	Force indexing	Google's sitemaps overview; Bing on sitemaps in AI search
IndexNow	Tells participating engines (Microsoft Bing, Naver, Seznam.cz, Yandex, Yep) that a URL changed	Reach Google, which isn't among the participants	IndexNow

The mix-up with the worst side effect, in this guide's reading, is using robots.txt to hide a page. Blocking the crawl also stops the crawler from ever seeing a noindex rule, so Google says the URL can still appear in results, without a description, if other pages link to it.

Crawled or Discovered, but Not Indexed

Two states in Google Search Console's Page indexing report mark the gap between the stages. "Crawled - currently not indexed" means Google fetched the page but didn't index it, and it "may or may not be indexed in the future".

"Discovered - currently not indexed" means Google found the URL but hasn't crawled it yet, typically because crawling then was expected to overload the site.

For a single URL, URL Inspection shows Google's indexed version and why it could or couldn't index the page, and a live test shows whether the page might be indexable now.

Can AI Crawlers Read JavaScript? Why Critical Content Belongs in the Server HTML

Google and Bing document that their crawlers run JavaScript. OpenAI, Anthropic and Perplexity don't document it either way, and the main public evidence, from late 2024, showed OAI-SearchBot, PerplexityBot and Anthropic's training crawler not running it.

Rendering means running a page's JavaScript to build what a visitor would actually see. A page that loads its text with JavaScript has nothing to index until something renders it.

What Google and Bing document. Google renders with a recent version of Chrome, but a page may wait in a rendering queue: "a few seconds, but it can take longer than that", per Google's JavaScript SEO basics.

For its AI features, Google says it "is able to process content within JavaScript as long as it isn't blocked."

Bing renders with Edge, yet its guidelines list "Hiding critical content behind client-side rendering" among things to avoid, because "Content that cannot be reliably rendered may not be indexed or selected for grounding results."

What the AI search crawlers don't document. OpenAI's, Anthropic's and Perplexity's crawler pages say nothing about JavaScript or rendering (checked September 25, 2026). Every claim about them is observed from outside, not documented.

That evidence is Vercel and MERJ's analysis of Vercel's own network, published December 2024. It found "none of the major AI crawlers currently render JavaScript", naming OpenAI's crawlers, Anthropic's ClaudeBot and PerplexityBot among others. Claude-SearchBot wasn't in the list.

The same analysis found that Google's Gemini uses Googlebot's infrastructure and does render. It is third-party data from one network, and nearly two years old.

A second data point is searchVIU's November 2025 roundup of 23 AI crawlers, which puts the share that can't run JavaScript at 69%. It aggregates third-party sources, Vercel's among them, rather than measuring anew.

The rule comes from Google itself: "server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript."

In this guide's reading, the practical consequence is to build for the least capable search crawler, so that nothing a page depends on exists only in what JavaScript adds later.

The ways a site can produce its pages differ exactly on this point:

• Client-side rendering: the server sends a near-empty page and the visitor's browser builds the content with JavaScript. A crawler that doesn't run JavaScript sees little or nothing.
• Server-side rendering: the server builds the full HTML for each request, so every crawler gets the content.
• Static generation: the full HTML is built ahead of time and served as ready-made files, with the same result for crawlers.
• Prerendering: the page's JavaScript is run in advance and the finished HTML saved, so crawlers get the result without running anything.
• Dynamic rendering: serving a prerendered version to crawlers and the client-side version to people. Google calls it "a workaround and not a recommended solution", and recommends server-side or static rendering instead.

The safe rule for a JavaScript site: the main text, internal links, title, canonical and robots tags go in the HTML the server sends. A crawler that can't run JavaScript sees only that HTML. Comparing a page's source with its rendered version shows what such a crawler would miss.

Mobile-First Indexing: Google Indexes the Smartphone Version

Google indexes the smartphone version of a page: it uses "the mobile version of a site's content, crawled with the smartphone agent, for indexing and ranking", which Google calls mobile-first indexing.

The consequence is content parity. Google asks for the same content, headings, titles and robots meta tags on both versions, and says a site with less content on mobile can expect "some traffic loss", since indexing comes from the mobile site.

With a different robots meta tag on mobile, Google says it "may fail to crawl and index" the page. A responsive site, one page that adapts to screen size, has parity by default: Google notes its content and metadata are the same on both.

Site Speed and Core Web Vitals: A Ranking Signal That Relevance Outweighs

Google's ranking systems use Core Web Vitals, but relevance comes first. Google's page experience documentation says "Core Web Vitals are used by our ranking systems" and "There is no single signal".

It adds that Google Search "always seeks to show the most relevant content, even if the page experience is sub-par."

Core Web Vitals are Google's three measures of how a page feels to a real visitor, with "good" thresholds set by Google:

Metric	What it measures	Good
Largest Contentful Paint (LCP)	How long the main content takes to appear	Within 2.5 seconds
Interaction to Next Paint (INP)	How quickly the page responds to clicks, taps and key presses	Under 200 milliseconds
Cumulative Layout Shift (CLS)	How much the layout jumps while it loads	Under 0.1

They're measured on field data: the experience of real Chrome users, collected in the Chrome UX Report (CrUX). A page passes when it meets all three at the 75th percentile of page loads, per web.dev's Web Vitals guide.

PageSpeed Insights and Search Console's Core Web Vitals report both show that data. web.dev goes much deeper on measuring and improving each metric.

Outdated Technical SEO Advice, and What the Documentation Says

Several pieces of still-circulating technical SEO advice are out of date or were never supported, and the engines' own documentation says otherwise:

Advice you'll still see	What the documentation says	Source
First Input Delay (FID) is a Core Web Vital	Interaction to Next Paint replaced FID as a Core Web Vital on March 12, 2024	web.dev
Googlebot reads the first 15 MB of a page	Googlebot crawls "the first 2MB of a supported file type" and the first 64 MB of a PDF, measured uncompressed. Older write-ups still cite 15 MB	Googlebot documentation
Every site needs to manage crawl budget (how many URLs an engine will crawl on a site in a given time)	Google's guide is for sites with 1 million+ pages changing weekly, 10,000+ changing daily, or many "Discovered - currently not indexed" URLs	Google's crawl budget guide
Blocking Google-Extended removes a site from Google's AI answers in Search	Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search"	Google's common crawlers list
An llms.txt file helps Google's AI features	Google Search "doesn't use" llms.txt or similar files, including for its generative AI features	Google's guide to generative AI search
Load in under 2.5 seconds and keep URL slugs to 17-40 characters to get cited by AI	Both get passed around citing a study of 5 million cited URLs, which "identifies correlations, not causation" and didn't measure page speed	Semrush study, January 2026

The likely reason stale advice persists is that checklists copy each other: a metric or a limit gets written down once and repeated long after the source page changed. A check that names its source and date can be re-verified; one that doesn't can't.

Technical SEO Checklist: What to Check, in What Order, and What a Pass Looks Like

Check what can block a page from every engine first (access, status codes, indexability), then whether crawlers can read it without JavaScript, then discovery, then mobile and speed. The order is this guide's reasoning from the documentation above: a fast page that no crawler can fetch gains nothing from its speed.

Each row gives the check, what a pass looks like, how to check it, and the documentation it comes from.

Can every search crawler reach the page?

Check	Pass looks like	How to check	Source
robots.txt responds	Returns 200, or 404 (which means no rules); never a 5xx, which makes crawlers assume everything is blocked	Request the robots.txt file directly and note its status code	RFC 9309
robots.txt allows the five search crawlers	No rule blocks pages meant to be found for Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot or PerplexityBot, and Googlebot and Bingbot can fetch the JavaScript and CSS files a page needs to render	Read the rules that apply to each crawler's name, and to all crawlers, including rules on script and style file paths	Google technical requirements; JavaScript SEO basics; Google mobile-first indexing documentation; Bing Webmaster Guidelines; OpenAI, Anthropic and Perplexity crawler docs
Each search crawler actually gets the page	Requests from each of the five crawlers receive 200	Server logs filtered to verified requests: Googlebot by a reverse lookup of its IP address to a Google domain, or by Google's IP ranges; Bingbot with Bing's Verify Bingbot tool or its Bingbot IP list; the AI search crawlers by the IP lists OpenAI, Perplexity and Anthropic publish	Googlebot documentation; Bing's Verify Bingbot tool and Bingbot IP list; OpenAI, Anthropic and Perplexity crawler docs
Status codes are honest	Indexable pages return 200; permanent moves 301 or 308; 302 only for short-term moves; removed pages 404 or 410; no soft 404s; no 5xx	An audit crawler's status report (Screaming Frog or Sitebulb, for example); Search Console's Page indexing report	Google HTTP status codes documentation; Bing Webmaster Guidelines

Is the page allowed into the index?

Check	Pass looks like	How to check	Source
No accidental noindex	No noindex in a robots meta tag or HTTP header on pages meant to be indexed; pages meant to stay out use noindex, not only robots.txt	Read the page's HTML head and response headers; URL Inspection	Google noindex documentation
Canonicals agree	Each canonical points to a URL that returns 200 and is indexable, and matches the sitemap entry and internal links	Compare the canonical tag with the sitemap and link targets; URL Inspection's Google-selected canonical	Google canonicalization documentation
Google reports the page indexed	URL Inspection shows it indexed; the Page indexing report has no unexplained exclusions	Search Console URL Inspection and Page indexing report	URL Inspection help; Page indexing report help
Search generative AI setting is the intended one	Search Console's "Search generative AI" setting matches what the site intends; inclusion, the default, is needed to appear in Google's AI features	Search Console settings	Google's guide to generative AI search; Search generative AI control help

Can crawlers read it without running JavaScript?

Check	Pass looks like	How to check	Source
Critical content is in the server HTML	Before any JavaScript runs, the HTML contains the main text, title, internal links, canonical and robots tags	View-source or a raw fetch of the page, compared with the rendered page in a browser or URL Inspection	Google JavaScript SEO basics; Bing Webmaster Guidelines
Links are crawlable	Internal links are <a> elements with an href holding a real URL, not script-only click handlers	Inspect link markup in the server HTML	Google's link best practices
Page size fits Googlebot's limit	Uncompressed HTML under 2 MB, with the main content early	Size of the uncompressed HTML document	Googlebot documentation

Can engines find the page and hear about changes?

Check	Pass looks like	How to check	Source
Sitemap is clean	Lists only canonical, indexable URLs that return 200; lastmod changes only when content changes; submitted to Google and Bing	Search Console's Sitemaps report; the Sitemaps section of Bing Webmaster Tools	Google sitemaps overview; Bing on sitemaps in AI search
IndexNow engines hear about changes	New, changed and deleted URLs are sent via IndexNow (Google doesn't take part)	The site's or content system's IndexNow submission records; Bing Webmaster Tools	IndexNow; Bing Webmaster Guidelines

Does the page hold up on mobile, security and speed?

Check	Pass looks like	How to check	Source
Smartphone version is complete	Same main content, headings, titles and robots meta tags on mobile as on desktop	Compare the page as served to a smartphone with the desktop version	Google mobile-first indexing documentation
Pages are served over HTTPS	Every page loads over HTTPS	Search Console's HTTPS report	Google page experience documentation
Core Web Vitals pass on field data	LCP, INP and CLS meet the "good" thresholds at the 75th percentile	PageSpeed Insights or Search Console's Core Web Vitals report, both built on CrUX data	Google Core Web Vitals documentation; web.dev

Turning these checks into a repeatable audit on a real site, with its own tools and schedule, is separate work.

Sources

Sourced claims:

• Google Search technical requirements, Google Search Central. Cited for the three minimum requirements (Googlebot not blocked, HTTP 200, indexable content) and for meeting them not guaranteeing indexing.
• In-depth guide to how Google Search works, Google Search Central. Cited for the crawling, indexing and serving stages, rendering with a recent version of Chrome, and Google not guaranteeing it will crawl, index or serve a page.
• Optimizing your website for generative AI features on Google Search, Google Search Central. Cited for AI features needing an indexed page eligible for a snippet plus the Search Console inclusion setting, AI Overviews and AI Mode, JavaScript processed "as long as it isn't blocked", and llms.txt going unused.
• Search generative AI control, Search Console Help. Cited for inclusion being the default setting.
• Bing Webmaster Guidelines, Microsoft Bing (read September 25, 2026). Cited for AI answers sharing the classic crawling and indexing foundation, not hiding critical content behind client-side rendering or blocking Bingbot, 302s for moves of "less than 2 days", redirects over canonicals, and IndexNow.
• Googlebot, Google Search Central. Cited for Googlebot Smartphone and Desktop, the 2 MB and 64 MB limits on uncompressed data, and verifying Googlebot by reverse lookup or IP ranges.
• The new evergreen Bingbot, Bing Webmaster Blog, October 9, 2019. Cited for Bingbot rendering JavaScript with Microsoft Edge.
• Verify Bingbot tool and Bingbot IP list, Microsoft Bing (checked September 25, 2026). A tool that checks whether an IP address belongs to Bingbot, and a published list of Bingbot's IP ranges.
• Overview of OpenAI Crawlers, OpenAI. Cited for OAI-SearchBot feeding ChatGPT search, GPTBot and ChatGPT-User as separate crawlers, published IP ranges, and the absence of any statement about JavaScript.
• Does Anthropic crawl data from the web?, Anthropic, and its published IP list. Cited for Claude-SearchBot, ClaudeBot and Claude-User, crawler IP addresses, and the absence of any statement about JavaScript.
• Perplexity Crawlers, Perplexity. Cited for PerplexityBot surfacing sites in Perplexity search, Perplexity-User, published IP ranges, and the absence of any statement about JavaScript.
• Introduction to robots.txt, Google Search Central. Cited for robots.txt not being the way to keep a page out of the index, and blocked URLs still appearing in results without a description.
• RFC 9309: Robots Exclusion Protocol, IETF, September 2022. Cited for robots.txt rules not being access authorization, an unavailable (4xx) robots.txt allowing crawling, and an unreachable (5xx) one meaning complete disallow.
• Block Search indexing with noindex, Google Search Central. Cited for noindex only working when the page isn't blocked by robots.txt and is accessible to the crawler.
• What is URL canonicalization, Google Search Central. Cited for rel="canonical" being a strong signal, sitemap inclusion a weak one, and Google weighing other canonicalization signals too.
• HTTP status codes and network errors, Google. Cited for how Google treats 200, 301, 308, 302, 404, 410 and 5xx responses, and soft 404s.
• Learn about sitemaps, Google Search Central. Cited for sitemaps helping discovery.
• Keeping Content Discoverable with Sitemaps in AI Powered Search, Bing Webmaster Blog, July 31, 2025. Cited for accurate lastmod values and submitting sitemaps in Bing Webmaster Tools.
• IndexNow. Cited for the participating engines (Microsoft Bing, Naver, Seznam.cz, Yandex, Yep); Google isn't listed.
• Page indexing report, Search Console Help. Cited for the meanings of "Crawled - currently not indexed" and "Discovered - currently not indexed".
• URL Inspection tool, Search Console Help. Cited for the indexed version, live test and Google-selected canonical.
• Understand JavaScript SEO basics, Google Search Central. Cited for the rendering queue, Google not rendering JavaScript from blocked files, and the sentence recommending server-side or pre-rendering because "not all bots can run JavaScript".
• Dynamic rendering as a workaround, Google Search Central. Cited for dynamic rendering being "a workaround and not a recommended solution".
• The rise of the AI crawler, Vercel and MERJ, December 17, 2024. Third-party data from Vercel's network, cited for no major AI crawler rendering JavaScript at the time (OAI-SearchBot, ClaudeBot and PerplexityBot among those named) and Gemini using Googlebot's infrastructure.
• AI crawlers and JavaScript rendering, searchVIU, November 2025. An aggregation of third-party sources, cited for the 69% figure across 23 AI crawlers.
• Mobile site and mobile-first indexing best practices, Google Search Central. Cited for indexing from the smartphone version, content and metadata parity, robots meta tags, letting Google crawl page resources, and traffic loss from thinner mobile pages.
• Understanding page experience in Google Search results, Google Search Central (updated September 22, 2026). Cited for Core Web Vitals being used in ranking, there being no single page experience signal, relevance winning over sub-par page experience, and HTTPS as part of page experience.
• Understanding Core Web Vitals and Google search results, Google Search Central. Cited for the LCP, INP and CLS thresholds.
• Web Vitals, web.dev. Cited for the 75th percentile assessment and the Chrome UX Report powering PageSpeed Insights and Search Console's Core Web Vitals report.
• PageSpeed Insights, Google. Named as a tool that shows field data.
• Interaction to Next Paint is officially a Core Web Vital, web.dev, March 12, 2024. Cited for INP replacing FID.
• Optimize your crawl budget, Google. Cited for which sites need to manage crawl budget.
• Google's common crawlers, Google. Cited for Google-Extended not affecting inclusion or ranking in Google Search.
• How Do Technical SEO Factors Impact AI Search?, Semrush, January 5, 2026. Cited only for its own caveat that it "identifies correlations, not causation" and for what it measured, which didn't include page speed.
• Link best practices for Google, Google Search Central. Cited for Google generally only crawling links that are <a> elements with an href attribute.

Argued, not sourced (this guide's reasoning from the sources above):

• The core claim's synthesis: that AI search added more crawlers rather than new rules, drawn from Google's and Bing's documentation and the AI search vendors' crawler pages.
• That the engines differ mainly in capability, especially running JavaScript.
• Building for the least capable search crawler, as this guide's extension of Google's "not all bots can run JavaScript" sentence to the AI search crawlers.
• The checklist order: access and status, indexability, reading without JavaScript, discovery, then mobile and speed.
• That server logs filtered to verified IP addresses show what a real crawler receives, while a request that only fakes a crawler's user agent can't (supported by the IP lists and verification methods the vendors publish).
• That using robots.txt to hide a page is the crawl-versus-index mix-up with the worst side effect.
• Why stale technical advice persists.