Engineering Reference · Search Indexation

Technical SEO Glossary & Directive Reference

Production definitions, HTTP/robots directive behaviors, and concrete engineering thresholds for teams managing Google, Bing, and AI crawler indexation.

Search Directive & HTTP Status Matrix

Mechanism Where Applied Crawler Behavior Common Pitfall
Disallow: /path/ robots.txt Prevents crawling URL bytes Blocking pages that are already indexed traps them with URL-only snippets
X-Robots-Tag: noindex, follow HTTP Response Header Drops URL from SERP while passing link equity Blocked concurrently in robots.txt so bots never read the header
rel="canonical" HTML <head> / HTTP Link Hint consolidating duplicate variants to primary URL Canonicalizing ?page=2..N back to page 1 hides deeper items
IndexNow Webhook API / CMS Publish Hook Instant ping of newly published or updated URLs Pinging non-200 or noindex URLs wastes crawler trust signals
301 Moved Permanently Nginx / CDN Edge Permanently transfers equity and indexation target Leaving 3+ hop redirect chains or soft-302 on permanent moves
410 Gone Origin / Edge HTTP Signals intentional permanent removal faster than 404 Returning HTTP 200 "Not found" HTML wrapper (Soft-404)

Crawling & Logs

Crawl Budget

The combined limit of crawl capacity (host rate limit) and crawl demand (URL popularity and staleness) that Googlebot and Bingbot allocate to a domain per day.

Engineering rule: Keep valid HTTP 200 money-page share above 85% of total verified bot requests.

Crawling & Logs

Verified Bot Log Analysis

Inspecting raw Nginx, Apache, or Cloudflare access logs validated via reverse DNS to measure actual search engine hit distribution rather than simulated crawls.

Engineering rule: Filter fake User-Agents with rDNS / official IP JSON before measuring waste.

Crawling & Logs

IndexNow Ping Protocol

Lightweight HTTP webhook standard supported by Bing and search indexers that notifies crawlers immediately when URLs are created, updated, or deleted.

Engineering rule: Fire IndexNow asynchronously on CMS publish hooks, stock updates, and 301/410 changes.

JS & Rendering

Two-Wave JS Rendering (WRS)

How Google handles client-side SPAs: Wave 1 parses initial HTML immediately; Wave 2 executes JavaScript in headless Chromium hours or days later.

Engineering rule: Ship critical H1, copy, internal anchor links, canonical, and JSON-LD in raw curl HTML.

JS & Rendering

SSR / SSG / ISR Parity

Server-Side Rendering, Static Site Generation, and Incremental Static Regeneration strategies that eliminate empty loading shells for search crawlers.

Engineering rule: Audit raw curl responses vs rendered DOM to guarantee 100% content and link parity.

JS & Rendering

Core Web Vitals (LCP, INP, CLS)

Field performance thresholds in CrUX telemetry measuring loading speed (LCP <= 2.5s), main-thread responsiveness (INP <= 200ms), and visual stability (CLS <= 0.1).

Engineering rule: Optimize 75th-percentile real mobile user CrUX metrics rather than chasing lab-only 100 scores.

URLs & Structure

Faceted Index Bloat

Exponential explosion of low-value parametric filter and sort URLs competing for crawl budget and cannibalizing core category keywords.

Engineering rule: Promote validated high-demand filter combinations to static SEO URLs; noindex multi-attribute sorts.

URLs & Structure

Reciprocal Hreflang Graph

Bidirectional alternate annotations mapping equivalent regional/language pages across /en/, /ru/, /es/, /de/ with mandatory self-referencing and x-default tags.

Engineering rule: Every cluster page must confirm all peer links bilaterally on 200 OK canonical URLs.

URLs & Structure

Programmatic Inventory Gates

Automated data thresholds enforcing minimum active records and unique attribute density before permitting template pages to emit index, follow.

Engineering rule: Require >= 3 active listings and > 45% unique spec data before opening a template URL.

URLs & Structure

Contextual Hub-Spoke Silos

Intentional internal link topology routing PageRank from high-authority hubs and articles toward commercial landing and solution pages with <= 3 click depth.

Engineering rule: Keep every priority revenue page within 3 clicks of the homepage via clean anchor links.

Schema & Authority

Connected JSON-LD @graph

Linking WebSite, Organization, Service, Offer, TechArticle, Person, FAQPage, and BreadcrumbList via persistent @id URI references inside one @graph payload.

Engineering rule: Cross-link entities cleanly using consistent @id anchors rather than isolated fragmented snippets.

Schema & Authority

GEO & /llms.txt Indexing

Generative Engine Optimization paired with a curated Markdown /llms.txt endpoint helping AI retrieval crawlers resolve canonical URLs, service scopes, and pricing directly.

Engineering rule: Pair server-rendered comparison tables and JSON-LD with an up-to-date /llms.txt index.

Schema & Authority

YMYL & Verifiable E-E-A-T

Experience, Expertise, Authoritativeness, and Trustworthiness signals crucial in financial, medical, and legal search categories evaluated algorithmically and by raters.

Engineering rule: Attach named author profiles, reviewedBy entities, regulatory citations, and legal disclosures.

Schema & Authority

NAP Consistency & LocalBusiness

Exact alignment of Name, Address, Phone, coordinates, and branch URLs between site HTML/Schema and Google Business Profile / Apple Business Connect listings.

Engineering rule: Route every map listing to its dedicated branch page rather than the root homepage.

Schema & Authority

AEO & Answer-First Grounding

Structuring page sections with direct 40–60 word answers, comparison HTML tables, and verifiable pricing/entity facts so AI Overviews, ChatGPT Search, and Perplexity extract accurate citations.

Engineering rule: Place a clear factual summary directly under each H2 heading and keep figures synchronized with JSON-LD.

Crawling & Logs

Content Pruning & Helpful Signals

Auditing and consolidating thin, zero-traffic, or overlapping URLs (301/410) so site-wide helpfulness ratios recover during Google Core Updates.

Engineering rule: Merge cannibalizing posts into one canonical hub or return HTTP 410 for obsolete zero-backlink pages.

JS & Rendering

Islands Architecture & Selective Hydration

Delivering zero-JS static HTML for editorial and SEO blocks while deferring client hydration strictly to interactive calculators, filters, or forms (Astro/Nuxt/Next).

Engineering rule: Keep above-the-fold copy and JSON-LD in raw SSR HTML without waiting for client-side hydration.

URLs & Structure

BOFU Commercial Intent Clusters

Dedicated bottom-of-funnel landing pages targeting high-intent queries ("vs competitor", "pricing", "integration", "OEM supplier") built to convert organic search into pipeline.

Engineering rule: Separate commercial BOFU pages from educational blog guides via clear intent mapping and internal anchor CTAs.

URLs & Structure

Hreflang x-default & Market Subfolders

Bidirectional regional mapping pairing language-country ISO codes (en-GB, de-CH, es-MX) with an x-default fallback across subfolder, subdomain, or ccTLD architectures.

Engineering rule: Validate 100% reciprocal return tags across HTML head and XML sitemaps with zero non-200 targets.

Schema & Authority

Entity Co-Citation & Digital PR

Earning unlinked brand mentions alongside topical entities and authoritative editorial backlinks that strengthen both Google PageRank and LLM vector associations.

Engineering rule: Align brand anchors, founder profiles, and external PR mentions with Organization sameAs links.