Engineering Reference · Search Indexation
Technical SEO Glossary & Directive Reference
Production definitions, HTTP/robots directive behaviors, and concrete engineering thresholds for teams managing Google, Bing, and AI crawler indexation.
Search Directive & HTTP Status Matrix
| Mechanism | Where Applied | Crawler Behavior | Common Pitfall |
|---|---|---|---|
| Disallow: /path/ | robots.txt | Prevents crawling URL bytes | Blocking pages that are already indexed traps them with URL-only snippets |
| X-Robots-Tag: noindex, follow | HTTP Response Header | Drops URL from SERP while passing link equity | Blocked concurrently in robots.txt so bots never read the header |
| rel="canonical" | HTML <head> / HTTP Link | Hint consolidating duplicate variants to primary URL | Canonicalizing ?page=2..N back to page 1 hides deeper items |
| IndexNow Webhook | API / CMS Publish Hook | Instant ping of newly published or updated URLs | Pinging non-200 or noindex URLs wastes crawler trust signals |
| 301 Moved Permanently | Nginx / CDN Edge | Permanently transfers equity and indexation target | Leaving 3+ hop redirect chains or soft-302 on permanent moves |
| 410 Gone | Origin / Edge HTTP | Signals intentional permanent removal faster than 404 | Returning HTTP 200 "Not found" HTML wrapper (Soft-404) |
Crawling & Logs
Crawl Budget
The combined limit of crawl capacity (host rate limit) and crawl demand (URL popularity and staleness) that Googlebot and Bingbot allocate to a domain per day.
Engineering rule: Keep valid HTTP 200 money-page share above 85% of total verified bot requests.
Crawling & Logs
Verified Bot Log Analysis
Inspecting raw Nginx, Apache, or Cloudflare access logs validated via reverse DNS to measure actual search engine hit distribution rather than simulated crawls.
Engineering rule: Filter fake User-Agents with rDNS / official IP JSON before measuring waste.
Crawling & Logs
IndexNow Ping Protocol
Lightweight HTTP webhook standard supported by Bing and search indexers that notifies crawlers immediately when URLs are created, updated, or deleted.
Engineering rule: Fire IndexNow asynchronously on CMS publish hooks, stock updates, and 301/410 changes.
JS & Rendering
Two-Wave JS Rendering (WRS)
How Google handles client-side SPAs: Wave 1 parses initial HTML immediately; Wave 2 executes JavaScript in headless Chromium hours or days later.
Engineering rule: Ship critical H1, copy, internal anchor links, canonical, and JSON-LD in raw curl HTML.
JS & Rendering
SSR / SSG / ISR Parity
Server-Side Rendering, Static Site Generation, and Incremental Static Regeneration strategies that eliminate empty loading shells for search crawlers.
Engineering rule: Audit raw curl responses vs rendered DOM to guarantee 100% content and link parity.
JS & Rendering
Core Web Vitals (LCP, INP, CLS)
Field performance thresholds in CrUX telemetry measuring loading speed (LCP <= 2.5s), main-thread responsiveness (INP <= 200ms), and visual stability (CLS <= 0.1).
Engineering rule: Optimize 75th-percentile real mobile user CrUX metrics rather than chasing lab-only 100 scores.
URLs & Structure
Faceted Index Bloat
Exponential explosion of low-value parametric filter and sort URLs competing for crawl budget and cannibalizing core category keywords.
Engineering rule: Promote validated high-demand filter combinations to static SEO URLs; noindex multi-attribute sorts.
URLs & Structure
Reciprocal Hreflang Graph
Bidirectional alternate annotations mapping equivalent regional/language pages across /en/, /ru/, /es/, /de/ with mandatory self-referencing and x-default tags.
Engineering rule: Every cluster page must confirm all peer links bilaterally on 200 OK canonical URLs.
URLs & Structure
Programmatic Inventory Gates
Automated data thresholds enforcing minimum active records and unique attribute density before permitting template pages to emit index, follow.
Engineering rule: Require >= 3 active listings and > 45% unique spec data before opening a template URL.
URLs & Structure
Contextual Hub-Spoke Silos
Intentional internal link topology routing PageRank from high-authority hubs and articles toward commercial landing and solution pages with <= 3 click depth.
Engineering rule: Keep every priority revenue page within 3 clicks of the homepage via clean anchor links.
Schema & Authority
Connected JSON-LD @graph
Linking WebSite, Organization, Service, Offer, TechArticle, Person, FAQPage, and BreadcrumbList via persistent @id URI references inside one @graph payload.
Engineering rule: Cross-link entities cleanly using consistent @id anchors rather than isolated fragmented snippets.
Schema & Authority
GEO & /llms.txt Indexing
Generative Engine Optimization paired with a curated Markdown /llms.txt endpoint helping AI retrieval crawlers resolve canonical URLs, service scopes, and pricing directly.
Engineering rule: Pair server-rendered comparison tables and JSON-LD with an up-to-date /llms.txt index.
Schema & Authority
YMYL & Verifiable E-E-A-T
Experience, Expertise, Authoritativeness, and Trustworthiness signals crucial in financial, medical, and legal search categories evaluated algorithmically and by raters.
Engineering rule: Attach named author profiles, reviewedBy entities, regulatory citations, and legal disclosures.
Schema & Authority
NAP Consistency & LocalBusiness
Exact alignment of Name, Address, Phone, coordinates, and branch URLs between site HTML/Schema and Google Business Profile / Apple Business Connect listings.
Engineering rule: Route every map listing to its dedicated branch page rather than the root homepage.
Schema & Authority
AEO & Answer-First Grounding
Structuring page sections with direct 40–60 word answers, comparison HTML tables, and verifiable pricing/entity facts so AI Overviews, ChatGPT Search, and Perplexity extract accurate citations.
Engineering rule: Place a clear factual summary directly under each H2 heading and keep figures synchronized with JSON-LD.
Crawling & Logs
Content Pruning & Helpful Signals
Auditing and consolidating thin, zero-traffic, or overlapping URLs (301/410) so site-wide helpfulness ratios recover during Google Core Updates.
Engineering rule: Merge cannibalizing posts into one canonical hub or return HTTP 410 for obsolete zero-backlink pages.
JS & Rendering
Islands Architecture & Selective Hydration
Delivering zero-JS static HTML for editorial and SEO blocks while deferring client hydration strictly to interactive calculators, filters, or forms (Astro/Nuxt/Next).
Engineering rule: Keep above-the-fold copy and JSON-LD in raw SSR HTML without waiting for client-side hydration.
URLs & Structure
BOFU Commercial Intent Clusters
Dedicated bottom-of-funnel landing pages targeting high-intent queries ("vs competitor", "pricing", "integration", "OEM supplier") built to convert organic search into pipeline.
Engineering rule: Separate commercial BOFU pages from educational blog guides via clear intent mapping and internal anchor CTAs.
URLs & Structure
Hreflang x-default & Market Subfolders
Bidirectional regional mapping pairing language-country ISO codes (en-GB, de-CH, es-MX) with an x-default fallback across subfolder, subdomain, or ccTLD architectures.
Engineering rule: Validate 100% reciprocal return tags across HTML head and XML sitemaps with zero non-200 targets.
Schema & Authority
Entity Co-Citation & Digital PR
Earning unlinked brand mentions alongside topical entities and authoritative editorial backlinks that strengthen both Google PageRank and LLM vector associations.
Engineering rule: Align brand anchors, founder profiles, and external PR mentions with Organization sameAs links.