Indexing is the process where Google stores, organises, and catalogues a web page's content in its database so it can be retrieved and ranked when someone searches for a relevant query. Google's index contains hundreds of billions of pages and exceeds 100 million gigabytes in size. A page must be indexed to appear in search results — crawling alone is not enough.
If you publish 30 articles a month but only 20 get indexed, you are losing 33% of your content investment to pages that cannot earn a single organic click. Indexing is the most fundamental requirement in SEO — and one of the most overlooked when teams focus entirely on content and links.
What is indexing in SEO?
Indexing is the final step in a three-stage process Google uses to make web content searchable:
- Discovery — Google's crawler (Googlebot) finds the URL through links, sitemaps, or direct submission.
- Crawling — Googlebot fetches the page's content, reads its HTML, and processes linked resources (CSS, JavaScript).
- Indexing — Google analyses the content, determines its topic and quality, and stores it in the search index alongside ranking signals.
Crawling and indexing are often mentioned together but are distinct operations. Google crawls far more pages than it indexes. A page can be crawled repeatedly and never indexed if Google determines it lacks sufficient quality or uniqueness to be worth storing.
Google's index contains hundreds of billions of web pages and exceeds 100 million gigabytes in size. Despite this scale, Google is selective. Not every crawled page earns a place in the index — pages must demonstrate quality, uniqueness, and relevance to pass Google's indexing evaluation.
Why indexing matters for SEO
Four reasons indexing is the foundation of every other SEO effort:
- Ranking prerequisite. A page that is not indexed cannot appear in search results, regardless of its backlink profile, content quality, or technical setup. Indexing is the gate everything else passes through.
- Not automatic. Google's indexing is selective. Publishing a page does not guarantee it will be indexed. Thin content, duplicate content, and missing internal links are common causes of pages being crawled but not indexed.
- Speed affects competitiveness. For trending topics, news, and product launches, a page indexed within hours has a significant advantage over one that takes two weeks. Indexing speed is influenced by site authority, crawl frequency, and internal link structure.
- Index quality affects site-wide rankings. Too many thin or low-quality pages in the index drags down how Google evaluates the site overall. Maintaining a high ratio of quality indexed pages to total indexed pages is an underrated ranking factor.
How the indexing process works
Once Googlebot fetches a page, indexing involves four stages of processing.
1. Content processing
Google analyses the page's text, heading tags, links, schema markup, and metadata. It identifies the page's topic, primary keywords, language, and entity relationships (what the page is about and how it relates to other known entities in the knowledge graph).
2. Quality evaluation
Not all crawled pages are indexed. Google assesses each page for uniqueness, usefulness, and substance. Pages that are substantially similar to existing indexed content, or that lack depth, are frequently classified as "Crawled — currently not indexed" and excluded.
3. Index storage
Qualifying pages are stored with a package of signals: page text, internal and external links, structured data, freshness indicators, and quality scores. These signals determine how the page will compete for rankings when a relevant query is submitted.
4. Rendering (JavaScript sites)
For pages that rely on JavaScript to render content, Googlebot must execute the JavaScript before it can read the content. This two-wave indexing process — fetch HTML first, render JavaScript later — means JavaScript-heavy pages can experience indexing delays of weeks. Rendering the HTML on the server eliminates this delay.
Indexing status types — what each means
Google Search Console's Pages report shows six indexing statuses:
| Status | Meaning | What to do |
|---|---|---|
| Indexed | Page is in the index and eligible to rank | No action — monitor for traffic |
| Crawled — not indexed | Crawled but excluded, usually thin content | Improve content quality and uniqueness |
| Discovered — not crawled | URL known but not yet visited | Add internal links; submit via URL Inspection |
| Excluded by noindex | Intentional noindex directive | Confirm directive is intentional |
| Duplicate — not canonical | Another URL selected as the canonical version | Verify canonical tag is correct |
| Blocked by robots.txt | Cannot be crawled | Check robots.txt if page should be indexed |
Real indexing examples and fixes
1. Restaurant seasonal menu — "Discovered — not yet crawled"
A restaurant's seasonal menu page sat in "Discovered — currently not indexed" for four weeks. The page existed in the sitemap but had no internal links from the main navigation or any other page. After adding a link from the homepage and the main menu page, Googlebot crawled and indexed the page within three days.
2. Marketing agency blog — "Crawled — currently not indexed"
A marketing agency found 40% of their blog posts in "Crawled — currently not indexed" status. The posts averaged 300–400 words with no original data. After rewriting each post to 800–1,200 words with specific examples and original statistics, 85% were indexed within three weeks of the rewrites going live.
3. Home services company — maintaining high indexing rates
A home services company publishing 30 articles per month maintained over 90% indexing rates by following three practices: submitting articles to their XML sitemap immediately, linking each new article from at least two existing indexed pages, and maintaining content standards above 800 words with original research.
Indexing vs. crawling — what is the difference?
These two processes are closely related but distinct, with different controls and failure modes.
| Dimension | Crawling | Indexing |
|---|---|---|
| What happens | Googlebot fetches and reads the page | Google stores the page in its searchable database |
| Analogy | Librarian picks up a book | Librarian catalogues and shelves the book |
| Can fail independently | Yes — robots.txt blocks crawling | Yes — noindex or thin content blocks indexing |
| You control with | robots.txt, sitemaps | noindex tags, content quality |
| Check status in | GSC Crawl Stats report | GSC Pages (Coverage) report |
5 best practices to ensure consistent indexing
- Submit and maintain your XML sitemap. Add new URLs to your sitemap immediately after publishing. Submit the sitemap in Google Search Console and check for errors after each update. Google prioritises sitemap URLs for crawling.
- Build internal links to every new page. Aim for 3–5 internal links from existing indexed pages to every new URL. Pages with no internal links are often left in "Discovered — not yet crawled" indefinitely.
- Improve thin content rather than publishing more. A site with 100 deeply useful indexed pages will earn more traffic than one with 400 thin pages, half of which are never indexed.
- Use the URL Inspection tool for priority pages. After publishing a critical landing page or time-sensitive article, use GSC's URL Inspection tool to request indexing. This does not guarantee immediate indexing but puts the URL in Google's priority crawl queue.
- Monitor indexing rates monthly. Track the number of Valid pages in GSC over time. A declining trend without intentional noindexing is an early warning of a configuration issue or content quality problem.
Pages that render all content via client-side JavaScript present a significant indexing risk. Googlebot's two-wave rendering means JavaScript content may not be processed for weeks after the initial crawl. If your key content — headings, body text, structured data — only appears after JavaScript executes, you are betting on Googlebot's rendering queue. Use server-side rendering or static HTML for any content critical to rankings.
Common indexing mistakes to avoid
- Accidental noindex tags — a plugin or template update can add noindex to the wrong pages. Check GSC after every major CMS change.
- Blocking CSS and JavaScript in robots.txt — if Googlebot cannot fetch your stylesheets or JS files, rendering fails and indexing quality suffers.
- Orphan pages with no internal links — pages listed only in a sitemap, with no in-site links, are often deprioritised by Googlebot.
- Duplicate content without canonical tags — www vs non-www, trailing slash vs no trailing slash, and HTTP vs HTTPS create duplicate URL variants that compete with each other and confuse Google's canonical selection.
- Assuming submission = indexing — requesting indexing via GSC queues the page for crawling but does not guarantee Google will index it. Quality is the deciding factor.
Frequently asked questions
Established sites typically see new pages indexed within 2–7 days. New sites with low authority may wait 2–4 weeks. Submitting pages via Google Search Console and ensuring internal links from indexed pages accelerates the process.
Common causes include thin or duplicate content, accidental noindex tags, robots.txt blocking, no internal or external links pointing to the page, or insufficient unique value compared to existing indexed content on the topic.
Two methods: search site:yoursite.com/page-url in Google to see if it appears, or use the URL Inspection tool in Google Search Console for a definitive status with the specific reason for any exclusion.
Yes. Add a noindex meta robots tag for permanent removal, use GSC's Removals tool for temporary removal, or return a 404 or 410 status code. Noindex is the most reliable method for permanent de-indexing.
No. Google crawls far more pages than it indexes. Crawling is discovery and reading; indexing is the decision to store and make a page eligible for rankings. Low-quality, thin, or duplicate pages are often crawled but not indexed.
Related glossary terms
Sources
