What is crawling?

Crawling is the process of requesting and fetching resources from the web with an automated program. For Google Search, Googlebot fetches pages and other resources that Google has discovered. Discovery, fetching and inclusion in search are different observations.

Google’s explanation of Search separates crawling, indexing and serving results. Not every discovered page is fetched, and not every fetched page is indexed. A useful investigation establishes which stage has evidence of a problem.

Why crawling matters

A crawler needs access to obtain the content of a page. A login requirement, unavailable server or disallow rule can prevent that fetch. If a public service page should be discoverable, its actual response and access settings are worth checking before rewriting its text.

Access restrictions have different purposes. A robots.txt rule controls permitted fetching; noindex concerns inclusion in search. Google’s noindex guidance explains that a crawler must be able to fetch the page to see the directive. A blocked address can still appear in search using information found elsewhere, so blocking crawling is not a reliable removal method.

How discovery and fetching fit together

Discovery

Google can learn about an address from links, a sitemap or a previous visit. Sitemap guidance describes a discovery aid, not a guarantee that every listed URL will be crawled or indexed.

Fetching

A request produces a response, which may contain the page, a redirect or an error. Inspect that response before assuming a crawler saw the expected content. For JavaScript-dependent pages, also check whether the important text and links appear in the rendered version.

Google’s link guidance recommends anchors with usable href addresses and descriptive labels. A navigation control that works only through a script event is not an equally reliable discovery route. Add relevant connections, rather than an arbitrary number of links from unrelated articles.

Crawl scheduling

Google decides which discovered addresses to request and when. Its Search guidance describes responses and server access as part of this process. Publishing regularly does not establish a fixed crawl schedule, and a business cannot purchase a guaranteed crawl or indexing decision from Google.

What a crawler can request

A crawl can fetch a newly discovered page, revisit a known address or request resources used by a page. The relevant question is what was requested and what response it received, rather than whether a website has completed an assumed “full” or “deep” crawl mode.

The Crawl Stats report counts requests, including repeated requests and some page resources. Its totals are not a count of unique indexed pages, and its example URLs are not an exhaustive request log.

An illustrative diagnostic example

A business publishes a new service page, but the page has no useful incoming link. Add it to relevant navigation or an existing explanation, and check the sitemap if one is used. Then inspect the address and monitor the available evidence. This is an example of a discovery repair, not a customer story or a promise of a crawl within a stated number of hours.

If the page is already recorded as fetched, a missing search result needs a different investigation. More links do not fix every indexing, canonical or content issue.

Crawling and indexing need different evidence

Choose evidence for the question you are asking
EvidenceUseful questionLimit
Server request recordsWhat was requested and returned?A request record does not by itself establish indexing.
Crawl StatsIs Google reporting request or serving problems?Aggregate requests and samples do not establish every URL’s indexing state.
URL InspectionWhat does Google report for this address?Stored index information and a live test answer different questions.
Page indexing reportWhich reported groups are indexed or not indexed?A not-indexed reason is not automatically a diagnosis of poor content.

URL Inspection guidance distinguishes stored information from a live test. Use the appropriate result before deciding that a recent repair failed.

A practical crawl-access check

  1. Identify the exact public address and intended replacement if it redirects.
  2. Check whether relevant navigation or a sitemap exposes that address.
  3. Inspect the actual response and access restrictions. Confirm that the expected content is available without an unintended login or challenge.
  4. For a JavaScript-dependent page, compare the initial response with the rendered content.
  5. Use available crawl and URL-level evidence to separate discovery, fetch and indexing questions.
  6. Record any repair and the observation it addresses. Review the affected pages after changing a template or access rule.

Questions about crawl requests and server access

How long does Google take to crawl a new page?

There is no guaranteed deadline for your page. Google’s recrawl guidance says a request does not guarantee immediate inclusion or inclusion at all. Keep the request and observed status separate from a promised delivery date.

Can I force Google to crawl my site?

No. An authorized Search Console user can request a crawl of a managed URL. Repeated requests for the same URL do not accelerate crawling. Fix a confirmed access problem rather than using resubmission as a substitute for diagnosis.

What is the difference between crawling and rendering?

Crawling fetches the response. Rendering processes the page, including JavaScript, to produce the rendered content. Google’s JavaScript guidance describes both queues. Rendering is not a universal second visit with a fixed delay, and server-rendered content does not guarantee indexing.

Does crawling affect my server?

Crawler requests use server resources. If load is a problem, inspect actual traffic with your hosting team. Google’s current crawl-rate guidance describes temporary emergency measures and their search consequences. The Search Console rate-limiter deprecation announcement supersedes the old advice to adjust its crawl-rate control. Do not deliberately serve persistent errors as a routine SEO tactic.

Sources and editorial scope

The primary Google references linked above support the distinctions between discovery, fetching, rendering and indexing. Related entries explain indexing, index coverage and robots.txt in more detail. References were checked on October 12, 2026.

This revision was prepared with AI assistance for theStacc. It reports no customer results, live crawler experiment, indexing guarantee or named expert approval.