SEO

Website indexing

Website indexing is how a search engine adds a page to its database so it can appear in search results. To index a website, the page has to be reachable for the crawler, free of noindex, listed in the sitemap and linked from other pages. Being indexed does not guarantee rankings, but without it a page gets no search traffic at all.

How indexing works: crawling, indexing, ranking

Search engines handle a page in three steps, and each one can fail on its own.

  1. Crawling. The crawler finds the URL through a link, a sitemap or a manual request and downloads the page.
  2. Indexing. The search engine renders the page, works out what it is about, picks the canonical version among duplicates and decides whether the page is worth storing.
  3. Ranking. Only indexed pages compete for positions when someone searches.

Google does not promise to index every page it crawls. If a page looks like a duplicate, has little content of its own or is hard to reach, Google may crawl it and still leave it out of the index.

How to check if a page is indexed

The site: operator. Type site:example.com/page-url into Google. If the page shows up, it is indexed. If it does not, that is a reason to check further, not a final answer: the operator is approximate and often misses pages.

URL Inspection in Search Console. This is the reliable way. Paste the full URL into the search bar at the top of Search Console. The report shows whether the URL is on Google, when it was last crawled and which canonical Google chose.

URL Inspection shows the index status, the last crawl date and the canonical Google selected


The Pages report. For the site as a whole, open Indexing → Pages in Search Console. It shows how many pages are indexed, how many are not and the reason for each group.

The Pages report groups non-indexed URLs by reason

How to index a website in Google: step by step

1. Make sure the page can be indexed. It returns status 200, is not blocked in robots.txt, has no noindex in the meta robots tag or the X-Robots-Tag header and points to itself in the canonical tag. URL Inspection → Test live URL checks all of this in one go.

2. Add the page to your sitemap and submit it. The XML sitemap should list only canonical URLs that return 200. Submit it once in Indexing → Sitemaps. After that, Google rereads it on its own.

Submit the sitemap once – after that Google rereads it on its own

3. Request indexing for important pages. In URL Inspection, click Request indexing. This puts the page in the crawl queue. Google limits how many requests you can send per day, so save them for new and substantially updated pages. Sending the same URL again will not speed things up.

4. Link to the page from pages that are already indexed. Crawlers find new pages mainly through links. A page that no internal link points to is an orphan page: it may sit unindexed for months even if it is in the sitemap. Two or three links from relevant, frequently crawled pages usually do more than a manual request.

Request indexing puts the page in the crawl queue, but does not guarantee it gets indexed

5. Do not forget other search engines. In Yandex, submit the sitemap in Yandex Webmaster and use the page recrawl tool. Bing and Yandex both support IndexNow, a protocol that notifies them of new and updated URLs automatically.

How long it takes. A new page on a site Google already trusts can get indexed within a few days. On a new site or a large one with many similar pages it can take weeks. If a page has not been indexed after two or three weeks, the cause is almost always in the page or the site, not in the waiting.

Why pages are not indexed and how to fix it

The Pages report names the reason for every non-indexed URL. These are the statuses that come up most often.

StatusWhat it meansWhat to do
Discovered – currently not indexedGoogle knows about the URL but has not crawled it yetAdd internal links and check the server response time; on large sites, cut duplicates that waste the crawl budget
Crawled – currently not indexedGoogle crawled the page and decided not to index it for nowMake the page more useful and less like other pages on the site, strengthen internal links
Excluded by 'noindex' tagThe page asks not to be indexedRemove noindex if the page should be in search
Blocked by robots.txtThe crawler is not allowed to fetch the pageRemove the Disallow rule; to keep a page out of the index, use noindex instead
Duplicate, Google chose different canonical than userGoogle considers another URL the main versionMerge the duplicates or make the content clearly different
Alternate page with proper canonical tagThe page points to another canonical URLUsually fine: the main version gets indexed instead
Page with redirectThe URL redirects elsewhereFine for old addresses; remove such URLs from the sitemap and internal links
Not found (404) / Soft 404The page is missing or looks emptyRestore it or set up a 301 redirect to the closest live page

Not every URL in this report needs fixing. Redirects, alternate pages and deliberate noindex are normal. What matters is whether the pages that should bring traffic are indexed. On large sites, duplicates also eat into the crawl budget, and old addresses are best closed with a 301 redirect.

Who should watch indexing closely

Any site that relies on search traffic. Problems show up most often on online stores with filters and sorting, on sites after a migration or domain change, and on sites that publish many similar pages quickly. Checking the Pages report once a month and after every major release is enough to catch most problems before they cost traffic.

Frequently asked questions

How long does it take Google to index a new page?

Usually from a few days to a few weeks. Pages on established sites with good internal linking get indexed faster; new sites and pages deep in the structure take longer.

How do I get my website indexed by Google for free?

Add the site to Google Search Console, submit an XML sitemap and request indexing for key pages in URL Inspection. Then make sure every important page is linked from other pages on the site. None of this costs anything.

Can I force Google to index a page?

No. Request indexing only puts the page in the crawl queue. Google decides whether to index it based on the page's quality and technical signals.

Why is my page crawled but not indexed?

Most often the page looks too similar to other pages, has little content of its own or gets few internal links. Make the page clearly useful on its own and link to it from relevant pages, then request indexing again.

How do I see how many pages of my site are indexed?

Open Indexing → Pages in Google Search Console. The site: operator gives only a rough estimate.

We will check your site against the same points

We will see what holds the site back and tell you where to start. Free, within two business days.