How indexing works: crawling, indexing, ranking
Search engines handle a page in three steps, and each one can fail on its own.
- Crawling. The crawler finds the URL through a link, a sitemap or a manual request and downloads the page.
- Indexing. The search engine renders the page, works out what it is about, picks the canonical version among duplicates and decides whether the page is worth storing.
- Ranking. Only indexed pages compete for positions when someone searches.
Google does not promise to index every page it crawls. If a page looks like a duplicate, has little content of its own or is hard to reach, Google may crawl it and still leave it out of the index.
How to check if a page is indexed
The site: operator. Type site:example.com/page-url into Google. If the page shows up, it is indexed. If it does not, that is a reason to check further, not a final answer: the operator is approximate and often misses pages.
URL Inspection in Search Console. This is the reliable way. Paste the full URL into the search bar at the top of Search Console. The report shows whether the URL is on Google, when it was last crawled and which canonical Google chose.
URL Inspection shows the index status, the last crawl date and the canonical Google selected
The Pages report. For the site as a whole, open Indexing → Pages in Search Console. It shows how many pages are indexed, how many are not and the reason for each group.
The Pages report groups non-indexed URLs by reason
How to index a website in Google: step by step
1. Make sure the page can be indexed. It returns status 200, is not blocked in robots.txt, has no noindex in the meta robots tag or the X-Robots-Tag header and points to itself in the canonical tag. URL Inspection → Test live URL checks all of this in one go.
2. Add the page to your sitemap and submit it. The XML sitemap should list only canonical URLs that return 200. Submit it once in Indexing → Sitemaps. After that, Google rereads it on its own.
Submit the sitemap once – after that Google rereads it on its own
3. Request indexing for important pages. In URL Inspection, click Request indexing. This puts the page in the crawl queue. Google limits how many requests you can send per day, so save them for new and substantially updated pages. Sending the same URL again will not speed things up.
4. Link to the page from pages that are already indexed. Crawlers find new pages mainly through links. A page that no internal link points to is an orphan page: it may sit unindexed for months even if it is in the sitemap. Two or three links from relevant, frequently crawled pages usually do more than a manual request.
Request indexing puts the page in the crawl queue, but does not guarantee it gets indexed
5. Do not forget other search engines. In Yandex, submit the sitemap in Yandex Webmaster and use the page recrawl tool. Bing and Yandex both support IndexNow, a protocol that notifies them of new and updated URLs automatically.
How long it takes. A new page on a site Google already trusts can get indexed within a few days. On a new site or a large one with many similar pages it can take weeks. If a page has not been indexed after two or three weeks, the cause is almost always in the page or the site, not in the waiting.
Why pages are not indexed and how to fix it
The Pages report names the reason for every non-indexed URL. These are the statuses that come up most often.
| Status | What it means | What to do |
|---|---|---|
| Discovered – currently not indexed | Google knows about the URL but has not crawled it yet | Add internal links and check the server response time; on large sites, cut duplicates that waste the crawl budget |
| Crawled – currently not indexed | Google crawled the page and decided not to index it for now | Make the page more useful and less like other pages on the site, strengthen internal links |
| Excluded by 'noindex' tag | The page asks not to be indexed | Remove noindex if the page should be in search |
| Blocked by robots.txt | The crawler is not allowed to fetch the page | Remove the Disallow rule; to keep a page out of the index, use noindex instead |
| Duplicate, Google chose different canonical than user | Google considers another URL the main version | Merge the duplicates or make the content clearly different |
| Alternate page with proper canonical tag | The page points to another canonical URL | Usually fine: the main version gets indexed instead |
| Page with redirect | The URL redirects elsewhere | Fine for old addresses; remove such URLs from the sitemap and internal links |
| Not found (404) / Soft 404 | The page is missing or looks empty | Restore it or set up a 301 redirect to the closest live page |
Not every URL in this report needs fixing. Redirects, alternate pages and deliberate noindex are normal. What matters is whether the pages that should bring traffic are indexed. On large sites, duplicates also eat into the crawl budget, and old addresses are best closed with a 301 redirect.
Who should watch indexing closely
Any site that relies on search traffic. Problems show up most often on online stores with filters and sorting, on sites after a migration or domain change, and on sites that publish many similar pages quickly. Checking the Pages report once a month and after every major release is enough to catch most problems before they cost traffic.
Frequently asked questions
Usually from a few days to a few weeks. Pages on established sites with good internal linking get indexed faster; new sites and pages deep in the structure take longer.
Add the site to Google Search Console, submit an XML sitemap and request indexing for key pages in URL Inspection. Then make sure every important page is linked from other pages on the site. None of this costs anything.
No. Request indexing only puts the page in the crawl queue. Google decides whether to index it based on the page's quality and technical signals.
Most often the page looks too similar to other pages, has little content of its own or gets few internal links. Make the page clearly useful on its own and link to it from relevant pages, then request indexing again.
Open Indexing → Pages in Google Search Console. The site: operator gives only a rough estimate.