SEO

Robots txt

A file in the root directory of a website that sets rules for search engine crawlers and determines which URLs they are allowed to request. robots.txt controls website crawling, but by itself is not a universal way to prevent a page from being indexed.

What's inside

Main directives – User-agent, Allow and Disallow. They are used to set rules for a specific crawler or group of crawlers. The file can also specify the address of Sitemap.xml. Yandex has separate crawlers and directives related to processing content for Alice AI services.

It is important not to confuse a crawling restriction with an indexing restriction. For example, Google may learn about a URL closed via Disallow from other sources and display it in search results without the page content. To prevent indexing, Google recommends using noindex, while the crawler must be able to crawl the URL and see this directive.

What this means in practice

An error in robots.txt may accidentally block important sections of a website from search engines. After changing the site structure, CMS or indexing rules, the file should be checked manually.

Who it is critical for

Important for any website, especially a large one with many technical URLs and separate rules for search engine crawlers.

We will check your site against the same points

We will see what holds the site back and tell you where to start. Free, within two business days.