A file for crawler access preferences
Robots.txt contains instructions that participating crawlers use when requesting website resources. It can help manage requests to selected URL patterns, such as large collections of repetitive filter pages. It does not enforce access restrictions against every visitor or every automated client.
Google specifically warns against using robots.txt to hide a web page from search. A blocked URL can still appear when it is discovered elsewhere. For the distinction between crawling and removal, see Google’s robots.txt introduction.
Example: checking a website before launch
A staging environment may use restrictive crawler rules while the team builds and reviews content. When moving the finished site, examine the actual robots.txt response on the destination domain. Copying files successfully does not prove that the search access settings match the launch plan.
Compare the rules with representative URLs: the homepage, a service page, a glossary entry and the assets needed to render them. Keep the previous file available so an unintended change can be understood and reversed promptly.
Choose the control that matches the problem
If the aim is to keep a public page out of supported search results, consider noindex while allowing the crawler to discover that directive. If the content is confidential, require appropriate authentication and permissions. If repetitive URLs are consuming crawler resources, investigate the URL generation and navigation before adding broad exclusions.
- Document why each important rule exists.
- Check that it does not accidentally cover useful sections.
- Review crawl and indexing reports after the change.
- Keep sitemap references aligned with the intended public URLs.
A short, understood configuration is easier to maintain than a copied collection of rules whose original purpose has been lost.
Common questions about robots.txt
Can I list noindex inside robots.txt?
Google does not support noindex as a robots.txt rule. Use the supported page meta tag or HTTP response header and allow the crawler to request the resource.
Does allowing crawling guarantee indexing?
No. Access is one requirement. The page’s content, other indexing signals and the search engine’s own assessment still determine whether it appears in the index.
Related definitions
- Crawlability — Whether a search engine can discover and request pages and resources. It depends on factors including links, server responses, access restrictions and crawler directives.
- Noindex — A directive asking supporting search engines not to include a page in search results. The crawler generally needs access to the page to discover the directive.
- Indexability — Whether a page is eligible to enter a search engine’s index. A crawlable page may still be excluded, for example because of a noindex directive or the engine’s own selection.
- Sitemap — A structured list of relevant site URLs. XML sitemaps support discovery by search engines; an HTML sitemap helps people navigate the site.
Related SEO guides
Robots rules manage crawler access and need to be considered alongside the site’s URL volume and indexing intentions.
For help connecting these decisions with your website, explore our SEO services or start with an SEO audit.
Put it into practice
Explore our technical seo service or talk with Refined IT about applying this to your website or business systems.