Robots.txt configuration sets crawl rules for supporting automated clients. It is useful for controlling access to low-value crawl spaces, but it does not protect confidential information and is not a dependable removal method for an indexed page. Small rule changes can affect entire sections of a website.
Understand the boundary of the file
A disallow rule controls retrieval by crawlers that honour it. Google can sometimes know about a blocked URL through external references even without reading its content. If a public page must leave search results, a crawlable noindex instruction is generally more appropriate. Confidential files require authentication or another access control. Keep essential CSS and JavaScript accessible where they are required to understand public content. Robots.txt and indexability solve related but different problems.
Review rules against sample URLs
Open the file for the actual hostname and protocol in use, then list the user-agent groups and path patterns. Test representative service URLs, images, scripts and unwanted parameter URLs against the intended rules. Pay attention to broad patterns copied from staging or old plugins. Check the response code of robots.txt itself and review Search Console crawl findings. Keep a versioned change record with examples of URLs that must remain allowed, so future edits can be reviewed against concrete expectations.
Check each hostname separately when the website uses subdomains. A rule file belongs to its particular host and should not be assumed to govern a separate shop, portal or media host automatically.
Avoid an over-broad catalogue block
Suppose an equipment supplier wants to reduce crawling of combinations such as colour, size and sorting filters. Blocking the entire catalogue directory would also prevent access to valuable product pages. Start by documenting which combinations are genuinely useful, then select appropriate URL and crawl controls. If a noindex page also becomes blocked, Google may no longer retrieve its instruction. Recheck product and category access after deployment, and use log evidence to see whether unnecessary requests actually decline.
Technical SEO support can help review crawl rules against the intended public content.
Common questions
Can robots.txt hide customer documents?
No. It is a public instruction file, and clients can ignore it. Customer documents need enforced access control rather than crawler guidance.
Should noindex be written inside robots.txt?
Google does not support noindex in robots.txt. Use a supported page meta tag or response header and allow crawling to read it.
Terms in this guide
- Robots.txt — A file containing rules for supporting web crawlers. It manages crawling and is not a reliable way to keep confidential information private.
- Noindex — A directive asking supporting search engines not to include a page in search results. The crawler generally needs access to the page to discover the directive.
- Indexability — Whether a page is eligible to enter a search engine’s index. A crawlable page may still be excluded, for example because of a noindex directive or the engine’s own selection.
- Faceted navigation — Filters that let users narrow a large collection, such as products by size and colour. The resulting URL combinations need a deliberate crawling and indexing strategy.
Related SEO guides
Apply this to your website
Explore Technical SEO, SEO Audits for help planning and implementing the relevant changes, or discuss your website with Refined IT.