How the directive is discovered
Noindex can be delivered in a page’s robots meta tag or through an HTTP response header. Google must be able to request the resource to see the instruction. Blocking the same URL in robots.txt can prevent that discovery, leaving an externally linked URL eligible to appear without its page content.
A PDF can use an HTTP header because it has no HTML head for a meta tag. The appropriate implementation depends on the resource and hosting setup. Google explains both approaches in its noindex documentation.
When a business might use noindex
A public thank-you page may be useful after a form submission but offer little value as a search landing page. Noindex can express that preference while allowing ordinary visitors to use it. By contrast, a service description intended to attract enquiries usually needs to remain indexable.
During a redesign, maintain a written list of pages that should and should not appear in search. Review the live environment separately from staging: a setting copied during migration can unintentionally exclude the finished website.
Checking a suspected indexing problem
- Inspect the final URL and its response, including any redirects.
- Check both the HTML robots tag and HTTP headers for directives.
- Confirm that crawler access is allowed where noindex needs to be discovered.
- Use Search Console’s URL inspection to compare the live page with Google’s recorded state.
Removing the instruction is only one part of restoring eligibility. The page still needs useful content, internal links and a working response. Search systems must revisit and reassess it, so a saved setting does not cause an immediate search-result change.
Common questions about noindex
Does noindex protect confidential information?
No. It is a search instruction, not an access control. Private client records should require appropriate authentication and permissions even when they are excluded from search.
Should a noindex page remain in an XML sitemap?
A sitemap intended for search discovery should normally focus on the canonical pages you want indexed. Review conflicting sitemap entries when changing a page’s indexing preference.
Related definitions
- Indexability — Whether a page is eligible to enter a search engine’s index. A crawlable page may still be excluded, for example because of a noindex directive or the engine’s own selection.
- Robots.txt — A file containing rules for supporting web crawlers. It manages crawling and is not a reliable way to keep confidential information private.
- Sitemap — A structured list of relevant site URLs. XML sitemaps support discovery by search engines; an HTML sitemap helps people navigate the site.
- Google Search Console — Google’s tool for monitoring a website’s presence in Search, including performance, indexing reports and URL inspection.
Related SEO guides
Noindex and robots.txt serve different purposes; a crawler normally needs access to read the page’s indexing instruction.
For help connecting these decisions with your website, explore our SEO services or start with an SEO audit.
Put it into practice
Explore our technical seo service or talk with Refined IT about applying this to your website or business systems.