Robots.txt provides crawler instructions for a website. It is commonly used to allow or restrict requests to particular paths for crawlers that follow its rules.
It controls crawling, not access security. A blocked path can remain accessible to people, and robots.txt should not be used to protect confidential information.
Crawling versus indexing
Crawling is requesting content. Indexing concerns whether information is processed for inclusion in search results.
Blocking a URL from crawling does not reliably ensure it disappears from search. A search engine may know the address from links without being able to fetch the page.
How is noindex different?
A noindex instruction tells a supporting search engine not to include the content in its index. The crawler normally needs access to read that instruction.
Combining a crawl block with an inaccessible page-level noindex can therefore defeat the intended process. Review the complete configuration instead of treating the controls as interchangeable.
Where do problems occur?
A rule intended for a staging environment may be copied to the main website. Broad path patterns can also affect pages beyond the original intention.
Shared rules deserve careful review. A small configuration change can have consequences across many services, articles or product pages.
What should be checked before launch?
Inspect the actual public robots.txt file and the paths covered by its rules. Compare the configuration with the intended public content.
Also check page-level indexing settings. A clean robots file does not prove the page is indexable if other controls exclude it.
Can robots.txt make a page private?
No. It is not authentication or authorisation. Anyone able to request an unprotected URL may still access the content.
Use suitable access controls for private material. Avoid publishing sensitive information and assuming a crawler instruction supplies a security boundary.
How should changes be tested?
Define the intended effect before editing. Identify examples that should be allowed and examples that should be restricted, then verify the rules against those cases.
Keep a record of the change and its purpose. Future teams need to understand why a path was restricted before deciding whether to remove the rule.
When should a business investigate it?
Investigate when important pages cannot be crawled, after a migration or when a technical audit identifies unexpected restrictions. The business impact depends on the paths affected.
For a small site, the priority is usually to avoid accidental barriers. Complex crawl management is not automatically necessary simply because the file exists.
Further reading
For reference, see the supporting documentation. Explore related explanations of Noindex, XML sitemap, Crawl budget.
A practical example
Illustrative example: a website's staging environment restricts crawling. Before launch, the team checks the production configuration independently and confirms that public service pages can be requested. Private documents remain protected through access controls rather than robots.txt.
Why it matters
Understanding robots.txt prevents confusion between crawler management, indexing and security. Clear ownership and testing help avoid site-wide mistakes.
