All listed user agents share this rule group. A disallow rule controls crawling, not access or guaranteed removal from search. Use authentication for private content, and allow crawling when a page needs a noindex directive read.
Build one robots.txt rule group with user agents, paths and an optional sitemap URL.
All listed user agents share this rule group. A disallow rule controls crawling, not access or guaranteed removal from search. Use authentication for private content, and allow crawling when a page needs a noindex directive read.
User-agent: * followed by Disallow: /admin/ and Sitemap: https://example.com/sitemap.xml.One rule group is generated. Disallow is not authentication or guaranteed removal from search. Allow crawling when noindex must be read. Review entire-site blocks before deploying; this tool does not change your server.
The tool's browser code handles entered text or selected files locally. Optional URL imports and external resources can make network requests. This website also uses third-party scripts, including analytics and advertising. Read the privacy policy and use sample data when testing sensitive workflows.
Found a problem? Report a tool issue with the browser version and steps to reproduce it. Avoid sending private document content or credentials.
No. Rules are public and cooperative. Protect private content using server access controls.
Not reliably. A blocked URL can still be known to a search engine. Allow crawling where indexing directives need reading.
At the host root, such as https://example.com/robots.txt. Rules apply to that host and protocol context.