A robots.txt file communicates crawl rules to cooperating robots. Use the robots.txt generator to assemble one group with user-agent tokens, allow and disallow paths, and an optional sitemap URL. The tool creates text for review and download; it does not deploy changes to your site.
Start with a small shared group
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlThis example asks the general user-agent group not to crawl /admin/ and identifies a sitemap. Replace the example host with your real sitemap address. Rules are relative paths; the sitemap is an absolute URL.
The generator places all entered user agents in one shared group. If you need different rules for different agents, review and edit separate groups carefully. A matching specific group can affect which rules a crawler uses; do not assume a separate “allow everything” group retains the restrictions from a general group.
Keep crawling and indexing separate
Disallowing a path does not authenticate a visitor or erase the URL from search. A crawler may know a URL from links even when it cannot read its page. Likewise, if you block a page, its noindex directive may not be available for the crawler to read.
Google's robots.txt introduction describes these limitations. Choose server access controls for private content, and choose the appropriate indexing workflow for public pages that should not appear in search.
Avoid listing sensitive filenames as a substitute for protection: robots.txt is itself public. Test that server authorization works independently of the crawler rules.
Review an entire-site block
Disallow: / is broad. It blocks crawling under the root for the affected group unless applicable rules override it. The generator warns about this case. An accidental staging rule deployed to production can prevent intended crawling, so review the downloaded file before replacing a working one.
When no paths are entered, the tool emits an empty Disallow: directive for the group. This does not create an entire-site block.
Verify the deployment
Save the file at the host root, such as https://example.com/robots.txt, and open that exact URL. Check the HTTP status, content and host: a rule on one host should not be assumed to govern a different subdomain or protocol context.
Also open the sitemap URL and sample page URLs. Confirm that intended indexable pages return useful HTML, have the expected canonical URL and do not accidentally emit noindex. Use Search Console's inspection and reporting tools for actual crawling and indexing evidence; a generated robots file does not prove that a page is indexed.
For search-result wording, use the snippet preview. It supports metadata editing, while robots.txt supports crawl management. Neither is a ranking measurement.