A small mistake in robots.txt can stop a crawler from reaching pages you wanted people to find. The file is short, but it's worth reading every rule before publishing it.
The robots.txt generator helps you prepare user-agent rules, paths and a sitemap address. It creates the text; you still need to put it on your website.
Start with a simple file
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlThis asks cooperating crawlers in the general group to avoid /admin/. Replace the example sitemap address with your own.
Paths such as /admin/ are relative to the host. The sitemap address is a full URL.
The generator puts the user agents you enter into one shared group. If different crawlers need different rules, edit separate groups carefully. A crawler may select a specific group rather than combine it with your general rules.
Don't use it to protect private pages
Robots.txt is public, and visitors don't have to follow it. Private content needs server-side access controls.
Blocking crawling also doesn't guarantee that a URL disappears from search. A crawler can discover the address through other links. If it can't fetch the page, it may not see a noindex directive on it either.
Google's robots.txt introduction explains these limits. Decide whether you're trying to manage crawling, keep information private or remove a page from search before choosing the rule.
Double-check an entire-site block
User-agent: *
Disallow: /This blocks crawling under the root for that group, subject to any applicable overriding rules. A file intended for staging can cause trouble if it reaches the live site.
An empty Disallow: line is different: it doesn't block the entire site. Read the downloaded file before replacing the working version.
Check the live URL
Save the file at the host root, for example https://example.com/robots.txt. Open that address and confirm it returns the expected text with a successful response.
Check the exact host too. Don't assume rules on one subdomain also apply to another.
Finally, open your sitemap and a few important pages. Check for accidental noindex tags and incorrect canonical URLs. Search Console can help you inspect actual crawl and indexing results after deployment. Generating the file is only the preparation step.
