Generate a robots.txt file to control search engine crawling on your website. Specify which paths to allow or disallow for web crawlers.
Functions
Generate
Create robots.txt content
Disallow
Paths to block from crawlers
Sitemap
Optional sitemap URL reference
Steps
1
Enter paths to disallow for crawlers (one per line)
2
Optionally add your sitemap URL
3
Click "Generate" to create the robots.txt content
4
Copy the result and save as robots.txt in your website root
Use cases: Control search engine crawling, block sensitive paths from indexing, specify sitemap location for crawlers