User-agent: * Allow: / Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
What is this tool?
Generates a robots.txt file and an XML sitemap for a website, the two standard files search engines use to understand what to crawl and index.
How to use it
Enter your site's URL and crawling preferences, and the tool generates both files ready to upload to your website's root directory.
Running this once before launching a new site is a five-minute technical SEO step that's easy to forget entirely if it's not part of a standard pre-launch checklist.
Part of: Setting Up Basic SEO for a New Website
Next: OG / Twitter Card Preview
Check how your pages preview when shared on social media
More from SEO Tools:
Advertisement