Create and test a robots.txt file for your website, with ready-made setups, AI crawler blocking and sitemap lines. Free.
Let all crawlers visit the whole site.
User-agent: * Allow: / Sitemap: https://www.example.com/sitemap.xml
✔ Allowed by the rule “Allow: /” in the * group
Save the file as robots.txt and upload it to the top folder of your site, so it opens at yoursite.com/robots.txt. Nothing you type here is sent anywhere.
NinjaFunction's robots.txt Generator builds a robots.txt file for your website. Start from a ready-made setup such as WordPress or an online shop, add rules for any crawler, block well-known AI crawlers with one click, and add your sitemap address. A built-in tester checks any address against your rules the way Google does, using the longest matching rule. Copy the result or download robots.txt. Nothing is sent anywhere.
| Line | What it does | Example |
|---|---|---|
| User-agent | Names the crawler the rules are for. A star means all of them. | User-agent: * |
| Disallow | Asks the crawler not to visit this path. | Disallow: /admin/ |
| Allow | Makes an exception inside a blocked path. | Allow: /admin/help/ |
| Sitemap | Gives the address of your sitemap. | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Asks the crawler to wait between requests. Google ignores it. | Crawl-delay: 10 |
It is a small text file at the top of your website that tells search engines and other crawlers which parts of the site they may visit. It lives at yoursite.com/robots.txt.
Not reliably. It asks well-behaved crawlers not to visit a page, but a blocked page can still show up in search if other sites link to it. To keep a page out of search, use a noindex tag on the page, and do not block it in robots.txt, or Google will never see the tag.
No. It is a polite request that bad crawlers ignore, and anyone can read the file. Never list private folders in it. Protect private pages with a login instead.
Add a group for each AI crawler with Disallow: /. The Block AI crawlers button adds the well-known ones, such as GPTBot, ClaudeBot, Google-Extended and CCBot. This only works for crawlers that choose to follow the rules.
Upload it to the top folder of your site so it opens at yoursite.com/robots.txt. A robots.txt in a subfolder is ignored. Each subdomain needs its own file.
A star (*) matches any run of characters, so /*.pdf matches every PDF address. A dollar sign ($) means the address must end there, so /*.pdf$ matches only addresses that end in .pdf.
Google uses the longest matching rule, so a more specific rule wins. If an Allow and a Disallow are equally long, Allow wins. The test box on this page follows the same rules, so you can check an address before you publish.
Yes. A Sitemap line helps crawlers find all your pages. Use the full address that starts with https://. You can list more than one.
