Robots.txt & LLMs.txt Generator: one-screen crawler rules, compared
Reviewed by the OnlineFree.app team · Updated
Key points
- The Robots.txt & LLMs.txt Generator turns a domain, disallow paths and one AI policy into a robots.txt plus an llms.txt starter.
- Its default policy blocks GPTBot, Google-Extended, CCBot, Anthropic's crawler and Bytespider while allowing OAI-SearchBot and PerplexityBot.
- Robots.txt rules are voluntary, so genuinely private paths need authentication or server-level blocking instead.
- llms.txt is an unratified convention, so the generator's Pages section needs your own curated links before publishing.
- The tool never crawls your site, so it cannot discover paths for you — you supply the disallow list.
What the Robots.txt & LLMs.txt Generator does
It converts three inputs — a site origin, a list of disallow paths and one AI-crawler policy — into two ready-to-paste files: a complete robots.txt and a matching llms.txt starter. The Robots.txt & LLMs.txt Generator does this on a single screen, updating as you type, with no signup, no crawl and no build step.
The output appears as two monospace panels, each with a file-name header, a copy button and a download button, separated from the inputs by a dashed "crawl path" divider. Protocol and trailing slash are normalised, so example.com, https://example.com and https://example.com/ all produce the same Sitemap line and the same llms.txt heading.
Because nothing is fetched, the tool cannot guess which paths on your site should stay private — you supply those. That is the one piece of work it deliberately leaves with you.
How it compares to hand-editing or plugin builders
Hand-writing robots.txt works until you need to name individual AI crawlers. Misspell GPTBot as GPT-bot and you have written a rule nothing matches, because unknown user-agent tokens are simply ignored by crawlers that do not recognise them.
CMS plugins and SEO suites generate a robots.txt for you, but they usually cover only the classic crawlers, follow the rules the vendor chose, and rewrite the file on the next update. A generator hands you finished text you can read, edit and commit yourself. The trade-off is maintenance: you re-paste the file when your paths change.
Crawler-based scanners sit at the other end of the spectrum. They walk your site, surface orphan URLs and suggest rules — worth it for a large site, overkill for a five-page brochure. For most small sites, a reviewed robots.txt plus a curated llms.txt covers the job, and the rest of the free utilities on OnlineFree.app handle the surrounding chores.
Which AI crawler policy should you choose?
The default, block_training_allow_search, writes User-agent groups for GPTBot, Google-Extended, CCBot, Anthropic's crawler and Bytespider, while leaving OAI-SearchBot and PerplexityBot free to fetch pages for AI search results.
allow_all adds no AI-specific groups at all; every crawler falls under the single * group. Pick it if you want maximum visibility in AI answers and have nothing on the site you mind being collected.
block_all disallows every known AI crawler, including the search-flavoured ones. That suits paywalled, licensed or client-confidential material, but be clear about the trade-off: if OAI-SearchBot cannot fetch a page, that page generally will not be summarised or cited in ChatGPT's search results.
One useful nuance, documented by Google: Google-Extended governs AI training and grounding and is separate from the crawler that indexes pages for Google Search, so blocking it is not the same as removing yourself from search.
Writing disallow paths that behave as intended
One path per line, each starting with a slash. /admin/ blocks that directory, while /admin blocks the prefix and therefore also catches /administrator. Blank lines and lines beginning with # are ignored, so you can document rules inline, for example /cart # checkout flow, no crawl.
Wildcards are supported in the same way search engines interpret them: *.json$ blocks JSON files anywhere and the $ anchors the match to the end of the URL. Keep the list short. Every extra rule is another thing to maintain, and an over-broad pattern can quietly hide product or pricing pages. Never disallow /css/ or /js/ — rendering crawlers need those files.
Above all, remember the protocol is voluntary. Polite crawlers honour it; scrapers and unlisted bots may not. If a path genuinely must stay private, protect it with authentication or server-level rules rather than relying on Disallow.
Don't use robots.txt as a de-indexing tool either. Disallow stops crawling, not indexing, and a URL can still appear in results if it is linked elsewhere. Use a noindex meta tag or HTTP header when removal from search is the goal.
What llms.txt is, and what it is not
llms.txt is a proposed convention published at llmstxt.org: a Markdown file at /llms.txt that offers language models a curated map of your site. It is not a ratified standard, and no crawler is obliged to read it.
The generator writes a starter file — an H1 taken from your domain, a > summary line to rewrite, a "Do not crawl" section that mirrors your disallow paths, and a "## Pages" section with placeholder links. Those placeholders are the point: the value comes from selecting your ten or twenty most useful pages and describing them in plain language.
Publishing the placeholder version unedited is worse than publishing nothing, because it advertises pages that do not exist and gives an assistant nothing accurate to quote. Treat it as a scaffold, not a finished document.
Publish and verify in three steps
First, place both files at the web root so they resolve at https://yourdomain.com/robots.txt and https://yourdomain.com/llms.txt, then load each address in a browser. You want a 200 response with raw text — not a 404, not a CMS page, not a login prompt.
Second, check the classic traps: saving the file as robots.txt.txt on Windows, letting a staging copy overwrite production, or shielding /robots.txt behind basic authentication. If you maintain several domains and keep one exported copy per site, the Bulk File Renamer keeps those filenames consistent.
Third, verify rather than assume. The robots.txt specification in RFC 9309 is short and settles edge cases such as case-insensitive user-agent matching, and it states plainly that the protocol is a voluntary convention. Re-read your file whenever you add a private section or change the AI policy, and check server logs if a rule is protecting something commercially sensitive.
Frequently asked questions
Is the Robots.txt & LLMs.txt Generator free, and does it need an account?
It is free and needs no account, no email and no build step. You open the page, type a domain and any disallow paths, and the robots.txt and llms.txt panels update live with copy and download buttons. The tool never crawls your site, so nothing on your server is fetched or scanned while you use it.
Do I still need robots.txt if I publish an llms.txt file?
Yes. robots.txt is the established crawl-control protocol that search engines and AI crawlers actually check, while llms.txt is an optional, unratified convention that may or may not be read. Generate both, but keep your blocking rules in robots.txt and use llms.txt to describe the pages you want summarised or cited.
Which crawlers does the block-training-allow-search policy block?
That policy writes User-agent groups for GPTBot, Google-Extended, CCBot, Anthropic's crawler and Bytespider, while leaving OAI-SearchBot and PerplexityBot allowed. The result is that training-style collection is discouraged, but pages can still be fetched to answer AI search queries. It is the generator's default policy.
Can robots.txt reliably stop AI companies from using my content?
No. The protocol is voluntary and unenforced; compliant crawlers honour it, but scrapers, proxies and unlisted bots may ignore it entirely. Treat robots.txt as a polite signal, back it up with server-side user-agent blocking, rate limiting or authentication for anything genuinely sensitive, and confirm behaviour in your access logs rather than assuming compliance.
Can I use the generated files on WordPress, Shopify or a static host?
Yes. The output is plain text, so it works anywhere the files can sit at the site root: a static host, a WordPress install, a Shopify or Webflow domain. Copy the robots.txt and llms.txt panels into your host's file root, then load /robots.txt and /llms.txt in a browser to confirm both return raw file contents.