Robots.txt & LLMs.txt Generator: Tips and Common Mistakes

What does the Robots.txt & LLMs.txt Generator output?

The Robots.txt & LLMs.txt Generator turns one site origin, a list of disallow paths and a single AI-crawler policy choice into two ready-to-paste files: a complete robots.txt and a matching llms.txt starter in Markdown. Generation happens in the browser on one screen — no signup, no crawl of your live site, no build step.

The robots.txt half is the enforceable part: User-agent groups with Allow and Disallow rules for * plus each AI crawler your policy selects, and a Sitemap line derived from the origin you typed. The llms.txt half is documentation: an H1 built from the domain, a > summary line, a 'Do not crawl' section mirroring your disallow paths, and a '## Pages' section with placeholder links you replace with real URLs.

Because the preview updates on every keystroke, you can watch what adding a path or flipping the AI policy does before you copy anything. That matters most when you're writing rules for GPTBot, ClaudeBot, CCBot or PerplexityBot and would rather not guess at exact user-agent spellings.

Your first 60 seconds: domain, paths, policy

Start with the origin — example.com and https://example.com/ both work, because protocol and trailing slash are normalized automatically. That one value feeds the Sitemap line and the llms.txt header, so if your sitemap lives at a non-standard filename, adjust it once after copying.

Then add disallow paths, one per line. Blank lines and lines beginning with # are ignored, and an optional trailing comment after a path lets you record why it exists — useful six months later when nobody remembers why /cart was blocked. Precision beats volume here: Disallow: /admin also blocks /admin/, /administrator and /admin-panel, while Disallow: /admin/ covers only the directory and everything beneath it.

Wildcards work in mainstream crawlers. Entering *.json$ in the path box stops crawlers fetching raw JSON endpoints, because * matches any string and $ anchors the match to the end of the URL. RFC 9309 codifies both operators, so you aren't relying on undocumented behavior — though a handful of minor bots still only understand plain prefixes.

Which AI-crawler policy should you choose?

There are three options and one meaningful difference between them. allow_all writes no AI-specific rules, so ordinary search crawlers and AI agents get identical treatment. block_training_allow_search — the default — disallows GPTBot, Google-Extended, CCBot, anthropic-ai and Bytespider while leaving OAI-SearchBot and PerplexityBot allowed. block_all disallows every known AI crawler in the tool's list.

The default suits most publishers: you opt out of bulk model-training crawling while keeping the crawlers that can still surface and cite your pages in answer engines. If your revenue depends on being quoted by AI assistants, block_all removes a distribution channel; if your content is licensed or paywalled, the opposite trade-off may be correct.

Two honest caveats. Robots.txt is a voluntary protocol — polite crawlers honor it and impolite ones ignore it entirely, so it is a signal, not a lock. And vendors rename user-agents over time, so treat any generated list, including ours, as a snapshot and re-verify it periodically; as of 2026 the AI bot names are still shifting.

Common robots.txt mistakes to avoid

The mistake we see most is broken grouping. Every rule line beneath a User-agent line belongs to that agent until the next User-agent line appears, so a missing User-agent row silently attaches your rules to the wrong bot, and one group listing several agents shares a single rule set — exactly how the generator emits a group per policy tier.

The second classic is blocking assets. Disallowing /assets/, /wp-content/uploads/ or *.css stops crawlers fetching the files that make your pages render, which hurts how search engines evaluate them. Keep crawl rules for pages, directories and parameters; leave the CSS, JS and images alone.

Then there's the crawling-versus-indexing confusion. A Disallow line prevents a crawl of a URL; it does not guarantee the URL leaves an index, because a blocked page can still be listed if other sites link to it. Use noindex, canonicals and removal requests for indexing control — and remember noindex only works on pages crawlers are allowed to fetch.

Does llms.txt actually do anything yet?

Not in the enforceable sense. As of 2026, llms.txt is a proposal rather than a standard: it is a Markdown convention published at llmstxt.org asking site owners to describe their site for AI systems, and no major crawler is currently obligated to fetch or obey it. Robots.txt, by contrast, is standardized in RFC 9309 and honored by mainstream search crawlers.

That does not make the file pointless. The generated llms.txt is a compact, human-readable brief — an H1 from your domain, a > summary line, a 'Do not crawl' list mirroring your disallow paths, and a '## Pages' section where you paste the handful of URLs that best represent the site. Many teams publish it as cheap, structured guidance and as a statement of intent.

So use the two files for different jobs. The llms.txt earns its keep as documentation for people and for agents that choose to read it; the robots.txt earns its keep by actually restricting crawler access. Ship both, but don't expect the Markdown file to enforce anything on its own.

Verify before you ship

Upload robots.txt to the root of the exact origin — https://example.com/robots.txt — because crawlers do not look anywhere else, and each subdomain needs its own copy. After deploying, test a few representative URLs using the tools described in Google Search Central's robots.txt documentation, and confirm the Sitemap line matches the sitemap that actually exists.

Keep staging separate from production. We generate one file per environment and name them robots.staging.txt, robots.preview.txt and so on; if you batch those alongside other per-environment files, a Bulk File Renamer handles the renaming faster than doing it by hand.

Finally, don't let any generator, including ours, have the last word. Diff the output against your previous file, keep the result in version control, and re-check whenever a crawler you care about announces a new user-agent name. The rest of the free, browser-based OnlineFree.app toolset works the same way — quick output, with you doing the final verification.

References

• RFC 9309 — Robots Exclusion Protocol (IETF)
• Google Search Central — robots.txt documentation