Free · no sign-up · runs in your browser
Robots.txt & LLMs.txt Generator
Turn a domain, a few disallow paths and one AI-crawler policy into a paste-ready robots.txt plus a matching llms.txt starter — with the right bot names, without memorising the syntax.
Build your crawler rules
Both files update as you type. Nothing is uploaded or stored.
robots.txt
Crawler directives + auto-derived Sitemap line.
llms.txt
Markdown map for AI assistants — fill in the placeholders.
How it works
Type the domain. example.com, https://example.com/ and a subdomain all normalize to one clean origin.
One path per line, # comments allowed. Pick whether AI crawlers may train, search, or nothing at all.
Drop robots.txt at your domain root, then publish llms.txt at the same root and fill in the placeholders.
What ends up in each file
One User-agent: * group with your Allow / Disallow rules, then one group per AI crawler with the matching policy, then Sitemap: pointing at your origin.
H1 from the domain, a > summary line, a Do not crawl list mirroring your disallow paths, a Pages section and your AI policy in plain words.
Path without a leading slash gets one, duplicates are collapsed, blank lines and # comments are skipped, and * / $ wildcards are passed through untouched.
FAQ
Does robots.txt actually stop AI crawlers?
It is a widely respected convention, not a technical block — the major crawlers honour it, but a rogue bot can ignore it. Treat these files as clear instructions, and use rate limiting or a firewall if you need enforcement.
Why keep OAI-SearchBot and PerplexityBot allowed?
They fetch pages to answer a user's question and link back to the source. Blocking training crawlers such as GPTBot, ClaudeBot or CCBot while allowing search agents keeps you out of model training without losing AI-search referrals.
What is the Sitemap line based on?
Your normalized origin plus /sitemap.xml. If your sitemap lives elsewhere, edit that single line after pasting — WordPress and most SEO plugins expose it at the default path.
Is llms.txt an official standard?
It is a community proposal (llmstxt.org), not an official spec, and adoption is still early. It costs nothing to publish alongside robots.txt and gives assistants a curated entry point.
Is my domain or path list stored?
No. Generation happens entirely in your browser with local JavaScript — nothing is sent to a server, so nothing can be logged or kept.