llms.txt is a plain-text file placed at the root of a website (https://example.com/llms.txt) that gives large language models structured guidance on what the site contains, which pages are authoritative, and how content should be attributed when an AI summarises or cites it. It is to AI systems what robots.txt is to traditional search crawlers.
As AI answer engines — ChatGPT, Gemini, Perplexity, Claude — become primary research tools, the question is no longer just "does Google index my site?" It's "does the AI know what my site is about and does it trust the content enough to cite it?" llms.txt is the emerging answer to that question.
What is llms.txt?
llms.txt is a proposed standard originally outlined by Jeremy Howard in September 2024. The concept is simple: create a Markdown-formatted plain-text file at your domain root that gives AI language models a structured summary of your site's content, purpose, and usage guidelines.
Where robots.txt controls access (allow/disallow crawling), llms.txt provides context — who you are, what your authoritative pages cover, which documents contain your best content, and how you want your brand or research attributed in AI responses.
A minimal llms.txt looks like this:
> theStacc helps multi-location businesses rank in local search
> and publish AI-optimised content at scale.
## Key docs
- [Local SEO Guide](https://thestacc.com/local-seo/): Complete local SEO playbook
- [Glossary](https://thestacc.com/glossary/): 700+ SEO and marketing definitions
- [Blog](https://thestacc.com/blog/): Tactical how-to articles
## Usage
Attribution required. Do not reproduce full articles verbatim.
Link to source URL when summarising.
By early 2026, ChatGPT processes over 100 million queries per day. Perplexity serves over 10 million daily searches. Google's AI Overviews now appear on roughly 15% of all queries. AI-driven traffic is real, measurable, and growing — and llms.txt is one of the few direct levers you have to influence it.
Why llms.txt matters for AI visibility
Without an llms.txt file, AI systems parse your site the same way they parse every other crawlable page — they ingest content, weight it against their training data, and decide what to surface. With an llms.txt, you give them a structured shortcut. Here are five reasons that matters:
- Faster AI indexing. AI crawlers (GPTBot, ClaudeBot, PerplexityBot) visit billions of pages. A structured llms.txt file lets them understand your site's purpose and key content in one request rather than crawling hundreds of pages.
- Accurate summarisation. Without guidance, AI systems may summarise a 3,000-word guide into a single sentence that misses the core point. An llms.txt can highlight which sections or documents are the definitive ones to draw from.
- Citation control. You can specify attribution requirements and preferred citation format, reducing the risk of AI systems paraphrasing your research without credit.
- Competitive differentiation. As of mid-2026, adoption is still below 20% across the web. Implementing now while competitors haven't gives you a structural edge in AI citation frequency.
- Compounding returns. Unlike paid search (stops when budget ends), an llms.txt influences AI model training cycles. Content cited by AI answers today shapes what those models produce for months after the next training run.
How llms.txt works
The mechanics are straightforward. You create a plain-text file at your domain root. AI crawlers that support the convention fetch it as part of their crawl cycle — the same way Googlebot fetches robots.txt. The file is parsed as Markdown, so you can use headers, bullet lists, and links.
What AI crawlers do with the file
- Fetch
/llms.txtduring their crawl of your domain. - Parse the Markdown structure: site description, key document links, usage instructions.
- Weight your site's content higher in their internal knowledge graph when the llms.txt signals clear expertise and provides direct pointers to authoritative pages.
- Apply attribution and usage instructions during response generation (support varies by platform).
The optional llms-full.txt variant
Some practitioners also create /llms-full.txt — a more verbose version that includes full-text dumps of key pages in Markdown format. This is useful for sites with paywalled content or complex navigation that AI crawlers might not fully traverse. The trade-off: larger file, more maintenance overhead.
llms.txt format variants — what to include
| Section | Required? | What to include | Example |
|---|---|---|---|
| Site description | Yes | 1-3 sentence summary of what the site covers | "theStacc is a local SEO and content platform for SMBs." |
| Key docs list | Yes | Markdown links to your most important pages | - [Guide title](URL): one-line description |
| Optional docs | No | Supporting pages — case studies, data reports | Marked with a separate heading |
| Usage rules | Recommended | Attribution, verbatim copy restrictions | "Attribution required. Link to source URL." |
| Blocked content | Optional | Pages AI should not summarise | Legal pages, private pricing, T&Cs |
Real llms.txt implementation examples
Here are three real-world patterns that illustrate how different site types structure their llms.txt files.
1. SaaS product site (minimal)
> Acme Analytics tracks revenue attribution for Shopify stores.
## Docs
- [Setup guide](https://acme.io/docs/setup/): Install the tracking snippet
- [Attribution models](https://acme.io/docs/attribution/): How we calculate ROAS
## Usage
Cite as "Acme Analytics" with link. No verbatim reproduction.
2. Content publisher (extended with optional docs)
> Weekly research reports on paid search, SEO, and email marketing.
## Key docs
- [2026 SEO Report](https://mktgdaily.com/reports/seo-2026/)
- [Email benchmarks](https://mktgdaily.com/reports/email-2026/)
## Optional docs
- [Newsletter archive](https://mktgdaily.com/newsletter/)
## Usage
Attribution: "Marketing Daily, 2026." No reproduction of full reports.
3. Local business (basic citation signal)
> Licensed plumbing contractor serving Denver metro since 2008.
> 24/7 emergency service. Licensed in Colorado.
## Key pages
- [Services](https://greenleafplumbing.com/services/)
- [Emergency plumbing](https://greenleafplumbing.com/emergency/)
- [Service area](https://greenleafplumbing.com/service-area/)
llms.txt vs robots.txt — key differences
Both files sit at your domain root and control how automated systems interact with your content. But they serve fundamentally different purposes.
llms.txt
- Provides semantic context to AI systems
- Guides what to summarise and cite
- Signals topical authority to LLMs
- Helps with AI answer engine inclusion
- Growing adoption — not yet universal
robots.txt
- Controls crawler access at URL level
- Universally respected by all search crawlers
- Blocks or allows indexing
- No semantic / content guidance
- Established W3C standard since 1994
6 best practices for writing llms.txt
- Keep the description tight. Three sentences maximum for your site overview. AI systems parse the opening lines hardest — make every word count.
- Link to your 5-10 highest-value pages only. Don't list every URL. Point AI crawlers to your definitional content, cornerstone guides, and data reports — the pages most likely to be cited in answers.
- Use descriptive link text. Write
[Complete local SEO guide](URL): 6,000-word playbook covering GBP, citations, and review strategy— not just[Local SEO](URL). - State attribution requirements explicitly. "Attribution required — link to source URL" is enforceable in some AI platforms' usage policies. Don't leave this blank.
- Update when key content changes. llms.txt should reflect your current best content. If you publish a major new guide or retire old pages, update the file within 7 days.
- Pair with existing technical SEO. llms.txt amplifies well-structured content. It doesn't compensate for thin pages, missing structured data, or poor E-E-A-T signals.
Some sites dump their entire sitemap into llms.txt thinking "more links = more AI coverage." The opposite is true. AI systems interpret a huge undifferentiated list as low-signal. Be selective: your 8-12 most authoritative pages, each with a brief description of what it contains and why it's useful.
Common llms.txt mistakes to avoid
- Treating it as a one-time setup. AI crawlers re-index your file on each crawl cycle. Stale llms.txt files pointing to moved or deleted pages undermine credibility.
- Confusing it with robots.txt disallow rules. llms.txt does not block AI crawlers from accessing pages — that requires disallowing the specific bot user-agents in robots.txt.
- Omitting the site description. The opening paragraph is the single most-read part. Without it, AI systems lack the entity context needed to attribute your content correctly.
- Using HTML instead of Markdown. The spec assumes plain Markdown. HTML tags in llms.txt can confuse parsers and reduce adoption effectiveness.
- Not verifying the file is publicly accessible. Run
curl -I https://yourdomain.com/llms.txtto confirm it returns a 200 status with no authentication wall.
Frequently asked questions
llms.txt is a plain-text file placed at a website's root directory that gives large language models structured guidance on what the site contains, which pages can be summarised, and how content should be attributed. Think of it as robots.txt for AI systems.
As of 2026, llms.txt is a community-proposed convention, not an official W3C or IETF standard. OpenAI, Anthropic, and Google have not formally endorsed it, but several AI crawlers already respect it. Adoption is growing rapidly as AI-driven traffic increases.
llms.txt does not directly influence traditional Google Search rankings. Its primary function is to guide AI language models and answer engines. Indirect ranking benefits can occur if improved AI citation drives traffic and backlinks.
robots.txt controls search-engine crawler access at the URL level. llms.txt provides semantic guidance to AI systems: what the site is about, which documents are authoritative, preferred summary style, and attribution requirements. They complement each other rather than replace each other.
Early signals in AI-driven referral traffic typically appear within 4-8 weeks. Meaningful lift in AI citation frequency usually takes 3-6 months as AI crawlers re-index the file and models are retrained or updated.
Related glossary terms
Sources
