The new web ingredient that may or may not improve your stack.
You may have started hearing the term “llms.txt” floating around in marketing and tech circles lately. Maybe an SEO tool flagged it as a missing file on your website. Or perhaps a developer mentioned it. Maybe you just spotted it in a LinkedIn post and filed it under “things I’ll Google later.”
Either way – you’re here now. Let’s sort it out.
First: what is it?
Think of your website like a well-stocked library. Your sitemap is the complete catalogue – every book, every shelf, every page. Your “robots.txt” file is the “staff only” sign on certain doors, telling automated bots which areas they’re not allowed into.
“llms.txt” is something new: it’s meant to be the librarian’s recommended reading list – a plain-text file that tells AI tools “start here, these are the most important pages.”
Jeremy Howard (co-founder of Answer.AI) proposed it in September 2024, and the idea is simple enough: AI assistants have limited “attention” (context windows, to be exact), and figuring out what’s actually useful on a complex website is hard for them. A well-written llms.txt file can give AI a curated shortcut.
How is it different from robots.txt?
Good question and worth understanding, because these two files are often confused.
robots.txt has been around since 1994 (yes, really – it’s older than Google) and it tells crawlers what not to access. It’s about permissions and boundaries.
llms.txt is almost the opposite. It’s not about restriction – it’s about guidance. It’s saying: “Hey AI, here’s the good stuff.”
A quick comparison:
robots.txt | llms.txt | |
|---|---|---|
| Purpose | Stop crawlers going where they shouldn’t | Guide AI to your best content |
| Around since | 1994 | 2024 |
| Officially standardised? | Yes (finally, in 2022!) | No – still a community proposal |
| Supported by Google? | Yes | No – they’ve explicitly said so |
That last row is important. We’ll come back to it.
So… who’s actually using it?
This is where it gets interesting – and a little muddy.
Companies that have implemented llms.txt:
→ Anthropic (the company behind Claude AI)
→ Stripe
→ Cloudflare
→ Vercel
→ Supabase
These are all developer-focused, technical documentation-heavy businesses. By late 2025, around 844,000 websites had added the file.
Companies that have publicly said they’re not using it:
→ Google. At a Search Central Live event in mid-2025, Google’s Gary Illyes stated plainly that Google doesn’t support llms.txt and has no plans to. Google’s John Mueller added that no AI system is currently using these files.
OpenAI, Anthropic, Microsoft, and Perplexity? They’ve all stayed conspicuously silent – no official confirmation either way.
There’s also evidence that some AI bots crawl llms.txt files – but crawling a file and actually using it to change how content is ranked or surfaced are very different things.
The misinformation loop you should know about
Here’s something we think is worth flagging, because it’s a pattern we’ve seen before in the SEO world.
SEO audit tools started flagging missing llms.txt files as an “issue.” Business owners saw the warning and felt anxious. WordPress plugin developers responded to demand and built auto-generators. More tools followed. The perception of necessity grew – even though there’s no confirmed evidence that any major AI platform actually uses these files to rank or surface content.
It’s a bit like buying a fancy “smart” doorbell because an app told you your house security score was low – even though no one had ever tried to break in.
We’re not saying llms.txt is useless. We’re saying: don’t let the anxiety spiral drive the decision.
So should your website have one?
Honestly? It depends – and for most small-to-medium businesses, it’s not a priority right now.
Here’s a simple way to think about it:
✅ Worth doing if you run a documentation-heavy site, a developer tools business, a SaaS platform, or a knowledge base – places where AI coding assistants and agent tools are likely to be pointing at your content directly.
⏸️ Not urgent if you’re a service business, retailer, or content site. The fundamentals (see below) will move the needle far more than a llms.txt file that the major platforms haven’t confirmed they’re reading.
If you do want to add one, it’s genuinely a 20-minute job for a developer – a simple Markdown file listing your key pages with short descriptions. Low effort, low risk, potentially useful down the track.
What actually drives AI (and search) visibility right now
Whether llms.txt becomes the next essential web standard or quietly fades out, the foundations of good website visibility don’t change. And these are the things worth investing in:
→ Clear structure – logical headings, sensible page hierarchy, clean URLs. Both humans and AI parse your site better when it’s well-organised.
→ Correct technical setup – valid HTML, proper metadata, accurate robots.txt configuration. If you want to appear in AI-powered search results, the key step right now is making sure you’re not blocking AI search crawlers in your robots.txt.
→ Mobile-first design – Google indexes the mobile version of your site. If it doesn’t work beautifully on a phone, you’re already behind.
→ Accessibility – semantic, accessible markup (proper headings, alt text, ARIA labels) isn’t just good practice for human users. OpenAI has explicitly stated that accessible sites are easier for its browsing agents to understand and navigate.
→ Great content – this one’s not going anywhere. Google’s recent algorithm updates have specifically targeted thin, AI-generated content without genuine expertise behind it. Original thinking, real experience, and genuinely helpful information are still the most durable visibility signals of all.
llms.txt at its best is a signpost pointing AI toward your content. But if the content it’s pointing to isn’t well-structured, accessible, and genuinely useful – the signpost doesn’t help much.
The bottom line
llms.txt is a real and legitimate proposal, not snake oil. It’s being used by serious companies, and it addresses a real problem. But it’s not yet a confirmed ranking factor for any major AI platform, and the loud noise around it has outpaced the actual evidence.
The smart move:
→ Don’t panic if you don’t have it
→ Don’t let an audit tool scare you into treating it as urgent
→ Do focus on the visibility fundamentals – they’ll serve you whether llms.txt becomes the next robots.txt or fades into obscurity
→ If you’re technically inclined (or have a developer on hand), add it when it’s convenient, it won’t hurt
Want to see how your site stacks up? We built a free AI Visibility Checker that tests your website’s AI readiness in about 30 seconds – including the robots.txt and llms.txt signals we’ve covered here. It’s a good starting point for understanding where you stand.
If what comes back raises questions about your marketing foundations, that’s exactly the kind of thing we help with.
Book a free 30-minute Marketing Discovery call and we’ll talk it through.
Leave a Reply