Markdown for Agents: Why AI bots prefer plain text

• A lean Markdown version delivered to AI bots on request
• Same URL and same content source, but less HTML dead weight from layout, navigation, and tracking
• A clean structure for bots through headings, paragraphs, lists, links, and embedded images
• Potentially easier content capture
• Moccu's open Django approach with the django-markdown-middleware instead of separate bot pages

To you, a website looks like a page. An AI bot, by contrast, opens a packed toolbox and tries to find the right screwdriver.

The problem for the AI crawler: modern websites are made up of around 90% code. Only about 10% of that is the actual content it wants to read. So while this structure makes sense for humans, for AI search engines it's a lot of unnecessary dead weight.

And dead weight costs. Because AI bots don't work with unlimited resources: compute is scarce, every request costs money, and extra code takes up context that's then missing elsewhere. The less a bot has to sift through, the more capacity is left for what matters: your content.

This is exactly where Markdown for Agents comes in.

The idea is refreshingly simple. Markdown prepares a page with the features that matter to bots: hierarchy, structure, and content with headings, paragraphs, lists, and links. An extensive HTML page becomes a compact content version. The bot doesn't first have to work out what's content and what's navigation, footer, or a cookie banner.

Markdown: No guarantee, but a real advantage for AI search

A Markdown concept for AI agents doesn't come with a 100% guarantee of being included in answers from ChatGPT, Perplexity, Gemini, or other answer systems. But it does simplify the crawling of your website through less dead weight, a clearer structure, and potentially more usable content per request.

It's not a hard rule, but the need is clear: AI bots have to take in as much relevant information as possible with each request. If you deliver your content in a clean, structured format today, you won't be waiting for others to set the standard.

That's why now is exactly the right time to provide a Markdown version of your website.

On-the-fly Markdown has a practical appeal: the website stays as it is. Only when a bot requests Markdown for Agents does the HTML page turn into a lean Markdown version. No second content world, no duplicate maintenance, no parallel editorial overhead.

Benefit 1: More content per request

Markdown for Agents lets bots process more content per request. Where an HTML page runs to 200 KB, the same page in Markdown often needs just 20 to 30 KB. The exact saving depends on structure, templates, and embedded elements. The direction, however, is clear: less code, more room for the actual content.

This matters especially for long articles, guide pages, category texts, or product information. In content commerce in particular, the point is to structure product and advisory content so it can be understood quickly and processed cleanly. When an AI crawler doesn't process the whole page because of token limits or internal cut-off logic, every bit of unnecessary dead weight gets expensive. Markdown for Agents can help deliver relevant content more compactly.

Benefit 2: Less noise in crawling

The bot has to guess less about which parts of the code are relevant to it. In HTML, a footer can look like text content, a navigation like a list of relevant links, and a cookie banner like perfectly normal page content.

Markdown for Agents largely takes such noise out of the equation. The actual semantic content no longer sits between layout code, navigation, footer, and other technical elements. That lowers the risk of content being misattributed or pulled from the wrong context.

Benefit 3: Better conditions for AI answers

This point is more strategic. Anyone who provides their content in Markdown ahead of their competitors puts themselves in a better position to be included in AI-generated answers more often – and more accurately. This is where Markdown for Agents touches the field of Generative Engine Optimization (GEO): content shouldn't just be findable, it should be processed as cleanly as possible by AI systems.

Does that mean Markdown for Agents automatically leads to more citations in ChatGPT, Perplexity, Gemini, or other answer systems? No – there's no solid basis for that yet. On the question of what influence Markdown delivery has on AI answers, the data is still thin. The systems not only all work differently, they're also constantly being updated to new models. At the same time, our AI traffic report shows why the question of AI answers is no longer abstract for website operators.

Markdown for Agents isn't a quick lever for more AI visibility. But it tidies up the toolbox that AI bots would otherwise have to search through themselves. Minimal intervention, hardly any side effects, clear technical advantages.

Two paths lead to Markdown for Agents. One of them immediately creates new work. The other doesn't.

Separate bot pages create new maintenance work

Before Markdown for Agents emerged, one obvious approach dominated for a long time: creating separate Markdown versions for important pages – .md files as a second version of the original web pages. These are often called grounding pages. They were meant to provide AI bots with the content in a particularly easy-to-capture form, but they sit alongside the actual website.

That can work. The price for it is double maintenance. A paragraph gets updated in the article, but not in the Markdown version. A link changes. A number is corrected. And just like that, the bot reads a different version than the site's visitors. In short: as soon as content lives in two places, it drifts apart sooner or later. So it's no surprise that Google themselves pointed out the “danger of parallel Markdown versions” on June 15, on their Search off the Record podcast.

For clearly defined cases, that may be manageable. For websites that have grown over time, it quickly becomes a second content world. And it brings another risk: if such grounding pages end up in the Google index, they can show up as duplicates of the actual pages.

Content negotiation: Same URL, different format

The second path is, in our view, more promising. With Markdown for Agents as a response to content negotiation, the AI bot requests the same URL as all other crawlers, but sends an additional signal with it: Accept: text/markdown. The server understands this and then delivers the Markdown version.

The same URL and the same source, but a different format. No duplicate content, no second editorial logic, no grounding pages that eventually go stale. The website remains the single source, with Markdown for Agents as the machine-friendly output format of the content.

Some hosts already do exactly this. Cloudflare converts HTML to Markdown for Agents on the fly when needed and delivers the result directly. Cloudflare also played a decisive role in coining the name “Markdown for Agents”. At Vercel, the same approach is called agent-friendly pages.

So Markdown for AI agents doesn't have to be a separate content world; it emerges from what's already there.

The catch: not every website runs on Cloudflare or Vercel. But even if your host can't take on the task, there are solutions at the application level itself.

And that brings us to Django.

How do you implement Markdown for Agents at the application level without touching the existing project structure?

Many CMSs already have this function built in or can be extended via plugins. For WordPress, for example, there's the plugin Markdown for Agents and Statistics. Anyone who, like us, uses Django gets a ready-made solution straight from us: the django-markdown-middleware, MIT-licensed and freely available.

Django isn't a classic CMS, but an established Python-based web framework. It often sits under the hood when websites need to do more than just manage pages. Instagram is a world-famous example. A middleware solution makes sense precisely for setups like these.

What the middleware does technically

The principle is deliberately kept lean. The middleware sits between the server and the delivery layer and responds only to requests with Accept: text/markdown. All other requests pass through unchanged, as if the middleware weren't there.

When it receives a request with Accept: text/markdown, it takes the fully rendered HTML response from the view and converts it into Markdown for Agents. In the background, the proven html-to-markdown library by kreuzberg-dev does the work. In concrete terms: no access to templates, models, or other content structures – the HTML page remains the only source. No duplicate data storage, no customized views, no separate grounding pages. Add it to the Django MIDDLEWARE list, and that's it.

Error pages, redirects, and JSON APIs stay out of scope. The middleware only kicks in for HTTP 200 responses with text/html. Anyone who has ever dealt with a middleware that reaches too deep into the system knows why that matters. A component that does only what it was built for causes far less trouble in operation.

Caching, images & config

So that Markdown for Agents isn't recalculated on every bot request, caching is built right in: with “MARKDOWN_MIDDLEWARE_CACHE_TIMEOUT” you can set how long the Markdown version stays stored. Cache invalidation runs via post-save signals, so the bot always reads the current version.

What the conversion outputs can also be adjusted. Inline SVGs and data URI images can be dropped as needed. And because logged-in users don't need Markdown, the middleware by default only applies to anonymous requests. That, too, can be changed.

Open source instead of a one-off solution

The django-markdown-middleware is tried and tested in live operation, MIT-licensed, and works with all common Django libraries. It was developed by Stephan Jäkel, our in-house senior developer. Or, as he puts it himself:

Markdown for Agents won't automatically get your website into AI answers. There's still a lack of reliable evidence for that, and the systems work too differently.

What can be said: Markdown for AI agents takes the dead weight out of crawling. Your site stays unchanged for humans; AI bots get a leaner version on request. Whether this leads to more frequent citations in AI answers – or ones less affected by hallucinations – remains to be seen. Perhaps it mainly helps you avoid falling behind as other websites follow suit. If you want to dig deeper, you'll find more from us on AI and content marketing.

Because Markdown delivery only happens in response to an explicit bot request, nothing changes about normal website operation. No new interface, no double maintenance, no visible change for your visitors. No side effects, no significant costs.

That's exactly why Markdown for Agents is about as close to a free lunch as the web ever gets. Just grab it.

Still have questions or need more explanation? We're happy to talk it through with you personally.