How to Make Your Website Readable for AI

Agent-Ready: For AI agents

Once you're citable for AI search, the next step is being usable by agents — automated systems that browse your site to complete a task. Structured around the 5 categories tested by isitagentready.com.

1. Discoverability — so agents can find what you have

An agent lands on your root and needs to quickly find: where's the sitemap, where's llms.txt, where's the MCP server, where's the API catalog. Three places to put this:

• robots.txt — Sitemap: directive; llms.txt isn't a standard robots directive but add it as a comment.
• HTTP Link response headers (RFC 8288) — agents find these via a HEAD request without parsing HTML.
• <link rel="..." href="..."> in HTML — fallback for clients that don't check headers.

Example Link header for Cloudflare Pages (public/_headers):

/*
  Link: </llms.txt>; rel="llms-txt"
  Link: </llms-full.txt>; rel="llms-txt-full"
  Link: </sitemap.xml>; rel="sitemap"
  Link: </.well-known/api-catalog>; rel="api-catalog"

Vibecoding.cz has all these Link headers globally — /.well-known/api-catalog is in application/linkset+json format per RFC 9727.

2. Content Accessibility — so agents can understand your content

HTML with ad banners, cookie pop-ups, and SPA hydration is a nightmare for agents. Three techniques:

• llms.txt as curator — a markdown index of sections and key pages. An agent reads it in a second instead of crawling the sitemap.
• llms-full.txt as full text — the entire site content as one large markdown for RAG indexing. Watch the size (vibecoding.cz is around 700 KB).
• Markdown content negotiation — when an agent sends Accept: text/markdown, the server returns MD instead of HTML. An advanced technique; most sites don't do this yet, but agents are starting to test it.

3. Bot Access Control — who can read what

Classic robots.txt plus the new Content Signals (IETF draft). The example below allows crawling by all bots, protects the admin section, and explicitly names the major AI bots (in case any of them ignore User-agent: *):

# robots.txt
User-agent: *
Allow: /
Disallow: /admin/

# Explicit opt-in for AI indexing and training
User-agent: GPTBot
Allow: /
Disallow: /admin/

User-agent: ClaudeBot
Allow: /
Disallow: /admin/

User-agent: PerplexityBot
Allow: /
Disallow: /admin/

User-agent: Google-Extended
Allow: /
Disallow: /admin/

User-agent: CCBot
Disallow: /

# Content Signals — usage preferences (apply globally)
Content-Signal: search=yes, ai-input=yes, ai-train=yes

Sitemap: https://example.com/sitemap.xml

Content-Signal values: search = classic indexing for search engines; ai-input = use as context for AI responses (RAG, citations); ai-train = model training. Each value is yes / no.

Web Bot Auth — authenticating bots via signed HTTP messages (an early draft exists). Practical implementation is premature; watch contentsignals.org for updates.

4. Protocol Discovery — so agents know what you can do

When a site offers actions (order, catalog search, event registration), agents need a machine-readable description of them. Two standards worth implementing today:

• /.well-known/api-catalog (RFC 9727) — an application/linkset+json list of available API endpoints. Vibecoding.cz has this; agents can find the MCP server, RSS feeds, sitemap, llms.txt there.
• MCP Server Card (/.well-known/mcp/server-card.json) — a JSON declaration of what the MCP server can do. An agent sees the available tools without needing an initial handshake. Vibecoding has an MCP server at mcp.vibecoding.cz with a Server Card declaring available tools.

Minimal MCP Server Card (abbreviated — see vibecoding.cz's full version at /.well-known/mcp/server-card.json):

{
  "$schema": "https://modelcontextprotocol.io/schemas/server-card/v1",
  "serverInfo": {
    "name": "vibecoding-events",
    "version": "1.0.0",
    "description": "MCP server for the Vibecoding.cz event calendar"
  },
  "transport": {
    "type": "streamable-http",
    "endpoint": "https://mcp.vibecoding.cz/mcp"
  },
  "tools": [
    { "name": "list_events", "description": "List upcoming events" },
    { "name": "search_events", "description": "Search by keyword" },
    { "name": "get_event", "description": "Event detail" },
    { "name": "submit_event", "description": "Submit a community event" }
  ]
}

Agent Skills and WebMCP are additional emerging standards (Anthropic Skills, browser-side MCP). They're still too early for most websites.

5. Commerce — agentic payments

Standards x402, MPP, UCP, and ACP handle situations where an agent wants to buy something or pay. x402 is the most mature (HTTP 402 Payment Required + crypto micropayments). For most websites this isn't relevant yet — if your agent doesn't make autonomous payments, monitor the space and revisit in a year.