AI21

Subjects

• A
  robots.txt rules for AI agentsGood work.All 11 AI agents checked may fetch every key page. No AI agent is named, so they all fall back to the * group.Extra credit: Add explicit User-agent groups for the AI agents you want, so your policy is a decision rather than a default.
• B
  llms.txt and llms-full.txtGood work./llms.txt found (7389 chars, 50 link(s)). No /llms-full.txt.Extra credit: Improve llms.txt: publish /llms-full.txt with the full docs inlined.
• A
  SitemapGood work.Sitemap index at https://www.ai21.com/sitemap_index.xml (from robots.txt) with 11 entries.
• A
  Server-rendered readable HTMLGood work.All 1 key page(s) return readable HTML without running JavaScript.
• F
  Markdown versions of key pagesHands in every page as HTML soup. Markdown was on the syllabus.1 of 1 key page(s) have no markdown version: /.Extra credit: Serve markdown for key pages, either at the same path plus .md or when the request sends Accept: text/markdown, and advertise it with <link rel="alternate" type="text/markdown" href="...">. Markdown cuts token cost and removes layout noise for agents.
• A
  Structured data (JSON-LD)Good work.Every key page has valid JSON-LD.
• F
  OpenAPI and API docs discoverabilityCouldn't find the API docs. Neither could the robot.No API spec at /openapi.json, /openapi.yaml, /swagger.json, /api/openapi.json, /.well-known/api-catalog and no API docs link on key pages or in llms.txt.Extra credit: Publish an OpenAPI spec at /openapi.json and link API docs from your homepage and llms.txt. If the product has no API, disable this check in config ("api-docs": { "enabled": false }).
• F
  MCP server discoverabilityNo MCP server in sight. Ironic, for an agent company.Key pages or llms.txt mention MCP, but no endpoint URL or discovery document was found.Extra credit: Document the MCP server URL and how to connect on a key page and in llms.txt. Discovery through /.well-known/mcp.json is still a draft proposal, so also document the server URL, transport, and auth in llms.txt and your docs.
• F
  Auth, signup, and bot-challenge frictionEverything worth reading is behind a login wall.1 of 1 key page(s) put friction in front of an agent: / (bot challenge page).Extra credit: Keep docs, pricing, and the product overview public. Let verified AI agents past bot challenges (most WAFs allowlist known agent user agents), and offer API keys or OAuth that an agent can use without a browser CAPTCHA.
• B
  Page weight and token costGood work.Key pages are light enough to read in one pass.