CrewAI

Subjects

• A
  robots.txt rules for AI agentsGood work.All 11 AI agents checked may fetch every key page. No AI agent is named, so they all fall back to the * group.Extra credit: Add explicit User-agent groups for the AI agents you want, so your policy is a decision rather than a default.
• A
  llms.txt and llms-full.txtGood work./llms.txt found (2827 chars, 27 link(s)). /llms-full.txt found.
• A
  SitemapGood work.Sitemap at https://crewai.com/sitemap.xml (from robots.txt) with 89 entries.
• A
  Server-rendered readable HTMLGood work.All 1 key page(s) return readable HTML without running JavaScript.
• B
  Markdown versions of key pagesGood work.Every key page has a markdown version.Extra credit: Advertise the markdown versions of / with <link rel="alternate" type="text/markdown">.
• A
  Structured data (JSON-LD)Good work.Every key page has valid JSON-LD.
• A
  OpenAPI and API docs discoverabilityGood work.Machine-readable API spec at /openapi.json. /.well-known/api-catalog found.
• A
  MCP server discoverabilityGood work.MCP discovery document at /.well-known/mcp/server-card.json.
• A
  Auth, signup, and bot-challenge frictionGood work.No login walls, bot challenges, or CAPTCHAs on 1 key page(s).
• A
  Page weight and token costGood work.Key pages are light enough to read in one pass.