# aperture robots.txt # # default policy: allow all crawlers (search and AI) to access the entire # marketing surface. /api/, /auth/, /dashboard/, /device-restriction are # application endpoints with no content value, so they stay disallowed for # every bot, including AI crawlers. # # AI crawlers are listed explicitly because: # (a) some bots (Google-Extended, Applebot-Extended, anthropic-ai) only # respect their own UA, not the User-agent: * fallback. They are # opt-in for AI training, separate from regular search indexing. # (b) per RFC 9309, a bot with its own named group ignores User-agent: * # entirely, so each named bot needs its own Allow + Disallow rules. # # Content-Signal declares aperture's preference for how content may be # used (contentsignals.org, IETF draft-romm-aipref-contentsignals): # - search=yes : show in search results # - ai-input=yes : use as input/context for AI agents at query time # - ai-train=yes : include in AI model training data # Documentation lives on a separate host and has its own crawl surface: # https://docs.aperturehq.ai/ (integration guides, scoring reference) # It has no robots.txt or sitemap of its own — see the note in the repo. # Agent auth policy: # /auth.md states that no agent registration or public API exists today, # so an automated client does not waste a discovery round finding out. # LLM context files (the fastest path to an accurate answer about aperture): # /llms.txt condensed context, one fetch # /llms-full.txt extended context, including the full fraud reference # Agent discovery (A2A): # /.well-known/agent-card.json what aperture is, and what it knows about # /.well-known/skills.json where that knowledge lives, per topic # /.well-known/agent-skills/index.json verifiable skill artifacts (SHA-256) # All are public, unauthenticated, descriptive only — they advertise no # callable endpoints and expose no API. User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ # === AI training crawlers === User-agent: GPTBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: ClaudeBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: anthropic-ai Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Google-Extended Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Applebot-Extended Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: CCBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Bytespider Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Amazonbot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Meta-ExternalAgent Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Meta-ExternalFetcher Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: FacebookBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: AI2Bot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: AI2Bot-Dolma Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: cohere-ai Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: cohere-training-data-crawler Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Diffbot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: omgili Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: omgilibot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Webzio-Extended Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: ImagesiftBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Timpibot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: DataForSeoBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: PetalBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: TurnitinBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: iaskspider/2.0 Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ # === AI search and answer crawlers (user-triggered real-time fetches) === User-agent: OAI-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: ChatGPT-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Claude-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Claude-SearchBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Claude-Web Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: PerplexityBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Perplexity-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: DuckAssistBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: YouBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: MistralAI-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ User-agent: Bingbot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /device-restriction Disallow: /dashboard/ Disallow: /auth/ Disallow: /api/ Sitemap: https://aperturehq.ai/sitemap.xml