# robots.txt for demandspring.com
#
# WHAT THIS REPLACES. Until launch the site served Cloudflare's managed robots.txt, which is
# 1,248 bytes of explanatory preamble about content signals and NOT ONE DIRECTIVE -- no
# User-agent, no Allow, no Disallow, and no Sitemap. It reads as policy and does nothing. The
# rights reservation it describes only bites when a content signal is actually set, and none
# were.
#
# THE SITEMAP LINE IS THE POINT. 715 URLs are listed in sitemap.xml and nothing pointed a
# crawler at it. That is the one thing a robots.txt on a launching site has to do.
#
# NOTHING IS DISALLOWED, deliberately. The pages that should stay out of the index say so
# themselves with a robots meta tag -- the three held services, the four service_pillar
# placeholder routes and the 404. That is the right mechanism: a Disallow stops a crawler
# READING a page, which also stops it seeing the noindex, so the two together are worse than
# either alone.
#
# CONTENT SIGNALS ARE NOT SET HERE, which is the same position the site is in today rather
# than a new one. Whether Demand Spring wants to reserve rights against AI training, AI input
# or search is the client's call, and it belongs in this file once made -- see
# https://contentsignals.org for the syntax.
#
# THE AI GROUP BELOW CHANGES NOTHING ABOUT WHO MAY CRAWL. `User-agent: *` already allowed
# every one of these and a fetch-as test on 2026-09-05 confirmed all of them get a full 200
# with byte-identical content -- Cloudflare is not gating them. Naming them makes an existing
# position legible instead of incidental, which is worth something the day someone asks what
# this site's stance on AI crawlers is and the answer has to be read out of an absence.
#
# READ THIS BEFORE ADDING A Disallow. A crawler that matches a named User-agent group uses
# ONLY that group and ignores `*` entirely. So a Disallow added to `*` below would NOT apply
# to any agent named in the AI group -- it has to be added in both places, or the group has to
# come out. That footgun is the whole cost of naming them; it is why they are one group rather
# than fifteen, and why nothing here is disallowed.
#
# AN AGENT THAT IS NOT NAMED IS NOT EXCLUDED. It falls through to `*` and is allowed. This
# list is the engines that actually drive citations for a North American B2B audience, not a
# permission boundary.
#
# A MAP FOR ANSWER ENGINES lives at /llms.txt -- H1, summary, and 86 curated URLs with each
# page's own description. Generated by scripts/emit-llms-txt.mjs; there is no robots.txt
# directive for it, hence this comment. See https://llmstxt.org.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-User
User-agent: cohere-ai
User-agent: YouBot
User-agent: CCBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://demandspring.com/sitemap.xml