A proposed robots.txt signal is a prompt to document who decides how public content may be used, while keeping search discovery separate.
A website's AI training preference and its visibility in AI search are separate decisions. Treating them as one switch can leave the wrong policy in place.
On October 30, OpenAI announced support for a shared signal in robots.txt for website operators to express preferences about foundational generative AI training. OpenAI says it aims to implement the signal in the coming months and will provide details when it does. Its existing crawler controls will continue to apply.
The operating question for a site owner is not simply “allow AI or block AI.” It is: which use are we deciding about, who owns that decision, and how do we know the published configuration reflects it?
A training preference is not a search setting
The proposed signal addresses a preference about using content for generative AI training. The October 30 Internet-Draft describes a machine-readable way to express that preference by path in robots.txt. It is an individual draft and a work in progress, not a finalized IETF standard.
The draft distinguishes a content-use preference from rules for managing crawler traffic. A signal about training is not, by itself, a command to stop every crawler from visiting a page. Nor does a crawler visit alone establish which use the content will have.
For OpenAI specifically, its crawler documentation describes separate settings for GPTBot, whose crawled content may be used for training, and OAI-SearchBot, which supports ChatGPT search. OpenAI says these settings are independent. Allowing a search crawler is not a guarantee of ranking or inclusion; it is one part of making a page eligible for that discovery path.
Other providers may use different controls. A shared preference is useful only to the extent that a provider implements and respects it. It is not a technical lock that prevents every possible collection or use of public content.
Make the policy an owned decision
Before changing a robots.txt file, put the business decision in plain language. For example: “We want public product documentation discoverable in search, and we want to separately decide how it is used for model training.” That sentence does not prescribe a particular setting. It gives the people responsible for content and infrastructure a shared question to resolve.
A small decision record can include:
- The sites and content paths covered.
- The intended treatment of search discovery and AI training as separate choices.
- The owner who can approve changes and the person who maintains the file.
- The providers and controls reviewed, including the date of the review.
- The existing rules that should remain in place until a deliberate change is approved.
Include the team responsible for the site and, where appropriate, the people responsible for content rights. A file deployed by engineering can still represent a business decision about material owned or licensed by someone else.
Check the actual file, then review again
Look at the robots.txt file served by each production hostname, not only a copy in a repository. Record the current rules and confirm which system publishes them. Multiple domains, redirects and deployment settings can make an assumed configuration different from the one a crawler receives.
Then compare that file with each relevant provider's current documentation. OpenAI's announcement says the proposed signal will sit alongside its existing controls, so deleting current rules on the assumption that a future signal has replaced them would be premature. Review any proposed change before deployment, check the served file afterward, and keep a record of what changed.
This is a small piece of AI governance, but it illustrates a larger operating habit: separate the decisions, name an owner, and verify what is actually running. A published preference is useful when the organization can explain what it means and maintain it as the standards and providers evolve.