Claude Opus 4 and 4.1 now have the discrete ability to end “abusive” or “harmful” conversations in consumer chat interfaces. Anthropic says the feature was developed as part of its exploratory work on the protection and well-being of its AI models. The company also envisions broader safety uses, although it does point out that having a model defensively terminate a chat is an extreme measure, intended for use in rare cases. “We’re working to identify and implement low-cost interventions to mitigate risks to model welfare,” Anthropic explains, qualifying it is unsure “such welfare is possible.”

In a research post, the San Francisco-based startup suggests “allowing models to end or exit potentially distressing interactions is one such intervention.”

The company emphasized that it “isn’t claiming that its Claude AI models are sentient or can be harmed by their conversations with users,” clarifying that it is “highly uncertain about the potential moral status of Claude and other LLMs, now or in the future.”

“Anthropic is essentially taking a just-in-case approach,” writes TechCrunch, citing the company’s April announcement that it was undertaking a study of model welfare, a subject world experts are also researching.

The new approach coincides with an update to Anthropic’s usage policy, effective September 15. The company previewed the usage changes in a news post about safeguards released earlier this month.

The Verge compares and contrasts the new and old policies, noting that “Anthropic previously prohibited the use of Claude to ‘produce, modify, design, market, or distribute weapons, explosives, dangerous materials or other systems designed to cause harm to or loss of human life,’” something the updated version expands on “by specifically prohibiting the development of high-yield explosives, along with biological, nuclear, chemical, and radiological (CBRN) weapons.”

The timing of Anthropic’s increased caution seems influenced, to some degree, by rising use of agentic AI, which imbues models with some autonomy, The Verge suggests.