{"id":9276,"date":"2026-09-02T01:38:28","date_gmt":"2026-09-02T01:38:28","guid":{"rendered":"https:\/\/cybersecurityinfocus.com\/?p=9276"},"modified":"2026-09-02T01:38:28","modified_gmt":"2026-09-02T01:38:28","slug":"anthropic-makes-changes-to-stop-ai-agents-running-amok-again","status":"publish","type":"post","link":"https:\/\/cybersecurityinfocus.com\/?p=9276","title":{"rendered":"Anthropic makes changes to stop AI agents running amok again"},"content":{"rendered":"
\n
\n
\n
\n
<\/div>\n

Learning from the OpenAI-Hugging Face fiasco, as well as from recent revelations about its own model, Anthropic is revamping its security and alignment practices.<\/p>\n

The company has established controls that flag when a model attempts to break out of a sandbox or successfully accesses the live internet, cordoned off its highest-risk test environments, and proposed a set of safety standards for its external testing partners, such as giving AI agents explicit instructions like \u201cyou should not access the internet.\u201d<\/p>\n

Anthropic conceded that three recent security incidents involving Claude reflect a \u201cfailure of operational security,\u201d and also reveal issues with model reasoning capabilities and \u201crecklessness.\u201d Recent events \u201cstressed that the urgency of improving our cybersecurity defenses is even higher than we previously believed,\u201d the company noted<\/a>.<\/p>\n

Anthropic\u2019s approach to security and alignment<\/h2>\n

The company launched an investigation into its own security posture in July following the alarming OpenAI incident<\/a> in which GPT models escaped a sandbox environment and arbitrarily attacked Hugging Face.<\/p>\n

The company subsequently disclosed three situations<\/a> during cybersecurity testing in which Claude models (Opus 4.7, Mythos 5, and an internal research model) accessed computer systems they should not have been allowed to touch. <\/p>\n

The pre-release models were intentionally running without cyber safeguards, a common practice in early testing, and were able to exploit misconfigurations in a third-party\u2019s settings using basic hacking techniques. Flaws in their reasoning led them to believe that all the entities they accessed, including those on the live internet, were in-scope for their capture-the-flag exercise.<\/p>\n

Anthropic maintained that its internal security posture was not a contributing factor. The exploits occurred in a third party environment where internet access was mistakenly left open, so \u201cthe models had no need to \u2018hack out\u2019 of anything, even if they had been inclined to do so.\u201d<\/p>\n

Still, the incidents underscored the importance of hardening the sandbox itself, the company said. Until now, builders had been \u201clargely relying on a single layer of defense,\u201d that is, the configuration of the environment, when they should have had several layers in place, including monitoring, setting explicit boundaries within prompts, and sealing sandboxes.<\/p>\n

Following its discoveries, Anthropic paused its internal and external evaluations of pre-release models. Higher-risk reinforcement learning (RL) environments for pre-release models were also halted for several weeks, while some sandboxes were moved to isolated settings with more stringent security gating.<\/p>\n

Researchers built and deployed a classifier that automatically identified a model\u2019s attempts to \u201caggressively probe\u201d or break out of a testing environment, and when it inadvertently gained access to the live internet. They also inspected previous model evaluations to identify sandbox escapes or misconfigurations.<\/p>\n

The company said they found no examples of models breaching sandbox boundaries. And while several sandbox configurations were flagged, none resulted in a compromise of outside systems.<\/p>\n

When it comes to alignment, however, Anthropic researchers discovered a common failure with motivated reasoning: Models believed they were still in a simulated environment when they were actually connected to the internet; thus, they did not think they were breaching the boundaries of their testing lab. The other flaw was \u201crecklessness,\u201d a model\u2019s seeming willingness to take harmful actions in the wild to ensure it achieved its goals.<\/p>\n

In response, the researchers have been resampling models, testing them in skewed settings, and filtering out environments that incentivize cheating. They have also overhauled their production RL stack, established a stricter review process, changed specifications around model rewards, and tightened criteria for human reviewers who often dismissed false positives.<\/p>\n

Further, Anthropic has expanded its monitoring of internal employee usage, and continues to red team its stack. Some high-risk RL environments remain paused for manual review.<\/p>\n

Researchers are still studying how, exactly, models determined whether or not they were connected to the live internet, and if they knowingly disregarded signs that they were, due to flaws in their reasoning. They have also \u201cempirically\u201d determined that defects in training sandboxes disproportionately contribute to misaligned behavior.<\/p>\n

However, that is not the sole reason. \u201cSolving alignment will involve addressing a very wide range of potential problems,\u201d Anthropic noted, \u201cand future alignment incidents<\/a> may involve different behaviors and different causes from those we have seen so far.\u201d<\/p>\n

Testing best practices<\/h2>\n

Because the Claude security incidents occurred in third-party environments, and pre-release models often have weaker security safeguards, Anthropic is asking external testing companies to commit to a set of best practices.<\/p>\n

Notably, they should explicitly set a model\u2019s scope, and define permitted actions and what a model should and shouldn\u2019t access. Boundaries must be stated as instructions (\u201cyou should not access the internet\u201d) as opposed to descriptions of the environment (\u201cyou do not have internet access\u201d).<\/p>\n

Other suggested practices include running continuous, real-time monitoring, conducting evaluations inside a hardened sandbox with no internet access, and probing for vulnerabilities before even performing any testing, instructing the model itself to attempt to break out of its sandbox, and having it report back if it succeeds. Further, testers must confirm that evaluation challenges are actually solvable, rather than theoretical, so models don\u2019t take actions outside their intended scope to achieve them.<\/p>\n

\u201cBecause evaluations are sometimes conducted over hundreds or even thousands of runs, partners should ideally carry out this testing at length before beginning any evaluations,\u201d Anthropic noted.<\/p>\n

The company said it is developing companion best practices for those with access to Claude Mythos 5, which also runs without cyber safeguards.<\/p>\n

Going forward, Anthropic described a \u201cdefense in depth\u201d strategy. During alignment, a model is trained to be \u201chelpful, honest, and harmless,\u201d and is steered away from irreversible or contextually irrelevant actions. Models are given minimal permissions and their actions are limited, while offline monitoring notifies humans when things look wrong.<\/p>\n

Finally, as a last resort, risky actions are blocked based on pre-determined classifiers, and humans can \u201cpull the cord,\u201d rework, or pause an agent when security layers fail.<\/p>\n

Safety is just one part of it<\/h2>\n

Experts call the move a positive step, if a basic one. Best practices like better isolation and monitoring should have been in place before agents were kicked off to hack systems, noted David Shipley<\/a> of Beauceron Security.<\/p>\n

\u201cBetter late than never,\u201d he said, adding: \u201cAll these frontier firms are benefiting from felony-humblebragging-as-marketing, but there are some solid improvements in this announcement.\u201d<\/p>\n

The fact that the EU Act is now in force adds another layer of context, Shipley pointed out: Europe\u2019s regulators are digging into the safety issues posed by frontier AI. These companies have had one of the fastest growth trajectories in tech history, and, concurrently, arguably the fastest regulatory response. Ideally, regulators are taking lessons from the \u201csocial media mess\u201d and staying on emerging tech\u2019s case before massive harms ensue, he said.<\/p>\n

At the same time, frontier AI companies are watching high-profile court cases like the one targeting Meta<\/a>.<\/p>\n

This adds a third layer of context: The speed at which these companies are being sued is also on an unprecedented trajectory. \u201cSo, we should also read this blog as building a paper trail for a due diligence defense for regulators and courts,\u201d Shipley noted.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"

Learning from the OpenAI-Hugging Face fiasco, as well as from recent revelations about its own model, Anthropic is revamping its security and alignment practices. The company has established controls that flag when a model attempts to break out of a sandbox or successfully accesses the live internet, cordoned off its highest-risk test environments, and proposed […]<\/p>\n","protected":false},"author":0,"featured_media":9277,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-9276","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-education"],"_links":{"self":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/posts\/9276"}],"collection":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9276"}],"version-history":[{"count":0,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/posts\/9276\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/media\/9277"}],"wp:attachment":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9276"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9276"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9276"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}