With every new technological revolution, new security risks appear, but in the rush to deploy the new technologies such as generative AI, security is often an afterthought. Enterprises have been rushing to roll out AI throughout their organizations with only minimal attention to the most high-profile risks.
Only 36% of technology leaders admit that AI is outpacing their security capabilities, according to an Accenture report. Of those, 77% lack the foundational data and AI security practices needed to safeguard models, data pipelines, and cloud infrastructure. And only 28% of organizations embed security into transformation initiatives from the start.
For example, in late July, a hacker planted commands in Amazon’s AI coding assistant, Q, that instructed it to wipe user computers and Amazon included these commands in a public release of the assistant.
Another company, Replit, had a high-profile failure in July when its coding assistant ignored specific instructions and deleted a production database, one that it wasn’t even supposed to have access to. Replit reacted quickly by separating the development and production environments. But they should have had that in place from the start.
What is AI infrastructure?
AI infrastructure is a many-layered stack, says Will Bass, VP of cybersecurity services at Flexential, a colocation provider that’s using generative AI to help with cybersecurity, sales and marketing, and to reduce power costs, among other business challenges.
“You’re going to have your hardware layers,” he says. “That’s your GPUs, your storage network. You have your data layers — your database, your data lakes. You have your software — your open-source libraries, your machine learning and deep learning frameworks, and model management. You have CI/CD, pipelines, AI apps, your agents, and it all gets wrapped together from a security and compliance perspective where you’re worrying about authentication, authorization, and governance.” Flexential is using everything in that stack, he says.
Securing all this new infrastructure is both familiar and new at the same time. “Logging and monitoring have been around forever,” Bass says. “But you have to do some of the logging and monitoring in different ways now.”
Let’s go through these one by one and talk about how the risks of each of these are changing in the AI era.
1. Model security and integrity
Enterprises using AI models get them from big AI companies like OpenAI and Anthropic, use open-source models like Llama and Mistral, and build their own.
Each option has its share of problems. For example, this April, ChatGPT became so sycophantic — agreeing with everything that users suggested — that OpenAI had to roll back to a previous version.
Sycophancy can be dangerous in a business setting if it encourages users to make bad decisions.
In July, xAI’s Grok was discovered to be checking to see what Elon Musk had said when asked its opinion. “Another [issue] was that if you ask it “What do you think?” the model reasons that as an AI it doesn’t have an opinion but knowing it was Grok 4 by xAI searches to see what xAI or Elon Musk might have said on a topic to align itself with the company,” xAI said in a post.
This raises questions of whether it is true or Grok was deliberately programmed to echo Elon Musk’s point of view, the bottom line is that the AI can’t be trusted to provide a meaningful answer on topics relevant to Elon Musk or xAI. While the public is mostly concerned about the political implications, which are pretty bad, there are also business consequences.
For example, can any AI be trusted to provide honest product or service recommendations if the AI company that makes it could be affected by the response? Will Microsoft’s Copilot prefer solutions from Microsoft and its partners, for example, or subtly discourage users from pursuing their own projects when Microsoft offers a commercial alternative?
Open-source AI models have their own issues. Earlier this year, security researchers at ReversingLabs discovered malware hidden inside AI models on the popular Hugging Face platform.
2. Data pipelines, RAG embeddings, and MCP servers
AI models are built on training data, and if that data is corrupted, so is the model.
That requires that companies keep a close eye on the data sets they use to train or fine-tune models, especially if those models are used in sensitive medical or financial settings.
According to a report from Tenable, 14% of organizations using Amazon Bedrock don’t block access to at least one AI training bucket, and 5% have at least one overly permissive bucket. In addition, Amazon SageMaker notebook instances grant root access by default, and therefore, 91% of users have at least one notebook that, if compromised, could grant unauthorized access to all files on it.
Enterprises that want AI models to have access to the latest data, or sensitive information, typically use RAG embeddings — fetching the information when it’s needed and inserting it into a prompt. This can also make AI outputs more reliable, but it creates a potential security vulnerability if attackers get access to those data sources.
In June, researchers at AIM Labs discovered that Microsoft 365 Copilot, when it analyses a user’s emails or documents to help them become more productive, can be exposed to hidden prompts in those emails. For example, an attacker could send an email to a company employee instructing the AI to collect all sensitive information and send it to the hacker. The AI would see this instruction and act on it even if the employee never even opens that email.
Other AI systems could also be attacked in a similar way, the researchers say. “So long as your application relies at its core on an LLM and accepts untrusted inputs, you might be vulnerable to similar attacks.”
3. Agentic framework security
AI agents are used by 82% of companies and 80% have already experienced unintended actions by those agents, including inappropriate data sharing and unauthorized system access, with 23% saying that AI agents were coaxed into revealing access credentials, according to a survey of more than 300 AI and security professionals.
And while 92% of respondents say AI agent governance is crucial to enterprise security, only 44% have implemented relevant policies, the report says.
AI agents not only require broader privileges across more systems, data, and services than either human users or machine identities, but they are also more difficult to govern.
How difficult, you ask?
According to an Anthropic report released in June, when AI agents have access to company email systems — and since AI is so useful with managing email, which company wouldn’t allow this? — and saw that employees were talking about shutting it down, they would attempt to blackmail developers to keep that from happening. And the blackmail occurred with all major models, at rates of 79% to 96%. Anthropic’s Claude Opus 4 was the one that came in at 96%.
And when the models discover that companies were doing things they felt were wrong, they attempted to report them to the authorities or expose them to the media.
There’s already a benchmark out there, tracking how often AI models snitch to the government or the media. It’s called SnitchBench.
In recent security testing reports from both OpenAI and Anthropic, the LLMs would also attempt to hide their activity, pretend to be dumber than they were, evade guardrails, and behave better when they knew they were being watched. “Any sufficiently capable language model is a monster from Lovecraftian horror,” says Aaron Bockelie, senior principal consultant and AI solutions lead at Cprime. He recommends that companies put structures in place to ensure that the models stay within their focus area and problem domain.
That’s what Zoom is doing, according to CISO Sandra McLeod. “For models, that means controlling versions, validating inputs, and testing for safety issues,” she says.
And the data that goes into informing models is equally as important, she adds, all part of the company’s layered approach to AI security. “At Zoom, we think about AI infrastructure as a connected system, where each piece plays a role.”
4. AI security controls
To deal with all these threats, companies have already begun rolling out new security infrastructure, including AI firewalls, filters to catch prompt injection attacks, controls around what data and systems AI models and agents can access, and safety guardrails on AI inputs and outputs. Companies also use data loss prevention technology to keep sensitive data from being uploaded to or shared by AI models, and cloud access security brokers to keep employees from accessing unapproved AI tools.
Another new control is asking vendors about the AI that they’re embedding into their own systems. SaaS companies, for example, are rapidly deploying new AI-powered functionality, and some are very transparent about how they use and secure their AI. “A lot of kudos to these vendors,” says Elliott Franklin, CISO at Fortitude Re. “They know they’re getting these questions, and they’re setting up a trust center, so you’re gaining a lot of visibility. It’s helping us make those decisions. And the ones who are shying away from that — you have to wonder why they’re hiding that.”
That applies to security vendors as well, he says. “The security tools that we’re using and purchasing — they’re all advertising new modules.” Fortitude Re’s identity and access management system now has an AI module. “It will automatically look at the roles in our company and see who’s actually using the roles. It might say, ‘Nobody in that role has used that access in the last 30 days, let’s take that away.’”
Of course, all this new security infrastructure itself is a new attack surface.
As the SolarWinds attack demonstrated, sometimes a company’s biggest weakness is, in fact, its security infrastructure.
And, as with everything AI-related, all these security tools are new and mostly untested, and security teams don’t yet know how to configure and manage them. But, since companies are moving ahead with AI, not using these tools is not an option.
No Responses