In their race to achieve productivity gains from generative AI, most organizations overlook the security implications of doing so, instead favoring hopes of game-changing innovations over sound security practices.
According to a study from the World Economic Forum conducted in collaboration with Accenture, 63% of enterprises fail to assess the security of AI tools before deployment, introducing a range of risks to their enterprise.
That includes both off-the-shelf AI solutions and in-house implementations created in collaboration with software development teams that, according to Tricentis’ 2025 Quality Transformation Report, are overwhelmingly focused on improving delivery speed (45%) over enhancing software quality (13%) — even as a third (32%) of respondents to Tricentis’ survey admit that poor quality software will likely result in more frequent security breaches or compliance failures.
And more frequent those breaches and failures are. Cisco’s latest Cybersecurity Readiness Index, published on May 7, found that 86% of organizations had experienced an AI-related security incident in the past year. Less than half (45%) believe their organization has the internal resources and expertise to conduct comprehensive AI security assessments.
Most common overlooked AI security risks
The failure to adequately test AI systems before deployment exposes organizations to a range of vulnerabilities that differ significantly from traditional software risks, according to experts quizzed by CSO. Here are some of the most prevelant.
Data exposure
AI systems often process large volumes of sensitive information, and without robust testing, organizations may overlook how easily this data can be leaked, either through unsecured storage, overly generous API responses, or poor access controls.
“Many AI systems ingest user data during inference or store context for session persistence,” says Dr. Peter Garraghan, chief executive and co-founder of AI security testing vendor Mindgard. “If data handling is not audited, there is a high risk of data leakage through model output, log exposure, or misuse of fine-tuned datasets. These risks are exacerbated by LLM [large language model] memory features or streaming output modes.”
Model-level vulnerabilities
These include prompt injection, jailbreaks, and adversarial prompt chaining. Without rigorous testing, models can be manipulated to bypass output constraints, leak sensitive data, or perform unintended tasks.
“These attacks often exploit flaws in the model’s alignment mechanisms or its reliance on token-level reasoning,” Garraghan, a lecturer at the UK’s Lancaster University, explained.
Model integrity and adversarial attacks
Without testing for adversarial manipulation or poisoned training data, it’s easy for attackers to influence how an AI model behaves, especially if it’s being used to support business decisions or automate sensitive tasks.
Jano Bermudes, COO of the global cyber consultancy CyXcel, says: “Attackers can manipulate input data to deceive AI models, causing them to make incorrect decisions. This includes evasion attacks and data poisoning.”
Systemic integration risks
AI models are frequently deployed as part of larger application pipelines, such as through APIs, plugins, or retrieval-augmented generation (RAG) architectures.
“Insufficient testing at this level can lead to insecure handling of model inputs and outputs, injection pathways through serialized data formats, and privilege escalation within the hosting environment,” Mindgard’s Garraghan says. “These integration points are frequently overlooked in conventional AppSec [application security] workflows.”
Access control failures
AI tools often plug into wider systems and, if misconfigured, can give users or attackers more access than intended. This might include exposed API keys, poor authentication, or insufficient logging that makes it hard to spot abuse.
Runtime security failures
AI systems may exhibit emergent behaviors only during deployment, especially when operating under dynamic input conditions or interacting with other services.
“Vulnerabilities such as logic corruption, context overflow, or output reflection often appear only during runtime and require operational red-teaming or live traffic simulation to detect,” according to Garraghan.
Compliance violations
Failing to ensure AI tools meet regulatory standards can lead to legal repercussions.
For example, regulatory violations might occur due to unauthorized data processing by AI tools or outages from untested model behaviours under scale.
Broader operational impacts
“These technical vulnerabilities, if left untested, do not exist in isolation,” Mindgard’s Garraghan says. “They manifest as broader organizational risks that span beyond the engineering domain. When viewed through the lens of operational impact, the consequences of insufficient AI security testing map directly to failures in safety, security, and business assurance.”
Sam Peters, chief product officer at compliance experts ISMS.online, sees widespread operational impacts from organziations’ tendency to overlook proper AI security vetting.
“When AI systems are rushed into production, we see recurring vulnerabilities across three key areas: model integrity (including poisoning and evasion attacks), data privacy (such as training data leakage or mishandled sensitive data), and governance gaps (from lack of transparency to poor access control),” he says.
Peters adds: “These issues aren’t hypothetical; they’re already being exploited in the wild.”
Test against delivery
The rush to implement AI puts CISOs in a stressful bind, but James Lei, chief operating officer at application security testing firm Sparrow, advises CISOs to push back on the unchecked enthusiasm to introduce fundamental security practices into the deployment process.
“To reduce these risks, organizations should be testing AI tools in the same way they would any high-risk software, running simulated attacks, checking for misuse scenarios, validating input and output flows, and ensuring that any data processed is appropriately protected,” he says.
To mitigate these risks, organizations should implement comprehensive testing strategies, such as:
Penetration testing: Simulating attacks to identify vulnerabilities
Bias and fairness audits: Ensuring AI decisions are equitable and non-discriminatory
Compliance checks: Verifying adherence to relevant regulations and standards
By integrating security testing into the AI development lifecycle, organizations can harness the benefits of AI while safeguarding against potential threats.
“Before deploying AI tools, organizations should be conducting threat modelling specific to AI systems, red-teaming for adversarial inputs, and robust testing for model drift and data leakage,” ISMS.online’s Peters says. “At the same time, they should integrate AI-specific controls into their risk management and compliance programs.”
Peters adds: “This is where the new ISO/IEC 42001 standard can really help. It provides a framework for governing AI responsibly, including guidance on risk assessment, data handling, security controls, and continuous monitoring.”
Other experts, while validating the need for security testing, argued that a different approach needs to be applied in testing the security of AI-based systems.
“Unlike regular software testing, you can’t just look at the code of a neural network to see if it’s secure,” Inti De Ceukelaire, chief hacker officer at crowdsourced security provider Intigriti, tells CSO. “Even if it’s trained on clean, high-quality data, it can still behave in strange ways. That makes it hard to know when you’ve tested enough.”
AI tools often offer a complex solution to a simple problem. Testers might focus only on what the tool is supposed to do and miss other things it can do. “For example, a translation tool could be tricked into opening a PDF with malicious code or accessing internal files and translating them for someone outside the company,” De Ceukelaire explains.
Organizations should consider implementing adversarial testing frameworks designed specifically for AI.
“This includes static model analysis, dynamic prompt fuzzing, integration-layer attack simulation, and runtime behavioural monitoring,” Mindgard’s Garraghan says. “These practices should be embedded into the AI deployment lifecycle in the same way that DevSecOps practices are integrated into software CI/CD pipelines.”
No Responses