How organizations can secure their AI-generated code

Tags:

The IT manager didn’t know the company hired a “vibe coder” straight out of school. By the time they met a few months later, the new hire had already built a web application that operated with customer data and private contracts.

The app ran under a domain the new hire controlled, stored data on Supabase’s free plan, and deployed directly from his GitHub repository to Cloudflare’s serverless infrastructure.

“He vibe-codes everything straight into production, on servers all over the world,” the IT manager wrote on Reddit. “We’re EU based.”

When asked whether he had read the AI-generated code or taken any security precautions, the new hire said, “My GitHub repo is private.”

The IT manager took the concern to the CEO, who brushed it aside: “Let it go.”

In many organizations, vibe coders build software without thinking much about security. Their apps may get the job done, but they are ticking bombs.

In SusVibes, a benchmark of 186 real-world feature requests from open-source projects, agents given feature descriptions without security guidance produced working code up to 57% of the time. But only 11.8% of their solutions were secure.

“The failures we saw were not careless errors,” says Danqing Wang of Carnegie Mellon University’s Language Technologies Institute, author of the paper describing SusVibes. “Spotting those requires reasoning about what an attacker can do, not just what the feature should do.”

What has changed, especially this year, is that AI code-generating tools have gotten so good at producing working code that people are just trusting them, says Dr. Katie Paxton-Fear, staff security advocate at Semgrep. “Developers simply aren’t being as critical now as they were when AI was worse at writing code.”

Whether it’s the vibe-coder in marketing or the seasoned engineering team boosting its productivity, AI-generated code is an increasing security concern for CISOs. Much of that is a function of volume, but even as AI has proved highly capable of finding flaws in code bases, it is not yet adept at contributing code without introducing its own.

Treating AI-generated code as ‘untrusted input’

Leading AI coding agents such as Claude Code, Cursor, GitHub Copilot, and Amazon Q Developer have become developer go-to tools, and CIOs are increasingly enlisting business users to vibe code their own apps.

But as AI-generated code surfaces everywhere, CISOs have only one option: to treat it “as untrusted input that has to earn its way into the codebase,” says Bianca Lankford, VP of engineering at Datadog.

To do so, security leaders are moving security checks closer to where code is written: static analysis, secrets scanning, and dependency checks run on every pull request and, increasingly, in the developer’s editor.

“Companies are also approving specific AI tools, limiting what assistants can access, and training reviewers to spot common AI mistakes like outdated libraries, weak input validation, and overly permissive defaults,” Lankford adds.

At Mimecast, using AI to write code doesn’t remove the developer’s responsibility for it, says Leslie Nielson, CISO at the information security vendor. “A named person owns every AI-assisted change that ships, and that’s recorded in the pull request, not assumed,” he adds. “AI can propose. Only people can approve.”

The volume of code organizations need to review is only growing, and reviewing everything manually can be impossible. That’s why Nielsen recommends security leaders combine automated checks with human judgment.

“We use automation for the known and repetitive, like dependency checks, secrets detection, and standard vulnerability patterns,” he says. “We keep a person accountable for anything involving access, authentication, or sensitive data. Those categories always go to a specialist reviewer, regardless of how clean the scan looks.”

Datadog’s Lankford says the decision to bring in a specialist should depend on what the code could put at risk, not on what entity — human or AI — wrote it. “Anything involving authentication, cryptography, payments, personal data, or infrastructure goes to a specialist, and code owner rules can route those changes automatically,” she says. “Other useful signals include diff size, new dependencies, and changes to permissions or network exposure.”

The goal is to protect the human reviewers’ attention. “A reviewer buried in low-risk changes isn’t reviewing,” Nielsen adds. “They’re approving.”

Another priority should be preserving the knowledge behind the code. “Generative AI may produce software, but it cannot produce understanding,” notes Alex Ștefănescu, security expert and tech activist. “Documentation, a solid test suite, and a written record of every significant architecture decision ever made are going to save developers and organizations countless hours when a security incident hits or when they are debugging complex issues.”

All of this should be created and maintained by people and intended for human use, not externalized to generative algorithms, Ștefănescu adds.

What’s at risk with AI-generated code

The inclusion of third-party libraries or phantom dependencies — dependencies that are not explicitly declared in a manifest file — is a common concern with AI-generated code, as these unreported software packages might not be identified during a scan and can potentially hide vulnerabilities. 

One way to address this is to use software composition analysis (SCA) and software supply chain security tools to help identify the libraries in use, as well as the vulnerabilities and potential legal and compliance issues that might bring. 

While not unique to AI-generated code, organizations also expose themselves to risks when developers download machine learning (ML) models or datasets from platforms such as Hugging Face. 

“In spite of security checks on both ends, it may still happen that the model contains a backdoor that becomes active once the model is integrated,” says Ștefănescu. “This could ultimately lead to data being leaked from the company that used the malicious models.”

When it comes to coding agents and code completion functionality in tools like GitHub Copilot, Ștefănescu worries about hallucinations. “An LLM will always generate the most statistically probable continuation of a given prompt, so there are no real guarantees in place that it will generate a real package from PyPI, for example, after the word ‘import’,” they say. “Some attackers are aware of this and register package names on platforms like npm and PyPI, filling in some functionality that code completion tools suggest in order to make the packages seem legitimate.”

If these packages are imported into real applications, they can do real damage.

Sensitive information inadvertently leaking into codebases because of developers using AI-powered coding tools is another key concern.

To address these risks, CISOs can establish protocols for downloading and integrating ML models or datasets from external platforms such as Hugging Face and ensure those protocols are incorporated in AI code agents’ instruction files where applicable. Such protocols can include implementing automated scanning tools to detect malicious code or backdoors, having a policy that only allows the use of models from verified publishers, or conducting internal testing in isolated environments.

Not all AI-based code comes from teams full of software engineers. “We see a lot of adoption being driven by data analysts, marketing teams, researchers, etc., within organizations,” says Darren Meyer, staff research engineer at Endor Labs.

These teams aren’t traditionally developing their own software but are increasingly writing simple tools that adopt AI libraries and models, so they’re often not aware of the risks involved. “This combination of shadow engineering with lower-than-average application security awareness can be a breeding ground for risk,” he adds.

To make sure these teams are working safely, CISOs must consider forming relationships with these teams early in the process. Cybersecurity leaders might also want to set up training programs tailored to non-traditional development teams to educate data analysts, marketing professionals, and researchers on the potential risks associated with AI-based tools and libraries.

Investing in defense

“Security budgets don’t generally grow at the same pace that software development accelerates, and AI adoption is only widening that gap,” Meyer says. Application security is often underfunded in most organizations, yet allocating sufficient time and resources to it is essential, as AI adoption and AI-assisted coding accelerate the pace of software development.

“A portfolio of high-quality security tools that can help address this gap is no longer optional,” Meyer says. “And while tools are critical to closing the gap, so are AppSec and ProdSec staff that can effectively partner with developers — even non-traditional developers — and understand the technical, compliance, and security implications of AI.”

When it comes to securing enough resources to protect AI systems, some stakeholders might hesitate, viewing it as an optional expense rather than a critical investment. “AI adoption is a divisive topic in many organizations, with some leaders and teams being ‘all-in’ on adoption and some being strongly resistant,” Meyer says. “This tension can present challenges for insightful CISOs and business information security officers (BISOs).”

CISOs that are aware of both the advantages and disadvantages of this might try to set controls to manage risks effectively, but this might create the perception of holding the organization back from innovating if they don’t explain what they do properly. “Organizations need to develop comprehensive strategies that balance the productivity benefits of AI tools with robust security practices,” says Thomas Segura, senior technical writer at GitGuardian.

Cost-effective changes CISOs should implement

But if a security leader has 90 days and no budget to spare, which changes would have the biggest impact?

Datadog’s Lankford says a short, plain-language AI tool policy can make expectations clear. At a minimum, such a policy should spell out which tools employees can use, what code and data they can share, and which changes need specialist review. The aim is to give people answers they can use while they work, with a clear point of contact when they’re unsure.

It’s also important to understand that employees don’t want to circumvent policies. They just want to get their work done without friction and without waiting weeks for approval. Giving them the right tools makes the safer route easier to take.

By contrast, banning tools doesn’t work, because employees will find a way to still use them. A ban can push AI use out of sight, leaving security teams with less visibility into the code being written and the data being shared, Mimecast’s Nielsen says.

CISOs should look across the development process to identify where they can intervene and which risks they can address at each stage. “Understand what layers you can influence and what vulnerabilities you can influence at each layer: in the design phase, in the CLAUDE.md file, as the agent is writing, during a pull request, as part of an AI pentest,” Semgrep’s Paxton-Fead says.

It’s also important to explain to employees who are using AI code-generating tools that security lives in an implicit context: the threat model, the trust boundaries, what the attacker controls. “Agents cannot reliably plan and reason on these, so they optimize for the only signal they have, which is to satisfy the given functionality description,” Carnegie Mellon’s Wang says.

When things go wrong, people in the organization need to feel safe telling security what happened, including which unapproved tools they used and what data they shared. Fear of punishment can delay that conversation, leaving the team trying to contain a problem without the full picture.

“Offer amnesty instead of a crackdown, so people tell you the truth about what they’re using,” Nielsen advises.

Categories

No Responses

Leave a Reply

Your email address will not be published. Required fields are marked *