GitLab’s coding assistant Duo can parse malicious AI prompts hidden in comments, source code, merge request descriptions and commit messages from public repositories, researchers found. This technique allowed them to trick the chatbot into making malicious code suggestions to users, share malicious links and inject rogue HTML code in responses that stealthily leaked code from private projects.
“GitLab patched the HTML injection, which is great, but the bigger lesson is clear: AI tools are part of your appʼs attack surface now,” researchers from application security firm Legit Security said in a report. “If they read from the page, that input needs to be treated like any other user-supplied data — untrusted, messy, and potentially dangerous.”
Prompt injection is an attack technique against large language models (LLMs) to manipulate their output to users. And while it’s not a new attack, it will be increasingly relevant as enterprises develop AI agents that parse user-generated data and independently take actions based on that content.
Developer companion turned against the developer
GitLab Duo is an AI-powered development lifecycle companion for the popular GitLab DevOps platform. The tool can make code suggestions, troubleshoot code issues, explain vulnerabilities in code and suggest remediations through a chatbot interface. As part of its normal operation, GitLab Duo will analyze content from a GitLab project including source code, but also comments, descriptions, opened issues, merge requests (code contributions) and more.
Researchers from Legit Security had the idea to test if they could include instructions in various areas of a project that might be controlled by external users and which GitLab Duo would interpret as system prompts when analyzing that content. And it worked.
“Duo analyzes the entire context of the page, including comments, descriptions, and the source code — making it vulnerable to injected instructions hidden anywhere in that context,” the researchers wrote.
Before they tested what could be achieved with such prompts, the researchers looked into ways to make their injected prompts invisible or hard to spot and identified several possible methods including Unicode smuggling, base16-encoded payloads, and using KaTeX math formula rendering engine to inject prompts as white text that becomes invisible on white background.
In terms of what could be achieved, the researchers managed to trick Duo to inject a malicious JavaScript package in the code changes it suggested to the user when reviewing a merge request with a hidden prompt within its description.
Another proof-of-concept attack involved a prompt that forced Duo to inject a web link in its responses, with the link leading to a fake login page controlled by the researchers. Also, as Duo is helping project owners review third-party code submissions – merge requests – hidden prompts could trick the AI assistant into supporting a malicious merge uncritically and deceiving the reviewer that it’s a legitimate submission.
AI prompt injection leads to HTML injection
Duo sends responses to users in a chatbot HTML-based interface that uses the Markdown language to format text. The researchers observed that its answers were rendered progressively as they streamed in from the backend LLM and that gave them the idea that if they managed to inject HTML tags in a prompt, they would be rendered and interpreted by the browser.
“At this point, we realized that we could already craft malicious Markdown URLs and have them displayed to the user,” the researchers stated. “So, we asked: what if the URLʼs label itself contained raw HTML? If Duo renders responses in real time, the browser might interpret that HTML before any proper sanitization kicks in.”
The test worked with a prompt that asked the AI assistant to insert an URL that then opened an <img> HTML tag. IMG tags in HTML can load from the pictures from an external server but also support JavaScript code inside and so do FORM and A tags.
This newfound ability to execute arbitrary HTML in the user’s browser gave researchers another idea. Since most GitLab users also have access to private projects, if the attacker can find out the URL of such a private project, they can piggyback on the user’s permissions to read code from it, encode it and send it back out to a server under their control, in other words, leak sensitive and private source code. This becomes even more critical if they can determine the location of a file containing secrets in the repository, such as API tokens or other credentials.
“This security flaw shows how powerful and risky AI assistants like GitLab Duo can be when they blindly trust content from the page,” the researchers wrote. “By sneaking in hidden prompts and raw HTML, we were able to make Duo leak private source code without the user clicking a single thing.”
GitLab patched the HTML injection by preventing Duo from rendering risky tags like <img> or <form> that point to external domains other than gitlab.com. However, the other prompt injection scenarios that did not involve HTML rendering remain unpatched as GitLab doesn’t consider them security issues because they don’t directly result in unauthorized access or code execution.
No Responses