{"id":116,"date":"2024-08-30T14:10:22","date_gmt":"2024-08-30T14:10:22","guid":{"rendered":"https:\/\/cybersecurityinfocus.com\/?p=116"},"modified":"2024-08-30T14:10:22","modified_gmt":"2024-08-30T14:10:22","slug":"llms-fueling-a-genai-criminal-revolution-according-to-netcraft-report","status":"publish","type":"post","link":"https:\/\/cybersecurityinfocus.com\/?p=116","title":{"rendered":"LLMs fueling a \u201cgenAI criminal revolution\u201d according to Netcraft report"},"content":{"rendered":"
\n
\n
\n
\n
<\/div>\n

Like seemingly everyone else, threat actors are increasingly adopting generative artificial intelligence (genAI) as a business tool. Recent findings by security researchers at Netcraft have revealed what it called \u201ca mass universal scaling up of genAI being used as a content creation tool for fraudulent websites.\u201d<\/p>\n

In a blog post<\/a> published Thursday, Netcraft noted that it has been identifying thousands of websites per week that use AI-generated content, with steady growth in the technology\u2019s use. In late July, however, there was a spike in the number of sites that continued into the first week of August before subsiding.<\/p>\n

Netcraft attributed this to a single threat actor who was setting up fake shopping sites and using genAI to write product descriptions.<\/p>\n

\u201cThis and the broader growth in activity between March and August appears to indicate a mass universal scaling up of genAI being used as a content creation tool for fraudulent websites, with a notable spike showing in the realm of online stores,\u201d Netcraft said in its post.<\/p>\n

Malicious content is becoming more convincing<\/h2>\n

\u201cThis has led to an abundance of malicious websites, attracting victims not only because of the sheer volume of content but also because of how convincing that content has become.\u201c<\/p>\n

It is no longer possible, the report said, to decide that a website or email is legitimate simply because it\u2019s written in professional English.<\/p>\n

However, there can be clues in the email or on the site. Netcraft said that sometimes threat actors accidentally include large language model (LLM) outputs in the fraudulent emails. For example, a phishing email it encountered, claiming to contain a link to a file transfer of family photos, also included the phrase, \u201cCertainly! Here are 50 more phrases for a family photo.\u201d<\/p>\n

\u201cWe might theorize that threat actors, using ChatGPT to generate the email body text, mistakenly included the introduction line in their randomizer,\u201d Netcraft said. \u201cThis case suggests a combination of both genAI and traditional techniques.\u201d<\/p>\n

Telltale evidence still shows which phishing emails are fake<\/h2>\n

Another phishing email it viewed would have been credible \u2014 had it not been for the sentence at the beginning, which included the LLM introduction line, \u201cCertainly, here\u2019s your message translated into professional English.\u201d\u00a0And a fake investment website touting the phoney company\u2019s advantages looked good, except for the headline saying, \u201cCertainly! Here are six key strengths of Cleveland Invest Company.\u201d<\/p>\n

\u201cThere\u2019s no honor among thieves, of course,\u201d Netcraft observed. \u201cJust as criminals are happy to siphon credentials from other phishing sites, we\u2019ve observed that when they see a convincing LLM-generated template, they may replicate the content almost verbatim.\u201d<\/p>\n

For example, the \u201cCleveland Invest\u201d website text was replicated (complete with LLM response) from another fake text created for \u201cBritannic Finance\u201d. In this case, the threat actor appeared to then use an LLM to adjust the text, using synonyms for some terms.<\/p>\n

Netcraft has also seen LLM-generated sites for fake shops and fake pharmacies designed for search engine optimization (SEO), to pull in more victims. Again, it cited a site on which the LLM\u2019s response to the request was leaked on the site, with discussion points followed by \u201cthis outline should give you a good start \u2026\u201d, and a reminder to include SEO keywords in the title, headings, and body of the text.<\/p>\n

And all this is just the tip of an ever-growing iceberg. \u201cThe report speaks to only one area of cyber threat that\u2019s being augmented by generative AI capabilities: gaining initial access to a victim, namely through phishing,\u201d said\u00a0Brian Jackson, principal research director at Info-Tech Research Group.<\/p>\n

\u201cUnfortunately, that\u2019s only one small part of the full scope of augmented threats we\u2019re seeing, thanks to LLMs,\u201d Jackson says. \u201cWhole new taxonomies of cyber threat techniques are being added to threat frameworks thanks to LLMs.\u201d<\/p>\n

LLMs are being used to conduct reconnaissance<\/h2>\n

Examples include using LLMs to conduct reconnaissance, such as searching and summarizing a potential victim\u2019s publicly available materials and potential vulnerabilities. \u201cOpenAI has banned state-sponsored accounts<\/a>\u00a0for doing exactly this,\u201d Jackson says. \u201cThen, there is the attempt to exploit LLMs themselves through prompt injection and jailbreak, etc.\u201d He pointed to an exhaustive list of techniques via MITRE ATLAS<\/a>.<\/p>\n

This coincides with Netcraft\u2019s findings. \u201cThere are many more [examples], with conclusive evidence pointing to the large-scale use of LLMs in more subtle attacks,\u201d the post said. \u201cThe security implication of these findings is that organizations must stay vigilant; website text written in professional English is no longer a strong indicator of its legitimacy. With genAI making it easier to trick humans, technical measures like blocking and taking down content are becoming increasingly critical for defending individuals and brands.\u201d\u00a0<\/p>\n

And, said Jackson, \u201cfrom my perspective, it\u2019s not the same old threats being augmented with AI that are most alarming. We already have defined techniques to help mitigate those. Rather, it\u2019s the net new cyber threats from generative AI that could really catch organizations off guard.\u201d<\/p>\n

\u201cAs we\u2019ve already seen, most of us expect that when an executive video calls us, we can trust that it\u2019s really them giving us instructions, Jackson says. \u201cThat\u2019s just no longer the case, as generative AI can effectively make deepfakes with limited available training data.\u201d<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"

Like seemingly everyone else, threat actors are increasingly adopting generative artificial intelligence (genAI) as a business tool. Recent findings by security researchers at Netcraft have revealed what it called \u201ca mass universal scaling up of genAI being used as a content creation tool for fraudulent websites.\u201d In a blog post published Thursday, Netcraft noted that […]<\/p>\n","protected":false},"author":1,"featured_media":117,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-116","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-education"],"_links":{"self":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/posts\/116"}],"collection":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=116"}],"version-history":[{"count":0,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/posts\/116\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=\/wp\/v2\/media\/117"}],"wp:attachment":[{"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=116"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=116"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cybersecurityinfocus.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=116"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}