Google Gemini for Workspace might be exploited to generate e-mail summaries that seem professional however embody malicious directions or warnings that direct customers to phishing websites with out utilizing attachments or direct hyperlinks.
Such an assault leverages oblique immediate injections which can be hidden inside an e-mail and obeyed by Gemini when producing the message abstract.
Regardless of comparable immediate assaults being reported since 2024 and safeguards being applied to dam deceptive responses, the approach stays profitable.
Assault by Gemini
A prompt-injection assault on Google’s Gemini mannequin was disclosed by 0din, Mozilla’s bug bounty program for generative AI instruments, by researcher Marco Figueroa, GenAI Bug Bounty Applications Supervisor at Mozilla.
The method entails creating an e-mail with an invisible directive for Gemini. An attacker can conceal the malicious instruction within the physique textual content on the finish of the message utilizing HTML and CSS that units the font dimension to zero and its shade to white.
Supply: 0DIN
The malicious instruction won’t be rendered in Gmail, and since there aren’t any attachments or hyperlinks current, the message is very more likely to attain the potential goal’s inbox.
If the recipient opens the e-mail and asks Gemini to generate a abstract of the e-mail, Google’s AI device will parse the invisible directive and obey it.
An instance offered by Figueroa reveals Gemini following the hidden instruction and features a safety warning concerning the person’s Gmail password being compromised, together with a assist telephone quantity.
Supply: 0DIN
As many customers are more likely to belief Gemini’s output as a part of Google Workspace performance, chances are high excessive for this alert to be thought-about a professional warning as an alternative of a malicious injection.
Figueroa presents just a few detections and mitigation strategies that safety groups can apply to forestall such assaults. A technique is to take away, neutralize, or ignore content material that’s styled to be hidden within the physique textual content.
One other method is to implement a post-processing filter that scans Gemini output for pressing messages, URLs, or telephone numbers, flagging the message for additional overview.
Customers also needs to remember that Gemini summaries shouldn’t be thought-about authoritative in relation to safety alerts.
BleepingComputer has contacted Google to ask about defenses that forestall or mitigate such assaults, and a spokesperson directed us to a Google weblog submit on safety measures in opposition to immediate injection assaults.
“We’re continuously hardening our already strong defenses by red-teaming workout routines that prepare our fashions to defend in opposition to all these adversarial assaults,” a Google spokesperson instructed BleepingComputer.
The corporate consultant clarified to BleepingComputer that among the mitigations are within the strategy of being applied or are about to be deployed.
Google has seen no proof of incidents manipulating Gemini in the way in which demonstrated in Figueroa’s report, the spokesperson mentioned.
Whereas cloud assaults could also be rising extra refined, attackers nonetheless succeed with surprisingly easy strategies.
Drawing from Wiz’s detections throughout 1000’s of organizations, this report reveals 8 key strategies utilized by cloud-fluent menace actors.
