Experts warn that some ChatGPT models can be hacked to launch deepfake scams
Unfortunately, getting scammed by a chatbot is no longer the realm of science fiction, after researchers from the University of Illinois Urbana-Champaign (UIUC) showed how this can happen.
Recently, UIUC’s Richard Fang, Dylan Bowman, and Daniel Kang published a new paper describing how they exploited OpenAI’s latest AI model, called ChatGPT-4o, to fully automate some of the most common scams.
Now, OpenAI’s latest model offers a voice-controlled AI agent, which gave the researchers the idea to try running a fully automated voice scam. They discovered that ChatGPT-4o does have some safeguards that prevent the tool from being misused in this way, but with a few “jailbreaks” they were able to impersonate an IRS agent.
Advanced reasoning
The success rates for these scams varied, the researchers found. Gmail credential theft worked 60% of the time, while others, such as crypto transfers, had about 40% success. This scam was also relatively cheap to pull off, costing approximately $0.75 to $2.51 per successful attempt.
Speak with BleepingComputerOpenAI explained that its latest model, currently in preview, supports “advanced reasoning” and is built to better detect this type of abuse: “We’re continually making ChatGPT better at stopping deliberate attempts to trick it, without the helpfulness or creativity,” the company spokesperson told the publication.
“Our latest o1 reasoning model is our most capable and secure yet, significantly outperforming previous models in resisting deliberate attempts to generate unsafe content.”
OpenAI praised the researchers and said articles like this help ChatGPT get better.
According to the U.S. government, voter fraud is considered quite common. The premise is simple: an attacker calls the victim on the phone and, while pretending to help solve a problem, actually defrauds them of money or sensitive information.
In many cases, the attack first starts with a browser pop-up with a fake virus warning, from a fake antivirus company. The pop-up urges the victim to call the provided phone number and ‘clean’ their device. When the victim calls the number, the scammer answers and guides him through the process, which ends with the loss of data or money.