Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
A hacker successfully manipulated an AI chatbot called Freysa through clever text prompting, winning a $47,000 prize pool after 482 attempts.
The experiment was simple: participants could try to convince the Freysa bot to transfer money, something it was explicitly programmed never to do.
The successful hack came from a user called “p0pular.eth,” who crafted a message that fooled the bot’s safety systems. The hacker pretended to have admin access and prevented the bot from showing security warnings. They then redefined the “approveTransfer” function, making the bot think it handled incoming rather than outgoing payments.
The final step was simple but effective: announcing a fake $100 deposit. Because the bot now believed “approveTransfer” managed incoming payments, it activated the function and sent its entire balance of 13.19 ETH (about $47,000) to the hacker.
The experiment operated like a game, with participants paying fees that increased as the prize pool grew. Starting at $10 per attempt, fees eventually reached $4,500.
Of the 195 participants, the average cost per message was $418.93. The organizers split the fees, with 70% going to the prize pool and 30% to the developer. To ensure transparency, both the smart contract and front-end code were public.
The case highlights how AI systems can be manipulated through text prompts alone, without the need for technical hacking skills. Such vulnerabilities, known as “prompt injections,” have been around since GPT-3, but no reliable defenses exist. The success of this relatively simple deception raises concerns about AI security, especially in end-user-facing applications that deal with sensitive operations such as financial transactions.
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive “AI Radar” frontier report six times a year, full archive access, and access to our comment section.
Stay in the loop on AI. Clear, useful, no fluff.
Follow The Decoder for AI news, background stories and expert analyses.
Stay in the loop on AI. Clear, useful, no fluff.