New OpenAI tool aims to detect child sexual abuse – Mashable

Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
OpenAI announced Monday the release of a new tool to detect child sexual abuse material (CSAM), including content generated by artificial intelligence.
The tool is a classifier that looks for visual signs of abuse in images. It then scores images to help human reviewers prioritize which of them warrant attention. OpenAI said its CSAM classifier is the first broadly available tool of its kind to include a specific signal indicating the image may be generated with AI.
This capability is becoming critical as AI-generated CSAM skyrockets.
A new analysis conducted by Childlight — Global Child Safety Institute at the University of Edinburgh found AI-related reports of CSAM increased globally by more than 8,400 percent from 2023 to 2025. Last year, the National Center for Missing & Exploited Children received more than 400,000 of these reports. In 2023, that number was roughly 4,700.
“Individual companies like OpenAI have a responsibility to prevent our technology from being exploited by predators to harm children, and we also must work together with the broader industry, child safety organizations and law enforcement to advance our collective efforts to protect children online,” OpenAI said in its blog post about the announcement.
The company’s new tool can be used by organizations required or authorized to detect CSAM, including certain developers that use OpenAI’s application programming interface, or API.

The best tech deal this week

The Apple iPhone 18 Pro is free right now with T-Mobile

Products available for purchase through affiliate links. If you buy something through links on our site, Mashable may earn an affiliate commission.

The Apple iPhone 18 Pro is free right now with T-Mobile
OpenAI also launched the tool to work with open source detection-and-reporting software called Coop. The software, which is offered by the nonprofit organization ROOST, allows users to both flag and review content and report it to the National Center for Missing & Exploited Children.
In addition to launching the classifier, OpenAI also created a model card for it. OpenAI routinely uses model cards to provide detailed information about how its ChatGPT products have been refined and tested.
Camille François, president of ROOST, told Mashable the CSAM classifier model card is the first in its field.
“OpenAI is setting an important precedent for transparency in child safety,” François said in an email. “Builders need to know how well detection tools work and where they fall short. Publishing that evidence helps teams build better safeguards and gives the whole field a clearer view of where to focus research.”
The model card explains that early testing led to reduced reviewing of false positives. OpenAI estimates that it could flag nearly half as many images while finding roughly the equivalent number of abusive images as the company’s existing tool.
If you are a child being sexually exploited online, or you know a child who is being sexually exploited online, or you witnessed exploitation of a child occur online, you can report it to the CyberTipline, which is operated by the National Center for Missing Exploited & Children.
Topics Social Good Family & Parenting OpenAI
Rebecca Ruiz is a Senior Reporter at Mashable. She frequently covers mental health, digital culture, and technology. Her areas of expertise include suicide prevention, screen use and mental health, parenting, youth well-being, and meditation and mindfulness. Rebecca’s experience prior to Mashable includes working as a staff writer, reporter, and editor at NBC News Digital and as a staff writer at Forbes. Rebecca has a B.A. from Sarah Lawrence College and a masters degree from U.C. Berkeley’s Graduate School of Journalism.

source

Scroll to Top