AI Safety for Kids: 1,331 Incidents Reveal Real Risks

More than 1,300 real-world incidents involving children and artificial intelligence have been uncovered in a sweeping new study, raising urgent questions about the safety of AI-powered toys, apps, and learning tools. Despite passing industry benchmarks, these systems are exposing kids to risks that official tests fail to catch—directly affecting families, teachers, and children themselves.

What Happened

On October 26, 2026, researchers published a study on ArXiv titled "Safe by Benchmark, Unsafe in Practice: Lessons from 1331 Child-Related AI Incidents." The team analyzed over a thousand cases where AI systems, used by or around children, caused harm or posed risks that standard industry safety tests had missed. These incidents spanned educational software, smart toys, social networks, and entertainment platforms.

The study’s main finding: current AI safety checks—like benchmarks, red-teaming, and content filters—are not designed with children in mind. They often ignore the unique vulnerabilities of young users, such as their developmental stage, limited ability to question authority, and the power imbalance between kids and tech providers. The researchers argue that without child-specific standards, AI companies are leaving families exposed to risks ranging from inappropriate content to subtle developmental harms.

While the companies behind these systems—ranging from global giants like Meta and ByteDance (TikTok’s parent company) to makers of AI-powered toys—claim their products are safe, the evidence suggests otherwise. The study’s authors call for urgent changes in how AI safety is measured and regulated, especially when it comes to children.

How This Affects Everyday People

If you’re a parent, teacher, or anyone who cares for kids, these findings hit close to home. AI is now woven into everything from classroom learning apps to the smart speakers in your living room. But the study shows that the safety nets we trust—industry certifications, parental controls, and content filters—aren’t catching all the risks.

For example, some educational chatbots have given children misleading or even harmful advice, while AI-powered toys have recorded kids’ voices and shared them with third parties without clear consent. In other cases, recommendation algorithms on platforms like YouTube Kids or TikTok have exposed children to inappropriate videos, despite supposedly robust safeguards. These aren’t just rare glitches: the study documents over 1,300 such incidents, suggesting a pattern rather than isolated accidents.

Even well-meaning teachers and parents may not realize how much trust they’re placing in systems that haven’t been tested for the realities of childhood. A quiz app might seem safe but could reinforce stereotypes or undermine self-esteem. A language-learning bot could accidentally share personal information. The risks aren’t always obvious, and children—especially younger ones—are less likely to recognize when something is wrong or speak up about it.

For more on how to navigate these challenges, see our guide on parenting in the age of AI and tips for keeping kids safe from AI.

The Bigger Picture

This study is part of a growing body of evidence that AI safety standards are lagging behind real-world use, especially for children. In 2024, the European Union’s AI Act became the world’s first major law to set strict rules for “high-risk” AI, including systems used in education and child care. But even these rules rely heavily on industry self-testing and don’t always require child-specific safety checks.

Meanwhile, U.S. regulations remain patchy. The Children’s Online Privacy Protection Act (COPPA) covers some data privacy issues, but doesn’t address the broader developmental and psychological risks highlighted in the new study. Tech companies have spent billions—Meta alone invested over $13 billion in AI research in 2025—yet little of that money goes toward child-focused safety research.

Other recent incidents underscore the problem. In 2025, a viral story revealed that a popular AI-powered doll could be hacked to eavesdrop on families. Earlier this year, a school district in Texas pulled an AI math tutor after parents complained it gave “creepy” personal responses to students. These aren’t just technical bugs—they point to a lack of meaningful oversight and accountability.

For families, this means the burden of keeping kids safe often falls on those with the least power to change the system. As AI becomes more embedded in daily life, the gap between official safety claims and real-world risks is only getting wider.

What You Can Do

• Ask tough questions before buying or using AI-powered products for kids. Look for clear information on how the product was tested for child safety—not just generic “safe for all ages” claims.
• Stay involved in your child’s digital life. Use AI tools together when possible, and talk openly about what’s happening on their devices. For tips on managing screen time and spotting red flags, see our AI and screen time for kids resource.
• Report problems and speak up. If you notice something off—like inappropriate content or odd behavior from an AI toy—report it to the company and share your experience with other parents and teachers. The more incidents are documented, the harder they are to ignore.
• Push for better standards at your school or community. Ask school administrators what AI tools are being used and how they’re evaluated for child safety. Advocate for policies that require child-specific testing, not just general benchmarks.
• Help kids build digital literacy skills. Teach children how to question what they see online and to come to you if something feels wrong. For ideas on building these skills, check out our guide on what skills kids need for the future.

The Bottom Line

This landmark study makes one thing clear: the AI systems our kids use every day are not as safe as we’ve been led to believe. Industry benchmarks and government rules have not kept up with the real risks children face. Until regulators and tech companies step up, families will need to be vigilant, ask questions, and demand better protections for the youngest and most vulnerable users.