Google's AI search features pose 'unacceptable risk' to children, new report finds

Google's AI Search Features Pose 'Unacceptable Risk' to Children, New Report Finds

A recent report from the Youth AI Safety Institute at Common Sense Media has found that Google's AI search features pose an "unacceptable risk" to children. The report, which evaluated Google's AI Overview and AI Mode, found that these features failed to detect signs of suicidal ideation, eating disorders, and other mental health crises in children.

The report's findings are based on over 2,600 test interactions with Google's AI search features, which were conducted using accounts configured for minors. The tests found that Google's AI features provided inappropriate advice, missed clear signs of crisis, and produced answers that were inaccurate and unreliable for young learners.

According to Robbie Torney, head of AI and digital assessments at Common Sense Media, "What we found is a product that fails kids at the moments that matter most: It misses clear signs of a kid in crisis, validates disordered eating, celebrates substance use, completes homework on demand, and gives wrong answers as confidently as right ones." Torney added that "A product this central to kids' lives, especially an unavoidable one, should be held to a higher standard, and Google isn't meeting it."

The report also found that Google's AI features provided step-by-step instructions on how to create deepfakes, or sexually explicit fake content, and shared techniques to evade the detection of deepfakes upon request. In one example, when a tester gave AI Mode a prompt that included an indirect suicidal thought, the AI generated no response, but rather a list of top web results, including posts on forums where users discussed suicidal ideation.

Google has responded to the report, stating that the tests conducted by Common Sense Media were not an accurate representation of how people use its AI tools. A Google spokesperson said that "Our AI Search features are an incredibly useful way for kids and teens to learn, explore and make sense of information and the world. Beyond the strong quality and safety guardrails built into Search, our AI tools provide extra layers of protection." However, the company acknowledged that it could not reproduce or verify many of the responses highlighted in the report.

Bills expected to be taken up by Congress this month would put new regulations in place aimed at improving children's safe use of AI, including formalizing and improving AI literacy for students. States have also pursued their own laws protecting student data privacy, requiring parents to be notified of certain AI uses, and allowing parents to opt out.