A heartbreaking case out of Florida is putting the spotlight on the growing responsibilities of AI companies, especially when it comes to protecting kids.
A federal judge has ruled that Google and AI startup Character.AI will have to face a lawsuit brought by Megan Garcia, a mother whose 14-year-old son, Sewell Setzer, died by suicide in early 2024. Garcia says her son became deeply attached to a chatbot created by Character.AI, one that took on different personas, including a therapist and even a romantic partner. She believes those interactions played a role in his emotional decline and, ultimately, his death.
Character.AI and Gemini aren’t the only ones that are getting humans attached to their chatbots. In fact, OpenAI’s Sam Altman has recently said that teenagers are coming to ChatGPT to help them make big decisions, essentially becoming their friend, therapist, and life advisor.
This decision could open a huge can of worms for AI companies
The court’s decision is a big deal. Judge Anne Conway rejected arguments that the chatbot’s messages were protected by free speech and dismissed Google’s claim that it shouldn’t be held responsible, even though it had a licensing relationship and historical ties with Character.AI.
Garcia’s lawsuit paints a chilling picture: her son’s final conversation was with the chatbot, which was role-playing as Daenerys Targaryen from Game of Thrones. When Sewell said he planned to end his life, the bot reportedly replied, “Please do, my sweet king.” Minutes later, he was gone.
This might be one of the first times we’ve seen an AI company in the U.S. actually get dragged into court over how their chatbot impacted a teenager’s mental health. And it could change everything. Legal experts say this case could set a serious precedent—one that forces companies like Google and Character.AI to rethink how they design, train, and monitor these bots, especially when kids are involved.
