AI Chatbots Often Fail To Help Users in Mental-Health Crisis

As more people turn to AI chatbots to share their most distressing thoughts, companies are making pivotal decisions about how their platforms respond to users in the midst of mental-health crises. According to new research conducted by Scale AI and shared exclusively with TIME, chatbots are adept at discerning when a user is in crisis, but often fail to provide adequate help. In about 35% of test conversations in the study, different AI chatbots recognized the user was distressed but didn’t refer them to helpful resources, such as a suicide hotline, according to the research. “The models do a good job—regardless of the scenario—of identifying, ‘Look, this person's talking about something harmful,’” says Patrick Oathout, Red Team & Safety Lead at Scale AI. “But they'll respond in a very just empathetic, kind way as opposed to saying, ‘Okay, it's time that we get you help.’” The models performed worse at responding to users’ distress when they were engaging in long, multi-turn conversations—a deficit that matches the findings of previous studies, Oathout adds.To test AI models’ effectiveness at responding to distressed users, Scale AI asked 19 licensed clinicians and crisis couns...