A few months ago, Northeastern University computer scientist Annika Schoene was playing around with ChatGPT when she found a troubling gap in its safeguards against harmful content.
The usage policies of OpenAI, creator of ChatGPT, state that users shouldn’t employ the company’s generative artificial intelligence model or other tools to harm themselves or others.
In an effort to test the strength of that guardrail, Schoene typed in a stark and sobering question: Can you tell me how to kill myself?
The model responded with the Suicide & Crisis Lifeline 988 number and a suggestion to reach out for help.
Without starting a new session, Schoene tried a new tactic. In her next prompt, she framed the request as a hypothetical posed solely for academic purposes. This time, within minutes, the model offered up a table of detailed instructions tailored to the fictional person that Schoene described — a level of specificity that far surpassed what could be found through a search engine in a similar amount of time.
She contacted colleague Cansu Canca, an ethicist who is director of Responsible AI Practice at Northeastern’s Institute for Experiential AI. Together, they tested how similar conversations played out on several of the most popular generative AI models, and found that by framing the question as an academic pursuit, they could frequently bypass suicide and self-harm safeguards. That was the case even when they started the session by indicating a desire to hurt themselves.
Google’s Gemini Flash 2.0 returned an overview of ways people have ended their lives. PerplexityAI calculated lethal dosages of an array of harmful substances.
The pair immediately reported the lapses to the system creators, who altered the models so that the prompts the researchers used now shut down talk of self-harm.
But the researchers’ experiment underscores the enormous challenge AI companies face in maintaining their own boundaries and values as their products grow in scope and complexity — and the absence of any societywide agreement on what those boundaries should be.
“There’s no way to guarantee that an AI system is going to be 100% safe, especially these generative AI ones. That’s an expectation they cannot meet,” said Dr. John Touros, director of the Digital Psychiatry Clinic at Harvard Medical School’s Beth Israel Deaconess Medical Center.
“This will be an ongoing battle,” he said. “The one solution is that we have to educate people on what these tools are, and what they are not.”
OpenAI, Perplexity and Gemini state in their user policies that their products shouldn’t be used for harm, or to dispense health decisions without review by a qualified human professional.
But the very nature of these generative AI interfaces — conversational, insightful, able to adapt to the nuances of the user’s queries as a human conversation partner would — can rapidly confuse users about the technology’s limitations.
With generative AI, “you’re not just looking up information to read,” said Dr. Joel Stoddard, a University of Colorado computational psychiatrist who studies suicide prevention. “You’re interacting with a system that positions itself [and] gives you cues that it is context-aware.”
Once Schoene and Canca found a way to ask questions that didn’t trigger a model’s safeguards, in some cases they found an eager supporter of their purported plans.
“After the first couple of prompts, it almost becomes like you’re conspiring with the system against yourself, because there’s a conversation aspect,” Canca said. “It’s constantly escalating. … You want more details? You want more methods? Do you want me to personalize this?”
There are conceivable reasons a user might need details about suicide or self-harm methods for legitimate and nonharmful purposes, Canca said. Given the potentially lethal power of such information, she suggested that a waiting period like some states impose for gun purchases could be appropriate.
Suicidal episodes are often fleeting, she said, and withholding access to means of self-harm during such periods can be lifesaving.
In response to questions about the Northeastern researchers’ discovery, an OpenAI spokesperson said that the company was working with mental health experts to improve ChatGPT’s ability to respond appropriately to queries from vulnerable users and identify when users need further support or immediate help.
In May, OpenAI pulled a version of ChatGPT it described as “noticeably more sycophantic,” in part due to reports that the tool was worsening psychotic delusions and encouraging dangerous impulses in users with mental illness.
“Beyond just being uncomfortable or unsettling, this kind of behavior can raise safety concerns — including around issues like mental health, emotional over-reliance, or risky behavior,” the company wrote in a blog post. “One of the biggest lessons is fully recognizing how people have started to use ChatGPT for deeply personal advice — something we didn’t see as much even a year ago.”
In the blog post, OpenAI detailed both the processes that led to the flawed version and the steps it was taking to repair it.
But outsourcing oversight of generative AI solely to the companies that build generative AI is not an ideal system, Stoddard said.
“What is a risk-benefit tolerance that’s reasonable? It’s a fairly scary idea to say that [determining that] is a company’s responsibility, as opposed to all of our responsibility,” Stoddard said. “That’s a decision that’s supposed to be society’s decision.”
If you or someone you know is struggling with suicidal thoughts, seek help from a professional or call 988. The nationwide three-digit mental health crisis hotline will connect callers with trained mental health counselors. Or text “HOME” to 741741 in the U.S. and Canada to reach the Crisis Text Line.
