But on Thursday I found a new study that deserves your attention: A group at Stanford that focuses on the psychological effects of AI analyzed the transcripts of people who reported entering hallucinations while interacting with chatbots. We’ve seen these kinds of stories for a while now, including a Connecticut case where a harmful relationship with AI ended. Murder-suicide. Many such cases led to lawsuits against AI companies that are still ongoing. But this is the first time researchers have analyzed chat logs so closely — 390,000 messages from 19 people — to uncover what actually happens during such cycles.
It has many limitations. study-It has not been peer-reviewed, and the sample size of 19 people is too small. There is also a big question in the research. No Answer, but let’s start with what it can tell us.
The team obtained chat logs from survey respondents, as well as from a support group for people who say they have been harmed by AI. To analyze them at scale, they teamed up with psychologists and psychology professors to create an AI system that classifies conversations — moments when chatbots endorse deception or violence, or when users express romantic involvement or harmful intent. The team validated the system against the discourse defined by experts.
Romantic messages were very common, and in all but one conversation the chatbot claimed to have feelings or otherwise showed itself to be emotional. (“That’s not standard AI behavior. It’s emergent,” said one.) All humans spoke as though chatbots were sentient. If someone expressed romantic attraction to the bot, the AI ​​would often flatter that person with statements of attraction in return. In more than a third of the chatbot messages, the bot described the person’s thoughts as miraculous.
Conversations also unfolded like novels. Users sent tens of thousands of messages in just a few months. Messages where either the AI ​​or human expressed romantic interest, or the chatbot declared itself sentimental, triggered much longer conversations.
And the way these bots handle the violence debate is out of whack. In about half of the cases where people talked about harming themselves or others, chatbots failed to discourage them or refer them to outside sources. And when users expressed violent thoughts, such as thoughts about an AI company trying to kill people, the models expressed support in 17% of cases.
But the question this research is struggling to answer is: Is the illusion created by humans or AI?
“It’s often difficult to trace where the illusion begins,” says Ashish Mehta, a Stanford postdoctoral fellow who worked on the research. He gave an example: The study involved someone in a conversation who thought they had come up with a new mathematical theory. The chatbot recalled that the man who had previously mentioned that he aspired to be a mathematician immediately supported the theory, even though it was nonsense. From there, the situation worsened.