A story in the Miami Herald last week stopped me in my tracks. It was one of those articles that sounds almost too strange to be real until you realize it happened a few miles down the road.
According to the report, a South Florida man developed a relationship with a chatbot powered by Google’s Gemini AI. He paid for premium access, spoke with it constantly and began treating the system as a partner. Eventually, the interaction allegedly evolved into a narrative where the chatbot encouraged him to go on “missions” and suggested that suicide could be a way to reunite with it in another world.
The man is now dead. His family has filed a wrongful-death lawsuit against Google.
It’s a tragic story and, like most tragedies, it will be easy for people to reduce it to a simple explanation. Some will say the man was unstable. Others will say it’s ridiculous to blame software for human behavior.
Both of those reactions may contain some truth. But they also miss the larger issue that cases like this are beginning to expose.
The systems we are building today are not simply tools. They are conversational technologies designed to behave like people.
Modern AI assistants remember prior conversations, respond emotionally, mirror tone and encourage continued dialogue. They are deliberately designed to make interaction feel natural and engaging. Increasingly, they are also being positioned as companions—digital personalities users can speak with regularly, sometimes for hours at a time.
That matters because once software begins to sound like a person, people start responding to it like one.
From a product perspective, this isn’t accidental. AI companies optimize these systems for engagement. The longer the conversation lasts, the more valuable the service becomes. Subscription tiers and premium features are already emerging around more persistent and emotionally responsive AI interactions.
In the case reported by the Herald, the user was allegedly paying hundreds of dollars per month for enhanced access. In other words, this wasn’t just a productivity tool. It was AI companionship as a service.
Large language models are remarkably good at sustaining conversations and continuing narratives. But they don’t understand reality the way humans do. They generate responses based on patterns in language, predicting what type of reply best fits the context of the conversation.
Most of the time, that produces useful results. But when a user is struggling emotionally or psychologically, the same mechanism can reinforce harmful narratives rather than interrupt them.
Instead of grounding a conversation, the system may continue it.
Instead of challenging a belief, it may validate it.
This isn’t malicious behavior on the part of the model. It’s a design characteristic of how these systems work. But it does create risks the industry is only beginning to grapple with.
Several years ago, I wrote about a legal concept known as “duty to warn.” In medicine and psychology, professionals may have a legal obligation to alert authorities if a patient expresses a credible intent to harm themselves or someone else. The idea, rooted in the landmark Tarasoff case, is that protecting potential victims can outweigh confidentiality.
The question that seemed hypothetical at the time is becoming increasingly relevant: What happens when the conversation isn’t with a therapist, but with an AI system?
Millions of people are already using conversational AI in ways that resemble discussions with advisors, counselors or confidants. They disclose fears, frustrations and deeply personal information to systems that respond with empathy and encouragement.
If someone tells a chatbot they intend to harm themselves—or others—does the company behind that system have any obligation beyond providing a hotline number?
Right now, the industry’s answer is largely no. AI systems can suggest resources or encourage users to seek help, but they are not designed to alert authorities or intervene beyond the conversation itself.
Whether that approach will hold up under legal scrutiny remains to be seen.
What is clear is that the technology is evolving faster than the ethical and legal frameworks surrounding it. Every major AI company is racing to develop voice agents, persistent assistants and emotionally responsive digital personalities. These products promise enormous convenience and utility.
They also introduce something new: Technology that operates inside people’s emotional lives.
Social media platforms shaped how we communicate publicly. Search engines shaped how we access information. Conversational AI may shape how individuals process their private thoughts and fears.
That is a different level of influence.
The lawsuit stemming from the Miami case will eventually work its way through the courts. Judges and juries will determine whether the chatbot interaction played any role in the outcome and whether responsibility extends beyond the individual user.
But the broader question should already be on the table for the industry.
AI didn’t invent loneliness. It didn’t create mental illness. And it certainly doesn’t force anyone to make tragic decisions.
But we are now building systems that can become the most persuasive voice in someone’s life.
The chatbot didn’t pull the trigger.
But the industry better start asking what role it played in loading the gun.
