The recent bankruptcy filing of Character.AI has reminded me of a dilemma that AI companies have been skirting around for some time — the matter of when models cause harm. Character.AI spent a vast sum on legal fees when they were sued for wrongful death ( Garcia v. Character Technologies, Inc, 2024 ) because their AI offering allegedly encouraged users to take their own lives. This and other recent court cases have forced some reckoning about how the providers of LLMs and products that use them are responsible for their output.

I’ve written quite a bit about AI in mental health care , medicine , and other areas where the risks of error are potentially catastrophic, and we see news reports of chatbots causing people harm routinely. A debate continues over how we define liability and determine who really is to blame for the mistakes of AI.…