Character.AI has introduced new security features and policies for building and interacting with the AI-powered virtual personalities it hosts. The new measures are intended to make the platform safer for all users, but especially for young people. The update includes more control over how minors interact with the AI chatbot, more content moderation, and better detection of the AI discussing topics like self-harm.
Although not quoted in the blog post about the update, Character AI linked to the announcement in a post on . . His family has now filed a wrongful death lawsuit against Character.AI, citing a lack of safeguards for the AI chatbots as contributing to his suicide.
We are heartbroken by the tragic loss of one of our users and would like to express our deepest condolences to the family. As a company, we take the security of our users very seriously and we continue to add new security features that you can read about here:…October 23, 2024
AI chat guardrails
Character AI’s post outlined several new safety features for the platform. For example, if the model detects keywords related to suicide or self-harm, it will display a pop-up directing the user to the National Suicide Prevention Lifeline and related resources. The AI will also be better at spotting and removing inappropriate content in a conversation, with particular sensitivity when users are under 18.
Presumably minors would already have limited content in conversations, but Character.AI may have increased that sensitivity even further. In cases where that may not be enough, entire chatbots have been removed.
“We perform proactive detection and moderation of user-created characters, including using industry-standard and custom blocklists that are updated regularly. We proactively, and in response to user reports, remove characters that violate our Terms of Service,” Character.AI explains. in his post. “Users may notice that we recently removed a group of characters that were flagged as infringing, and these will be added to our custom block lists in the future.”
Other new features are more about helping land users. So you’ll see a notification when you’ve spent an hour on the platform asking if you want to continue, so you don’t lose track of time. You’ll also see more prominent disclaimers emphasizing that the AI is not a real person. Such disclaimers are already in the conversations, but Character.AI wants to make them impossible to ignore.
These safety features are the flip side of how Character.AI has made interacting with chatbots more like talking to a real person, including voices and the two-way conversations available with the Character Calls feature. Still, the company likely wants to ensure its services are as secure as possible, and its moves could inform how others in the space are shaping their own AI chatbot characters.