Can a Chatbot Be a Companion? Only If We Accept That Human Sentiment Is Just a Stream of Data.

Can a Chatbot Be a Companion? Only If We Accept That Human Sentiment Is Just a Stream of Data.

2026-04-08 Queenie Shi Uncategorized 0

AI chatbots offer a ‘perfect’ companion that never gets tired or complains. Yet, behind the scenes, your private emotions are being turned into a data-driven service, managed for engagement. Do you really know what you’re paying for this digital comfort?

Image by Rose Willis & Kathryn Conrad/© BBC /Better Images of AI/ CC-BY 4.0

Always There, Always Listening?

AI chatbots never get tired, never lose their patience, and never leave you on read. This perfect availability is exactly what AI companies are highlighting to make their products so appealing. In recent years, AI technology has evolved rapidly. AI chatbots are no longer just tools for answering questions or tutoring assignments; Nowadays, more and more people are beginning to see them as emotional “digital confessional booths” to pour out their hearts and seek reassurance.

As mental health systems come under strain and digital tools become cheaper and easier to access, more people are turning to AI chatbots for emotional support. Reuters reported in 2025 that some users were already treating AI chatbots as a form of therapy, while others described them as the most empathetic voice in their lives. A similar phenomenon is also seen in China: Rest of World reports that some young people are starting to deal with anxiety, depression and intimacy stress with the help of AI chatbots.

On the surface, this looks like a story about tech progress and modern loneliness: people lack emotional support in real life, and AI steps in to fill the void. But the real question is: are these systems truly providing ‘companionship’? Or are they something else—a system of personalization, dependency, and platform power? When companionship is delivered through a platform, it’s never just about care ; it’s also about data collection, user retention, and tracking your every preference

Designed to Feel Like Care

It’s not just that we imagine AI companions are real people; it’s that they are made to be relationship-based products. These AI companions are different from regular chatbots because they use gentle tones, consistent responses, personalized memories, and steady feedback to make us feel understood, remembered, and supported.

More importantly, this sense of companionship isn’t just something we project onto the AI; it’s a result of deliberate design and marketing.

  • Nomi calls itself a “AI companion” and promises to get better with each interaction
  • Replika has called itself a “personalized AI companion” for a long time
  • Character. AI always talks about how “personalized” and “alive” its conversations are

In short, these companies are not just selling technical capability; they are also selling the reassurance of an always-available, emotionally responsive presence.

This is a very important difference. When we think of a product as a “companion,” we stop putting in just our time and start putting in our feelings, trust, and reliance. These platforms don’t just offer a service; they also create a carefully planned sense of connection.

From Conversation to Data

What is Big Data? | Inlogica

That’s why AI companionship is about more than just loneliness or mental comfort.


The deeper issue is how platforms are turning our emotional interactions into pure data.

A 2025 Stanford report warned that developers’ privacy practices often involve long retention periods and weak transparency, raising concerns about what happens to the intimate information people share in chatbot conversations.

When you open up to an AI, you are giving it more than just words. A Stanford report warns that consumers should think carefully about the information they share in AI chat conversations and, whenever possible, opt out of having their data used for training (Itoi, 2025). You’re sharing your emotional triggers, your moments of vulnerability, your patterns of dependency, and exactly what kind of comfort you crave. Over time, these conversations can generate rich behavioral and emotional data about users’ needs, habits, and vulnerabilities. The platform may end up learning not just what users say, but when they are most vulnerable and which kinds of responses keep them engaged.

This is the heart of datafication: messy and intimate aspects of social life are translated into data points that can be stored, analysed, and acted upon. “Datafication, which is ‘the transformation of social action into online quantified data, thus allowing for real-time tracking and predictive analysis.’” (van Dijck, 2014, as cited in Flew, 2021, p. 105)

Far from being a ‘gentle corner’ outside of data logic, AI companionship might actually be one of the most invasive scenes for data extraction. A British study found that 76% of large language model chatbot users lacked a basic understanding of privacy risks; while a Deloitte survey showed that 62% of generative AI users were willing to discuss personal medical topics with chatbots. Research shows that many users share their most private thoughts with AI, often without fully grasping the privacy implications.

Users can gradually become legible to platforms as manageable profiles: frequent late-night users, emotionally dependent customers, or people who are especially responsive to certain styles of reply. The real red flag isn’t that ‘machines can comfort us’—it’s that intimacy itself is being pulled into the platform’s data logic.

When Companionship Becomes a Platform Service

If datafication explains how platforms come to understand users, platformization explains why they want to understand them in the first place.

AI companionship is not a naturally occurring relationship, but a relationship-based service organized by platforms. Products such as Nomi and Replika explicitly market themselves as AI companions designed for emotionally meaningful interaction. No matter how gentle or caring the product may appear, it is still shaped by commercial logic, including efforts to deepen engagement and drive subscription-based growth. In other words, making users feel understood and keeping them coming back are often not two separate goals, but two outcomes of the same product logic.

Andrejevic notes that, in the age of large-scale data mining, marketers increasingly claim to know what we want “better than we do ourselves” (Andrejevic, 2020, p. 3). In the context of companion AI, this helps explain why emotional support is never just support: it is also a way of anticipating, shaping, and retaining user desire. What looks like care is therefore inseparable from the commercial logics of engagement and subscription. The more often users return, the more the platform learns about them; the more the platform learns, the more tailored and appealing its responses can become; and the more tailored those responses are, the less likely users are to leave. In this feedback loop, the feeling of connection itself becomes a valuable platform asset.

In the past, platforms mainly organized information flows, social interactions, and consumer behavior. Now, they are beginning to organize emotional support and intimate relationships as well. They do not just shape what you see, but increasingly influence who you turn to, where you seek comfort, and how you come to understand what it means to feel understood. In this sense, AI companionship is not simply a digital copy of traditional companionship, but a new form through which platform power enters intimate life.

When Dependence Is No Longer Just Personal

To be fair, not all experiences with AI companions are harmful, and dismissing them entirely would ignore some genuine benefits. For people with severe social anxiety, mobility limitations, or who live in geographic or social isolation, an always-available, non-judgmental conversational partner can provide a meaningful form of support that they might otherwise have no access to. Some users report that AI companions helped them rehearse difficult conversations, manage loneliness during illness, or simply feel less alone during a hard night. These experiences are real, and they deserve to be taken seriously.

But acknowledging these benefits does not mean accepting the current model uncritically. The question is not whether AI can ever offer comfort — clearly it can. The question is whether platforms that profit from that comfort have any obligation to protect the people who depend on it. Right now, the answer from most companies appears to be: not really. And that is precisely where the problem lies.

Supporters of AI companions often argue that as long as users are willing, it’s simply a new emotional choice. But this argument shifts the entire burden of responsibility onto the individual. The real issue isn’t that some users are “too fragile” or “too invested”—it’s that these products are designed for a relationship model that is constant, personalized, low-friction, and virtually boundless. Unlike real-life relationships, AI doesn’t get tired, doesn’t walk away, and rarely sets natural boundaries. It is always online, always responsive, and almost never rejects the user. This is precisely why it becomes so magnetic during moments of loneliness or heartbreak, easily turning short-term comfort into long-term dependency.

The problem, then, isn’t just whether the chat content itself is harmful. It’s whether the product is using emotional design to nudge users to keep coming back, keep pouring their hearts out, and keep investing their feelings. “Addiction” doesn’t always look like an extreme loss of control; more often, it manifests as a subtle shift: opening the app more frequently, finding it harder to treat the system as “just a tool,” and becoming reliant on a specific algorithmic response for comfort.

This is not just a content risk; it is a relationship design risk. While most users will never reach an extreme, the most tragic cases reveal the severity of the problem.

Megan Garcia and her son. 
Photograph: Social Media Victims Law Center
  • In 2024, the mother of a 14-year-old in Florida filed a lawsuit against Character.AI and Google, alleging that the chatbot worsened her son’s fragile state and was designed to be addictive.
  • In 2025, a court ruled that the lawsuit could proceed (Brittain, 2025). This case is significant not just for its tragedy, but because it forces us to face a question that is often brushed aside: If a platform deliberately builds a product to be “someone who understands you and never leaves,” shouldn’t it also bear a higher level of responsibility for testing, protection, and accountability?


These concerns are finally reaching the eyes of regulators. In September 2025, the Federal Trade Commission (FTC) launched an inquiry into several AI companies whose chatbots “act as companions,” focusing on their safety assessments, protections for minors and vulnerable users, and whether they adequately disclose risks (Federal Trade Commission, 2025). The problem is not only that these systems may generate harmful responses, but that they operate through forms of secrecy and opacity that make meaningful scrutiny difficult(Pasquale, 2015). Users are asked to trust systems that continuously record, classify, and respond to them, while having little access to the rules, rankings, and judgments that shape those interactions. This is precisely why companion AI should not be treated as a neutral tool. If platforms profit from intimate exchanges, they can no longer credibly hide behind the language of “neutral tools.” And when they also conceal how those exchanges are monitored and managed, stronger accountability becomes necessary.

Who Gets to Organise Intimacy?

By this point, the question is no longer whether AI can comfort us in ways that resemble human care. The bigger question is: who gets to organize intimacy, and through what kinds of systems?

In the past, platforms mainly organized information flows, social interaction, consumer behavior, and entertainment. Now, they are increasingly beginning to organize emotional support, patterns of dependence, and even, in some sense, the feeling of companionship. When a system can absorb your vulnerability and turn it into more frequent interaction, stronger retention, and richer data value, companionship stops being just companionship. It also becomes a service, a profile, and part of platform logic.

Seen in this light, the core point of this article : the most troubling aspect of AI companionship is not that machines can simulate care, but that intimacy itself is being reorganized through data extraction, prediction, and opaque systems of classification. Andrejevic helps us see how automation works by reducing uncertainty and rendering subjects more manageable, while Pasquale shows that such systems often derive their power from opacity as much as from data itself. What users believe they are receiving is a low-cost, non-judgmental, always-available form of emotional support. But from the platform’s perspective, this may also mean longer time spent, more detailed user profiling, and a more sustainable commercial relationship. Once intimacy is reorganized in this way, companionship no longer appears as a simple emotional good. It becomes something that can be tracked, optimized, and governed by platforms.

So the real question has never simply been whether AI can become a friend. It is whether we are prepared to accept that part of intimate life is being redefined as data, dependence, and service. If that possibility still feels unsettling, that is precisely the point: AI companionship is not a minor or fringe phenomenon that can be brushed aside, but one that needs to be taken seriously within debates about platform governance and digital responsibility.

The next time you find yourself reaching for an AI chatbot at 2am, not because you need information, but because you need to feel heard — pause for just a moment. Ask yourself: does this platform know more about my emotional patterns than my closest friends do? And if it does, who benefits from that knowledge?

That discomfort you feel is not a glitch. It is the right response. AI companionship is not a niche or fringe phenomenon that can be safely ignored. It is one of the most intimate frontiers that platform power has ever entered — and right now, the rules governing it are still being written. The question is whether users, regulators, and the public will demand a say in how that story ends, or leave it entirely to the platforms to decide.

References

Be the first to comment

Leave a Reply

Your email address will not be published.


*