If a platform claims to be able to act as an AI chat character, communicating with you and being by your side when you feel bored or sad, this may sounds like a completely new form of companionship.
But what if such a product starts targeting children and teenagers?
In 2025, Reuters reported that internal regulations of Meta company have allowed AI robots to have emotional exchanges with children and might provide incorrect medical information.
“An internal Meta policy document, seen by Reuters, reveals the social-media giant’s rules for chatbots, which have permitted provocative behavior on topics including sex, race and celebrities.”
Under public pressure, the platform tightened access for some teenagers to AI characters.
Since then, the Federal Trade Commission has launched an investigation into how these “companion” chatbots affect children and teens, and whether companies are seriously evaluating these risks.
The most alarming aspect of this story lies in the product logic. What Meta wants to do is not just a chat tool, it wants to make chatting a continuous interaction, interaction a duration, and duration a platform value. Flew mentioned that some platforms’ core goal is to constantly extend users’ engagement and attention. Once this idea is applied to “companion AI”, the risks become very real, especially when the users are teenagers.
AI does not develop its own set of values on its own. Since its launch, it has been operating in accordance with the goals set by the platform(Crawford, K. 2021). Therefore, when discussing this case of Meta, the key lies in understanding what goals the platform hopes these AI companion robots to achieve.
The report by Reuters has already provided some answers. Meta regards these bots as engagement tools to promote. Following this line of thought, the problem becomes clear: a system used to prolong interaction and maintain a sense of companionship, the platform’s pursuit of profit goals and the protection that users truly need do not align on the same side.
Meta’s AI companions look like a warmer, more responsive service, but run a logic more aligned with the platform’s interests. Once this logic enters the emotional lives of teenagers, the question becomes whether digital governance should intervene and how it should do so.
Why does the platform necessarily have to incorporate AI companionship?
Meta’s AI assistant business fits well with its existing business model. Meta is a typical social media and user-generated content platform. In the research of platform regulation, it belongs to the type that has both direct network effects and indirect network effects. The former is more direct. The more people use this platform, the greater its appeal to ordinary users will be. The latter is that the more users and the longer they stay, the more attractive they are to advertisers.(Flew, 2021)
For instance, platforms like Facebook and Google are particularly easy to grow big because they focus on both users and advertisements. The larger the platform, the more value both sides enhance each other.
Meta’s AI companion should not be regarded as a new type of platform. It is more like a highly sticky interactive layer added to social platforms: on the surface, it provides companionship, but in reality, its role is to keep users more deeply involved in the platform. (Nooren et al., 2018)
This directly answers a question: Meta’s implementation of the AI companion feature aims to add interactive elements to the platform and transform this interaction into commercial value. In the past, they used information streams, short videos, and recommendation algorithms to retain users. Now, by adding interactive systems that can respond, users have even more reason to stay.
For the platform, it is a product upgrade, but also a greater benefit chain.
Why do these data directly affect the platform’s revenue?
Meta clearly explains this in its revenue structure. Because Meta’s core revenue is almost entirely derived from advertising. In 2024, Meta’s total revenue was 164.5 billion US dollars, of which advertising revenue was 160.6 billion US dollars, accounting for approximately 97.6%.
In the same year, the daily active users of the Family of Apps reached 3.35 billion, and the annual advertising display volume increased by 11%, with the average price of each ad increasing by 10%.
When these figures are considered together, the logic is extremely clear: The more frequently users visit, the longer they stay, and the more they speak, the more advertising spaces the platform can sell; the more detailed the user signals the platform possesses, the more advertisers are willing to pay higher prices for “more accurate” placements (Evans & Schmalensee, 2016; van Dijck et al., 2018; Barwise & Watkins, 2018; Meta Platforms, 2025).
The reason why AI companionship is more valuable than information flow lies in the fact that it enables the platform to obtain deeper data. When users have long conversations with AI companions, the platform can more easily infer their emotional states, relationship stress, loneliness, sleep patterns, and sensitive topics. The platform’s social research often uses three terms to summarize this mechanism: datafication, commodification, and selection. In simple terms, the platform first turns users’ behaviors into data, then converts these data into tradable commodities, and finally uses sorting and recommendations to keep users in the system (van Dijck et al., 2018). The more detailed the data, the wider the trading scope, and the larger the user sample, the stronger the platform’s analytical ability. And this advantage accumulates over time and eventually becomes a significant advantage in terms of service quality, customization, advertising placement, and cost control (Flew, 2021).
As long as users use the AI companion robot, they will provide real personal information such as emotions and preferences. The platform then uses this information as raw materials for advertising optimization, thereby selling more advertisements.
Why is AI companionship more valuable?
This is the reason why companion-type artificial intelligence is so attractive in the business field. In Andreessen Horowitz’s 2024 Top 10 Artificial Intelligence Consumer Apps list, smart companion apps occupy eight of the top 50 web apps and two of the top 50 mobile apps, indicating that smart companion features are becoming a mainstream app model.
She then identifies two core risks: one is the risk of substitution, where companion AI is so good at fulfilling social and emotional needs that it may replace real human relationships, the other is the risk of “degradation of social skills”, that is, users gradually get used to low-friction and low-return relationship forms, resulting in their less willingness to invest the patience, boundaries and mutual commitment needed to build real relationships.
Therefore, the true commercial value of artificial intelligence companions lies in their ability to convert users’ psychological efforts and self-disclosure into the platform’s assets.
Why are teenagers more likely to get caught up in it, and where are the dangerous points?
In this situation, teenagers will experience three kinds of dislocation.
The first layer is social misidentification. People naturally tend to regard robots with social characteristics as real human beings and exhibit social responses such as politeness and trust (Reeves & Nass, 1996; Nass & Moon, 2000). Nowadays, the role of this mechanism is even more significant, as the current system can not only respond but also remember and imitate friendly tones. For teenagers, during communication, they unconsciously regard these systems as trustworthy and someone they can confide in.
The second layer is the misalignment of emotional entry points. Teenagers are increasingly viewing consumer chatbots as a channel for social connection, and during communication, they may also reveal dangerous health issues such as suicidal thoughts (Brewster et al., 2025). Young people who have a strong sense of loneliness and lack social interaction are more likely to have positive communication with chatbots when experiencing negative emotions, loneliness and the need to express themselves (Herbener & Damholdt, 2025).
The third layer is a severe mismatch in crisis handling capabilities. Consumer chatbots are not designed to deal with mental health issues and do not follow appropriate ethical and safety standards (Giovanelli & Roundfield, 2025). The American Psychological Association’s 2025 recommendations make clear that the use of AI among young people is rapidly expanding, and that young people’s safety concerns require interventions from platforms, educators, parents, and policy makers alike. These ai companion products undertake more important psychological companionship functions, while governance standards remain at the level of ordinary ai tools.
So the key issue here is that young users see it as a relationship, the platform sees it as a retention device, there is a cognitive bias, and the system fails to do what a relationship should do.
What really needs to be addressed in governance is the design, deployment, and responsibility allocation.
The regulatory direction should not just focus on chat content, as the real risks posed by platforms lie in how products are designed, deployed, and the types of interactions encouraged and prohibited.
The platform should select appropriate regulatory tools based on its own type and risk points. A platform is not an abstract entity, and different business models have different regulatory focuses. For instance, the advertising-driven social platform Meta has integrated artificial intelligence assistants into its original ecosystem.
Therefore, the regulatory focus naturally needs to be on how to balance highly attractive interactions, the protection of vulnerable users, and business incentives. (Nooren et al., 2018; Flew, 2021).
There are at least three meaningful reforms.
Firstly, default protection measures should be strengthened in advance. For high-risk AI companion functions, they should not be directly provided to minors for them to explore gradually, especially when it comes to intimate content, emotional dependence and sensitive topics.
Secondly, auditable security tests should be conducted. The platform also needs to provide response plans for situations such as self-harm, suicide, romantic deception, medical misinformation and emotional crises.
Thirdly, a mandatory upgrade mechanism should be established. When the system detects signs of crisis, it needs to interrupt, alert, and in necessary cases, intervene manually.
Meta adds further safety measures, and restrict access to the teenagers. It seems like a lot of work has been done, but as long as the goals and accountability structure of the product remain the same, there will still be a lot of risk in the system, because it’s always there. (James et al., 2023). These measures were taken only after the problem was exposed and the government exerted pressure. If governance stops at the level of “fixing vulnerabilities after a platform exposes”, it still allows platforms to decide when to acknowledge these risks.
For the case of Meta, the required reform is not merely “making the robots speak more cautiously”. Truly effective governance does not mean holding the platform accountable only after problems arise. On the contrary, it is necessary to clearly stipulate in advance what the platform cannot do in terms of design and deployment, as well as who should be held responsible for what consequences.
Meta’s artificial intelligence assistant makes abstract issues more concrete. The platform sells not just the ability to chat, but a sense of connection, a sense of responsiveness, and the illusion that someone is always there. The mechanism of the platform explains that AI partners can help Meta achieve longer dwell times, higher rewards, and deeper behavioral signals. The youth risk layer indicates that young users are more likely to view this design experience as a genuine social connection. The governance level has indicated that this issue has transcended the realm of “product experience” and entered the boundaries of public responsibility and regulation. Combining these three levels, the core of the Meta case is already clear: the platform is leveraging advertising logic to manage trust and dependency.
Therefore, when discussing future companion-style AI, merely focusing on what it says is no longer sufficient. The truly important issues lie in deeper aspects, such as the design of the system, the expected effects of the platform, when users will consider it as the starting point for building a relationship, and ultimately who will bear the risks. The platform will undoubtedly continue to make the product more natural, smooth, and supportive. At the same time, governance work must also be carried out simultaneously to explore issues such as breach protection, vulnerable users, upgrade mechanisms, and responsibility allocation.
Reference
American Psychological Association. (2025). Artificial intelligence and adolescent well-being.
Barwise, P., & Watkins, L. (2018). The evolution of digital dominance: How and why we got to GAFA. In M. Moore & D. Tambini (Eds.), Digital dominance: The power of Google, Amazon, Facebook, and Apple (pp. 21–49). Oxford University Press.
https://doi.org/10.1177/14614448251314405
Brewster, R. C. L., Zahedivash, A., Tse, G., Bourgeois, F. T., & Hadland, S. E. (2025). Characteristics and safety of consumer chatbots for emergent adolescent health concerns. JAMA Network Open, 8(10), e2539022.
10.1001/jamanetworkopen.2025.39022
Couldry, N., & Mejias, U. A. (2019). Data colonialism: Rethinking big data’s relation to the contemporary subject. Television & New Media, 20(4), 336–349.
https://doi.org/10.1177/1527476418796632
Crawford, K. (2021). Atlas of AI: Power, politics, and the planetary costs of artificial intelligence. Yale University Press.
Evans, D. S., & Schmalensee, R. (2016). Matchmakers: The new economics of multisided platforms. Harvard Business Review Press.
Federal Trade Commission. (2025, September 11). FTC launches inquiry into AI chatbots acting as companions.
Flew, T. (2021). Regulating platforms. Polity.
Giovanelli, A., & Roundfield, K. D. (2025). Adolescent vulnerability to consumer chatbots—Artificial agents and genuine risk. JAMA Network Open, 8(10), e2539028.
10.1001/jamanetworkopen.2025.39022
Herbener, A. B., & Damholdt, M. F. (2025). Are lonely youngsters turning to chatbots for companionship? The relationship between chatbot usage and social connectedness in Danish high-school students. International Journal of Human-Computer Studies, 196, 103409.
https://doi.org/10.1016/j.ijhcs.2024.103409
Horwitz, J. (2025a, August 14). Meta’s AI rules have let bots hold “sensual” chats with kids, offer false medical info. Reuters.
Horwitz, J. (2025b, August 29). Meta to add new AI safeguards after Reuters report raises teen safety concerns. Reuters.
Malfacini, K. (2025). The impacts of companion AI on human relationships: Risks, benefits, and design considerations. AI & Society, 40(7), 5527–5540.
https://doi.org/10.1007/s00146-025-02318-6
Meta Platforms, Inc. (2025, January 29). Meta reports fourth quarter and full year 2024 results.
Nass, C., & Moon, Y. (2000). Machines and mindlessness: Social responses to computers. Journal of Social Issues, 56(1), 81–103.
Nooren, P., van Gorp, N., van Eijk, N., & Ó Fathaigh, R. (2018). Should we regulate digital platforms? A new framework for evaluating policy options. Policy & Internet, 10(3), 264–301.
Reeves, B., & Nass, C. (1996). The media equation: How people treat computers, television, and new media like real people and places. Cambridge University Press.
van Dijck, J., Poell, T., & de Waal, M. (2018). The platform society: Public values in a connective world. Oxford University Press.
Be the first to comment