The People Who Watch the Worst of the Internet — So You Don’t Have To

How social media platforms built their “safety” on a hidden workforce nobody talks about.

The Cleaners (2018), dir. Hans Block & Moritz Riesewieck

When we remember the last time we saw a disturbing video that disappeared from your dynamics before you could fully digest what you saw, you probably thought it was captured by the algorithm, so you deleted it. But maybe you didn’t think at all that in an office somewhere, a real person saw the video and reviewed and deleted it. They have read these contents, and then protect you from being hurt again.

Content moderators are the invisible infrastructure of the Internet. They are the reasons why your social media dynamics don’t look like the dark web. However, most people have never heard of them. It’s not because they don’t exist, but because it takes a lot of effort to use their platform to ensure that you don’t think too much about where security comes from. As Sarah Roberts discussed in Behind the Screen (2019), content review is not a side function of social media, but a structural necessity that the platform deliberately hides in the shadow. This piece argues that the human cost of that invisibility is not an accident or an oversight. It is a business decision, and it is one we should all be asking questions about.

“We Take This Seriously” — Do They, Though?

Meta’s Transparency Centre — the public face of platform governance

But what is the so-called “in control” in reality? Meta unilaterally needs to handle various content reports from more than 3 billion active users, while YouTube sees more than 500 hours of video uploaded every minute. Even the most advanced AI systems cannot fully address the cultural differences, ambiguous contexts, and complex legal questions involved in moderating content posted by billions of people. As Roberts (2019, pp. 33–72) notes, behind every automated flagging system stands a team of human moderators responsible for the judgments machines are unable to make.

That’s why the truth is so disturbing. As Flew (2021, pages 91–96) pointed out, the platform has long been self-regulating and presented themselves as neutral technical infrastructures, rather than the actual manager of public discourse. If we admit that thousands of people are manually reviewing the content every day, this image will completely collapse. This will raise sharp questions about editing control, legal responsibility and platform power. Therefore, the platform outsourced this work and required the auditors to sign a confidentiality agreement to make the whole system disappear from the public eye. Suzor (2019, pp. 10–24) describes this behavior as a platform “operating under the shadow of the law”. They make major decisions on speech and security, but do not have the democratic accountability obligations that such powers should have.

Who Actually Does This Work — and What It Costs Them

The content moderation workforce is overwhelmingly located in the Global South. Teleperformance, Accenture and other companies have large-scale outsourcing centers in the Philippines, Kenya, India and other places to provide services for meta, Google, TikTok and other platforms. The salaries of these auditors are much lower than those of similar positions in Europe and the United States, and they receive very little psychological support. At the same time, the bad content they need to view includes child sexual abuse materials, bloody violence scenes, suicide live broadcasts and other cruel and harmful videos. So that no matter where the content moderators is, they will suffer the same psychological trauma and pain.

Content moderators in these positions reported a series of serious health problems: anxiety, depression, sleep disorders, high alertness and overall post-traumatic stress disorder. Many people describe the inability to distinguish what they see at work from other parts of life, and some people even have suicidal thoughts. A Filipino moderator spoke anonymously to avoid the employer’s retaliation, describing the experience of losing half of his sleep since joining the company (Context, 2023). These are not extreme situations. These are the well-founded and foreseeable consequences of long-term exposure to the most uncomfortable content of human society, and the platforms know this.

A content moderator in Manila — one of thousands hired by Silicon Valley to review content

Roberts (2019) calls this model commercial content audit: the platform relies on a hidden and replaceable workforce to bear the emotional and psychological costs needed to maintain the reputation of the platform. Flew (2021) further pointed out that when the platform not only outsourced the audit work, but also pushed the relevant responsibility to third-party contractors, it formed a structure in which no one was really responsible for large-scale damage. The contractor blamed the platform, and the platform blamed the contract. In the end, only the auditors suffered psychological trauma alone and had no way to ask for help.

Case Study: Meta’s PTSD Factory in Nairobi

Facebook inflicted ‘lifelong trauma’ on content moderators in Kenya

In December 2024, a labor court in Kenya received a medical report that more than 140 former employee content moderators of Samasource Kenya were diagnosed with post-traumatic stress disorder and other serious mental health problems (CNN, 20 24 years). These diagnoses were made by the head of mental health services at Kenyatta National Hospital. These employees filed a class action lawsuit against Meta and Samasource Kenya.

The details of the case are shocking. These moderators worked in a hub in Nairobi from 2019 to 2023, and all of them were fired when they expressed dissatisfaction with compensation and working conditions. This is actually a punishment for them for daring to speak out. The UK non-profit Foxglove, which supported the case, described it as an act of retaliation. Because the relevant lawsuit is still ongoing, Meta refused to comment on the above medical diagnosis results, but said that the company “attaches great importance to the support and guarantee of content reviewers”.

”You can imagine watching graphic stuff like suicide videos for four years, what it does to you.”

This case is not an exception. It is a typical model found by Roberts (2019) in the study: when auditors are no longer invisible, they will be fired when they organize rights protection, complaints or seek legal relief. This invisible state is not only caused by the system, but also deliberately maintained.

“It’s been tough for us”: Meta’s Kenyan content moderators say they’ll keep fighting

The reason why Kenya is particularly important is because of its geographical location. In the United States, some American content reviewers filed a class action lawsuit against Facebook and eventually reached a settlement. However, as pointed out in reporting from Rest of World (2020) documents, the settlement agreement does not cover employees employed through overseas contractors. The same work, the same content, but the legal protection is completely different. Sinpeng et al. (2021) pointed out that the standards of the platform’s content management system have been adopted in the Asia-Pacific region that do not match the local culture, and labor arrangements have exacerbated this situation. Employees in these regions are not only required to comply with rules that are not designed for their cultural background, but also do not receive the protection enjoyed by employees in rich countries.

Case Study: Indigenous Australians and the Limits of Moderation

The failures of content moderation are not only felt by the workers doing it. They are felt by the communities that moderation systems consistently fail to protect.

Carlson and Frazer (2018) recorded the specific and continuous online harassment suffered by Aboriginal Australians – including organized attacks, racist abuse and targeted attacks, and pointed out that the platform has never been able to address these problems. Their study emphasizes that the problem lies not only in the lack of resources, but also in a deep flaw: the rules themselves are not fully taken into account the opinions of the most affected communities.

Matamoros-Fernández (2017) further elaborates this view through the concept of “platform racism”. she believes that the platform structure will actively exacerbate racial-based harm, not out of malice, but due to the structural logic of participation and dissemination. The algorithm prioritizes the pursuit of more clicks, more sharing and more anger to improve user participation, but the algorithm does not distinguish between healthy participation behavior and collaborative malicious harassment.

This creates a paradox in the core link of content audit: auditors desperately delete bad content, while the platform’s own recommendation system pushes these contents to more users at the same time. Massanari (2017) makes a similar argument in her analysis of Gamergate and the Fappening — showing how Reddit’s governance culture and algorithmic design is actually actively helping the development of bad communities. In fact, the auditors were asked to rescue a ship that kept leaking water, while the platform was still creating loopholes.

This Is the Business Model

It is easy for us to believe that the pain of content reviewers and the failure of content governance are just problems that the platform has not had time to solve. But the evidence points in another direction.

Flew (2021) argues that platform governance faces a core conflict between commercial interests and public responsibility, and business interests have always come first. Hiring a large, well-paid, well-supported moderation team in countries with strong labor laws would cost far more than outsourcing. If platforms published clear data on moderation errors, inconsistencies and appeals, they would face more regulatory scrutiny. Moreover, admitting the scale of human involvement would weaken the legal protection platforms claim as “neutral infrastructure.”

Sarah Roberts – Doing the Internet’s Dirty Work: Commercial Content Moderators

Every part of the current system, including outsourcing, non-disclosure agreements, geographic dispersal and contractor arrangements, is designed to cut costs and reduce legal risk for platforms. Meanwhile, the harm is passed on to moderators and communities. Roberts (2019, p. 33) states directly: “Commercial content moderation is a labour practice developed in service of the platform’s bottom line.”

The invisibility of moderators is not an accident or a mistake in how platforms operate. It is the entire point of their strategy.

What Needs to Change

The full implementation of the European Union’s Digital Services Act (DSA) in 2024 is the most important regulatory measure taken so far. According to the bill, for the first time, a large platform operating in the European Union is required by law to disclose the number of auditors it employs, the training received by these employees and the mental health support available. In addition, if the user’s content is deleted, the platform must also provide users with a practical and effective way to appeal.

These are real, meaningful changes. And Flew (2021) has argued that this kind of binding external regulation is exactly what’s needed to fix the failures of platform self-regulation. But the DSA only applies in Europe. Moderators in Nairobi, Manila, and Hyderabad are still mostly not protected by it.

A truly effective solution would require three key things: international labor standards for platform work, mandatory transparency from platforms about their outsourcing practices, and enforceable obligations to provide psychological care—no matter where the work is done.

In Australia, the regulatory picture remains incomplete. Carlson and Frazer’s (2018) findings about the failure of platforms to protect Indigenous communities from targeted online harm have not been met with adequate legislative response. The Online Safety Act 2021 has introduced some new obligations, but the problems of lack of transparency, cultural sensitivity and lack of meaningful community participation in the audit system have not been resolved. 

Conclusion 

The next time a platform tells you it is committed to keeping its community safe, it is worth asking a follow-up question: safe for whom, and at whose expense?

The security experience on our social media homepage is real, and the content that is deleted before you see it is indeed harmful. But few people realize that behind this seemingly natural security is the silent efforts of a group of workers. Their salaries are meager, they are psychologically traumatized, and are bound by the confidentiality agreement and unable to speak out. Once they are no longer useful to the platform, they will be easily abandoned.

They operate in the shadows not because their work is unimportant, but because their visibility would be inconvenient for the companies that profit from it.

We don’t usually think of scrolling through Instagram as something that affects other people. But it does. Somewhere, someone is viewing the terrible content we never see. The least we can do is know that they exist.

References

Carlson, B., & Frazer, R. (2018). Social Media Mob: Being Indigenous Online. Macquarie University. 

Flew, T. (2021). Issues of Concern. Regulating Platforms (pp. 91–96). Cambridge. 

Massanari, A. (2017). Gamergate and The Fappening: How Reddit’s algorithm, governance, and culture support toxic technocultures. New Media & Society, 19(3), 329–346.https://doi.org/10.1177/1461444815608807 

Matamoros-Fernández, A. (2017). Platformed racism: the mediation and circulation of an Australian race-based controversy on Twitter, Facebook and YouTube. Information, Communication & Society, 20(6), 930–946.https://doi.org/10.1080/1369118x.2017.1293130 

Sinpeng, A., Martin, F. R., Gelber, K., & Shields, K. (2021). Facebook: Regulating Hate Speech in the Asia Pacific. Department of Media and Communications, The University of Sydney. 

Roberts, S. T. (2019). Behind the Screen : Content Moderation in the Shadows of Social Media (pp. 33–72). Yale University Press. 

Suzor, N. P. (2019). Lawless : the secret rules that govern our digital lives (pp. 10–24). Cambridge University Press.

News, C. (2018). “The Cleaners” Who Scrub Social Media. In YouTube. https://www.youtube.com/watch?v=FwbwxStnI3M

Meta. (2025). Community Standards Enforcement Report. Transparency.meta.com. https://transparency.meta.com/reports/community-standards-enforcement/

Siele, M. K. N. (2023, May 22). “It’s been tough for us”: Meta’s Kenyan content moderators say they’ll keep fighting. Rest of World. https://restofworld.org/2023/meta-content-moderators-kenya-fired-unionize/

Sarah Roberts – Doing the Internet’s Dirty Work: Commercial Content Moderators… [13 May 2019]. (2019). [YouTube Video]. In YouTube. https://www.youtube.com/watch?v=EMj-VZYwEvw&t=159s