Hate speech, online harm, and why the platform review is always late
Word count : 1906
Figure 1. Diagram of smartphone damage and digital vulnerability. As a guide, this chart aims to lead to an in-depth discussion on the problem of online injury in South Korea.Source: Photographer Fotografia Lui Vlad, taken from Unsplash. This figure is only a conceptual indication, not a documentary of the incident site.
https://unsplash.com/photos/cracked-smartphone-screen-on-a-repair-mat–O12MiReddU
Introduction
Many people tend to portray the Internet as a “free territory”, an ideal space for everyone to speak out, connect and participate in public discussions. This vision is still appealing, but the other side of online life has become unavoidable. It is these channels that give people the right to express themselves that also make harassment, humiliation and precise attacks against individuals easy. These injuries are often very immediate and open, and once they occur, they are difficult to eradicate them completely. This is exactly why the current digital platform discussion always falls into a core game: how can we substantially reduce the real harm while defending freedom of expression (Flew, 2021; Sinpeng et al., 2021)?
The Deepfake crisis that broke out in South Korea in 2024 provided an extremely heavy footnote to this dilemma. It is reported that a large number of pornographic fake images targeting women and underage girls circulated wildly in Telegram chat groups, which ignited public anger and triggered police investigations and legal revisions (Kim & Lee, 2024; Yim, 2024). The profound impact of this case is not that the content itself is more “eye-catching”, but that the harm spreads rapidly in a socialised way and quickly out of control. When the mainstream society began to pay attention to this matter, many victims had already fallen into the abyss of humiliation, fear and loss of reputation.
This is also why Korean cases are typical samples of hate speech, online harms and moderation. Even if it does not necessarily conform to the traditional definition of hate speech in a narrow sense of law, it obviously falls into the broader category of online harm. It reveals the core of the problem: the key is not the existence of harmful content itself, but the way these online spaces are organised, the extremely slow response of the platform, and how the damage can easily reach far without intervention. This article aims to show that this storm should not only be regarded as shocking news about new technologies, but should be understood as a typical case of online damage diffusion and platform audit failure (Flew, 2021; Roberts, 2019; Sinpeng et al., 2021 ).
Section 1: Online harms are more than “offence”
The starting point for understanding this problem is that we must stop simply defining harmful content as “uncomfortable”. The weight of the word “offence” is too light, and it covers up the seriousness of the problem. Harmful content can do much more than make people feel bad. It can turn individuals into hostile onlookers by creating a sense of shame, isolation and insecurity. The study of harmful expression clearly points out that this is by no means a simple emotional offence, but a real injury. This kind of injury may break out instantly or continue to accumulate in subsequent life (Sinpeng et al., 2021).
The case of South Korea accurately confirms this. The reason why pornographic forged images is fatal is not only because of its falsehood or pornography, but also because it directly destroys a person’s social reputation, exposes him to endless harassment, and makes online participation, which should be normal, shocking step by step. Victims tend to be banned, escape from the digital space, or fall into the endless fear of “who has seen it and who is spreading”. Even if the original file is finally deleted, the humiliating branding will not be erased. Once something is spread, copied, used as talkative or forwarded among peers by the social circle, the damage caused has long gone beyond the image itself.
This explains why narrow legal interpretations are often pale. Legality is important, but it is not the only measure of injury. The real core question is: what consequences do these contents cause in real life? Does it exacerbate the fear of society? Is it inducing more aggressive behaviour? Does it make specific users feel that digital space is no longer safe? If the answer is yes, then what we are facing is uncompromising online harm, not just “indecent speech” or “in vulgarity” (Flew, 2021; Sinpeng et al., 2021).
Section 2: Why harmful content keeps circulating
This leads to another fundamental question: since the platform has repeatedly promised to deal with harmful content, why can they still be boom and continue to circulate? One cruel reason is that the existence of rules does not mean the provision of real protection to users. The literal standard of community and reporting of most of the platforms explicitly forbids abuse, harassment, exploitation, as well as any other form of detrimental behaviour. However, practically speaking, the cold rules have a vast disparity with the experience of users.
Numerous dangerous elements are extremely contextual. Difficult areas that automated audit can hardly access include the micro-manipulation of language, the shunning of local culture, and the complicated social relations. The platform’s official definition of “violation” often fails to accurately cover the harm perceived by users, resulting in the continuous spread of content in the gaps of the rules. The other reason that one cannot overlook is the mentioned fatigue of reporting in the study, the “reporting fatigue.” By repeating the action of reporting again and again, and then realizing that the current situation is not getting better, users will eventually lose their belief in the system and will no longer trust the intervention possibility (Sinpeng et al., 2021).
This is the case clearly shown in the trism in South Korea. The anger of people is caused by the content, but also by the sense of unity that “the platform could have acted earlier and harder but chose silence”. Once users realise that the reporting and audit system cannot provide substantial protection, the whole Defence mechanism will accelerate the collapse. This gives more windows for the spread of harmful content, while victims have to bear the bitter consequences of system failure alone (Kim & Lee, 2024; Yim, 2024).
Figure 2. Comparison of the number of Deepfake sexual crimes filed in South Korea in 2021 and 2024.Source: The author is self-made according to the statistics of the South Korean police cited by Reuters.
Section 3: Moderation is more than deleting content
For this reason, we must re-examine the content moderation instead of simply enominating it with “deleting posts”. Behind the moderation is: where the bottom line of social acceptance is, how quickly the damage can be discovered, who will be prioritised for protection, and who must suffer while waiting. In other words, it is by no means a simple technical action, but a form of governance with real power (Flew, 2021).
Roberts (2019) deeply analysed this point in his research. She pointed out that moderation is a highly labour-dependent and chaotic work, which usually lags behind the uploading of content and even large-scale dissemination. This is especially fatal in the Korean case. Once the fake image enters the fission-spreading chat environment, the damage has been fully launched before the moderation intervenes. The victim might have undergone the entire process of being recognised, humiliated, and intimidated before the complaint is accepted. As a processing task to the platform, the moderation is a late consolation to the victim; but as a “processing” task to the platform, it is a late consolation.
I believe that this is also the source of the lost trust of the public in the platform. The dissemination of damage is usually immediate, whereas the feedback of the moderation is never timely. It is not merely whether the content has finally been deleted or not, but whether the system has acted suitably before the spread of the damage. The platform can still allow users to suffer and still maintain standards, since the platform might be too clumsy, ambiguous, or merely not in touch with the actual pain of the users (Roberts, 2019).
Figure 3. The process of transforming online harmful content into social harm indicates: covering image circulation, peer forwarding, victim exposure, user reporting and lagging platform removal.Source: The author is based on the research of Roberts (2019), Sinpeng et al. (2021) and the report of related events in South Korea.
Section 4: Platform responsibility should mean more than “we have rules”
Finally, the platform should be the one to shoulder the responsibility of the foothold of the discussion. By the time the platform has become the infrastructure of free communication, they are not only able to discuss freedom of speech in an abstract plane, but are able to blind themselves to the systematic harm being perpetuated in their systems. One point that is harshly critical is that the platform is usually surprisingly effective when it comes to combating commercial concerns like copyright protection, yet it falters when it comes to combating harmful material like harassment and abuse (Flew, 2021). This opposition demonstrates that it is not about the technical means complying with the norm, but what is valued on the platform in the scale of interests.
The Korean case makes this dereliction of duty extremely clear. The public’s dissatisfaction is not only aimed at the rumour-mongers and disseminators, but also directly at the platform itself: is its cooperative attitude active enough? Does he really care about those broken lives? When the space it operates becomes a hotbed of humiliating others, how can it continue to maintain the illusion of “neutral intermediary” (Kim & Lee, 2024; Yim, 2024)? The responsibility not only stops at the users who post, but also extends to the platform environment where harm can be repeated at a low cost and high efficiency.
Therefore, the responsibility of the platform should not be limited to “after-the-scent liquidation”, but should be reflected as: the platform must redesign and operate its system in a way that reduces foreseeable harm. This does not mean that every incidental harm can be eliminated, but at least it means that the platform should no longer regard these repeated evils as some kind of “accident”. If a specific pattern of abuse continues to reappear in the same space, the platform cannot be completely separated from this result in terms of morality and governance logic (Flew, 2021; Woods & Perrin, 2021).
Figure 4. South Korea’s response to the Deepfake crisis in 2024: including public indignation, platform pressure, police investigation and legal revision.Source: The author is self-made based on Reuters reports from August to September 2024.
Conclusion
When discussing solutions, “duty of care” is a very inspiring perspective. This concept requires us not only to ask whether the content has been deleted, but also to ask whether the platform has injected security genes into the service design from the beginning to reduce those foreseeable damage (Woods & Perrin, 2021). For South Korea, this implies that we should continue to pose the question: Is the reporting channel open? Does it have an agile complaint handling? Are the vulnerable groups adequately safeguarded? Is the audit system design logic relevant in hedging the actual damage within these spaces?
In essence, I think the Deepfake scandal in South Korea is a typical case of “injury running to win the audit”. It is not the sophistication of the technology itself that matters, but the disintegration of the responsibility of the platform: dangerous information can be removed on the social level even before the platform actually reacts to it. This is also the reason why the harm in online communication cannot be considered a minor incident in digital communication. They are actually core propositions about how the platform is governed or why the governance fail (Flew, 2021; Sinpeng et al., 2021).
From this perspective, the focus of the case is no longer a novel technological panic, but an older and more difficult proposition: what should we do when the space of expression that originally touted freedom is alienated into a slaughterhouse of hostility, humiliation and harm? If so, content review will no longer be a marginal technical patch, but will become the core ruler to judge whether a platform is still qualified to call itself a public communication space (Flew, 2021; Roberts, 2019).
References
Flew, T. (2022). Regulating platforms. Polity Press.
Kim, J., & Lee, J. (2024, September 3). South Korea police launch probe into whether Telegram abets online sex crimes, Yonhap reports. Reuters. https://www.reuters.com/world/asia-pacific/south-korea-police-launch-probe-into-telegram-over-online-sex-crimes-yonhap-2024-09-02/
Roberts, S. T. (2019). Behind the Screen: Content Moderation in the Shadows of Social Media (1st ed.). Yale University Press. https://doi.org/10.12987/9780300245318
Sinpeng, A., Martin, F. R., Gelber, K., & Shields, K. (2021). Facebook: Regulating Hate Speech in the Asia Pacific. Department of Media and Communications, The University of Sydney.
Woods, L., & Perrin, W. (2021). Obliging platforms to accept a duty of care. Oxford University Press, Oxford, 93-109.
Yim, H. (2024, August 29). South Korea to ask Telegram, other social media firms to help tackle digital sex crimes. Reuters. https://www.reuters.com/technology/south-korea-ask-telegram-other-social-media-firms-help-tackle-digital-sex-crimes-2024-08-28/
Yim, H. (2024, September 26). South Korea to criminalise watching or possessing sexually explicit deepfakes. Reuters. https://www.reuters.com/world/asia-pacific/south-korea-criminalise-watching-or-possessing-sexually-explicit-deepfakes-2024-09-26/
Be the first to comment