In August 2024, South Korea faced a severe crisis involving AI-generated deepfake sexual abuse content. A large number of male students stole photos of their female classmates from social media platforms, used artificial intelligence to create fake pornographic images, and distributed them by school in a Telegram group with over 220,000 members (Human Rights Watch, 2024). The victims included middle school students, college students, teachers, and even female soldiers. The South Korean Ministry of Education received reports from 196 schools, 179 of which have been referred to the police for investigation (Lee, 2024).
The incident came to light thanks to a high school student. After learning that five female students at his school had been victimized, he drew inspiration from COVID-19 contact-tracing tools and, in just two hours, created a crowdsourced “deepfake map” that marked over 500 affected schools (Lee, 2024). South Korean feminists shared the map on X with an English translation, and its views quickly surpassed 3 million. The Telegram group in question is still active, though the map was subsequently removed.
The stark difference between a teenager’s ability to identify an issue in the shortest time with only a map and the neglect of major platforms over the years is the article’s central focus. Platforms are not neutral mediums, as scholar Nicholas Suzor (2019) has pointed out, but they are a form of private government, as they are able to establish rules and can choose to disregard the safety of some users. The present deepfake crisis in South Korea is a classic case of the existing mess in governing platform safety.
Platforms Are Governments—Just Unelected Ones
In order to comprehend why the deepfake crisis has been bursting in such a manner, we should initially ask the question that many individuals fail to answer: Who, really, determines the rules of cyberspace?
It is the government that many people automatically think is the case. After all, it is the role of parliaments to make laws, regulatory authorities to enforce them, and courts to adjudicate disputes. However, in his book, In Lawless: The Secret Rules That Govern Our Lives (2019), Nicolas Souzol claims that such an impression is deeply misguided. As a matter of fact, the largest technological platforms are the ones that actually dictate what we are able to say, see, share, and do in the online world. Telegram, Meta, and Google companies have silently set the rules of behavior online by means of user agreements, community guidelines, and algorithms. These platforms, according to Sousol, are functioning as private governments, exercising enormous power over our online lives with no transparency, accountability, or democratic legitimacy that the government should enjoy.
Platforms, as he observes, do not control us in the way that a state controls its citizens. They set the limit of behavior in the form of product design, the content we see thanks to algorithms, and the spread of information thanks to policies. Their choices are not only based on so-called community values but also involve advertisers, governments, and lobbying groups, and include personal value judgments of the platform at the same time. This paradigm of privatization is very powerful and can be invisible to the masses.
In his lecture, an associate professor at the University of Sydney, Jonathan Hutchinson (2026), pointed out the asymmetry of information as the most obvious problem with this model. The platform has total influence over users. Functionalities that seem to provide control to the users, like privacy settings, consent options, and reporting features, are usually created to address platform liability, not individual rights. In principle, users have many rights. However, in reality, the question of their actual implementation always lies in the hands of the platform.
Telegram’s Rules Were a Political Choice
Telegram never acted in the South Korean deepfake pornography scandal, not due to any technical inability to remove such materials or even identify them, but simply because of the design of its products and its philosophy of governance. This negligence of duty was not only not avoided but even condoned by Telegram itself.
Ever since its inception, Telegram has followed a set of principles: maximum privacy, minimum moderation, and no compromise. Its founder, Pavel Durov, has clearly indicated that it is a principled dedication to free speech—in 2024, he wrote that the user experience at Telegram is based upon our mission of ensuring that users are safe even in totalitarian regimes (IEEE Spectrum, 2024). In practice, group chats on Telegram lack an end-to-end encrypted option, require only a phone number for identity checks, and do not actively scan the content (Norton, 2025; Green, 2024).
According to the records in the NPR article about this crisis, the above-mentioned circumstances make Telegram structurally well-suited to spread non-consensual deepfakes: easy to distribute, hard to track, and hard to take away (Kuhn, 2024). After the Korea Communications Standards Commission formally pressured Telegram, the platform agreed to remove only 25 specific pieces of content and symbolically announced the implementation of a “zero-tolerance policy.” Compared to the relevant groups with hundreds of thousands of members, 25 pieces of content are clearly insignificant. This reaction is not real regulation enforcement. It is merely a show of compliance in a system that the platform itself created, a system that was deliberately designed to avoid regulation.
The theory by Sour helps us to realize the nature of this phenomenon. Telegram does not make technical decisions about encryption, verification, and moderation, but policy decisions that are biased and predetermined, which the platform is biased towards. In this case, Telegram was always on the side of the users who wanted to communicate in an uncontrolled space, totally disregarding the fact that the privacy of the victims was being consistently violated and their personal rights were being grossly violated.
Platforms Protect What They Value
The most interesting case to learn about platform governance in practice is not South Korean but is instead an experiment that researchers at the University of Washington carried out in 2024.
Qi Wei and his team (2024) posted 50 AI-generated non-consensual intimate images on the X platform and reported them in two variants. Half of them were reported under the platform section of non-consensual nudity, and the remaining half were reported for copyright violations. The findings were shocking: within 25 hours, 100 percent of the pictures that were reported as violating copyright rights were removed, but no pictures that were reported as a violation of privacy were dealt with in three weeks.
The photographs, the medium, and the damage to the characters were the same, but the results were the opposite. The essence of the matter lies in which rule the platform preferred to prioritize.
This is a perfect example of a phenomenon of private governance introduced by Suzor. It is not that platforms are technically unable to respond to copyright violations with appropriate speed and disregard privacy violations, but instead a policy decision, which explicitly reveals which interests they prioritize. The copyright claims are usually backed by institutions with access to legal resources and commercial influence, whereas the privacy rights are usually associated with ordinary people, particularly women who do not have a voice and budgetary support. The severity of the moderation activities of a platform can be seen as a hierarchy of interests.
Hutchinson (2021) also mentioned the work of Terry Flue in his lecture, stating that platforms always put themselves in the role of a conduit of information and not a publisher of content, but in reality, they perform the functions of an editor and a regulator. The deepfake crisis has fully broken this illusion. Neither Telegram nor X is a neutral transmission channel; they are de facto governing bodies. Their decisions on who to protect have far-reaching consequences, and such biased decisions have directly resulted in the creation of a great number of victims.
Who Gets Governed Most Harshly
Even the people already on society’s periphery are habitually denied an essential privilege: access to effective digital protection. Marwick and Boyd (2018), in the article on the problem of privacy in marginalized populations, explicitly indicated that such an ability to be secure is also a structural privilege. That platform governance has never achieved equality for all.
The situation in South Korea is a stark illustration of this injustice. South Korean police say that the number of reported deepfake cases alone has surged from 156 in 2021 to 297 as of July 2024. The vast majority of victims are women and girls, many of whom are minors.
This issue has also been widely reported in mainstream media, which highlights how advances in AI technologies have exacerbated the spread of non-consensual intimate images online.
A study by Umbach et al. (2024), published in the proceedings of the CHI Conference on Human Factors in Computing Systems, reveals an even more startling statistic: 99% of victims of non-consensual deepfake pornography are women. These women are in no way at fault; they simply did what platforms have consistently encouraged users to do—share everyday photos on social media, all of which were taken in ordinary social settings.
However, Marwick and Boyd (2018) point out bluntly that it has become increasingly difficult to “opt out” of data-driven systems today. For Korean women, abandoning social media means being forced to disconnect from their normal social circles and professional lives. The business models of the platforms are based exactly on the active participation, but regarding the governance structures, they are virtually unprotective.
Suzor is critical in noting that the technical constraints of the platforms are not the gist of the problem, but the dire absence of accountability mechanisms. Platforms are free to choose the rules to follow and those to disregard. When such decisions, over and over again, are made in favor of corporate interests rather than the rights of individual users, and when this tendency causes more damage to women than to men, it is no longer a technical limitation but a natural consequence of this system of governance functioning as it should.
Limits of Existing Privacy Laws
In September 2024, the South Korean National Assembly enacted a new law due to the anger of the population regarding the increase in the number of digital sex crimes. The legislation punishes the viewing and possession of sexually explicit deepfake material, and the violator can be sentenced to up to 3 years in prison or a fine. In the past, the laws only banned the creation of such content with the view to distributing it without specifically banning its simple viewing or possession (Reuters/Asahi Shimbun, 2024).
According to academic studies, non-consensual synthetic images have emerged as the main form of digital sexual violence. This is the background of the legal change and emphasizes that the evils of deepfakes will continue even after legal fines (Jung and Noh, 2025).
The current privacy regulations, including the GDPR of the EU, were developed in the context of conventional data processing. They find it difficult to adjust to the novel pattern of data use of generative AI, revealing a gap between the obsolete legislation and the danger of AI (European Parliamentary Research Service, 2020).
Research findings indicate that there are significant differences between the regulation of platforms and national regulation. Laws are also trailing and incoherent. They cannot keep up with updates to the platforms. Platforms are moderated in real-time, which creates loopholes in law enforcement and undermines the safeguarding of digital rights (Flew, 2021).
The Accountability Gap
Suzor is not proposing the end of platform power, but he only hopes that platform power would address the legitimacy expectations of public institutions, which would be transparency, consistency, due process, and real accountability to users. His idea of digital constitutionalism would impose the human rights standard upon platforms, which must comply with the law, and not just out of goodwill.
Based on the case of South Korea, the fundamental problem is to analyze the way platforms such as Telegram meet their accountability requirements. This needs to be done through platforms that show that their enforcement procedures are effective, not just by being in existence of community guidelines, but also by ensuring that the guidelines are implemented fairly.
The fact that copyright and privacy enforcement are seen as separate issues that should be managed instead of a minor technical problem should be perceived as a failure of regulation, and platforms should be made responsible for the legal implications. Telegram is also a clear example of the imbalance in governance because it favors the freedom of certain users at the expense of other users in matters of their privacy and security.
Despite the fact that laws that defend the victims are already in place, the platforms that actually govern the digital space have chosen the other option. The same harms will keep on happening unless the discrepancies between the promises made to the people by a platform and its real performance of its duties are open to legal scrutiny by the people and not by the platforms themselves.
References
European Parliamentary Research Service. (2020). The impact of the General Data Protection Regulation (GDPR) on artificial intelligence.
https://www.europarl.europa.eu/RegData/etudes/STUD/2020/641530/EPRS_STU%282020%29641530_EN.pdf
Flew, T. (2021). Regulating platforms (pp. 72–79). Polity Press.
Green, M. (2024, August 25). Telegram is not really an encrypted messaging app. A Few Thoughts on Cryptographic Engineering.
https://blog.cryptographyengineering.com/2024/08/25/telegram-is-not-really-an-encrypted-messaging-app/
Human Rights Watch. (2024, August 29). South Korea’s digital sex crime deepfake crisis.
https://www.hrw.org/news/2024/08/29/south-koreas-digital-sex-crime-deepfake-crisis
Hutchinson, J. (2026). Week 4: Issues of concern: Privacy, security and digital rights [Lecture]. ARIN6902 Digital Policy and Governance, University of Sydney.
IEEE Spectrum. (2024, October 14). Telegram may not be as secure as it claims.
https://spectrum.ieee.org/telegram-security
Jung, S., & Noh, H. (2025). From grassroots advocacy to AI governance: Lessons from South Korea’s 2024 deepfake sexual abuse crisis on democratising knowledge and policy. International Journal of Comparative and Applied Criminal Justice.
https://doi.org/10.1080/01924036.2025.2596578
Kuhn, A. (2024, September 6). South Korea investigates Telegram over alleged sexual deepfakes. NPR.
https://www.npr.org/2024/09/06/nx-s1-5101891/south-korea-deepfake
Lee, H. (2024, August 30). Deepfake map made by middle school student goes viral nationwide. The Korea Times.
https://www.koreatimes.co.kr/southkorea/society/20240830/deepfake-map-made-by-middle-school-student-goes-viral-nationwide
Norton. (2025, March 14). Is Telegram safe for the average user?
https://us.norton.com/blog/privacy/is-telegram-safe
Qiwei, L., Geeng, C., Kohno, T., Roesner, F., & Thomas, K. (2024). Reporting non-consensual intimate media: An audit study of deepfakes [arXiv preprint 2409.12138].
https://arxiv.org/abs/2409.12138
South Korea police launch probe into whether Telegram abets online sex crimes, Yonhap reports. (2024, September 2). Asahi Shimbun/Reuters.
https://www.asahi.com/ajw/articles/15411279
South Korea to criminalize watching or possessing sexually explicit deepfakes. (2024, September 27). Asahi Shimbun/Reuters.
https://www.asahi.com/ajw/articles/15443130
Suzor, N. P. (2019). Lawless: The secret rules that govern our digital lives. Cambridge University Press.
Umbach, R., Henry, N., Beard, G. F., & Berryessa, C. M. (2024). Non-consensual synthetic intimate imagery: Prevalence, attitudes, and knowledge in 10 countries. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24).
https://doi.org/10.1145/3613904.3642382
Be the first to comment