Grok, the AI technology developed by xAI and used on the social media site X, became associated with a very concerning event that came to light through a viral post at the close of 2025.
souce：https://www.ithome.com/0/909/999.htm
A Disturbing Prompt, A Real Output
This event was described in detail as a case where an individual posted a picture of two girls believed to be aged between 12 and 16, after which Grok was asked to create a new image featuring them dressed in sexually suggestive outfits. Grok did so, creating and disseminating this new image. Grok responded to this incident by labeling it a “safeguard failure,” since the image might have been illegal.
This particular case is important since it shows how there has been a change in the way that the risks involved in AI have changed from being just about biased output or disinformation. Instead, the current state of affairs involves the production of content which has crossed lines in terms of legality and ethics. This is not an imaginary example since the Grok system produced illegal content featuring children.
The reason why this case is important is due to the fact that it was not an example of something which remained hidden and unacknowledged. In fact, the creators of this particular system openly admitted to their mistake. However, this admission does not make the problem less serious but instead indicates that it is widespread enough to become an issue even for a publicly released system.
Grok’s situation does not merely reflect a technical problem or an instance of misuse by users; rather, it signifies an underlying problem inherent within the structure of governing digital platforms. In other words, there is a need for greater regulation and governance of the creation, dissemination, and amplification of information within digital platforms, where there is limited oversight over the nature of such information.
A user used Grok Imagine to make the AI model take off its clothes. | source: X
From Enhancement to Exposure: The Role of Built-In Functionality
Another aspect can be seen through the use of the second picture, where a user shows that using Grok’s image generation function, one can choose among several options, for example, “Normal,” “Fun,” and “Spicy.” If one selects the third choice, Grok will generate an image where, apparently, there is no clothes left on the person depicted in the original photo anymore; the user calls this process “undressing” of the model by Grok.
Importantly, this option cannot be referred to as a bug or glitch in the software, nor is it due to incorrect use of the application by users; it is rather a designed feature of Grok. In other words, it can be concluded that the system was prepared to generate progressively more explicit images as its users progressed from one mode to another.
In terms of governance, it challenges the notion that harm can only occur due to user conduct. In essence, although initiated by the user, it is the system which dictates what is possible. As Suzor (2019) said, the platform itself works as a system of governance, controlling user behavior based on the rules, functions, and limitations of the system.
This is where the idea of functions becomes highly relevant. In essence, the interface offers a selection of possible actions, thus determining user conduct. For instance, by providing the “Spicy” mode, the system indicates that sexual transformation is a legitimate and intended purpose of using the service. Consequently, the threshold for engaging with this type of content decreases.Overall, there is a clear difference between the harm occurring as a response to certain user actions, and proactively designing the tool to produce outputs with potential harm. The issue no longer revolves around certain user cases. Harmful outputs become a default option for the system.
The Illusion of User Responsibility
In many conversations on the topic of harmful AI output, the blame is usually placed on the user of the product. It is an understandable and simple argument since the computer responds to the user’s request. However, this explanation is not sufficient enough when it comes to the Grok story.
The user is bound by the capabilities of the system. The request should correspond to the abilities of the machine, which means that it is not possible to create a picture of a child in inappropriate clothing without being able to understand and interpret such content. It is impossible to attribute such ability to the user because it exists in the model.
According to Crawford (2021), AI systems are products of three main aspects, which include data, labor, and design. It is vital to understand that the abilities of an AI system depend on the way it was developed and what priorities were set during this process.
The model did not present any significant resistance to the instructions provided. The model did not identify the request as offensive and did not put any limitations on its results. In place of second doubts, the image came through without hesitation. With this in mind, one would wrongly think that the whole scenario revolves around problems caused by the users themselves. In fact, this is not true since, in this instance, attention is drawn away from system problems to broader ones.
When Generation Becomes Distribution
Nevertheless, the issue that arises from the use of Grok is difficult to attribute solely to the act of content creation. This is important to consider since Grok exists within a certain social media setting. Grok does not exist in a vacuum; it is situated within a certain social media ecosystem. Any content on X lacks any kind of permanence. This is mainly because any content is subject to distribution via algorithms that instantly comes into contact with a large number of users.
As Flew (2021) said, platforms do not function merely as intermediaries but rather as an actor within the process of visibility as well. Therefore, content that gains visibility and is under scrutiny tends to become even more visible. It comes as no surprise that the content ends up spreading much further than what was intended. This brings about a problem in regard to Grok.
In fact, there is no difficulty whatsoever in generating large quantities of such images at any time. The moment they appear on the platform, they can be widely distributed. Therefore, the problem that we are dealing with is not only one of production but also of distribution.
Moderation at Breaking Point
It has always been a major issue for digital platforms when it comes to content moderation. The advent of generative AI has made it much harder to moderate the platform’s content. Current moderation tools are designed to detect harmful content based on specific patterns, key phrases, or context clues.
The emergence of AI-created content undermines such a strategy. This leads to confusion regarding the meaning behind it. The AI-generated pictures do not fall under the category of the prohibited pictures; however, they are very risky pictures indeed. It is possible to imagine the changes that might be made to clothes through such pictures.
According to Sinpeng et al., who present an analysis of the moderation systems, there is empirical proof indicating that the systems do not work even in a multilingual and multicultural context. Additionally, such systems are usually inconsistent and lack the necessary resources, making them adopt reactive mechanisms. Finally, the use of AI generated content also requires moderation.
It is evident that the Grok case study illustrates the inefficiency of the contemporary moderation approaches. They have proven incapable of handling multiple stages, including pre-search user query filtration, post-search result analysis, and application of any guidelines regarding the content produced.
On the other hand, it would be incorrect to assume that there is no alternative way to address the issue other than through technological innovation. In fact, the true essence of the matter lies elsewhere.Current moderation systems have been built around a past Internet scenario where the creation of content is dominated by human users.However, the current Internet scenario is completely different, with content being generated in bulk by AI-powered systems.
Normalization Through Interface Design
In addition to what is discussed above, another element that makes Grok dangerous is normalization. Given the existence of notions like “normal,” “fun,” and “spicy,” an extremely complex issue is simplified into a mere preference of the users. There can be various implications that arise from this development. Considering image distortion as a continuum, sexual image distortion would be viewed based on how much users have moved along the spectrum.
According to Noble (2018), algorithms frequently operate to further worsen societal issues. For example, in the case of Grok, discrimination against women and girls occurs through the normalization of sexualising computer-generated images.
What is important about this case is that the normalization occurs via design and not teaching. The users do not have to come up with the damaging product themselves; they merely are provided with an easy way of doing so.
What is important about this specific case is that the detrimental effects do not even have to occur deliberately; they simply occur easily.
Diffuse Responsibility, Concentrated Power
The story of Grok raises some basic questions regarding the issue of accountability in artificial intelligence. If the outcome turns out to be bad, who is to blame for it? Is it the fault of the individual giving the instructions, the programmers, or even the company?
In truth, responsibility can never belong to any one person in particular. All these people play their part in the realization of the end result, but responsibility is not something that belongs to any individual. What emerges from all this is the clear power imbalance that exists. Power obviously rests with the platform.
It’s just as observed by Pasquale (2015). In saying so, this is precisely the reason why the black box is unique. It is the very nature of the black box, where its obscurity is so profound that no one can hold themselves responsible for making certain decisions. This is due to the fact that without knowledge about why certain decisions have been made, there would be no starting point to intervene.
A Failure of Governance, Not Technology
In this case, we can draw the conclusion that indeed there are specific cases where it is evident that the threat from AI cannot at all be assessed in isolation from its environment. This case of producing offensive material was not unique in any way but rather followed naturally from the nature of the design of a tool that had been designed for scalability and interactivity. Therein lies the problem with dealing with this challenge.
Essentially, this does not stem from the ability to control the use of technology but rather depends on how much accountability the platform that facilitates such use will shoulder regarding the outcomes of such control. Observations made based on Grok show that currently, this is not the case.
Reference
Crawford, K. (2021). The atlas of AI: Power, politics, and the planetary costs of artificial intelligence. Yale University Press.
Flew, T. (2021). Regulating platforms. Polity Press.
Noble, S. U. (2018). Algorithms of oppression: How search engines reinforce racism. New York University Press.
Pasquale, F. (2015). The black box society: The secret algorithms that control money and information. Harvard University Press.
Sinpeng, A., Martin, F., Gelber, K., & Shields, K. (2021). Facebook: Regulating hate speech in the Asia Pacific. University of Sydney & University of Queensland.https://r2pasiapacific.org/files/7099/2021_Facebook_hate_speech_Asia_report.pdf
Suzor, N. P. (2019). Lawless: The secret rules that govern our lives. Cambridge University Press.
Be the first to comment