Introduction
Have you ever wondered why, when ordering takeout on your phone, the price sometimes appears higher for you than for your friends—even when everything else is identical, including the restaurant, dishes, and delivery address? Or why, even after joining up for a membership, do the meal deals you order frequently increase in price?
This isn’t a fiction of your imagination; it’s a classic case of food delivery platform deploying algorithms to engage in “data-driven price discrimination.” At the same time, photos you casually post on social media may already have been “identified” and “recorded” by some system without your knowledge. Facial recognition technology is also gradually blurring the lines between public safety and privacy.
In today’s society, when digital platforms are ubiquitous, algorithms have become exceedingly linked to our daily lives. “Algorithm” was once a technical term used nearly exclusively by computer scientists, but it is increasingly appearing more frequently in common life. We appear to be becoming more reliant on these clever systems, yet we hardly grasp how they truly work. Behind these phenomena is a same core issue: the algorithmic black box.
What is black box
The term “algorithm black box” generally refers to a state in which the internal logic of an algorithmic system is invisible or difficult to explain.(Pasquale, 2015).
Between the input and output of machine learning, there exist unexplainable and untraceable technological and mechanism barriers, which prevent humans from fully, clearly and verifiably understanding its decision-making process and causal basis (Pasquale, 2015). It’s as if everything is hidden inside a black box, and you can neither see nor touch it.
China: Price Discrimination on Food Delivery Platforms
In China, the long-standing practice of “algorithmic price discrimination” based on mainstream food delivery platforms is a classic example of the black-box operation of algorithms.
（Figure1.What can we learn from the design of Chinese delivery apps? Britta Cheng. Source: author’s own illustration）
The core logic of food delivery platforms is as follows: through automated data collection, they record 28 categories of user data including mobile phone models, purchase frequency, geographic location, willingness to pay, and even swiping speed and use algorithms to create precise user profiles. Users are then categorized into “price-sensitive” (such as new users and infrequent users) and “price-insensitive” (such as long-time users and frequent users) groups, enabling differentiated pricing—raising prices and reducing discounts for the latter group to achieve “precise exploitation”.
And the core of all this is precisely the role played by algorithmic black boxes. The platform will not disclose the specific logic of its pricing algorithm, nor will it inform users “why your price is higher”. Consumers have no way of knowing which user group they have been categorized into, nor can they question the basis for the algorithm’s judgments.
Even when users do notice price discrepancies, platforms often attribute them to “caching discrepancies,” “location errors,”or “differences in account discounts,” making it difficult for consumers to gather evidence or seek redress. After all, the algorithm’s computational process is hidden behind the platform’s technical systems, which ordinary users cannot see or trace.
Notably, in this case, the algorithmic black box primarily operates within market relationships, with its impact concentrated at the level of economic interest distribution. However, this seemingly “low-risk” application is actually gradually reshaping the power structure between consumers and platforms. By enhancing information asymmetry, personalized pricing, and non-appealable processes, it converts traditional markets’ relatively balanced interactions into a platform-dominated, one-way power structure.
US: Clearview AI’s Problematic Facial Recognition
If the “big data price gouging” in China’s food delivery industry is related to consumer fairness, then the case of Clearview AI demonstrates the risk of the abuse of algorithmic black boxes in the field of public security after the integration of digitalization and AI.
Clearview AI is a New York-based startup founded in 2017. Its business model is simple yet highly controversial: using automated web-scraping technology, the company collected over 3 billion facial photos from social media platforms such as Facebook, Instagram, and Twitter, as well as various public websites, without anyone’s consent. It then built the world’s largest facial database, trained facial recognition models using AI algorithms, and sold its facial search engine to law enforcement agencies for use in identifying suspects. In this process, the risks associated with algorithmic black boxes have been magnified to an extreme degree.
In March 2026, the United States exposed a typical “algorithmic injustice”, which became the most shocking incident in the controversy history of the company
A woman from Tennessee was arrested after being mistakenly identified as a suspect in an interstate bank fraud scheme due to a false match by a facial recognition system. Despite having never been to the location where the crime occurred and possessing a clear alibi, law enforcement detained her for several months based on the algorithm-generated “match result” until the case was ultimately dismissed.
(Figure2.Angela Lipps,50,was first arrested in Tennessee on July 14. Source:CNN）
Even more ironically, the very law enforcement agency using the system—the West Fargo Police Department—had purchased the Clearview system on its own initiative, and senior leadership was completely unaware of this until the error occurred. Furthermore, Clearview’s algorithmic “black box” poses a significant risk of privacy breaches. Since facial data is sensitive personal information that cannot be changed, once leaked or misused, users will face the irreversible risk of identity theft.
The news about Clearview highlights a key issue with algorithmic black boxes: on the one hand, individuals have no insight into how the algorithm generates matching results; on the other hand, the enforcement officers are often influenced by “automation bias” in practice, tending to place excessive trust in algorithmic outputs without questioning their accuracy. They overlook Clearview’s disclaimer in its terms of service stating that “results are for reference only,” which prevents errors from being promptly detected or traced, ultimately leading to an erroneous trial.
This case not only highlights the inherent risks of flawed algorithms, but also reveals how the “black box” nature of algorithms can be amplified by institutional structures and practices—when inexplicable technological outcomes enter systems of power, the consequences can shift from mere “technical errors” to substantive infringements on individual freedoms.
Comparative Analysis: Commercial vs. Governance Algorithms
The two scenarios above show us the different traits of algorithmic black boxes.
The main thing they have in common is that they both rely on large-scale data collection and are highly opaque, which makes it hard for individuals to dispute the choices made by algorithms.
The key difference is that the two have quite different types of risks and effects on society. The algorithms of food delivery platforms mainly serve business interests, their risks mostly show up as economic inequality, which leads to economic losses; while the facial recognition system is used as a part of the national governance system, its impact extends to citizens’ rights and freedoms.
This disparity indicates that the algorithmic black box is not a simple, single issue but rather takes on different forms in different institutional environments. Just as Kitchin (2017) pointed out, algorithms should be regarded as socio-technical systems, and their impact depends on the specific application scenarios and power structures.
Core Dilemma: Why the Algorithmic Black Box Unbreakable?
( Source: Google image）
- 1.Technology
Complex machine learning models (such as deep neural networks) have highly nonlinear structures, making their decision-making processes difficult to be explained (Burrell, 2016).
The pricing algorithms of food delivery platforms integrate various factors such as user profiles, supply and demand relationships, and profit targets. The calculation process is extremely complex.
Similarly, The facial recognition algorithm of Clearview, whose basis for judgment is sealed in a huge parameter matrix, can only be understood by machine operations. Even if the information is made public, ordinary people who lack the necessary knowledge basis will still struggle to fully comprehend the algorithm (Burrell, 2016)
This technical complexity and lack of interpretability make “opening the black box” exceptionally difficult.
- 2.Business
Enterprises use the algorithmic black box as their “core weapon” to achieve economic benefits. In practice, firms typically refuse to disclose algorithm details to safeguard their financial interests, making algorithm transparency difficult to achieve (Pasquale 2015).
The “data-driven price discrimination” practiced by food delivery platforms is essentially a strategy to maximize profits through an algorithmic black box – exploiting information asymmetry to implement differentiated pricing for different users, without having to bear the pressure of public opinion regarding fairness. Clearview, meanwhile, uses algorithmic black boxes to conceal the illegality of its data collection and algorithmic flaws, thereby selling services to law enforcement agencies and reaping massive profits.
Algorithms are a key competitive advantage for businesses, if they are completely exposed, competitors may copy them and harm the company’s market supremacy. Also, being open about problems like data misuse and algorithmic prejudice could lead to fines from regulators and censure from the public.
Consequently, they all have no motivation to willingly open their algorithmic black boxes and attain transparency; instead, they intentionally increase the opacity of their black boxes.
- 3.Regulation
At the moment, AI, algorithms, and dataization are developing much faster than the rules that govern them are being improved. Because of this, there aren’t any clear regulations that can be followed right now for numerous parts of algorithm requirements.
The Personal Information Protection Law, which has been enacted in China, mandates transparency in algorithmic decision-making. However, it does not specify the methods to ensure transparency or the penalties for lack of transparency. The EU’s Artificial Intelligence Act mandates the registration of high-risk AI systems, but there are still grey areas regarding the disclosure of algorithmic details.
On the one hand, accountability is unclear: platforms use “automated algorithmic decision-making” as a justification to evade responsibility, enforcement agencies overlook disclaimer clauses, all making it difficult to trace accountability. So when algorithmic “black boxes” lead to decision-making errors, who bears the responsibility? Is it the algorithm developers, the companies, the users, or the regulatory authorities?
On the other hand, the blurred ethical boundaries have made things worse: algorithms do not automatically get rid of inequity; instead, they unintentionally amplify the existing unfair structures. These variables are highly correlated with race, class, gender, etc. in the real world, which are often difficult to solve through technological means and lack clear ethical norms for regulation.
Approaches to the Algorithmic Black Box
(Figure3.Explainability: Cracking open the black box, Part 1.Manu Joseph. Source: author’s own illustration)
In this context, the academic community has proposed algorithmic auditing as a middle ground. This means that an outside group would look at algorithms to find any biases or unfairness (Sandvig et al., 2014). This method lessens some of the business risks that come with fully exposing algorithms, but it only works if auditing standards and data are available, which is hard to do in practice because of differing standards or built-in hostility (Kroll et al., 2017).
From a technical perspective, the development of Explainable Artificial Intelligence (XAI) offers new possibilities for alleviating the black box nature of algorithms. For example, using local explanation models (such as LIME or SHAP), researchers can explain individual decision results, improving system comprehensibility (Ribeiro et al., 2016). However, these methods frequently only yield “approximate explanations” rather than a thorough reveal of the underlying logic of the model, so their application in high-risk settings is still limited.
It is important to note that there are inherent contradictions in the aforementioned governance approaches. For example, increasing transparency may undermine corporate competitiveness, but reinforcing regulation may hinder technical innovation. The regulation of algorithmic black boxes should not be viewed as a single technological challenge, but rather as a multifaceted process comprising trade-offs between technology, market, and institutions (Kitchin, 2017).
Conclusion
The problem of algorithmic “black boxes”has never been a defect in technology itself, it should be attributed to the abuse of technology and the absence of rules by humans. The main goal of algorithms and data-driven systems is to make life better and more equitable. When algorithms stop being neutral tools and start being an unseen force to decide who gets what, what rights people have, and how society should work, what should we do?
The future approach to governance may not entail completely “opening the black box”, but rather establishing a governing structure with multiple layers. By combining limited transparency, independent audits, and accountability mechanisms, we may can protect innovation while mitigating the potential harm that algorithm poses to social fairness and individual rights. Ultimately, it’s not the machines’ will that matters, but human responsibility!
Reference
Pasquale, F. (2015). The black box society: The secret algorithms that control money and information. Harvard University Press. https://doi.org/10.4159/harvard.9780674736061
Burrell, J. (2016). How the machine “thinks”: Understanding opacity in machine learning algorithms. Big Data & Society, 3(1). https://doi.org/10.1177/2053951715622512
Kitchin, R. (2017). Thinking critically about and researching algorithms. Information, Communication & Society, 20(1), 14–29. https://doi.org/10.1080/1369118X.2016.1154087
Sandvig, C., Hamilton, K., Karahalios, K., & Langbort, C. (2014). Auditing algorithms: Research methods for detecting discrimination on internet platforms. In Data and discrimination: Converting critical concerns into productive inquiry.
Kroll, J. A., Huey, J., Barocas, S., Felten, E. W., Reidenberg, J. R., Robinson, D. G., & Yu, H. (2017). Accountable algorithms. University of Pennsylvania Law Review, 165(3), 633–705.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why should I trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1135–1144). https://doi.org/10.1145/2939672.2939778
Standing Committee of the National People’s Congress. (2021). Personal Information Protection Law of the People’s Republic of China.http://www.npc.gov.cn/npc/c2/c30834/202108/t20210820_313088.html
European Parliament & Council of the European Union. (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act).The Act Texts | EU Artificial Intelligence Act
Images:
What can we learn from the design of Chinese delivery apps? | by Britta Cheng | UX Collective,Mar 17, 2018,from https://uxdesign.cc/what-can-we-learn-from-the-design-of-delivery-apps-in-china-e5af4675cec8
Explainability: Cracking open the black box, Part 1 – KDnuggets,By Manu Joseph,From,https://www.kdnuggets.com/2019/12/explainability-black-box-part1.html?utm_source=Pinterest&utm_medium=organic
CNN. Mar 29, 2026.Police used AI facial recognition to arrest a Tennessee woman for crimes committed in a state she says she’s never visited,from https://edition.cnn.com/2026/03/29/us/angela-lipps-ai-facial-recognition
Be the first to comment