Imagine being denied a loan, rejected for a job, or flagged as a health risk by a system that won’t explain why. This is called algorithmic governance: the use of computers to determine decisions that affect your daily life, with no humans involved in the decision-making and no accountability (Flew, 2021, p. 79). Today, algorithms are the new “gatekeepers” to opportunity, and they categorize you into groups that you created and behaviours that you will exhibit based on your social media feed or your credit score.
Algorithmic governance is one of the most significant forms of changing the game of who has the most power in our society. Since algorithms can look at large quantities of data, and because they can make many thousands of decisions per second (a rate that far exceeds the ability of humans), they have great efficiency and predictability. However, they create a new type of obscurity and partiality horizontally that threatens two key principles of democracy: openness and equality. I want to delve into how algorithmic governance functions, why it is a concern for your digital rights, and what actions may need to be taken to prevent these systems from negatively impacting the public good.
Figure 1: Source: Missouri School of Journalism
What Is Algorithmic Governance?
In its simplest definition, algorithmic governance pertains to making use of computational procedures that allow machines to perform automated work and arrive at choices that would otherwise have been made manually (Flew, 2021, p. 79). Think of it as rule by recipe: instead of a person looking at evidence and making a choice, a computer follows a set of instructions to reach conclusions about who gets what, when, and how.
There has been a tremendous increase in the area of algorithmic governance. In the healthcare sphere, AIs in the present day are used to help diagnose illnesses and provide priority to patients to receive treatment. Recidivism algorithms are used to predict risks and drive sentencing in criminal justice. Automated systems in the financial sector identify creditworthiness and identify fraud. The commonality between these applications is that they give up much of the social power of the technical systems that are not fully understood by most of the population (Suzor, 2019, p. 10).
The major difference between today’s use of algorithmic governance and earlier computerization is the extent of machine learning in today’s systems. Today’s systems use large amounts of data to look for patterns and develop a predictive system from those patterns through the development of statistical correlations; whereas, they do not have an understanding of cause and effect. Legal Scholar Frank Pasquale describes this system as a “black box” where money and information are governed by secret algorithms (Suzor, 2019, p. 11), with the term black box used to describe how completely opaque the decision making process of such systems is to both their developers and end-users.
A Real World Warning from the NHS
The UK’s National Health Service exemplifies the promises and dangers associated with the deployment of healthcare Artificial Intelligence (AI) within healthcare settings. With nearly 7 million individuals awaiting medical treatment and over 100,000 vacant positions, the NHS has turned toward AI to assist in identifying disease sooner, in analyzing radiological images more quickly, and to manage other non-clinical tasks (De Costa, 2024). Dr. John Bell, an advisor to the British government, believes that AI can transform the way that clinicians provide care through improved diagnostic precision and personalized treatments.
Figure 1: Dr. Bell. Source: https://www.bbc.com/news/articles/c6233x9k4dlo
However, there are already real-world problems beginning to arise from the rapid development and implementation of AI in clinical environments. Patients have been leaving their physicians’ offices due to fears about how the practice will utilize their private information via the use of AI and potentially miss necessary treatment altogether (De Costa, 2024). Dr. Caroline Green at Oxford University’s Institute for Ethics in AI states that physicians are currently utilizing untrained tools such as ChatGPT, which places the risk of incorrect guidance to patients. Furthermore, these systems “can hallucinate,” creating fictitious information and could be created from data sets that do not represent the diversity of all patient populations, thereby exacerbating current inequities related to race and sex.
Thus, the reason that democratic governance is essential becomes apparent. The Health Foundation has urged for the creation of a national plan to equitably implement AI into clinical environments and regulatory agencies continue to fall behind in terms of regulating rapidly developing technologies (De Costa, 2024). As Dr. Paul Campbell from the Medicines and Healthcare Products Regulatory Agency explained, “As a regulator, we must balance appropriate oversight to protect patient safety with the agility needed to respond to the particular challenges” – an issue that has yet to be resolved (De Costa, 2024). Therefore, the NHS example illustrates that if there is no true accountability to stakeholders, no genuine transparency regarding the development and utilization of AI, and no confidence from patients, then AI poses an enormous threat to the patients whom it is intended to aid.
What we Lose for Convinience
Algorithmic governance is being driven toward through promises of increased efficiency, improved accuracy, and lower cost. Governments see AI as a means of implementing a more neutral approach to the allocation of limited resources, particularly in relation to patient care. Government agencies view automated decision-making as a solution to reducing backlogs within their bureaucracy while removing human bias from decision-making. Technology vendors also market their products as neutral arbiters that can replace flawed human decision-making with objective data-driven results.
Figure 1: Manual Processes. Source: Forbes
However, under this rhetoric of technology neutrality lies a serious risk. As media scholar Kate Crawford states in her book The Atlas of AI, the impact of the decisions being made by today’s algorithms will be significantly larger than those made by humans (Crawford, 2021, p. 1), which include harm to people and the planet. Some examples of these impacts are the environmental damage caused by large energy-using data centers used to run algorithms; the exploitation of workers throughout the globalized supply chain; and the erosion of democracy at the hands of monopolistic capitalist interests. Automating government is not simply creating faster ways of doing things but fundamentally changing how power is exercised.
It is clear from this point of view why there is a strong correlation between digital rights and the dependency upon the algorithm systems to function. That is because the success of algorithm systems depends on the successful collection and extraction of very large amounts of quality personal data, including your browsing history, location, social connections, purchases, consumption patterns, and most importantly, biometric data, including your face and/or voice print. This data extraction and surveillance capabilities allow for what Shoshana Zuboff refers to as “surveillance capitalism,” an economy that is based solely upon the gathering of human experience and subsequently turning it into commerce (Flew, 2021, p. 72). Your data becomes the material from which these systems are built, and then they transform your lived experiences into profit and power sources for someone else.
The Accountability Gap
The most concerning issue related to algorithmic governance is the lack of accountability it allows. When a human bureaucrat denies your application, you typically receive some form of rationale or justification for the denial. If you do not agree with the denial, you can ask for an explanation, appeal to a supervisor or take legal action. On the other hand, if you are denied by an algorithm you receive an unexplainable response — “the computer says no” (Suzor, 2019, p. 10)
This opacity matters because algorithms are not neutral. They incorporate the assumptions and biases of the creators, the constraints of their training data, and the interests of the institutions that implement them. Researchers show in studies of AI-assisting hiring tools like these that discrimination against women and minorities can be reproduced through AI systems based on historic employment data, while also creating a real-world automation of inequality (Singhal et al., 2024). The bias within the AI models is difficult to identify or challenge since the bias exists within the model’s complex neural networks.
New transparency rules are being enacted in hopes of addressing issues of discrimination through requiring high-risk AI systems to provide the reasoning behind their choices (Flew, 2021, p. 134); however, many believe transparency alone will not resolve this issue. Due to the nature of automated systems that create harm through their processes, the use of humanistic ethical frameworks to analyze the creation of such negative consequences is ineffective (Shahid et al., 2025, p. 5). They instead propose developing a posthumanist relational ethics that recognizes the role of algorithms in shaping social realities. Using this lens of thinking about algorithms, accountability for the actions of these automated decision-making systems would require considering more than simply looking beyond the ‘black box.’ It would be necessary to consider how our views on responsibility should change as we begin to work together with both humans and machines on making decisions.
Toward Democratic Algorithmic Governance
Therefore, to resolve issues related to this type of problem-solving, there is a need to move from technology-based solutions toward the concept of democratic AI, which refers to population-based participation in developing definitions as to how algorithmic systems function. Rather than simply informing populations that algorithms exist, democratic AI offers the opportunity for those impacted by decision-making processes to evaluate decisions made, receive clarification on the logic behind them, and have input on the value base used when designing automated systems.
A number of principles are present in recent research. To begin with, there should be true human supervision and not nominal. The presence of a human in the loop does not matter when all that person has is no power, data, or time to substantially consult the algorithmic recommendations (Singhal et al., 2024). Second, it should be contestable, and citizens should have access to opportunities to oppose algorithmic judgments and have a remedy in the event of failure. Third, bias and a lack of equity may be detected through regular auditing by independent bodies and compliance with equity standards (Singhal et al., 2024).
These are the principles that are now being integrated into regulatory frameworks. The EU AI Act sets up risk-based criteria for AI applications, with a set of tighter standards imposed on the high-stakes areas of healthcare and criminal justice (Flew, 2021, p. 134). Most recently, regulations of the Office of the National Coordinator of Health Information Technology in the United States require transparency in healthcare AI, where developers are required to share how their systems operate and what data they use (Crawford, 2021, p. 2). These developments represent important steps, but critics argue they remain insufficient for addressing the structural power imbalances that algorithmic governance creates.
Your Rights in an AI World
Algorithmic governance is neither positive nor negative. It is simply an instrument of governance based on the values and interests of those who create and apply it. The question is therefore whether we should allow such systems to be opaque instruments of control or require them to provide transparency and democratic accountability in their operation.
The potential for harm from unchecked algorithmic power is substantial. The use of this type of technology has the capacity to solidify inequality through automation, erode personal privacy, and centralize decision-making authority in the hands of corporate entities and government agencies that lack democratic accountability. However, there are also legitimate opportunities for AI to enhance human abilities, decrease drudgery, and possibly lead to fairer decisions than biased human bureaucrats. Achieving these opportunities while avoiding the potential for harm will depend upon continued public participation, robust regulatory structures, and our ability to delay implementation of automation when the value being pursued undermines democracy.
As individuals, one of the first steps we can take is to ask ourselves two questions regarding the systems that are determining our lives: What systems are creating decisions related to my life? What data do those systems rely upon? When I am negatively impacted by a system that is making decisions about my life, what individual or group(s) are held accountable? As citizens, we can advocate that algorithmic governance should be governed under the same standards of transparency, accountability, and opportunity to participate as would be expected from human authority. If software is capable of controlling aspects of people’s lives, then the people must have control over the software.
References
Crawford, K. (2021). The atlas of AI: Power, politics, and the planetary costs of artificial intelligence (pp. 1–21). Yale University Press.
De Costa, K. (2024, August 7). AI in Healthcare? What are the risks for the NHS? BBC. https://www.bbc.com/news/articles/c6233x9k4dlo
Flew, T. (2021). Regulating platforms (pp. 72–79, 134–166). Polity Press.
Shahid, F., Agarwal, D., & Vashistha, A. (2025). One style does not regulate all: Moderation practices in public and private WhatsApp groups. Proceedings of the ACM on Human-Computer Interaction, 9(2), 1–30. https://doi.org/10.1145/3711042
Singhal, A., Neveditsin, N., Tanveer, H., & Mago, V. (2024). Toward Fairness, Accountability, Transparency, and Ethics in AI for Social Media and Health Care: Scoping Review. JMIR medical informatics, 12, e50048. https://doi.org/10.2196/50048.
Suzor, N. P. (2019). Lawless: The secret rules that govern our digital lives (pp. 10–24). Cambridge University Press. https://doi.org/10.1017/9781108666428
Be the first to comment