In February 2026, eight people died in Tumbler Ridge, Canada. The ChatGPT account of gunman Jesse Van Rootselaar was marked by the OpenAI internal system as early as 8 months ago.
His dialogue involved the scene of gun violence and many employees were aware of it. Someone suggested contacting the police. But OpenAI just blocked his account.
Their reason? The threat “did not meet the internal threshold.”
Eight months later, he pulled the trigger.
This Is Not a Gun Control Article
This article is about a stranger, more uncomfortable question:
When an AI knows someone might kill people, does it have a legal duty to speak up?
Right now, in most countries around the world – the answer is no. That answer has real consequences. Real deaths. Real families destroyed.
Let me walk you through why this happened, and what needs to change.
The Black Box: We Don’t Know Where the Threshold Came From
OpenAI said the threat “didn’t meet the threshold. “Who set the threshold? Based on what data? How often is it updated? No one knows.
This is exactly the question raised by Frank Pasquale in 2015. He pointed out in The Black Box Society that the algorithm system deliberately remained opaque, and the company refused to disclose internal operations on the grounds of “trade secrets”. Outsiders, including regulators and the public, cannot know how the algorithm makes decisions. They just have to trust the company. And trust, as we saw in Tumbler Ridge, is not always justified.
This is exactly the case in the Tumbler Ridge case. The numbers in a black box determine the life and death of eight people. Pasquale warned us about this exact scenario. When critical decisions are locked inside black boxes, accountability disappears. No one has to explain what happened. No one can be held responsible when things go wrong.
Power Asymmetry: The AI Knows, You Can’t Ask
OpenAI employees are not policemen, psychologists and risk assessment experts. They are technicians. But the company gave them to judge whether the threat was real or role-playing.
What’s the problem?
In 2021, Kate Crawford gave the answer in The Atlas of AI. Crawford studied the power structure of the AI system, and she pointed out that AI is not a neutral tool. It embodies the values, assumptions and prejudices of the creator. More importantly, AI has created a profound asymmetry of power.
What OpenAI had:
· Full access to the shooter’s conversation history
· Full knowledge of their own internal threshold settings
· Full ability to change those settings at any time
· Full authority to decide whether to contact police
· Full legal protection (no law required them to report)
What the potential victims had:
· Zero knowledge that a threat had been detected
· Zero ability to ask OpenAI to review the decision
· Zero legal right to be informed about the risk
· Zero recourse after the shooting happened
This is a system designed to protect the company, not the public.
This right to know but not act, to have information but not to take responsibility is the dark side of what Crawford called “extraction logic”. AI extracts our data, extracts our dialogue, and extracts our weaknesses. But when we ask the system to be accountable, the company says: “We’re just a tool. We don’t make the final call. Don’t blame us.”
Except in Tumbler Ridge, they did make the final call. They decided not to report. And eight people died.
Automated Culture: Moral Questions Become Technical Questions
Why do we give the task of “judging danger” to AI?
Mark Andrejevi summarized and analyzed this phenomenon. He put forward the concept of “automated culture”. Andrejevic does not mean “we have a lot of automation tools”. What he means is that we are used to outsourcing decision-making to machines.
For example, let the algorithm decide what news we watch, who gets the loan, and which post will be deleted; now even let the algorithm decide what is dangerous enough. Andrejevic issued a warning: When we outsource decision-making, we also outsource judgement. We also outsourced moral responsibility.
In Tumblr Ridge, OpenAI employees rely too much on the “threshold”. They won’t ask, “Is this person really dangerous?” They just care about the standard.
That shift is a classic symptom of automated culture. We turn moral questions into technical questions. We turn human judgment into a yes/no checkbox on a dashboard, when the checkbox says “no,” we walk away.
Governance by Algorithms: Who Gave Algorithms the Authority?
Algorithms are not just passive tools. They are becoming much bigger. They are becoming governors. They are making decisions that used to belong to humans, and we are letting them.
Natascha Just and Michael Latzer made this argument in their 2017 paper Governance by Algorithms. They write that algorithms don’t just process information, they actively construct reality through a process they call “algorithmic selection.”
Every day, the algorithm decides what you can see; what information is considered important and what will be filtered; which sounds are amplified and which are suppressed.
In Tumbler Ridge, OpenAI decided that this particular threat was not worth reporting to anyone. Not to police. Not to mental health services. Not to anyone.
This is called “algorithm selection”. The algorithm screened the information, and it classified the gunman’s dialogue as “not up to standard”. Then, the technicians accepted the judgement. The algorithm governed their behavior.
But the problem is that the algorithm selection is not neutral. It is based on the designer’s assumptions and priorities. If the assumption is wrong, the whole governance system will collapse.
When it fails, the people who are harmed including potential victims of violence have no right to appeal. No right to ask for a review. No right to even know why. The algorithm’s decision is final.
The Regulatory Gap: Where Is the Law?
Where is the law in all of this? Where are the elected officials and regulators?
Terry Flew addresses this question directly. Flew studies how governments can and should regulate digital platforms. He identifies a core tension that makes this work so difficult: innovation versus accountability.
Platform companies want to maximize the freedom of innovation. Regulation will slow down their development, push up costs, and push jobs to other countries. The government hopes to protect the public from foreseeable harm. They believe that some basic rules are necessary to prevent disasters like Tumbler Ridge. There is a huge gray area between the two, where enterprises can do almost whatever they want.
The Tumbler Ridge case exposed just how dangerous that grey zone can be when it comes to public safety.
Consider what Canada currently lacks:
· No law requires AI companies to report threat information.
· No standard definition of “trusted threat”
· No institution to supervise the AI audit system
· No channel for victims to be held accountable.
Flew wrote that the core challenge of platform supervision is to “keep up with the pace of technological development”. The law is progressing slowly, and it will take several years to pass new legislation, study its impact and make adjustments. And artificial intelligence will have new models, new functions and new risks every month.
The result is a permanent gap. The law is always several years behind the technology. And in those years, companies operate in a near-vacuum.
They set their own internal rules. They make their own judgments about what matters. They ignore their mistakes when it is convenient or when fixing them would cost too much.
There are no consequences for getting it wrong.
Until eight people die. Then there are still no consequences, because no law was broken. The company just says “we followed our internal policies” and moves on.
Would Reporting Have Stopped the Shooting?
The answer is: not necessarily. We cannot know for sure.
Police might have investigated. They might have found no immediate evidence of a concrete plan. The shooter might have been interviewed and then released.
But we will never know the answer to this question.Because we never tried. The system never gave anyone the chance to try.
The point is not guaranteed success. The point is that we will never know, because the decision was made by a private company with no legal obligation to act. We had a moral obligation to at least try.
What Needs to Change? Three Concrete Fixes
Fix One: Mandatory Reporting Laws
Artificial intelligence companies must clearly assume the legal obligation to report credible threats of violence to law enforcement. It’s not complicated. We have developed similar regulations for other industries, such as psychology, teachers and social workers. It’s time to include artificial intelligence companies in this list.
The law should include:
· Determine the definition of “trusted threat”. This standard should be formulated by independent experts.
· Widely applicable law. Any artificial intelligence system that handles user-generated content should be included in the scope of supervision.
· Formulate practical punishment measures-. Failure to report the situation that leads to the occurrence of harm should make the company and possible individual decision-makers criminally liable.
Fix Two: Independent Audits
We cannot simply trust what AI companies tell us about their safety systems.
Independent audits shall require third-party audits every year; random spot checks shall be conducted every year, and auditors shall review the handling of reported cases (including unreported cases); each company shall be forced to publish summary data on the number of reported threats, the number of reports and why some threats were not reported; and finally, the auditors should be given the executive power.
Fix Three: Minimum Human Review Standards
Algorithms should never be the final decision-maker in life-and-death situations.
What that means in practice:
· High risk flags must be reviewed by trained human moderators. · Companies cannot rely solely on “internal thresholds” to avoid responsibility. · Moderators need proper training. · Create escalation pathways.
Objection: Won’t This Destroy Privacy and Free Speech?
Somebody will say: mandatory reporting will make people afraid to use AI. People will self-censor. Innovation will slow down. Privacy will be destroyed. Free speech will be chilled.
These are valid concerns. They deserve to be taken seriously.
But we need to consider the pros and cons.
Option one: We protect privacy above all else. We do not require AI companies to report threats. But then, as we saw in Tumbler Ridge, threats slip through. People die. Eight families never see their loved ones again.
Option two: We create a narrow mandatory reporting law. Some false alarms get reported to police. Some users feel uncomfortable and stop using AI for certain topics. Some privacy is lost. But some lives are saved.
We do not have to choose between the extremes. There is a middle ground. There are careful, narrow policy designs that balance competing values.
Not every conversation needs to be reported. Only credible, specific, and imminent threats of violence should trigger mandatory reporting. That is a much narrower category than “anything that sounds mean” or “anything political.”
But the standard must exist. The duty must exist. Because right now, the standard is whatever each company decides internally – and we have seen where that leads.
As Kate Crawford says: AI’s power must come with AI’s accountability. You cannot have one without the other.
As Terry Flew says: regulation is not the enemy of innovation. Innovation without regulation is not freedom. It is a breeding ground for chaos, harm, and eventually a public backlash that leads to much heavier regulation.
Better to design thoughtful rules now, before the next disaster.
What You Can Do Now
This article is not meant to make you panic about AI. It is not meant to make you delete your ChatGPT account.
It is meant to inform you about a real problem that already exists.
Here is what you can do starting today:
First, share this story. The more people know about the “digital confession” problem.
Second, ask your elected representative a simple question: Does our country have laws requiring AI companies to report credible threats of violence to police?
Third, use AI carefully. Understand how the system works before you trust it with your secrets.
Fourth, support organizations that are pushing for AI accountability.
👇 The comments section is yours I have given you my argument. Now I want to hear yours. Do you think AI companies should be required to report threat information to police? Here are three quick options:
· 👍 Yes, even if it means some false alarms. Saving lives is worth the cost.
· ❤️ No, privacy and free speech matter more. Mandatory reporting would chill honest conversation.
· 🤔 Somewhere in between. I think there is a middle ground.
Pick one. Or ignore the options and just write down what you think.
I will read every comment. I will respond to as many as I can.
Let’s have a real conversation about this. Because the next Tumbler Ridge could be just one algorithm decision away.
Refernences
Andrejevic, M. (2019). Automated media. Routledge. (Chapter 3, pp. 44–72)
Crawford, K. (2021). The atlas of AI: Power, politics, and the planetary costs of artificial intelligence. Yale University Press. (pp. 1–21)
Flew, T. (2021). Regulating platforms. Polity. (pp. 79–86)
Just, N., & Latzer, M. (2017). Governance by algorithms: Reality construction by algorithmic selection on the Internet. Media, Culture & Society, 39(2), 238–258.
Pasquale, F. (2015). The black box society: The secret algorithms that control money and information. Harvard University Press. (Chapter 1, pp. 1–18)
Be the first to comment