Designed to Hook You: The Attention Economy Is Not Neutral — It Is Built to Harm

You open TikTok to look at salad recipes.
Five videos later, you’re watching someone explain why eating under 800 calories a day is fine.

You didn’t search for this. You didn’t ask for it.
The algorithm led you there — one small step at a time.

This is not a personal failing. This is a design decision.

Platforms like TikTok do not passively reflect what users want. They actively shape what you see, feel, and believe — because keeping you emotionally activated is how they make money.

That makes this a governance problem, not a personal one.

“Engagement” Doesn’t Mean What You Think

The algorithm is not asking what you enjoy. It is finding what you cannot look away from.

The result is not a neutral feed. It is an engineered environment, built around your psychological vulnerabilities, optimised for profit.
The longer you stay, the more certain it becomes.

And here is the problem: what keeps people watching is not what makes them feel good.

Emotionally activating content — material that triggers anxiety, insecurity, or outrage — consistently outperforms neutral content in generating watch time. A video that makes you feel bad about your body is more likely to keep your thumb still than one that doesn’t.

This is only possible because of how thoroughly platforms track behaviour.

What van Dijck (2014, as cited in Flew, 2021, p. 105) calls datafication — “the transformation of social action into online quantified data, thus allowing for real-time tracking and predictive analysis” — helps explain why this happens.

How the System Actually Works

Taken together, these mechanisms form an engagement-optimisation system — a feedback loop that continuously aligns content with what holds attention, rather than what informs or supports users.

This reflects what Just and Latzer (2017, as cited in Flew, 2021, p. 110) call algorithmic governance — the ways in which automated, data-driven processes come to shape both what we see and how we act, operating through rules and routines that “both limit activities and create new room for maneuver” (Just and Latzer, 2017, p. 244, as cited in Flew, 2021, p.110).

Three Platforms. Three Scales. One Logic.

These mechanisms are not unique to TikTok. They show up wherever engagement is the primary metric.

Individual level: TikTok and algorithmic drift

In 2021, the Wall Street Journal built 100 automated test accounts on TikTok, each programmed to linger on content matching a hidden interest — including sadness and depression. No searches. No follows. Just watch time.

One account, nicknamed “kentucky_96,” saw its first sad video at video 15, three minutes in. Thirty-three minutes later, after 224 videos:

93% of all content served was about depression, anxiety, heartbreak, and self-harm — without it ever asking for any of it (Barry, West & Wells, WSJ, 2021).

Community level: Reddit and toxic technocultures

Researcher Adrienne Massanari’s study of Reddit shows the same logic at community scale. Reddit’s karma-and-upvote system provided what she calls “fertile ground” for toxic technocultures to take hold (Massanari, 2017, p. 330).

Communities built around harassment and misogyny, like those behind #Gamergate, thrived precisely because outrage drives upvotes. The platform didn’t create the hate. But it systematically rewarded it.

Societal level: YouTube and radicalisation pathways

At population scale, the same dynamic produces radicalisation pathways. Ribeiro et al. (2020) audited over 330,000 YouTube videos and found consistent user migration from moderate content toward far-right communities — through incremental algorithmic steps, not deliberate searching.

What researchers have called a filter bubble is more precisely an engagement-driven epistemic drift: the algorithm optimises toward homogeneity because agreement and validation generate more watch time than challenge and discomfort. Over time, your sense of what is normal, credible, and true slowly recalibrates around what keeps getting served to you.

These platforms differ in design and governance. But what drawing on Gillespie and Van Dijck (as cited in Massanari, 2017, p. 336), calls “platform politics” — the assemblage of design, policies, and norms that encourage certain kinds of cultures and behaviors to coalesce on platforms while implicitly discouraging others — operates identically across all three.

When the same optimisation logic reproduces harm across platforms with entirely different designs, the problem cannot be attributed to any single platform’s choices. It is systemic.

The harm is not incidental. It is structurally embedded in what these systems are built to maximise.

Why Platforms Won’t Fix This

The people scrolling through salad recipes are not stupid. They are not weak. They are operating inside a system designed, by some of the most sophisticated engineers in the world, to find and exploit their psychological threshold. They never had a fair chance.

Platforms know about these harms. They have internal research documenting them. They have been sued over them.

They respond with the same script: community standards, content moderation teams, ongoing improvements.

But content moderation and algorithmic optimisation are two completely different systems — and the second one consistently overrides the first.

Content moderation asks: does this content break our rules?

Algorithmic optimisation asks: will this content make the user stay?

A video promoting disordered eating can pass every content check and still be aggressively served — because anxious users tend to watch to the end. These two systems don’t conflict. They simply don’t talk to each other.

AI scholar Kate Crawford argues that because AI systems “are ultimately designed to serve existing dominant interests,” artificial intelligence is, in her words, “a registry of power” (Crawford, 2021, p. 8). The real question is not how they work technically, but “what is being optimized, and for whom, and who gets to decide” (Crawford, 2021, p. 9).

The answer: advertising revenue, for shareholders, decided entirely by the company.

Massanari found the same logic on Reddit — administrators were “loathe to make any concrete changes” (Massanari, 2017, p. 342) despite documented harm, because of a deep reluctance to alienate any audience “no matter how problematic, as it will mean less traffic and ultimately less revenue” (Massanari, 2017, p. 340).

And users have almost no recourse. Suzor (2019) shows that platform terms of service reserve what he calls “absolute discretion” to operators — leaving users no meaningful option except to leave entirely (Suzor, 2019, pp. 10–24).

As Flew (2021, p. 99) observes, the largest platforms “increasingly function as public infrastructure even as they involve private companies” — shaping how billions access information and form political views, while remaining accountable primarily to shareholders rather than to the public they serve.

That is the governance gap. That is where the harm lives.

What Regulation Is Trying — And Why It’s Not Enough

Three regulatory approaches have emerged. Each targets a different layer of the problem.

Transparency — EU Digital Services Act (2022): platforms must conduct algorithmic risk assessments and submit to independent audits. Meaningful — but audits depend on platform cooperation, and Pasquale (2015, pp.1–18) warned a decade ago that algorithmic systems function as “black boxes” — their inner workings hidden from users and regulators alike, with proprietary methods shielded by trade secrecy law even when those systems make decisions with profound consequences for people’s lives. That protection remains largely intact.

Harm removal — Australia’s Online Safety Act (2021): the eSafety Commissioner can investigate complaints and require platforms to remove severely harmful content. While the Act also introduces proactive “Safety by Design” expectations, its enforcement mechanisms remain largely reactive — triggered by harm that has already occurred, targeting content rather than the algorithmic logic that amplified it.

Duty of care — Woods (2021) argues platforms should face a statutory duty of care modelled on workplace health and safety law — obliged to identify and manage foreseeable risks in their systems, much as an employer must ensure a reasonably safe environment for those affected by their operations. The logic, as this article extends it, is similar to product liability: the burden of demonstrating safety should rest with the platform, not with regulators trying to prove harm after the fact.

All three, however, share the same fundamental limitation: they are reactive rather than structural.

They make harm more visible, more removable, or more compensable. None of them require platforms to change what their algorithms actually optimise for.

As long as engagement maximisation remains the commercial objective, the incentive to produce harmful outcomes remains baked into the system.

The question regulation has not yet answered — and is not currently designed to answer — is whether a platform has the right to engineer an emotional environment around your vulnerabilities in the first place.

Your attention is the product. Your wellbeing is a cost they have chosen not to pay.

The regulatory frameworks we have built are mostly trying to limit the damage after the fact. That is not enough.

This is not a personal failing. It is a governance failure — one that requires us to move beyond reactive regulation toward something more fundamental: the right to an information environment that does not treat your vulnerabilities as a revenue stream.

Remember the people who opened TikTok to look at salad recipes.

They are still scrolling.

The algorithm already knows what it’s going to show them next.

The only question left is who gets to decide whether that’s acceptable.

References

Barry, R., West, J., & Wells, G. (2021, July 21). Investigation: How TikTok’s algorithm figures out your deepest desires [Video]. The Wall Street Journal. https://www.wsj.com/video/series/inside-tiktoks-highly-secretive-algorithm/investigation-how-tiktok-algorithm-figures-out-your-deepest-desires/6C0C2040-FF25-4827-8528-2BD6612E3796

Crawford, K. (2021). The atlas of AI: Power, politics, and the planetary costs of artificial intelligence. Yale University Press.

Flew, T. (2021). Issues of concern. In Regulating platforms (pp. 98–127). Polity Press.

Massanari, A. (2017). #Gamergate and The Fappening: How Reddit’s algorithm, governance, and culture support toxic technocultures. New Media & Society, 19(3), 329–346. https://doi.org/10.1177/1461444815608807

Pasquale, F. (2015). Introduction: The need to know. In The black box society: The secret algorithms that control money and information (pp. 1–18). Harvard University Press. http://www.jstor.org/stable/j.ctt13x0hch.3

Ribeiro, M. H., Ottoni, R., West, R., Almeida, V. A. F., & Meira, W. (2020). Auditing radicalization pathways on YouTube. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (pp. 131–141). ACM. https://doi.org/10.1145/3351095.3372879

Suzor, N. P. (2019). Who makes the rules? In Lawless: The secret rules that govern our digital lives (pp. 10–24). Cambridge University Press.

Woods, L. (2021). Obliging platforms to accept a duty of care. In M. Moore & D. Tambini (Eds.), Regulating big tech: Policy responses to digital dominance (pp. 93–109). Oxford University Press.