The hidden data machine behind every ‘eerily accurate’ recommendation
Source of image: Author’s own
When I first arrived in Sydney as a student, I hadn’t searched for anything at all. But a few hours later, Xiaohongshu (also known as RedNote) was filled with information about renting houses in Sydney, travel guides and experience sharing.
A few weeks later, TikTok began to recommend “people I might know” to me – classmates I had met once, neighbors with whom I had never exchanged contact information, whom I had never searched for.
The real question is not “Is it eavesdropping on me?”, but why can the platform know so much without my knowledge and what right does it have to do so.
It doesn’t need to listen — it already knows
Source of image: Make Use Of (2021)
Platforms do not need “eavesdropping” to know so much; they rely on data accumulated through everyday use.
The researchers from Northeastern University systematically analyzed over 17,000 Android applications and found no application that would activate the microphone without authorization (Choffnes et al., 2018). From a technical perspective, large-scale monitoring is also almost impossible to hide for a long time. Antonio García-Martínez, a former product manager at Facebook, once calculated that if audio data is continuously transmitted to the server, the data generated by only the American users alone each day would be extremely huge, approaching the storage limit that the platform itself can withstand (García-Martínez, 2017).
In other words, the platform usually doesn’t need to “hear” what you said. It can already know enough from other signals.For example:
- Your IP address and approximate location
- What kind of content do you spend more time on
- Your device ID and app usage traces
This is exactly what Flew (2021) refers to as “datafication”. Our everyday actions are continuously transformed into data that platforms can collect, analyze, and use. The time you spend scrolling and what you look at are recorded to form a profile of you.
TikTok’s official documentation openly acknowledges that even if you’ve never tapped “Allow Location,” the platform can still determine your approximate location using your IP address and other device signals (TikTok, 2024). In other words, we assume that “not granting permission” means “not being tracked,” but these are two different things. You might think that simply turning off a permission switch is enough, but the platform never relies on just one path.
Your data does not stay where you gave it
Location is just one type of data. For instance, when you browse content on Xiaohongshu, the platform will record what you have viewed and for how long you have stayed on certain content. Then, these pieces of information are used for making recommendations and running advertisements.
At first, this might just seem like “the recommendations are getting more tailored to you”. But these data are not necessarily used only in one place. They can be initially used for content recommendations, and later they might be incorporated into advertising profiles or even be used for other commercial purposes. As a result, the issue is not just what content you will see, but who will continue to use these data and what consequences it will bring.
Case study: When ordinary app data becomes insurance data
Source of image: Tiwari (2025)
While the previous examples involved merely personalised content recommendations—where platforms use your data to determine what you see—the case of Allstate and Arity demonstrates that when the same logic is applied to the insurance industry, the consequences are entirely different. Data no longer merely influences the content you see; it directly determines how much you pay in premiums and even whether you can purchase insurance at all.
In January 2025, the prosecutor of Texas filed a lawsuit against the insurance company Allstate and its subsidiary Arity. Arity secretly embedded tracking software into some everyday apps, continuously collecting users’ location, movement and driving data on their mobile phones without their knowledge, and then using these data for insurance-related purposes (Office of the Attorney General of Texas, 2025). When users opened these apps, they thought they were just using a ordinary tool, but they didn’t know that they had also activated a data collection system that was constantly running.
This case matters because it reveals a crucial step in platform data logic:
- The app that users downloaded was an ordinary one;
- The embedded software in the app continuously collected location and behavior signals;
- These signals were analyzed, classified, and linked to driving or mobile patterns;
- Finally, these data were transferred to another completely different business scenario – insurance pricing and risk assessment.
More importantly, this was not an accidental data leak, but a designed data collection and utilization method without clear notification and user consent. Originally, the location and movement data left during daily use were later used to assess driving behavior and might affect insurance pricing.
Such systems can also make mistakes in truly important areas. Due to the difficulty of accurately distinguishing drivers from passengers based solely on mobile phone data, such “driving scores” may also make misjudgments (Gruber, 2025). A mobile phone moving at 80 kilometers per hour in the system’s eyes is just a data point moving at high speed – whether it belongs to the driver or the passenger in the back seat. This also indicates that the system is fundamentally unable to truly understand a person’s specific situation. The complex situations created by human are compressed into a few predictable numbers, and these numbers influence our decisions in real life.
This is precisely the moment when, as Nissenbaum (2018) puts it, “contextual integrity” is compromised. Nissenbaum argues that privacy is not just about whether secrets have been leaked, but also about whether information continues to circulate in the context in which it was originally shared. In this case, users provided data merely to use a particular app, but that location information and behavior patterns entered an insurance company ‘s risk assessment model in unknown ways. The data itself has not changed, but its use has. The data is used in ways that the user neither foresaw nor approved.
In 2024, the US Federal Trade Commission sued data broker Mobilewalla on the grounds that it collected and sold location data without users’ consent. (Federal Trade Commission [FTC], 2024).This demonstrates the maturity of the data broker industry, which integrates data from various sources to create marketable user profiles. As Flew (2021) pointed out, contemporary platforms rely on the continuous collection and analysis of user data, and the daily behaviors of users are also constantly transformed into tradable assets in this process.
You clicked “I agree” — but did you really choose?
When you download a new application, you have most likely done the same thing: quickly scroll to the bottom, click “Agree”, and then continue. Because in many cases, if you don’t do this, you will often be unable to continue using this app.
But is this “Agree” valid?
Research shows that if ordinary users carefully read the privacy policies of all the apps on their phones, it would take them hundreds of hours each year (McDonald & Cranor, 2008).These documents are often lengthy, complex, and do not always help users make truly informed choices (Attorney-General’s Department, 2023). Most of the time, privacy policies are not intended for users to truly understand, but rather to make them agree quickly.
Source of image: Khan (2021)
This is precisely the “take-it-or-leave-it” predicament as Suzor (2019) described: Users are not faced with whether they truly agree to these terms, but whether they should give up using this service. In this situation, “agreement” is more like a formal legitimization mechanism rather than meaning that users truly have substantive choice rights.
In other words, the issue is not whether users click “agree”, but whether the platform can obtain the legitimacy to handle user data merely based on such a weak form of consent. If privacy is regarded as a fundamental right, it should not be entirely based on a contract that users can hardly negotiate and find difficult to truly understand. As emphasized in Article 12 of the Universal Declaration of Human Rights, privacy should not be subject to arbitrary interference (United Nations, 1948).
“Just delete the app” is not a solution
Source of image: Gonçalves (2026)
Whenever someone raises the issue of digital privacy, the same answer is always given: Then you don’t use them, that’s all.
But today, what does “not using” digital services mean?
This “either accept or exit” choice is inherently unequal. Marwick and Boyd (2018) pointed out that privacy and surveillance do not occur in the same way for everyone; for many people on the margins of society, achieving privacy is particularly difficult. With the increasing prevalence of data-driven systems, the “privilege” of exiting these systems is becoming increasingly difficult to achieve.
Moreover, digital platforms have already permeated our daily lives. From life consumption, social connections, rental information, government services, online banking, entertainment and communication, everything relies on digital platforms to function. Therefore, completely shifting the responsibility for privacy protection back to users essentially means making individuals pay for the lack of design and regulation by the platforms.
So, “just delete the app” is not the answer.
If exiting is not equally feasible for everyone, then the privacy issue cannot be merely understood as a personal choice issue.
Why law still struggles to protect us
So, can the existing legal framework protect us?
One of the most influential privacy protection frameworks currently is the EU’s General Data Protection Regulation (GDPR). It enhances the standards for platforms to handle personal data by implementing principles such as purpose limitation and data minimization and strengthening the requirements for legitimate processing conditions like informed consent (European Union, 2016).
However, the problem lies in that when the collection, circulation, and use of data are already very complex and opaque, users’ “consent” may not truly limit these data from being further analyzed and reused. In other words, the existing legal framework is not completely ineffective, but it often still struggles to keep up with the complex and cross-contextual logic of data flow.
The situation in Australia is more complex. The 2025 privacy law reforms did bring about some progress; for example, the law established new avenues for redress in cases of serious privacy breaches and provided individuals with clearer safeguards. (Argon Law, 2025).However, Kemp (2023) points out that these reforms do not solve the fundamental problems.Platforms and data brokers continue to profit from user behavioral data,and in terms of targeted advertising, the existing reforms basically still use the original data usage methods.
From a governance perspective, the problem is not simply that platforms have “done something wrong”, but that the regulatory framework itself is deficient. Flew (2021) argues that platform governance requires a three-pronged approach: platform self-regulation, government oversight and citizen participation. In the field of digital privacy, relying solely on platforms to set their own rules is far from sufficient. Frankly, self-regulation is rarely effective on platform, government regulations are often slow to respond, ordinary users have no idea how these systems work in the background. Therefore, digital privacy is not just a matter of individual decision, but a matter of public governance, encompassing the power of platforms, regulatory rules, and institutional design.
The real question is not how it knows, but why it is allowed to know
Back to the night I first arrived in Sydney.
My Xiaohongshu knew where I was, and TikTok remembered who I had met. An unknown data broker may have packaged my location data and sold it to the next buyer. And I, like countless other ordinary users, knew almost nothing about any of this.
As Suzor (2019) pointed out, real change may require a reassessment of the mechanisms that govern the Internet.In the digital era, what role do platforms play in our lives, how should this power be curbed.And at the heart of it all is a simpler question that every user deserves to ask:
Why is it allowed to know so much?
References
Argon Law. (2025). How has Australian privacy law changed in 2025? https://argonlaw.com.au/legal-articles/australia-privacy-changes-2025/
Attorney-General’s Department. (2023). Privacy Act Review Report. Australian Government. https://www.ag.gov.au/sites/default/files/2023-02/privacy-act-review-report_0.pdf
Choffnes, D., Pan, E., & Ren, J. (2018). Are your apps watching you? New study reveals privacy issues with thousands of Android apps. Northeastern University.
European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 (General Data Protection Regulation). EUR-Lex. https://eur-lex.europa.eu/eli/reg/2016/679/oj
Federal Trade Commission. (2024, December 3). FTC takes action against Mobilewalla for collecting and selling sensitive location data (Press release). https://www.ftc.gov/news-events/news/press-releases/2024/12/ftc-takes-action-against-mobilewalla-collecting-selling-sensitive-location-data
Flew, T. (2021). Regulating platforms. Polity Press.
García-Martínez, A. (2017, November 10). Facebook’s not listening through your phone. It doesn’t have to. WIRED. https://www.wired.com/story/facebooks-listening-smartphone-microphone/
Gruber, J. (2025). Texas sues Allstate, alleging it violated data privacy rights of 45 million Americans. The Record. https://therecord.media/texas-sues-allstate-data-privacy-cars
Kemp, K. (2023). Proposed privacy reforms could help Australia play catch-up with other nations. But they fail to tackle targeted ads.
Marwick, A. E., & boyd, d. (2018). Understanding privacy at the margins: Introduction. International Journal of Communication, 12, 1157–1165.
McDonald, A. M., & Cranor, L. F. (2008). The cost of reading privacy policies. Journal of Law and Policy for the Information Society, 4(3), 543–568.
Nissenbaum, H. (2018). Respecting context to protect privacy: Why meaning matters. Science and Engineering Ethics, 24(3), 831–852.
Office of the Attorney General of Texas. (2025). Attorney General Ken Paxton sues Allstate and Arity for unlawfully collecting, using, and selling over 45 million Americans’ driving data (Press release). https://www.texasattorneygeneral.gov/news/releases/attorney-general-ken-paxton-sues-allstate-and-arity-unlawfully-collecting-using-and-selling-over-45
Suzor, N. P. (2019). Who makes the rules? In Lawless: The secret rules that govern our lives (pp. 10–24). Cambridge University Press.
TikTok. (2024). Location information on TikTok. TikTok Support. https://www.tiktok.com/support/faq_detail?id=7543897457726593542&category=web_account
United Nations. (1948). Universal Declaration of Human Rights. https://www.un.org/en/about-us/universal-declaration-of-human-rights
Gonçalves, S. (2026). Portugal approves restrictions on social media access for children. Reuters. https://www.reuters.com/world/europe/portugal-approves-restrictions-social-media-access-children-2026-02-12/
Khan, S. (2021). Implementing a dynamic “terms & conditions” page with animation and progress tracking in React Native. Medium (Nerd For Tech). https://medium.com/nerd-for-tech/implementing-a-dynamic-terms-conditions-page-with-animation-and-progress-tracking-in-react-56e27677041b
MakeUseOf. (2021). What are iPhone and iPad permissions, and how do they work? https://www.makeuseof.com/iphone-ipad-permissions-how-do-they-work/
Tiwari, A. (2025). Allstate reports $213 million in August catastrophe losses. The Insurer. https://www.theinsurer.com/ti/news/allstate-reports-213-million-in-august-catastrophe-losses-2025-09-19/
Be the first to comment