Algorithmic Agency Asymmetry: When AI makes decisions for you, you don't even have the right to object
A wise society should not allow invisible systems to shape people's choices, rewards, and behaviors without equipping them with effective means to observe, question, and correct such influences. With the advancement of artificial intelligence, society is sliding down a dangerous slope, rapidly moving from experimenting with and integrating AI to relying on it, and eventually even becoming addicted to it. However, one of the most critical questions is whether policymakers are aware of this shift.
In general, asymmetry means that the two parties in a relationship are not on equal footing. In digital life, "algorithmic asymmetry" describes a deeper imbalance between two parties: one can observe, model, test, and refine its algorithms, while the other primarily bears the consequences brought by those algorithms. This imbalance has now permeated fields such as recruitment, lending, insurance, education, policing, media, and daily attention architectures. Its consequence is the asymmetry of algorithmic agency, meaning users cannot recognize and resist the improper influence of algorithms on their own situations.
The Three Layers of "Cognitive Shackles" Imposed by Algorithms
This algorithmic asymmetry can be explained from three dimensions.
The first dimension is opacity, which refers to the fact that organizations that design, deploy, or purchase algorithmic systems typically have a better understanding of the systems' goals, thresholds, incentive mechanisms, and vulnerabilities than the people interacting with them. The "opacity problem" explains why this gap persists: some systems are deliberately hidden to protect intellectual property, some require specialized training to comprehend, and others are difficult to interpret even for experts. When a system is hard to inspect, its outputs often appear more objective than they actually are, leading to the "black box fallacy."
The second layer of algorithmic asymmetry is historical bias amplification. Algorithms learn from the past world, including past prejudices or exclusions. Even seemingly neutral systems can reproduce pre-existing patterns of inequality in the data. Biased past inputs, used as training materials, are eventually output in the form of predictions, scores, or recommendations, and appear neutral simply because they are computational results. In reality, this is just old hierarchies re-emerging through a more modern, streamlined interface.
The third layer is recursive systems. Systems are usually not deployed once and for all; instead, users continuously train these systems. Every click, pause, prompt, path selection, purchase action, and hesitation becomes data. Recommendation systems are designed to learn from these signals and adjust accordingly, but this is not the end of the cycle. With these learnings, the systems shape what we see next, determine what feels normal, what seems relevant, and sometimes even what feels desirable, while their goals remain ambiguous to end users. In other words, we train the systems, and the systems in turn train us. "Algorithmic drift" refers to this co-evolutionary relationship between users and platforms.
When Algorithms "Live" on Your Behalf
The agency of artificial intelligence refers to the capability to judge, choose, and take actions in meaningful ways, as well as to understand the various forces that influence one's own choices.
Asymmetry of agency arises when organizations use digital systems — such as personalized feeds, targeted advertising, dynamic pricing, recommendation engines, and risk scoring — to test, measure, and optimize influences and outcomes on a large scale. Marketing has always sought to shape behavior; the difference today lies in precision and feedback mechanisms: organizations can observe individual behaviors in real time, divide crowds into increasingly granular categories, continuously conduct A/B tests, and adjust what each person sees, how they pay, or what offers they receive. In contrast, individuals usually only have access to the surface-level information of the system: a feed, a score, a price, a recommendation, or a rejection, with no knowledge of how their data is used, which goal is being optimized, and how their choices are being guided.
This is crucial because people will adapt to what the system rewards. In recruitment, it is no longer just about job seekers polishing their resumes to please recruiters; automated screening tools and AI ranking systems may reward certain specific signals while hiding the logic behind them. A study by the University of Washington found that after large language models ranked more than 550 real resumes, they showed a preference for resumes with names associated with white individuals in 85% of cases, and never favored resumes with names associated with black men. In the field of education, the 2020 grade controversy in the UK demonstrated how algorithmic models translated school-level history into individual grades: Ofqual (the Office of Qualifications and Examinations Regulation) lowered the internal assessment scores of about 40% of students, triggering strong public backlash and eventually forcing the government to retract this decision.
Furthermore, newer AI tools bring additional risks. Researchers at Stanford University tested the performance of seven widely used AI detectors using samples from native and non-native English speakers. The results showed that in the non-native speaker samples, AI detectors incorrectly classified 61.22% of the articles as AI-generated, indicating that some students are more vulnerable to suspicion or punishment simply because of their writing styles. Similar phenomena have emerged in digital life and work. Facebook's famous 2014 News Feed experiment on 689,003 users showed that changes in users' exposure to positive or negative posts affected the emotional language they used afterward. In the retail industry, Amazon warehouse workers have also reported that they must meet speed-based metrics without knowing how these metrics are calculated. Reports and studies on algorithmic management in Amazon warehouses have also explored this phenomenon. These cases reveal a deeper problem: digital systems do not just classify behaviors after the fact. They also teach people which words to use, which risks to avoid, which emotions to express, and which metrics to pursue. When organizations shape the conditions under which people think, act, and make decisions, while individuals only experience these conditions as scores, grades, messages, goals, or prices, algorithmic agency asymmetry takes on political significance.
Policies Must Go Beyond Empty Slogans
Therefore, policies must rebalance this relationship. First, legislators should require meaningful notification and explanation when impacts occur. Users should know when they are interacting with AI, when content is synthetic, and when an important decision is influenced by an automated system. The logic behind the European transparency obligations in Article 50 of the EU's AI Act points in the right direction. The OECD AI Principles also state the same point at a broader level: people need sufficient information to understand the outcomes and question them when necessary.
Second, governments should require enforceable impact assessments before algorithmic systems enter high-risk areas such as employment, education, housing, insurance, healthcare, welfare, and policing. Some existing approaches provide a foundation for this, such as Canada's Algorithmic Impact Assessment, Ontario's Human Rights AI Impact Assessment, and Europe's Fundamental Rights Impact Assessment for high-risk AI systems. Recent failures demonstrate that stronger safeguards are critical. In the UK, the Court of Appeal ruled in "R (Bridges) v Chief Constable of South Wales Police" that the South Wales Police Force's use of real-time automatic facial recognition technology was illegal. In Detroit, Robert Williams was wrongfully arrested due to a facial recognition misidentification, a case documented by the American Civil Liberties Union. Therefore, before deployment, agencies should assess the potential impacts of AI systems, such as rights violations, harm to vulnerable groups, and the distribution of errors, while also evaluating the necessity of human oversight, appeal mechanisms, and remedial measures, and conducting public reports as much as possible.
Third, human oversight must be genuine, effective, trained, and protected. In many institutions, the power of "human intervention" is often restricted when employees face pressure to trust the outputs of the system. Australia's Robo-Debt scheme showed how automated welfare debt calculations can harm people when officials treat system-generated claims as authoritative. In the R (Bridges) v South Wales Police case, the UK Court of Appeal ruled that the use of real-time facial recognition was illegal, partly because of insufficient safeguards around discretion, data protection, and fair impacts. The UK Post Office's "Horizon" scandal exposed similar failures: people trusted the outputs of flawed software over the firsthand experiences of hundreds of sub-postmasters. The value of Article 14 of the European AI Act lies in its requirement that personnel conducting human oversight of high-risk AI systems must understand, monitor, interpret, override, or interrupt the system. Any agency using AI with significant impacts should appoint responsible reviewers, train them to identify automated biases, and grant them real authority to stop harmful outputs.
Fourth, regulation should not stop once the system is released. Models drift, circumstances change, and incentive mechanisms shift. A system that seems acceptable during testing can become discriminatory or manipulative once it interacts with real people. Therefore, post-deployment monitoring, logging, independent auditing, and incident reporting should become legal obligations. The US National Institute of Standards and Technology's AI Risk Management Framework and the post-market monitoring provisions in the AI Act recognize this point. The Pro-Social AI Index can be used to map, measure, and monitor the impacts of AI systems on humans and their environments.
Fifth, certain practices deserve to be banned. Systems designed to exploit vulnerabilities, distort behavior through deceptive design, or manipulate children and other vulnerable groups deserve to be prohibited, rather than just given mild guidance. Article 5 of the EU AI Act bans certain manipulative and exploitative uses, drawing a necessary firm line. A healthy digital society cannot rely solely on information disclosure; it must pay attention to whether the underlying design is intended to undermine judgment.
Algorithmic literacy should be regarded as civic infrastructure. If only developers, vendors, and compliance teams understand how these systems operate, the problem of power asymmetry will persist even under good regulation. Citizens, teachers, judges, journalists, clinicians, and public administrators all need practical literacy about synthetic media, ranking systems, behavior steering, the right to challenge, and the limitations of model outputs. The fourth European article on AI literacy is a helpful signal and should be developed into a broader public mission. In addition to AI literacy, now is the time to invest in dual literacy to ensure users can be aware of the interaction between individual perception, behavior, and the influence of artificial assets on them.
Ultimately, algorithmic agency asymmetry is not an isolated technical issue, but a structural imbalance in who can perceive, shape, and resist the power of algorithms. One party learns faster, continuously tests, and intervenes quietly; the other adapts with partial opaque information. Good policies cannot completely eliminate this asymmetry, but they can narrow the gap in the most critical areas by making automated impacts visible, challengeable, auditable, and governable.
This article is from the WeChat official account "Internet Law Review", author: Cornelia Walter, published with authorization from 36Kr.