What is the hidden truth behind outstanding employees quietly bypassing AI?
At the end of 2022, a multinational bank approved an AI system to automate routine credit decision-making. On the surface, this was exactly the so-called proactive move that experts had long been urging enterprises to adopt. During the pilot phase, the system processed applications faster than human staff, made fewer errors, and promised significant cost reductions. The bank's risk management department endorsed it, the compliance department signed off on it, and the CEO even publicly promoted the system as strong proof of the bank's march into the future.
Six months on, the AI system continued to score users' credit applications and generate recommendations, but most application documents still required manual review. Exceptions were also piling up. Measures originally designed as temporary safety safeguards, such as dual-review thresholds and manual confirmation flags, gradually solidified into permanent process steps. Although the AI system was integrated into the bank's workflow, in reality, most final decisions were never made by it.
The justifications are all too familiar: the organization is not fully ready, the model needs further optimization, employees require more training, and change is always difficult. Yet the technology is already practically viable, the economic benefits are clear, and the AI system has received formal and enthusiastic approval from the organization. The only thing it never truly obtained is "empowerment" — the full backing of the organization to make decisions within clearly defined boundaries.
Stories of a similar nature are playing out across many industries. A multinational manufacturing enterprise launched an AI-based demand forecasting system, but its planning department still stuck to old workflows and only treated the AI's forecasts as reference points. A healthcare network introduced an AI-assisted triage system that shortened patient waiting times and improved triage consistency, and clinicians recognized the accuracy of the AI model, yet they still insisted on manually reviewing most of the AI's recommendations. As a result, the triage system was mostly used for archiving rather than supporting decision-making.
The problem is not that the pilot project failed, but that accountability and legitimacy within the organization failed to evolve in tandem to support the application of the AI system. What I call a "legitimacy architecture" refers to a consensus within an organization about who has the authority to make decisions and whose decisions can be trusted. Many organizations are continuously raising their level of intelligence, yet they have not reshaped their organizational structures around this change.
This Scenario Is Far From New
The situation described above seems novel because the technology is new, but from an organizational perspective, it is not new at all. Previous waves of automated decision-making, such as ERP (Enterprise Resource Planning) systems and credit scoring, exposed the same organizational inertia: enterprises introduced systems capable of making better decisions, yet retained their existing human decision-making authority structures, which often led to the rejection of decisions made by the systems.
In the 1990s, when ERP systems (such as those from Oracle and SAP) emerged, they promised centralized, transparent management and operational optimization across the entire enterprise. Nestlé USA's experience implementing an ERP system is highly illustrative. Prior to that, the company's various departments operated independently, each responsible for its own production, costs, and supplier contracts. When the ERP system required departments to give up this autonomy and follow unified business processes, employees who were still accountable for local business outcomes naturally had good reasons to insist on making their own decisions. Even though the system generated unified, consolidated forecasts, the accountability mechanisms at the time rewarded local adjustments to — or "override" of — the system's recommendations. As Tom Davenport of Babson College pointed out, the company treated the ERP system at the time purely as a technology deployment, rather than as a comprehensive management process reshaping.
Almost simultaneously, the same pattern played out with automated credit scoring. Although models were faster and more reliable than humans at quantifying risk, the prevailing practice was "human intervention." Studies on auto loans show that compared with manual underwriting, algorithmic underwriting not only delivered higher loan profits but also reduced default rates, and the most significant performance gap between the two occurred precisely in segments where loan officers, out of self-serving considerations, fiercely guarded their discretionary power. Other studies found that loan officers manipulated input data and scoring results to push borderline applications across the approval threshold, in order to earn performance bonuses tied to transaction volume. It looked like approval authority had been delegated to the model, but ultimate responsibility still rested with the relevant human staff.
The airline industry's experience with revenue management systems offers a useful contrasting case. These systems established lasting authority because airlines gave them strong support by adjusting the allocation of rights and responsibilities. Airlines clarified the scenarios suitable for human intervention, standardized the escalation process for issues, adjusted performance evaluation methods, and established the professional legitimacy of "following the system." The system's authority was not established because of the algorithm's perfect performance, but because airlines changed their organizational governance mechanisms for automated systems.
The underlying logic shared by these positive and negative cases is consistent: when the evolution of decision-making logic outpaces the adjustment of related governance mechanisms such as accountability and legitimacy, the system can never establish authority. Changes in governance models are often accompanied by a shift of power. Davenport, Erik Brynjolfsson, and others have pointed out this phenomenon before. This article reveals the root cause behind it: as long as individual employees are still required to take responsibility for the outcomes of system operations, decision-making authority will never truly shift.
Enterprises must resolve this disconnect between rights and responsibilities. Otherwise, employees will continue to bypass AI systems — not out of resistance, but purely out of self-protection.
No amount of training or communication can bridge this gap. Enterprises may successfully implement changes, yet permanently relegate the technology to an advisory role. They may provide employees with ample psychological safety, but they may still penalize employees who follow the system's instructions when decision errors occur. As long as accountability remains targeted at individuals, people will engage in self-preserving behaviors.
There are other explanations for why people fail to delegate decision-making authority to AI, but they often point to the wrong countermeasures. For example, some argue that organizational inertia creates resistance, but in AI applications, what we often see is superficial compliance: dashboards are running, workflows have incorporated the system, and employees are all using it. Yet the AI system never truly gains decision-making power.
Research on "automation complacency" suggests that humans tend to defer to machines too easily. But this phenomenon does not apply here; in the face of AI systems, people do not blindly follow, because the company's accountability system makes it professionally risky to do so.
Research on the "productivity paradox", while pointing out that information technology investments fail to translate into performance improvements, mostly does not explain the organizational mechanisms that lead to this outcome.
Ethan Bernstein's research on transparency shows that employees under surveillance will hide their actual operations to avoid scrutiny, but they also use explicit compliance mechanisms, such as archived exception operations, escalation procedures, and multi-level review processes, as tools to maintain their own control.
What Is the Truth?
In most cases, enterprises adopt some kind of AI system and promote it vigorously in internal corporate demonstrations, but in reality, whenever AI and humans disagree, humans quietly prevail in the end. Out of prudence, experts always reshape the boundaries of AI systems: adding more review links, slightly lowering trigger thresholds, increasing the number of archived "override" operations (actions that bypass regular workflows), and enabling new flagging mechanisms for sensitive cases. Individually, each of these settings and adjustments is justifiable, but taken together, the consequences are catastrophic.
UPS's deployment of its ORION route-planning system vividly illustrates this tension. When the system was first rolled out in 2013, many drivers treated the routes it generated as advisory suggestions, preferring to rely on their own local experience and customer relationships. Over time, UPS adjusted its performance evaluation criteria: drivers were no longer compared against their own past performance, but assessed against the routes set by ORION, with the system tracking their compliance with the routes. As a result, "following the algorithm" became a legitimate part of the drivers' job responsibilities.
Recently, I observed the same dynamic at another multinational logistics company that handles high-value cargo transportation for enterprise clients. During the pilot project, an AI route-planning system drastically reduced cargo delays and improved asset utilization, and the company immediately approved the system for adoption. Dispatchers privately admitted that the system often generated better routes than the ones they devised themselves. However, after the full rollout, dispatchers kept adjusting routes in their daily work, for long-standing clients, cargo with internal political importance to the company, and drivers they trusted more. At first, these adjustments were seen as "fine-tuning", but over time, the AI system no longer functioned as a decision engine, and its outputs only served as the starting point for discussing subsequent plans.
When senior management pushed for full adoption of the AI route-planning system, dispatchers neither argued that the system was flawed nor refused to use it. Instead, they added measures: flagging sensitive cargo, adding escalation links, and lowering the thresholds for triggering reviews. One dispatcher explained the logic to me with disarmingly frank candor. He said: "If a decision made by the system goes wrong, my name is the one signed on the report; if a mistake is made in a decision I made myself, at least I can explain it." This dispatcher was neither troubled by the AI system nor felt a lack of security; he simply accurately interpreted where the responsibility lay.
Research on professional identity threat shows that when practitioners' professional skills face automation substitution, they experience genuine anxiety about their social status, professional value, and future career prospects. This anxiety affects their behavior, and these behaviors are neither irrational nor trivial. However, these factors cannot explain a phenomenon I have observed: this pattern persists even when fear and identity threats have been resolved, change management has been effective, psychological safety is abundant, and employees openly admit that the system outperforms themselves. The crux of the problem lies in a structural exposure to risk: people who are required to follow the AI system's instructions bear the cost of the system's errors.
This problem is particularly acute for AI agent systems. Recommendation systems, at least, maintain the illusion of "human control": the system puts forward suggestions, humans make decisions, and the accountability subject is clearly visible (even if the responsibility should rest with others). Agent AI systems break this illusion. They do not just offer recommendations; they take direct action. They perform operations such as placing orders, approving requests, planning routes, signing contracts, and escalating issues within established parameters. These operations are authorized within the parameters, but often lack continuous human monitoring. When an AI agent acts within established parameters yet leads to adverse outcomes, accountability becomes an extremely challenging puzzle: were the parameters set incorrectly? Was the scope of authorization too broad? Or did the agent encounter a scenario unforeseen by its designers? These questions will inevitably be raised repeatedly in board meetings, court hearings, and regulatory filing documents. When AI systems only provide recommendations, gaps in accountability can be patched through workarounds; however, when AI agent systems take direct action, they can trigger large-scale ambiguity in legal responsibility attribution.
Shifting Accountability
Truly adopted automated systems leave no room for ambiguity: decision-making processes are streamlined, exceptions are drastically reduced rather than increased, rights and responsibilities are redistributed, and people clearly understand who (or what mechanism) is responsible for making decisions, who is responsible for explanation, and who bears the risks. This clarity can cause discomfort, as it forces a transformation in everyone's role; but in the long run, it brings stability, as rights and responsibilities are aligned.
Amazon's "Hands off the Wheel" initiative is a typical case of power transfer. Launched in 2012 as "Project Yoda", the initiative aimed to integrate demand forecasting, pricing, and procurement forecasting functions into workflow tools. Amazon separated "input rights" from "output rights": while business teams could set data and parameters, routine manual intervention was discouraged, and sometimes required pre-execution approval. The focus of vendor managers' responsibilities also shifted accordingly, from specific procurement and pricing decisions to reviewing input data, anomalies, and final outcomes.
This handover process was completed smoothly because responsibility was transferred along with authority. When results fell short of expectations, the focus was no longer on "who made the wrong decision", but on "which input, rule, or system design went wrong" and "who is responsible for fixing it". At the same time, employees' job content was redesigned to focus on product and project management work that is difficult to automate.
Technology deployment and organizational architecture must evolve in tandem; organizational adjustments should be designed before automated systems are deployed at scale, rather than patched up after resistance is encountered. For the governance of automated systems, management must answer four key questions. These questions may cause some discomfort, but they are critical to revealing gaps in accountability mechanisms. The four questions are:
(1) If this AI system operates successfully, who will lose their autonomous decision-making power?
(2) Who will gain authority, and over which decisions?
(3) When employees follow the AI system's decisions and actions, who will be held accountable for explaining the final outcomes?
(4) In the process of integrating AI systems into the business, who will bear the risks that may arise?
If organizations do not clearly answer these four questions before launching AI systems, the employees who actually use the systems later will, in order to safeguard their existing power, act to provide their own answers.
The true adoption of AI systems requires that employees do not see "following the system's decisions" as a professional self-sacrifice. Leaders must redesign job responsibilities to ensure that after routine decision-making is automated, human judgment shifts to higher-level work such as macro monitoring, boundary setting, and system optimization. Organizations should establish clear "guardrail" mechanisms and provide employees with corresponding tools so that they can explain decisions that were not made by themselves. In addition, the rollout of AI systems requires public commitment from leaders; leaders must be willing to treat early errors as opportunities for organizational learning, rather than attributing failures to employees.
The Core Competency That Creates Enterprise Differentiation
As the act of "exercising judgment" becomes codifiable and scalable, enterprises need to reshape their organizational architecture around AI systems. Unfortunately, many enterprises simply purchase intelligent systems and expect everything else to fall into place naturally. They treat each AI deployment as an isolated project, whose success or failure depends entirely on model performance, vendor selection, and the rigor of rollout execution. In these enterprises, the introduction of each new AI system triggers the same cycle: initial enthusiasm, subsequent limitation due to concerns, partial application, and finally a quiet shift to a reference-only auxiliary mode, where it settles into a steady, unremarkable state. Over time, employees realize that fundamental change will not happen, and managers learn to tone down their grand visions for AI applications before they can trigger organizational restructuring.
In contrast, truly wise leaders recognize that the core of the AI application game is not technology, but power. They can see where decision-making authority truly lies, then reshape workflows around the system's default settings, rather than just overlaying new systems on top of existing processes, and manage AI transformation without passing on learning costs to employees.
Process is the key here. Decision-making authority will only shift after organizations first set algorithms as the default operating mechanism, ensure that manual intervention operations are open and transparent, redesign employee roles around "pre-review" rather than "post-hoc questioning", and transfer accountability to system owners. Technology and organizational architecture must evolve in tandem.
Companies that have experienced the synchronized evolution of technology and organization are often better positioned to cope with the next wave of change. In my consulting and field research work, I have observed how enterprises manage subsequent changes properly: they use a common set of terminology and a list of questions, and draw on experience in process reshaping from previous transformations. They clearly remember: who lost autonomy, who (or which department) gained decision-making authority, who is responsible for explaining outcomes, and who bore the early trial-and-error costs. This does not mean that their next transformation will be completely painless, nor that political games will automatically diminish; but it is certain that compared with enterprises that have no such experience, those that do will start their march into the future from a higher starting point.
For this reason, enterprises should regard "decision-making authority transfer" as an organizational-level capability, rather than just a one-off challenge when deploying a certain AI system. Before deploying an AI model, enterprises need to clarify where decision-making authority will flow, what role human judgment will play in subsequent workflows, and how the organization will absorb learning costs as employees gradually trust the system.
If you resonate with this idea, it is likely that your company has already passed the "approval stage". You are in the subsequent phase, looking at an AI system that has quietly morphed into the "advisory" role described earlier. The steps outlined above are still indispensable for recommendations on the next deployment: treat approval as a redesign of employee roles, answer the four key questions before scaling up, and explicitly commit to who will bear trial-and-error costs before full rollout. But for AI systems that have already been deployed, the next step requires a different strategy: diagnose where business decision-making authority actually resides right now.
Most organizations tend to avoid talking about one thing: if AI performs as expected, the enterprise will need less of certain forms of human judgment. The narrative of "augmenting" (rather than replacing) human labor cannot obscure this reality. If an underwriting model can automatically approve 70% of routine cases, the role of underwriters can no longer be defined by case-by-case approval work. Executives' silence on this matter effectively acquiesces to the quiet retreat of human judgment.
Enterprises must make it clear that organizational restructuring around AI systems is not about eliminating human judgment, but about shifting human judgment to more important areas: assessing the risk quality of investment portfolios, monitoring model drift, and handling scenarios unsuitable for the system. This shift in judgment must be incorporated into performance evaluations. If employees are still rewarded for outcomes that are driven by the system, they will engage in defensive interventions out of self-protection.
Rights and responsibilities must be aligned. If a model is allowed to make decisions within established parameters, strict adherence to these parameters must be professionally legitimate. Leaders cannot attribute the system's early mistakes to individual employees.
The airline industry offers a useful case. The industry adheres to the principle of a "just culture", which encourages people to proactively disclose safety issues, while clearly distinguishing between acceptable errors and unacceptable behavior. EU law requires airlines to clearly explain how they ensure a