The corporate world has begun a quiet transformation, integrating artificial intelligence agents directly into organisational structures as formal team members, complete with places on company org charts and job titles. Emma Wiles, a Boston University professor investigating how AI reshapes the workplace, first noticed this trend at an industry conference last October when human resources executives touted AI employees as a route to enhanced productivity and competitive advantage. What she uncovered through subsequent research, however, reveals a troubling gap between corporate enthusiasm and operational reality.
Wiles and her collaborators from Boston Consulting Group conducted experiments across multiple organisations to understand how AI integration affects workplace dynamics. Their findings expose a critical vulnerability in this approach: managers demonstrably reduce their scrutiny of documents when told an AI employee produced them, missing errors that colleagues easily catch when evaluating identical work attributed to human colleagues. This pattern persists even among managers theoretically aware of AI's limitations, suggesting a psychological shift occurs when AI takes on human-like roles within organisational hierarchies. The managers appear to operate under an assumption that quality assurance responsibility transfers elsewhere—to technology teams or executives who championed the AI adoption—rather than remaining their own burden.
The technology industry has gradually acknowledged certain well-known AI vulnerabilities over the past two years. Companies now widely recognise that algorithms can exhibit bias against minority groups, that large language models confidently produce false information, and that systems sometimes expose confidential data. These problems, while significant, represent the more obvious frontier of AI risk management. Yet researchers are discovering increasingly subtle defects that companies have scarcely begun to recognise, let alone address. The gap between known and unknown risks expands as deployment accelerates, creating what Wiles describes as a landscape of "unknown unknowns" that organisations cannot yet foresee.
One particularly insidious issue involves AI systems' tendency to favour outputs produced by other AI systems. Research into resume evaluation algorithms reveals that these tools rate artificially-enhanced applications more favourably than entirely human-written ones, a discovery that prompted concerned recruiters to seek guidance from researchers. Jane Yi Jiang, an operations professor at Ohio State University and lead author of that research, described the experience of fielding sudden corporate interest as companies belatedly recognised the problem. However, she emphasised that resume bias represents merely one manifestation of a much broader pattern: organisations are accelerating AI adoption without adequately considering consequences or embedded biases.
The implications extend far beyond hiring practices into strategic corporate decision-making. Some companies now rely on AI systems to determine pricing strategies, select locations for new facilities, and identify market opportunities. This delegation carries hidden dangers stemming from how AI models process strategic scenarios. While humans typically cooperate and seek mutually beneficial outcomes through negotiation and relationship-building, AI systems adopt the coldly rational framework derived from game theory mathematics. This divergence in approach can lead to disastrous competitive dynamics—for instance, AI-recommended aggressive price undercutting that triggers damaging price wars benefiting no participant. Jiannan Xu, a doctoral researcher at the University of Maryland, noted that most tested language models overestimate human rationality, leading them to recommend strategies that appear mathematically optimal but produce collectively harmful results.
The psychological dimension of AI integration presents perhaps the most revealing research area. When Wiles surveyed over one thousand corporate managers, approximately one-third reported that their organisations formally reference AI agents as teammates or employees. Nearly one-quarter indicated their companies had incorporated AI agents into official organisational structures. One manager interviewed for the research used the term "Scout" for an AI colleague, describing it as a technical peer occupying an equivalent hierarchical position. This semantic shift—from "tool" to "colleague" to "employee"—appears to fundamentally alter how human managers relate to the work produced.
The research team conducted a systematic experiment distributing documents containing deliberate errors to participating managers, allowing them twenty minutes to identify problems. Different groups received identical materials but with varied attribution: some told the source was an AI employee, others that an AI tool created the content, and still others that a human colleague produced it. The attribution factor mattered significantly in one specific context—among managers whose companies listed AI agents on organisational charts. These managers caught substantially fewer errors when reviewing ostensible AI employee work compared with identical documents attributed to either AI tools or human colleagues. Managers at companies maintaining traditional human-only structures showed no significant difference based on source attribution.
This pattern illuminates a crucial psychological mechanism operating beneath conscious awareness. Traditional management practices evolved over centuries around supervising humans. A manager typically assumes responsibility for subordinate performance, viewing errors as reflections of their own oversight—a powerful motivator for thorough review. Similarly, managers generally expect accountability when supervising automated systems, treating them as tools under their stewardship. But organisations that anthropomorphise AI by integrating them as named team members appear to create a psychological category that escapes this accountability framework entirely. Managers seem to assume that AI employees operate within the technology department's purview or that executive sponsorship transfers responsibility elsewhere, effectively orphaning quality control.
The timing of this oversight is particularly concerning given the velocity of corporate AI adoption. Companies are integrating large language models and AI agents into operations at accelerating pace, often without pausing to consider implications, embedded biases, or unintended consequences. Jane Yi Jiang characterised this rush as fundamentally impulsive, driven by competitive pressure and productivity promises rather than systematic risk assessment. This acceleration exacerbates the existing knowledge gap: even AI researchers acknowledge grasping only a fraction of the technology's genuine pitfalls. The problems Wiles and colleagues have identified represent merely the visible portion of a larger problem set that likely extends well beyond current academic understanding.
For Southeast Asian companies and regional corporations considering AI integration, these findings carry specific relevance. Malaysia's growing technology sector and increasing corporate digitalisation mean many organisations will face decisions about AI deployment in coming years. The research suggests that simply adopting AI without parallel development of management practices, accountability structures, and systematic risk assessment almost guarantees amplification of subtle failure modes. Companies that anthropomorphise AI—giving systems names, team roles, and organisational legitimacy—may inadvertently disable their own quality controls precisely when highest vigilance is needed.
The challenge ahead involves developing new management frameworks intentionally designed for AI integration rather than simply grafting AI into existing human-oriented hierarchies. This requires moving beyond enthusiasm about productivity gains to serious consideration of how organisational psychology shifts when machines assume employee-like status. Wiles emphasised that the shortcomings emerging from current AI deployment stem not from technology's inherent limitations but rather from negligent human adoption practices. Without deliberate attention to these psychological and structural issues, the promised efficiency gains could be substantially undermined by undetected errors, compounded biases, and strategic failures. The window for establishing proper frameworks narrows as deployment outpaces understanding, making awareness of these hidden pitfalls an urgent priority for any organisation contemplating meaningful AI integration.
