To err is human; to forgive, divine. But when it comes to autonomous AI \u201cagents\u201d that are taking on tasks previously handled by humans, what\u2019s the margin for error?\u00a0<\/p>\n
At Fortune\u2019s recent Brainstorm AI<\/a> event in San Francisco, an expert roundtable grappled with that question as insiders shared how their companies are approaching security and governance\u2014an issue that is leapfrogging even more practical challenges such as data and compute power. Companies are in an arm\u2019s race to parachute AI agents into their workflows that can tackle tasks autonomously and with little human supervision. But many are facing a fundamental paradox that is slowing adoption to a crawl: Moving fast requires trust<\/a>, and yet building trust takes a lot of time.\u00a0<\/p>\n Dev Rishi, general manager for AI at Rubrik, joined the security company last summer following its acquisition<\/a> of his deep learning AI startup Predibase. Afterward, he spent the next four months meeting with executives from 180 companies. He used those insights to divide agentic AI adoption into four phases, he told the Brainstorm AI audience. (To level set, agentic adoption refers to businesses implementing AI systems that work autonomously, rather than responding to prompts.)\u00a0<\/p>\n According to Rishi\u2019s learnings, the four phases he unearthed include the early experimentation phase where companies are hard at work on prototyping their agents and mapping goals they think could be integrated into their workflows. The second phase, said Rishi, is the trickiest. That\u2019s when companies shift their agents from prototypes and into formal work production. The third phase involves scaling those autonomous agents across the entire company. The fourth and final stage\u2014which no one Rishi spoke with had achieved\u2014is autonomous AI.\u00a0<\/p>\n Roughly half of the 180 companies were in the experimentation and prototyping phase, Rishi found, while 25% were hard at work formalizing their prototypes. Another 13% were scaling, and the remaining 12% hadn\u2019t started any AI projects. However, Rishi projects a dramatic change ahead: In the next two years, those in the 50% bucket are anticipating that they will move into phase two, according to their roadmaps.\u00a0<\/p>\n \u201cI think we\u2019re going to see a lot of adoption very quickly,\u201d Rishi told the audience.\u00a0<\/p>\n However, there\u2019s a major risk holding companies back from going \u201cfast and hard,\u201d when it comes to speeding up the implementation of AI agents in the workforce, he noted. That risk\u2014and the No.1 blocker to broader deployment of agents\u2014 is security and governance, he said. And because of that, companies are struggling to shift from agents being used for knowledge retrieval to being action oriented.<\/p>\n \u201cOur focus actually is to accelerate the AI transformation,\u201d said Rishi. \u201cI think the number one risk factor, the number one bottleneck to that, is risk [itself].\u201d<\/p>\n Kathleen Peters, chief innovation office at Experian<\/a> who leads product strategy, said the slowing is due to not fully understanding the risks when AI agents overstep the guardrails that companies have put into place and the failsafes needed for when that happens.<\/p>\n \u201cIf something goes wrong, if there\u2019s a hallucination, if there\u2019s a power outage, what can we fall back to,\u201d she questioned. \u201cIt\u2019s one of those things where some executives, depending on the industry, are wanting to understand \u2018How do we feel safe?\u2019\u201d<\/p>\n Figuring out that piece will be different for every company and is likely to be particularly thorny for companies in highly regulated industries, she noted. Chandhu Nair, senior vice president in data, AI, and innovation at home improvement retailer Lowe\u2019s, noted that it\u2019s \u201cfairly easy\u201d to build agents, but people don\u2019t understand what they are: Are they a digital employee? Is it a workforce? How will it be incorporated into the organizational fabric?\u00a0<\/p>\n \u201cIt\u2019s almost like hiring a whole bunch of people without an HR function,\u201d said Nair. \u201cSo we have a lot of agents, with no kind of ways to properly map them, and that\u2019s been the focus.\u201d<\/p>\n The company has been working through some of these questions, including who might be responsible if something goes wrong. \u201cIt\u2019s hard to trace that back,\u201d said Nair.\u00a0<\/p>\n Experian\u2019s Peters predicted that the next few years will see a lot of those very questions hashed out in public even as conversations take place simultaneously behind closed doors in boardrooms and among senior compliance and strategy committees.\u00a0<\/p>\n \u201cI actually think something bad is going to happen,\u201d Peters said. \u201cThere are going to be breaches. There are going to be agents that go rogue in unexpected ways. And those are going to make for a very interesting headlines in the news.\u201d<\/p>\n Big blowups will generate a lot of attention, Peters continued, and reputational risk will be on the line. That will force the issue of uncomfortable conversations about where liabilities reside regarding software and agents, and it will all likely add up to increased regulation, she said.\u00a0<\/p>\n \u201cI think that\u2019s going to be part of our societal overall change management in thinking about these new ways of working,\u201d Peters said.<\/p>\n Still, there are concrete examples as to how AI can benefit companies when it is implemented in ways that resonate with employees and customers.\u00a0<\/p>\n Nair said Lowe\u2019s has seen strong adoption and \u201ctangible\u201d return on investment from the AI it has embedded into the company\u2019s operations thus far. For instance, among its 250,000 store associates, each has an agent companion with extensive product knowledge across its 100,000 square foot stores that sell anything from electrical equipment, to paints, to plumbing supplies. A lot of the newer entrants to the Lowe\u2019s workforce aren\u2019t tradespeople, said Nair, and the agent companions have become the \u201cfastest-adopted technology\u201d so far.<\/p>\n \u201cIt was important to get the use cases right that really resonate back with the customer,\u201d he said. In terms of driving change management in stores, \u201cif the product is good and can add value, the adoption just goes through the roof.\u201d<\/p>\n But for those who work at headquarters, the change management techniques have to be different, he added, which piles on the complexity.\u00a0<\/p>\n And many enterprises are stuck at another early-stage question, which is whether they should build their own agents or rely on the AI capabilities developed by major software vendors.\u00a0<\/p>\n Rakesh Jain, executive director for cloud and AI engineering at healthcare system Mass General Brigham<\/a>, said his organization is taking a wait-and-see approach. With major platforms like Salesforce<\/a>, Workday, and ServiceNow<\/a> building their own agents, it could create redundancies if his organization builds its own agents at the same time.\u00a0<\/p>\n \u201cIf there are gaps, then we want to build our own agents,\u201d said Jain. \u201cOtherwise, we would rely on buying the agents that the product vendors are building.\u201d<\/p>\n In healthcare, Jain said there\u2019s a critical need for human oversight given the high stakes.\u00a0<\/p>\n \u201cThe patient complexity cannot be determined through algorithms,\u201d he said. \u201cThere has to be a human involved in it.\u201d In his experience, agents can accelerate decision making, but humans have to make the final judgment, with doctors validating everything before any action is taken.\u00a0<\/p>\n Still, Jain also sees enormous potential upside as the technology matures. In radiology, for example, an agent trained on the expertise of multiple doctors could catch tumors in dense tissue that a single radiologist might miss. But even with agents trained on multiple doctors, \u201cyou still have to have a human judgment in there,\u201d said Jain.\u00a0<\/p>\n And the threat of overreach by an agent that is supposed to be a trusted entity is ever present. He compared a rogue agent to an autoimmune disease, which is one of the most difficult conditions for doctors to diagnose and treat because the threat is internal. If an agent inside a system \u201cbecomes corrupt,\u201d he said, \u201cit\u2019s going to cause massive damages which people have not been able to really quantify.\u201d<\/p>\n Despite the open questions and looming challenges, Rishi said there\u2019s a path forward. He identified two requirements for building trust in agents. First, companies need systems that provide confidence that agents are operating within policy guardrails. Second, they need clear policies and procedures for when things will inevitably go wrong\u2014a policy with teeth. Nair, additionally, added three factors for building trust and moving forward smartly: identity and accountability and knowing who the agent is; evaluating how consistent the quality of each agent\u2019s output is; and, reviewing the post-mortem trail that can explain why and when mistakes have occurred.\u00a0<\/p>\n \u201cSystems can make mistakes, just like humans can as well,\u201d said Nair. \u201c But to be able to explain and recover is equally important.\u201d<\/p>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":" To err is human; to forgive, divine. But when it comes to autonomous AI \u201cagents\u201d that are taking on tasks previously handled by humans, what\u2019s the margin for error?\u00a0 At Fortune\u2019s recent Brainstorm AI event in San Francisco, an expert roundtable grappled with that question as insiders shared how their companies are approaching security and […]<\/p>\n","protected":false},"author":325,"featured_media":17263,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_uag_custom_page_level_css":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[123],"tags":[],"class_list":["post-17262","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-business"],"yoast_head":"\nIntegrating agents into the workforce<\/h2>\n
Who\u2019s watching the agent?<\/h2>\n