Automating a Broken System: The Hidden Risk of Agentic AI

Beyond The Chatbot

Agentic AI is rapidly moving beyond the chatbot. For the past few years, most people’s experience of generative AI has followed a relatively simple pattern: we give an AI model a prompt, it processes the information available to it, and it gives us a response. That is extremely useful and fun, but it is not really what we mean when we talk about an AI agent.

The next stage is different. We are beginning to give AI systems responsibilities, tools, access to organizational information, memory of what has happened, and the ability to take action. We are also beginning to connect these agents together so that several specialized agents can coordinate their activities toward an outcome that would previously have required a team of people.

There is enormous potential in this. There is also a danger that I don’t think we are discussing nearly enough. Agency.

If we take an existing system of work, with all its bottlenecks, queues, handoffs, functional silos, poorly designed measures, centralized decisions and unnecessary bureaucracy, and simply replace or augment human activity with AI agents, we have not necessarily improved the system. We may simply have given a poorly designed system the ability to operate much faster.

That should concern anyone interested in continuous improvement because the technology may be new, but the systemic problems are remarkably familiar.

Key Takeaways for Leaders

1. AI can automate work. It cannot fix the system through which that work flows. Faster execution does not eliminate constraints, queues, excessive WIP or poorly designed processes. It can simply accelerate them.
2. The system, not the agent, is the unit of analysis. What matters is how humans, agents, processes, information, technology, constraints and feedback interact to create value across the whole system.
3. Faster output does not necessarily mean better Flow. AI can generate work faster than the rest of the system can absorb it, creating more WIP, longer queues and delayed feedback rather than better outcomes.
4. The human in the loop is part of the system. Human oversight must be deliberately designed. Put humans into every decision and they can become the constraint; remove them entirely and errors can propagate at machine speed.
5. Don’t automate the dysfunction. Understand the system. Improve the Flow. Then decide where AI can create value. Continuous improvement should be part of the agentic architecture from the beginning, allowing the system not only to perform work but also to learn and improve how that work gets done.

Table of Contents

From AI Tool to AI Agent

It helps first to understand what we actually mean by an AI agent. A large language model (LLM) such as GPT, Claude or Gemini provides an underlying capability to interpret information, reason about it and generate a response. We might give a model a requirements document and ask it to identify risks, for example. The model processes the information and returns its analysis.

The interaction can be extremely valuable, but the model does not inherently own the work. It does not necessarily maintain an understanding of what happened previously, decide what needs to happen next, interact with other organizational systems, determine whether its previous recommendation was acted upon, or continue working toward an objective. In its simplest form, the interaction is essentially input, processing and output.

An AI agent adds additional capabilities around the model. A useful way of thinking about it is Model + Role + Context + State + Tools + Rules + Goals + Feedback. Instead of merely asking the model to produce an answer, we give the agent a purpose and some degree of responsibility within a system of work.

This changes the operating cycle considerably. Instead of simply thinking and answering, the agent can observe what is happening, reason about the situation, decide what action is appropriate, use a tool to take that action, observe the result and determine what should happen next. It can repeat that cycle until a defined condition has been satisfied, it encounters something outside its authority, or it requires human intervention.

The understanding of this matters because we have moved from AI being a tool used by someone doing the work to AI becoming an agent within the system through which the work gets done.

Building a Simple AI Agent for Product Development

Imagine we want to create an AI Requirements Analyst for software development. Its responsibility is to determine whether a proposed product feature is sufficiently understood to enter development. Rather than simply prompting a model like ChatGPT to “review these requirements,” we can give the agent an explicit role and responsibility for determining whether the customer need, acceptance criteria, constraints, dependencies and intended outcomes are sufficiently understood.

We can then establish rules governing how the agent performs that responsibility. Every proposed feature might need to identify the customer or user and the problem being solved. Acceptance criteria must be testable, dependencies identified, security implications considered and important ambiguities resolved. Critically, the agent should be instructed not to invent missing information merely because it can produce a plausible answer.

The agent also needs tools.

It might be allowed to read Jira, search Confluence, inspect GitHub, query architectural documentation, examine previous decisions, request clarification from a product manager and update the status of the work. The AI model provides much of the reasoning capability, while these tools give the agent the ability to interact with the environment in which the work exists.

It also needs to understand the state of that work.

Perhaps it determines that the customer and business outcome are understood and that acceptance criteria have been established, but authentication architecture, regulatory implications and compatibility with an existing API remain unresolved. Those are no longer simply observations contained in an answer. They become part of the state the agent uses to determine what should happen next.

Finally, the agent needs a stopping condition.

It should only declare the requirements ready when the mandatory conditions have been satisfied and critical uncertainties resolved. Otherwise, the work remains not ready and the agent either continues working, requests information or escalates the issue.

We now have something substantially different from a chatbot. The agent has a purpose, responsibility, knowledge, rules, tools, state, some degree of decision authority and explicit completion criteria.

From an Agent to an Agentic Team

Once we understand how to construct an agent, the temptation is to construct an increasingly capable one. We might ask the same agent to understand the customer, develop requirements, design the architecture, write the software, test it, perform a security review and deploy the result. Technically, we can attempt this, but organizationally we have created an AI version of the mythical employee who does everything.

There are obvious problems with that approach. The same entity can end up creating the work and evaluating its own work, grading its own homework. Responsibilities become ambiguous, context becomes enormous, independent challenge disappears and an incorrect assumption made early in the process can propagate throughout everything that follows.

Instead, we can create specialized agents with bounded responsibilities. A software delivery system might contain Product, Requirements, Architecture, Development, Testing, Security and Release agents. Each can have its own purpose, context, tools, knowledge, rules, authority and completion conditions.

A Team of Agents

Not Just AI Agents

At this point something important has happened. We are no longer simply designing AI agents. We are designing a system of work in which humans and artificial agents have different capabilities, responsibilities and decision authority. In many cases, we are effectively designing a new kind of organization.

Those agents now have to coordinate. The Development Agent cannot simply announce that it is finished. It needs to communicate the condition of the work in a way the Testing Agent can understand. That might include whether the code is complete, whether unit tests passed, whether static analysis succeeded, what known issues remain and whether required security checks have occurred.

The Testing Agent then evaluates the work against its responsibilities. If an acceptance criterion fails, the work returns to development. The agents have created a feedback loop in which the work moves through different states and can move backwards when new information reveals a problem.

That is considerably more sophisticated than several AI models talking to one another. We have created a system through which work flows.

And the moment we create a system through which work flows, all the things we know about systems start to matter.

Agentic AI Does Not Make the System Disappear

This is where I think some of the current enthusiasm for agentic AI risks becoming dangerously simplistic. There is an understandable fascination with what individual agents can do: how quickly they can write code, analyze information, create documentation, conduct research or complete administrative work. Those capabilities are very impressive, but organizational performance has never simply been the sum of the productivity of individual resources.

Suppose a Development Agent can produce software changes ten times faster than the Testing Agent can evaluate them. The Development Agent looks extraordinarily productive, but work begins accumulating in front of testing. Work in progress increases, lead time grows, feedback arrives later and defects remain undiscovered for longer. More work may be started while previous work remains unresolved.

We have not eliminated the constraint. We have simply moved it.

This is exactly the kind of mistake organizations have made for decades when they focus on resource efficiency rather than flow efficiency. We celebrate how busy or productive an individual resource appears while ignoring what is happening to the work as it moves through the whole system.

AI does not repeal queueing theory. If anything, the ability to create work at extraordinary speed makes understanding queues, WIP and Flow more important than before.

AI Could Become the Greatest WIP Generator We Have Ever Invented

Humans Are The Constraint

Humans provide a natural constraint on how quickly organizations can create work. There are only so many reports, requirements, product ideas, software changes, analyses and proposals people can produce in a day. AI removes much of that constraint.

An agent can generate dozens of product ideas, hundreds of requirements, thousands of test cases or enormous quantities of code. The cost of creating another piece of work can approach zero, but the cost of absorbing that work does not necessarily disappear with it.

Someone or something may still have to review it, validate it, integrate it, test it, approve it, deploy it, monitor it and maintain it. If those downstream capabilities do not increase at the same rate, AI has not necessarily created more value. It has enabled us to create inventory faster.

That inventory may be digital rather than physical, but from a Flow perspective the underlying problem is remarkably similar. Producing something is not the same as delivering value, and starting more work does not mean more work will finish.

The risk is that organizations will point proudly at enormous increases in AI-generated output while customer lead times, rework, complexity and coordination costs quietly increase around them.

Local Optimization Is Still Local Optimization

Now imagine that we give each member of our agentic team its own performance objective. The Product Agent is rewarded for identifying more opportunities, the Requirements Agent for specifications completed, the Development Agent for code produced, the Testing Agent for tests executed and the Release Agent for deployment frequency.

Every agent can improve its metric while the performance of the whole deteriorates.

Incentives are Human Nature

Again, there is nothing uniquely artificial about this problem. Human organizations have been doing it for decades. Functional departments optimize their own objectives, often with perfectly rational behavior from their own perspective, while value becomes trapped between them.

Agentic AI could make this worse because agents can optimize what we ask them to optimize with extraordinary persistence. If we encode simplistic targets into their goals, we should not be surprised when they pursue those targets.

A Development Agent instructed to maximize code output can produce more code. A Testing Agent rewarded for the number of tests executed can produce enormous test suites. A Product Agent measured by ideas generated can create an endless backlog.

The measures improve. Whether the system improves is another question entirely.

The lesson should already be familiar to anyone who has worked seriously with organizational measures: a measure should help us understand the system. The moment we turn it into a target for an autonomous actor, we change behavior around that measure. AI does not remove that problem. It potentially industrializes it.

I wrote an article about when local optimization may be desirable in a complex system.

Decision Authority Still Matters

Agentic systems also force us to think carefully about where decisions should be made. A centralized orchestration agent can appear attractive because it gives us control, but if every decision has to travel through one central point, we may simply recreate the bureaucratic bottlenecks of traditional organizations in software.

The opposite extreme is no better. Allowing every agent to make whatever decisions it considers appropriate can create fragmentation, conflicting actions and systemic risk. The answer is not simply centralization or decentralization, but the thoughtful distribution of decision authority according to knowledge, uncertainty, consequences and accountability.

Agents need sufficient autonomy to respond to local conditions while remaining constrained by the purpose and needs of the wider system. Humans also remain part of that architecture. An agent may be perfectly capable of generating tests, correcting a routine defect or deploying a low-risk change, while accepting a significant security risk, changing a fundamental architecture or making a regulated decision may appropriately remain a human responsibility.

The objective is not maximum autonomy. It is effective decision-making within the system.

Governance Cannot Be Left to the Agent

Govern Agents Independently of Reasoning

This is why governance also has to exist independently of individual agent reasoning. Imagine that a Development Agent completes a change, the Testing Agent approves it and the Release Agent prepares to deploy it. The fact that the agents collectively believe the work is ready does not necessarily mean that deployment should be permitted.

The wider system might require a successful security scan, zero critical vulnerabilities, architectural compliance and human approval for particular classes of change. Those conditions should not merely be suggestions available for an agent to reinterpret. They are constraints on what the system is permitted to do.

This creates an important distinction. Agents can reason about what they believe should happen, but governance determines what their reasoning is allowed to cause. As organizations grant agents increasing authority to interact with real systems, customers, finances, infrastructure and data, that distinction becomes increasingly important.

The Human in the Loop Is Part of the System

Human Oversight Is a Legitimate Concern

One of the most common concerns when people discuss agentic development is the need to keep a human in the loop. Software developers want to know whether an agent should be allowed to change production code without review. Architects worry about agents making design decisions based on incomplete context. Security teams want humans involved when significant risks are identified, while executives are understandably reluctant to allow autonomous systems to make consequential decisions without some form of human oversight.

These are legitimate concerns, but “human in the loop” can easily become another simplistic solution if we are not careful. Simply inserting a human approval step into an automated process does not necessarily make the system safer or better. The important question is not simply whether a human is involved, but why they are involved, what decision they are expected to make and whether they have the knowledge and capacity to make it effectively.

Humans Can Become the Constraint

If every decision generated by an agent has to wait for a human to review it, we may have created an extremely fast upstream process feeding a much slower downstream constraint. Agents can generate work faster than humans can possibly inspect it, the approval queue grows, feedback slows down and people eventually begin approving things superficially because the volume of work has overwhelmed their capacity to review it.

That is a systems problem, not an AI problem. We could build an extraordinarily sophisticated collection of AI agents capable of performing work in seconds and then create days of waiting because everything eventually arrives in somebody’s approval queue. From a Flow perspective, the consequences should be obvious. We have accelerated one part of the system without increasing the capacity of the system as a whole.

This is precisely why human oversight cannot simply be added as another stage in the workflow. If the human becomes the constraint, increasing the productivity of the agents feeding that constraint may actually make overall system performance worse. More work accumulates, lead time increases and the quality of the human review itself may deteriorate.

The Question Is Where Humans Belong in the Loop

A question we should be asking is where humans need to be in the loop, what information they need, what decisions genuinely require human judgment, and what authority should remain with them. A routine, reversible and well-understood decision may be safely delegated to an agent. A novel, ambiguous, high-consequence or irreversible decision may require human judgment. Between those extremes there will be many decisions where the appropriate degree of autonomy depends upon context.

Inside or Outside the Loop?

It is useful here to distinguish between humans being in the loop, on the loop, and sometimes deliberately outside the operational loop. A human in the loop participates directly in a decision before the system can proceed. A human on the loop supervises the system, observes its behavior and can intervene when necessary without becoming a mandatory approval gate for every transaction. Other activities may be sufficiently bounded, observable and reversible that agents can operate autonomously while humans monitor the performance of the overall system rather than approving every individual action.

This matters enormously for Flow. If we require human intervention everywhere simply because we distrust AI, humans become the constraint and queues form around them. If we remove humans everywhere simply because the technology is capable of acting autonomously, we risk allowing errors, misunderstandings and unintended consequences to propagate at machine speed. Neither maximum human control nor maximum AI autonomy is a sensible design principle.

The better principle is to design human and artificial participants as components of the same system. Decision authority should sit where the appropriate knowledge and judgment exist, while the degree of autonomy should reflect uncertainty, consequence and reversibility. Humans should not become expensive rubber stamps inserted into every automated workflow, and agents should not receive authority merely because they are technically capable of exercising it.

Humans Are Also Part of the Learning System

There is another reason humans remain important that goes beyond approval and governance. Human involvement should not exist solely to inspect what an agent wants to do. People also need to observe how the overall system is behaving, question the assumptions upon which it was designed, notice things the measures are not showing, interpret weak signals and recognize when the system itself needs to change.

This is where the relationship with continuous improvement becomes particularly important. An agent might be exceptionally good at operating within the rules and objectives it has been given while remaining completely unaware that those rules and objectives are producing undesirable systemic consequences. The ability to execute a process and the ability to question whether that process should exist in its current form are not necessarily the same capability.

The human role therefore begins to change. Instead of humans performing every activity or approving every automated decision, their contribution increasingly includes governing, observing, challenging and improving the system through which humans and agents collectively create value.

The human is not simply in the loop of the work. Humans also need to be in the loop of learning about and improving the system itself.

This Is Not Really a Software Development Problem

I have used software development as an example because it makes the structure of an agentic team relatively easy to see. We can imagine Product, Requirements, Architecture, Development, Testing, Security and Release agents passing work between one another, making decisions and responding to feedback. But there is nothing particularly unique about software in the underlying argument. Agentic AI is better understood as a participant within a wider socio-technical system rather than simply another piece of automation.

The Same Problem Appears in Manufacturing

Consider a manufacturing system. We could have agents monitoring customer demand, scheduling production, ordering materials, controlling inventory, inspecting quality, predicting maintenance requirements and coordinating logistics. Those agents could potentially react to information considerably faster than a conventional planning and control system. Yet if the production system itself contains excessive batch sizes, poor process capability, unreliable equipment, inappropriate inventory policies, long changeover times or badly designed incentives, adding AI does not make those conditions disappear.

An AI scheduling agent could produce a theoretically optimal production schedule every few seconds, but if the physical system cannot execute that schedule reliably, all we have created is a faster mechanism for rescheduling failure. An inventory agent could optimize the availability of individual components while inadvertently increasing inventory across the whole value stream. A maintenance agent could maximize machine availability while encouraging production of things the customer does not currently need. A quality agent could inspect defects faster without addressing the process conditions that continue creating those defects.

Anyone familiar with the Toyota Production System should recognize the distinction. The objective is not to make every machine run continuously, maximize every local resource or optimize every individual process independently. The objective is to create a system capable of delivering value through effective Flow while exposing abnormalities and creating the conditions in which problems can be understood and removed.

Service Systems Are No Different

The same reasoning applies to financial services, healthcare, insurance, supply chains, customer support and almost any other system of work. The work may be less visible than something moving along a production line, but there is still a system through which information, decisions and value must flow.

Every System Has a Flow

Imagine AI agents processing insurance claims. One agent extracts information, another assesses eligibility, another detects fraud, another calculates settlement and another authorizes payment. We can automate an extraordinary amount of that process, but if the underlying policies are contradictory, customers repeatedly provide information the organization already possesses, approvals exist because departments do not trust one another, or performance measures encourage claims handlers to minimize payouts rather than resolve legitimate claims effectively, agentic AI will inherit those conditions.

It may even make them worse because the dysfunctional process can now operate at enormous scale. The agents may execute every step precisely as designed while customers experience the same delays, unnecessary complexity and poor outcomes more efficiently than before.

This is an important point because we often associate automation with improvement. They are not the same thing. Automation changes how an activity is performed. Improvement changes the performance of the system.Sometimes automation achieves both, but we should never assume that one automatically produces the other.

The System, Not the Agent, Is the Unit of Analysis

This is why I think we should be careful about treating agentic AI primarily as a technology implementation. The important unit of analysis is not the agent. It is the system of work within which the agent operates. That system may contain people, AI agents, machines, software, policies, suppliers, customers, information, physical constraints and regulatory requirements. Value emerges from their interactions, not from the productivity of any one component.

This also changes how we should think about AI productivity. An individual agent can become dramatically faster or more capable without producing a corresponding improvement in the performance of the overall system. In some circumstances, increasing the capability of one component can actually make system performance worse because it generates work faster than the rest of the system can absorb it, shifts a constraint elsewhere, increases variation or encourages further local optimization.

That is exactly why understanding the system has to precede simply inserting agents into it.

The Questions of Continuous Improvement Remain the Same

Once we see the problem this way, the connection to continuous improvement becomes much clearer. Whether the work involves software moving through development, components moving through a factory, a patient moving through healthcare, a claim moving through an insurer or an order moving through a supply chain, the questions are fundamentally similar.

Where does value actually flow? Where does work wait? Where does information become distorted or delayed? Where are decisions being made without the necessary knowledge? Where is work being pushed into the system faster than it can be absorbed? Where are defects and rework being generated? What are our measures encouraging people and agents to optimize? Where are humans required because their judgment genuinely adds value, and where have we simply created another approval queue? Most importantly, how quickly can the system detect that reality differs from what we expected and learn from it?

Those questions mattered before generative AI existed, and they will continue to matter when organizations contain hundreds or thousands of autonomous agents. In fact, the speed at which agentic systems can act may make answering them considerably more important.

AI changes the capabilities available within the system. It does not change the need to understand the system.

Continuous Improvement Has to Become Part of the Agentic Architecture

This is where the connection to continuous improvement becomes particularly important for me. Continuous improvement is too often treated as a set of activities applied to an existing organization: run a Kaizen event, create a value-stream map, install some visual management, reduce waste and improve a process.

I have always regarded it as something much deeper. Continuous improvement is an organization’s ability to sense reality, expose problems, learn from what happens, change its behavior and continually improve the system through which value is created.

An agentic organization needs exactly the same capability. It should not merely execute work faster. It should be able to observe how the work itself is working.

That means understanding where queues are forming, where WIP is accumulating, where failures repeatedly occur, where handoffs create delay, where rework originates, where decision authority creates bottlenecks and where apparently sensible rules produce unintended consequences. It means monitoring Flow rather than merely activity and being able to recognize weak signals that indicate the system is behaving differently from what its designers expected.

The agentic system therefore needs at least two forms of feedback. One feedback loop operates the work: observe the situation, decide, act, examine what happened and adjust. Another operates on the system itself: observe patterns in how work is flowing, identify problems, develop hypotheses, experiment with changes, learn from the results and adapt the system.

That second loop is continuous improvement. Far from becoming less relevant because AI can automate more work, continuous improvement becomes part of the architecture required to make agentic systems sustainable.

Don’t Automate the Organization You Already Have

Perhaps the greatest risk is also the simplest. Organizations will take their existing operating models and automate them.

Existing departments will become collections of specialized agents. Existing handoffs will become API calls. Existing approval processes will become automated workflow gates. Existing KPIs will become agent objectives. Existing functional silos will become agent boundaries, and existing bureaucracy will execute at machine speed.

We could spend millions of dollars deploying AI only to create extraordinarily efficient versions of organizations that were poorly designed in the first place.

The question should therefore not simply be, “How can we use an agent to automate this activity?” Before we automate anything, we should understand the system in which that activity exists.

• Why does the work exist?
• What customer value does it create?
• Where does the information come from?
• Where does it go next? What dependencies exist?
• What feedback is required?
• Where are the constraints?
• Who needs authority to make which decisions?
• Which activities genuinely add value and which exist only because of the way we have historically designed the organization?

Only then should we ask how AI might change the system. Otherwise, we risk automating things that should never have existed in the first place.

The Goal Is Not Maximum AI Productivity

This may ultimately be one of the most important lessons. The objective of an agentic organization should not be to maximize the productivity or utilization of its agents. The objective should be to improve the performance of the whole system in creating value.

Sometimes that will mean an agent should be idle because there is no reason to create more work. Sometimes it will mean deliberately limiting how much work an agent can initiate because the downstream system cannot absorb it. Sometimes the fastest agent should slow down. Sometimes an agent should stop the entire process because it has detected something unexpected. Sometimes a human should intervene, and sometimes the most valuable thing the agentic team can do is reveal that the system through which it has been asked to work is fundamentally wrong.

These are not new lessons. They are the same lessons we have struggled to learn in human organizations about Flow, WIP, feedback, measures, local optimization, decision authority and continuous improvement.

What changes with AI is the speed and scale at which we can get them wrong.

AI Can Automate the Work. It Cannot Fix the System for Us.

AI agents will undoubtedly change how work gets done, and agentic teams may change organizations even more profoundly. Faster reasoning, cheaper execution and greater autonomy, however, do not remove the fundamental realities of systems.

• Work still flows through dependencies.
• Queues still form.
• Variation still matters.
• Local optimization can still damage global performance.
• Measures can still distort behavior.
• Information can still arrive too late.
• Decision authority can still reside in the wrong place,
• and rules can still create consequences nobody anticipated.

It’s Not About Replacing Humans

The great opportunity presented by agentic AI is therefore not simply replacing human activity with machine activity. It is the opportunity to reconsider how we design systems of work in which humans and artificial agents operate together. We should resist the temptation to encode a century of management dysfunction into those systems and then congratulate ourselves because they operate faster. Instead, we can design around Flow rather than utilization, purpose rather than activity, feedback rather than compliance, learning rather than prediction, and appropriately distributed authority rather than unnecessary control.

Most importantly, we can make continuous improvement a property of the system rather than an intervention performed occasionally upon it. An agentic system should be capable not only of performing the work, but also of helping us understand how that work flows, where problems emerge, where assumptions fail and where the system itself needs to change.

The greatest risk of agentic AI may not be that the agents fail. It may be that they work exactly as designed inside a system that was never designed particularly well in the first place.

Before we automate the work, we need to understand the system through which that work flows. The starting question should not be “Where can we deploy an AI agent?” but “How does this system actually work, where does value flow, what constrains it, and what should we improve before we automate it?” Otherwise, we are not necessarily transforming how value is created. We may simply be automating a broken system.

If your organization is exploring AI agents or agentic operating models, this is where I can help. We can examine the system before you automate it: identifying where AI can genuinely improve Flow, where it could create more WIP or simply move the constraint somewhere else, where humans need to remain in or on the loop, and how continuous improvement can become part of the agentic architecture from the beginning.

Don’t automate the dysfunction. Understand the system. Improve the Flow. Then decide where AI can create value.