As autonomous systems take on more of the work, people still need the visibility, authority, and context to own what happens next.

I recently hosted a panel at Teague with Jake Sprague of Overland AI and David Smith, a principal designer here at Teague, to talk about human agency in autonomous systems.

One of the things I kept coming back to was what it actually means for a person to remain in command as the system takes on more of the planning and execution. What do they still need to understand? When do they need to step in? And if they’re accountable for the outcome, how much authority do they need along the way?

Autonomy can reduce the number of decisions a person makes directly. It can interpret the environment, compare options, and act toward an objective without waiting for approval at every step. But delegating the work doesn’t delegate the responsibility. In most cases, accountability still sits with a person or an organization.

That changes what we need from the interface. An emergency stop or override is part of it, but it isn’t enough. The harder question is whether the person can understand what the system is doing, influence the plan before something goes wrong, and take over without having to reconstruct the entire situation from scratch. 

Our conversation surfaced four interconnected challenges: understanding what the system sees, having the authority to shape what it does, returning control to a person with enough context to act, and ensuring people can understand and explain the decisions they're ultimately responsible for.
 

CHALLENGE 01

Visibility: understanding what the system sees.

Autonomous systems don’t follow instructions in the same way traditional software does. With a conventional interface, a person selects an action and expects a predictable result. With autonomy, they might only provide the objective, leaving the system to interpret the environment, weigh alternatives, and determine how to proceed.

That creates a gap. The system is making a series of decisions, but the operator isn’t necessarily watching each one happen. They still need some way to understand what the system thinks is going on, what it plans to do next, and where it might be getting into territory it doesn’t handle well.

The system is making a series of decisions, but the operator isn’t necessarily watching each one happen. They still need some way to understand what the system thinks is going on.

This challenge became clear in Teague’s work with Overland AI on OverWatch. During early testing, operators sometimes plotted a vehicle’s route one waypoint at a time, even when the autonomy was capable of navigating to a destination on its own. At first glance, it looked like a lack of trust. But distrust was only part of the story.

The operators didn’t have a clear picture of what the system could handle, how how it would respond to a change in terrain, or when it might need help. There was also a cultural issue. In a military environment, a plan generally has to be communicated, reviewed, and defended. Telling an autonomous vehicle to “get over there somehow” doesn’t fit neatly into a chain of command built around deliberate planning.

So before someone can decide how much work to hand over to the autonomy, they need a better sense of its limits. The challenge is giving them that understanding without showing every calculation taking place underneath.

An autonomous vehicle might process thousands of signals and make thousands of small decisions in a short period of time. Showing all of that would be useless to the operator. Showing too little leaves them guessing. A lot of the design work sits somewhere in between: deciding what the person actually needs to see in order to understand the plan and recognize when it no longer makes sense.

Operators remotely monitor an Overland AI autonomous vehicle through OverWatch, using live vehicle views and mission maps to understand its surroundings and maintain control.

CHALLENGE 02

Authority: shaping what the system does.

Seeing what the system intends to do is only one part of the problem. The person also needs a meaningful way to change it.

This is where the phrase “human in the loop” can get a little vague. Technically, someone might be in the loop because they have access to an emergency stop or override. But if their only choices are to let the system continue or shut it down completely, that isn’t much of a working relationship.

Human agency depends on the ability to shape the plan before action begins, redirect it as conditions change, and correct the system when its reasoning no longer fits the situation. 

That becomes harder as autonomy scales.

Seeing what the system intends to do is only one part of the problem. The person also needs a meaningful way to change it.

OverWatch is being designed for a future in which one operator could coordinate several vehicles at once. Instead of driving each vehicle directly, the operator establishes objectives, evaluates routes, assigns tasks, and manages exceptions across a larger mission. Their role starts to look less like operating a machine and more like orchestrating a system.

That role starts to look less like driving and more like managing a team. The operator isn’t controlling every movement. They’re setting intent, reviewing the plan, and stepping in where their judgment is needed.

The autonomy might suggest several routes, coordinate assignments across multiple vehicles, or adapt when something changes in the environment. The operator can’t follow every movement in real time, and they probably shouldn’t have to. But they do need to know when the plan has changed in a meaningful way and have a practical way to correct it.

This gets even more important as generative AI moves into planning.

A future system could take a short mission objective and generate a coordinated concept of operations for multiple autonomous vehicles. That could replace hours of manually assigning tasks, entering coordinates, and sequencing actions. It could also produce a plan that sounds coherent in language and is completely wrong in practice.

I don’t think the answer is to ask someone to read pages of generated instructions and verify them line by line. That’s not how the mission is going to happen, so it probably isn’t the best way to understand whether the plan works.

The operator needs to see the plan spatially and over time. A map might reveal that two routes conflict, that a vehicle is being asked to cross terrain it can’t handle, or that the timing doesn’t work. Those problems can be difficult to notice in a block of text and immediately obvious once they’re visualized.

They also need to be able to change the plan directly. Maybe they move a route, reassign a task, or tell the system that a certain area is off-limits. The point isn’t simply to accept or reject what the AI produced. It’s to give the person enough leverage to work through the plan and make it better.
 

CHALLENGE 03

Handoff: returning control with context.

Autonomy is often treated as a binary state, with either the system or the human in control.

In practice, control moves continuously between them. The system might operate independently under ordinary conditions, request guidance when it encounters ambiguity, and return control completely when it can’t proceed. The difficult part is what happens at that moment.

The goal isn’t to keep the person watching constantly. It’s to bring them back in early enough, and with enough information, that their judgment can still make a difference.

Someone directly operating a vehicle understands what has happened over the last several seconds. Someone supervising several autonomous vehicles might have been focused elsewhere for twenty minutes. If the system suddenly asks that person to take over, it can’t just hand them the controls and expect them to be ready. It has to help them understand what they’re stepping into.

Where is the vehicle? What was it trying to do? What changed? Why did it stop? What does the system think the problem is, and how confident is it? The operator also needs to know what happens if they don’t respond right away. Without that context, the person might technically have control but still be several steps behind the situation.

The timing of these interruptions matters too.

If the system asks for help too often, the operator ends up managing the autonomy rather than benefiting from it. If it asks too late, the person might inherit responsibility for a situation they had no realistic chance to prevent.

That’s one of the tensions we’re still working through. You don’t want the system interrupting the operator every time it encounters something slightly unusual. But you also don’t want it waiting until the only remaining option is an emergency intervention.

The goal isn’t to keep the person watching constantly. It’s to bring them back in early enough, and with enough information, that their judgment can still make a difference.

CHALLENGE 04

Accountability: explaining what happened.

Autonomous systems can take action, but they can’t take responsibility.

If a mission fails, a vehicle makes an unsafe maneuver, or a system contributes to harm, accountability still falls to a person or an organization.
That means the interface has another job beyond helping someone manage the system in the moment. It also needs to help them understand how the decision was made and how the situation changed over time.

As Jake put it during the panel, “If it’s going to be my responsibility, I need the interface to help me make the right decisions. I need receipts.” I think “receipts” is the right word. If someone is going to answer for the outcome, they need to know what the system understood, what it recommended, where it was uncertain, and what alternatives were available. They also need to see what the person changed and how those changes affected the final plan.

If someone is going to answer for the outcome, they need to know what the system understood, what it recommended, where it was uncertain, and what alternatives were available.

That record matters after something goes wrong, but it can also help before it does. If the system shows that its confidence is low, the operator knows to take a closer look. If it shows several options, they can compare the tradeoffs. If the plan has changed repeatedly, they can understand how it ended up where it is instead of treating the latest version as though it appeared fully formed.

This starts to take us beyond the usual definition of usability. It’s no longer only a question of whether someone can operate the product. We also have to ask whether they can responsibly hand work over to it.Do they understand what the system is doing well enough to trust it? Can they recognize when it has reached the edge of what it knows how to do? Can they intervene in a way that actually changes the result? And if someone asks later why the system acted the way it did, can they give a credible answer?

These questions aren’t peripheral to the technology. They help determine whether the technology can be trusted with meaningful responsibility.
As autonomous systems become more capable, people will probably perform fewer of the individual actions themselves. They’ll spend more time setting objectives, evaluating plans, managing exceptions, and making the decisions the system can’t make on its own.

That’s still control, but it’s a different kind of control than we’re used to designing for. The person might not choose every action, but they still need to understand the system well enough to guide it and remain connected to the consequences.

Autonomy can execute the plan, but it can’t own the outcome. That responsibility remains ours.