As autonomous systems take on more of the work, people still need the visibility, authority, and context to own what happens next.

I recently hosted a panel at Teague with Jake Sprouse of Overland AI and David Smith, a principal designer here at Teague, to talk about human agency in autonomous systems—particularly autonomous off-road vehicles for defense applications.

One of the things I kept coming back to was what it actually means for a person to remain in command as the system takes on more of the planning and execution. What do they still need to understand? When do they need to step in? And if they’re accountable for the outcome, how much authority do they need along the way?

Autonomy can reduce the number of decisions a person makes directly. It can interpret the environment, compare options, and act toward an objective without waiting for approval at every step. But delegating the work doesn’t delegate the responsibility. In most cases, accountability still sits with a person or an organization.

That changes what we need from the interface. An emergency stop or override is part of it, but it isn’t enough. The harder question is whether the person can understand what the system is doing, influence the plan before something goes wrong, and take over without having to reconstruct the entire situation from scratch. Our conversation surfaced four parts of that problem: what the person can see, how much authority they have, what happens when control moves back to them, and how decisions are explained afterward.
 

CHALLENGE 01

Visibility: understanding what the system sees.

Autonomous systems don’t follow instructions in the same way traditional software does.
With a conventional interface, a person selects an action and expects a predictable result. With autonomy, they might only provide the objective, leaving the system to interpret the environment, weigh alternatives, and determine how to proceed.

That creates a gap. The system is making a series of decisions, but the operator isn’t necessarily watching each one happen. They still need some way to understand what the system thinks is going on, what it plans to do next, and where it might be getting into territory it doesn’t handle well.

The system is making a series of decisions, but the operator isn’t necessarily watching each one happen. They still need some way to understand what the system thinks is going on.

This challenge became clear in Teague’s work with Teague's work with Overland AI on OverWatch, the company's software for coordinating autonomous vehicles.

During early testing, operators sometimes plotted a vehicle’s route one waypoint at a time, even when the autonomy was capable of navigating to a destination on its own. At first glance, it looked like a lack of trust. As David put it during the panel, “Understanding precedes trust.” But distrust was only part of the story.

The operators didn’t have a clear picture of what the system could handle, how it would respond to a change in terrain, or when it might need help. There was also a cultural issue. In a military environment, a plan generally has to be communicated, reviewed, and defended. Telling an autonomous vehicle to “get over there somehow” doesn’t fit neatly into a chain of command built around deliberate planning.

Part of the response depends on the technology itself. Overland AI continues to refine its data-fusion tech to better predict a realistic route—or route options—to a destination that might be several kilometers away. Our design work addresses how operators understand and compare those routing options without having to analyze every detail. Which route is fastest? Which offers the best prospect of survivability, the lowest likelihood of detection, or the least risk of damaging the vehicle? Making those tradeoffs clear helps someone decide what to delegate and better communicate a more confident plan to the other people who depend on it.

The challenge is giving them that understanding without showing every calculation taking place underneath.

An autonomous vehicle might process thousands of signals and make thousands of small decisions in a short period of time. Showing all of that would be useless to the operator. Showing too little leaves them guessing. A lot of the design work sits somewhere in between: deciding what the person needs to see to understand the plan and recognize when it no longer makes sense.

Operators remotely monitor an Overland AI autonomous vehicle through OverWatch, using live vehicle views and mission maps to understand its surroundings and maintain control.

CHALLENGE 02

Authority: shaping what the system does.

Seeing what the system intends to do is only one part of the problem. The person also needs a meaningful way to change it.

This is where the phrase “human in the loop” can get a little vague. Technically, someone might be in the loop because they have access to an emergency stop or override. But if their only choices are to let the system continue or shut it down completely, that isn’t much of a working relationship.Human 

agency depends on the ability to shape the plan before action begins, redirect it as conditions change, and correct the system when its reasoning no longer fits the situation. That becomes harder as autonomy scales.

Seeing what the system intends to do is only one part of the problem. The person also needs a meaningful way to change it.

OverWatch is being designed for a future in which one operator could coordinate several vehicles at once. Their role starts to look less like operating a machine and more like orchestrating a system: setting objectives, evaluating routes, assigning tasks, and stepping in where judgement is needed.

The autonomy might suggest several routes, coordinate assignments across multiple vehicles, or adapt when something changes in the environment. The operator can’t follow every movement in real time, and they probably shouldn’t have to. But they do need to know when the plan has changed in a meaningful way and have a practical way to correct it.

This gets even more important as generative AI moves into planning.

A future system could take a short mission objective and generate a coordinated concept of operations for multiple autonomous vehicles. That could replace hours of manually assigning tasks, entering coordinates, and sequencing actions. It could also produce a plan that sounds coherent in language and is completely wrong in practice.

I don’t think the answer is to ask someone to read pages of generated instructions and verify them line by line. That’s not how the mission is going to happen, so it probably isn’t the best way to understand whether the plan works.

The operator needs to see the plan spatially and over time. A map might reveal that two routes conflict, that a vehicle is being asked to cross terrain it can’t handle, or that the timing doesn’t work. Those problems can be difficult to notice in a block of text and immediately obvious once they’re visualized.

They also need to be able to change the plan directly. Maybe they move a route, reassign a task, or tell the system that a certain area is off-limits. The point isn’t simply to accept or reject what the AI produced. It’s to give the person enough leverage to work through the plan and make it better.
 

CHALLENGE 03

Handoff: returning control with context.

Autonomy is often treated as a binary state, with either the system or the human in control.

In practice, control moves continuously between them. The system might operate independently under ordinary conditions, request guidance when it encounters ambiguity, and return control completely when it can’t proceed. The difficult part is what happens at that moment.

Someone directly operating a vehicle understands what has happened over the last several seconds. Someone supervising several autonomous vehicles might have been focused elsewhere for twenty minutes.

It has to help them understand what they’re stepping into, without adding to the cognitive load they're already managing.

The design challenge is deciding which boundaries the system should always enforce and where operators should be able to set the terms of delegation.

Where is the vehicle? What was it trying to do? What changed? Why did it stop? What does the system think the problem is, and how confident is it? The operator also needs to know what happens if they don’t respond right away. Without that context, the person might technically have control but still be several steps behind the situation. The timing of these interruptions matters too.

As Jake noted during the panel, “If it asks too often, operators stop trusting it because it's constantly interrupting them.” But waiting too long creates another problem: the operator may inherit responsibility for a situation they had no realistic chance to prevent.

The design challenge is deciding which boundaries the system should always enforce and where operators should be able to set the terms of delegation.

Some situations should require human intervention by default. If an operator asks a vehicle to reach a destination that has since been designated as off-limits, the system needs someone to resolve that conflict. This can be baked in at a system-level since it is a clear issue.

Other situations leave more room for judgment. If a vehicle encounters an obstruction, it may be able to find another way around. But how long should it try before asking for help? That decision might depend on the mission, the vehicle, or the operator's preferences. Whatever the parameters, the system needs to make them clear so the operator understands when and why intervention is required.

That’s one of the tensions we’re still working through case by case. There is a degree of fine-tuning based on the current situation and even operator preference. The common dividing line is often determining what can and should be baked into the handoff logic as a default versus what should be prompted to the user as parameters when planning a mission.
 

CHALLENGE 04

Accountability: explaining what happened.

Autonomous systems can take action, but they can’t take responsibility.

If a mission fails, a vehicle makes an unsafe maneuver, or a system contributes to harm, accountability still falls to a person or an organization.

That means the interface has another job beyond helping someone manage the system in the moment. It also needs to help them understand how the decision was made and how the situation changed over time.

As Jake put it during the panel, “If it’s going to be my responsibility, I need the interface to help me make the right decisions. I need receipts.”

I think “receipts” is the right word. If someone is going to answer for the outcome, they need to know what the system understood, what it recommended, where it was uncertain, and what alternatives were available. They also need to see what the person changed and how those changes affected the final plan.

If someone is going to answer for the outcome, they need to know what the system understood, what it recommended, where it was uncertain, and what alternatives were available.

That record matters after something goes wrong, but it can also help before it does. Visibility into uncertainty, alternatives, and changes to the plan helps operators decide when to intervene. And their responsibility for the outcome doesn’t remove the software provider’s responsibility to help them improve it.

The same autonomy that helps execute a mission could also help close that feedback loop: flagging recurring problems, capturing the context behind an operator’s correction, and proposing changes for review. Some improvements might be adjustments a team makes to how its own system behaves; others might require changes from the software provider. The interface should make that collaboration easier, so operators can help shape better behavior over time as well as answer for what happened.

That partnership can help the system improve over time. But the operator still needs to make sound decisions with the system as it exists today. This starts to take us beyond the usual definition of usability. It’s no longer only a question of whether someone can operate the product. We also have to ask whether they can responsibly hand work over to it.

Do they understand what the system is doing well enough to trust it? Can they recognize when it has reached the edge of what it knows how to do? Can they intervene in a way that actually changes the result? And when they review an outcome with their team or the software provider, can they explain what happened and help identify what needs to change?

These questions aren’t peripheral to the technology. They help determine whether the technology can be trusted with meaningful responsibility.

That’s still control, but it’s a different kind of control than we’re used to designing for. The person might not choose every action, but they still need to understand the system well enough to guide it and remain connected to the consequences.

Autonomy can execute the plan, but responsibility remains with the people and organizations that operate, design, and build it. The interface should support that partnership, helping them understand what happened and improve what happens next.