Design this Day | Episode 8

Behind the first AI that understands the physical world.

Leonardo Giusti joins Teague Futurist Devin Liddell to talk about how humans can wrestle agency back from AI, why designing for “polite” and “considerate” AI reveals a deeper design challenge, and how this work can inspire optimism—even among skeptics. Leonardo is the founder and design director at Archetype AI, a company building a completely different kind of AI. Before starting Archetype AI, he led design at Google's advanced technology and projects division where he worked on some of the company's most experimental ideas. Now, he is training an AI model using – not language – but sensor data from the physical world. 

Scene: You are driving on the highway on your way to a meeting, brake lights start stacking up. You ease off the gas and groan, another bottleneck.

"You are still on the fastest route."

You're going to be late, but then something strange, the slowdown doesn't get worse. In fact, traffic starts flowing again. You glance up at the digital sign stretched across the highway.

"AI traffic management enabled."

Up ahead, an on-ramp has temporarily stopped cars from merging onto the crowded highway. To your left, a carpool lane has opened up just long enough to ease the backup. Soon, you're on the open road again. You exit into a side street in an industrial part of town and glide through the quiet intersections. Suddenly, the traffic light ahead starts flashing red. You hit the brakes, you look around confused. There's no one waiting to cross. Then you see small brown ears bouncing in front of your car. You see it, a dog hopping onto the curb. The AI model guiding the traffic system saw the dog before you did. The light turns green and you continue on your way. This is the future of sensor-powered AI. Welcome to Design This Day, a podcast about the future, the futures we want, and the people working right now to make those futures real. 

Devin Liddell: Let's get into it, starting with what sparked Leo's curiosity in large language models, and training an AI model using not words but sensor data.

Leonardo Giusti: We pre-trained a model for sensor data, and that's how Archetype AI started basically. And now we are building Newton, we call it Newton, which is our foundation model designed for making sense of sensor signal and translate them into a form that we humans understand.

Devin Liddell: When you say making sense of sensor data, that's the headline version of Newton's superpowers, if you will, taking all of this data that would actually be incomprehensible to the human mind and then turning it into something that can be understood by those around it. Is that a good capture of it?

Leonardo Giusti: That's it. That's exactly what Newton tried to do, discover patterns in the physical world that are beyond our capability as humans and present them to us in a way that we can understand.

That's exactly what Newton tried to do, discover patterns in the physical world that are beyond our capability as humans and present them to us in a way that we can understand.

Devin Liddell: There have been times when I have attempted and probably fairly clumsily tried to explain some of the promises of AI, and this is one of the reasons I'm so interested in what you're doing with Archetype AI. And I'll get to the clumsy example. I will sometimes say, "Well, sometimes everyone who's had a dog probably has some experience similar to this, where you watch the dog try to drag a stick inside the house." And they bring it in broadside and it hits the sides of the door and you want to urge it, "Hey, if you just turned it the other way, you could bring it in." But my sense is that that's probably... When you talk about the superpowers of pattern recognition around data from all these sensors, it enables that type of insight that maybe we just even can't quite comprehend yet. Is that fair to say that some of the outcomes that you're after at Archetype AI are in some ways hard to wrap our heads around?

Leonardo Giusti: 100%. And I really like the example of the dog, by the way, because sometimes I feel like we are the dog.

Devin Liddell: Well, exactly, yes, we are 100% the dog in that scenario.

Leonardo Giusti: We're trying to bring this stick inside and we can't and from our limited point of view, we can't understand why we can do it. And that's why I think AI is so powerful in this way because it can give us these superpowers to actually see beyond our limitation in terms of sensing capability that we have, but also cognitive capability and overcome this limitation so that we can make better decisions on how to organize a boarding process or a factory floor or any other processes. This goes at the core of the vision that we have at Archetype AI.

Devin Liddell: If we are the dog, then Leo is saying that Archetype AI could help us see and anticipate the problem before we reach the door and advise us on how to bring the proverbial stick through the doorway on the first try. We talk a lot on this podcast about which jobs are good for AI and which jobs are good for humans. But Leo is asking a different question altogether. Unlike many new AI companies, Leo is not seeking to replace boring or repetitive tasks that humans don't want to do. Instead, he says Archetype AI will help humans see what we otherwise could not.

Leonardo Giusti: It's really trying to see AI as an instrument to understand the world around us. And this is very important, I believe design implication, as well, because if you look at today AI landscape, very often AI is designed as an assistant that takes the form of a chatbot or a robot. And in this context, we interact with AI by delegating a task. I tell you to do something, you go out in the world, physical or digital, to do something for me and you accomplish this task on my behalf.

So if we actually change the perspective dose, when AI serves as an interpretation layer for the physical world, like AI as an instrument, things are actually different. We actually use AI actively to explore and discover the world around us. And so AI ultimately empowers us and augments us to make better decisions. And so the agency, the ability to affect change in the world goes back to people. Instead of delegating it, we take it back in a way. And so that's why we are proposing from a design point of view, a new metaphor, shifting from the agent, which is the typical metaphor for automation to the idea of lenses.

We actually use AI actively to explore and discover the world around us. And so AI ultimately empowers us and augments us to make better decisions.

Devin Liddell: A lens is the metaphor Leo uses to describe the different versions of Newton, his company's foundation model. For example, there could be a lens for optimizing warehouse operations or managing supply chains in real time. Or remember in a previous episode we talked about the thousands of sensors installed at Dallas-Fort Worth International Airport. Maybe Newton could offer a lens that helps keep flights on time. As Leo described what this kind of AI could do, it was easy to see how powerful it might be. These lenses could not only surface at meaningful patterns, but also help builders make important design decisions that actually improve outcomes. One of Leo's current projects is working with the city of Bellevue, right here in Washington state, to build a lens focused on traffic safety.

Leonardo Giusti: We are working with the city of Bellevue where they have a lot of traffic cameras at intersections, and they're very interesting in improving road safety. And so they want us to build a lens that receives as input the camera feed plus the signal coming from the traffic lights, pedestrians, go, stop when it turns red. And by combining these two things, the lens can report back what they have defined, like a near-miss, an accident that almost happened but didn't happen. And so basically this lens is designed to receive this input and create a heat map at a different intersections where these little near-misses are happening so that basically the city now has a map of the city where the areas that are the most dangerous ones, and so they can actually intervene and go there and make the changes before something bad actually happens.

Devin Liddell: And I'm assuming that going back to your earlier point, the heat map that the lens produces is a form of pattern recognition that would otherwise be entirely invisible to traffic planners, for example?

Leonardo Giusti: Yes.

Devin Liddell: They wouldn't actually see it, okay.

Leonardo Giusti: You can think AI as an instrument, like a microscope or a telescope or X-rays. What it does is able to identify these patterns in the world that you cannot directly access. With AI, we can just augment this capability that we have to understand the world around us, but ultimately, the agency, as you said, is back to us. We are, in the end, the ones making the decision on what to do with this information.

Devin Liddell: Interesting. This is a purely devil's advocate question. I'll just give you a super rudimentary example I sometimes struggle with myself. When I was a teenager, my mother had no idea where I was most of the time. I actually can see where my family is on my phone. I can see their approximate location. I wrestle with that, and I often sometimes try not to look up where they are and simply ask them... An example of my two teenage sons, I'll say, "Well, what are you up to and when might you be home?" And so forth. When I actually could just look up their personalized location and surmise for myself what they're up to. But I guess what I'm after, is there a scenario in which we actually have more sensor data than is good for us?

Leonardo Giusti: I think there are probably scenarios in our everyday life where we want to retain our ability to connect as humans without relying too much on sensor data or automation in general. And I think it's a very complex question and honestly, I don't know where is the boundary between what's good and what's bad. What we're trying to do at Archetype, we are trying to start from something where we know there is value, and this is the industrial world. So we are working with partners in the context of safety in construction sites or manufacturing process, logistics, all these industrial use cases where we know there is a problem that we can help solve it with AI. And so I think it's a good starting point. At the very beginning, you want to be in a place that is a little bit more constrained and you can start understanding what are this technology? How actually are they really working? And especially with AI.

I think there are probably scenarios in our everyday life where we want to retain our ability to connect as humans without relying too much on sensor data or automation in general. 

Devin Liddell: What Leo is saying is that Archetype can do more than just optimize workflows and make construction sites safer for people. He sees even bigger possibilities, like predicting when machines will fail before they actually do. Take wind turbines, for example. Right now, operators often deal with costly downtime when the turbines break down, but in the future, AI could pinpoint the exact moment a part will need replacing, cutting downtime and keeping energy flowing. If you scale that kind of solution across industries, it would be a win-win for profit and sustainability.

Leonardo Giusti: I'm not an optimistic guy. I'm Italian. I'm very pessimistic, but I do have some hope here that AI can really unlock a new industrial revolution. The idea that we can arrive to an era where the way we make things is way more efficient than we do today, and we also shift the conversation from, and this is a big word, right now, we're a lot talking about sustainability, maybe with AI we could talk about an era of abundance for everyone because we are getting so good and make use of the resources of our planet that there will be more for everyone. Harvesting resources and using them efficiently is a very, very hard problem, and there is a lot of network effects and complexity that we don't completely understand as human. And maybe with AI as an instrument, it can allow us to see these complexities, being very optimistic.

Devin Liddell: Well, I love it. Well, and I want to ask about that some more in the sense that when you were first working in developing Newton, were there moments when you were taken aback by its superpowers?

Leonardo Giusti: There was a moment personally. I was really like, "Wait, what? It does what?" And we showed that we train our model with a lot of sensor data, motion sensors, temperature sensor, all sorts of different things. And then we asked the model to predict things that it has never seen before. And we gave the, I think it was some data from the electric grid of some city in Turkey that is public available. The model, without any training, was able to predict a blackout. I said, "How does it do that?"

Devin Liddell: Newton was able to generalize what it had learned about the world, apply it to a data set that it had never seen before, and accurately predict the behavior of a piece of infrastructure. This ability to generalize is what differentiates a model like Newton from older AI models that need task-specific training, but Newton's analytical abilities were not restricted to earth. It also made forecasts about celestial phenomena that to the human mind seemed completely unrelated to the original dataset.

I do have some hope here that AI can really unlock a new industrial revolution. 

Leonardo Giusti: And there are a few other things, it was able to predict sunspots, those phenomenon in the sun, which are completely [inaudible 00:14:45] and nothing to do with each other. And we don't know exactly, there are some hypothesis in the paper about the model has developed some sort of representation, deep representation of how things work in the physical world beyond the specific sensor. And so at the end of the day, maybe the model has understood some sort of basic understanding of the world of physics, in a way, and how things work, and now it's able to do all this prediction and detections. But that was a moment where we were like, "Oh, that's actually quite interesting," even from a philosophical point of view. As it discover a new general equation of the physical world, this is just very speculative and question, I don't know the answer, maybe not, but it's interesting to think about it.

Devin Liddell: I'm assuming this must challenge your deeply entrenched pessimism as an Italian, right? A model that could unlock all new understandings of the world. That's extraordinary. When you think about what you're doing now at Archetype AI through the lens of design, what are some of the biggest design challenges you're trying to solve in terms of where you're going with Archetype AI?

Leonardo Giusti: In terms of design challenges, there is a straightforward interaction design challenge, which is how do you interact with an AI that is in the world nearby you, alongside you? If you really think about that, when we interact with other people, for example, in the physical world, we are not only using language, right? You and I, if we were here together and we were trying to do something on a desk, we look at each other in the eye, we point at things. There is a lot of nonverbal activities, communication that help us to anchor that language into what it actually means. That's why, for example, it's very weird sometimes to interact with these embodied AI agents, Alexa and so on, because they really don't know what's going on around them. And you have to describe everything with language, to the point when it's ridiculous. And so one of the biggest challenges that we have is how do you, beyond language, can orient the model toward the things that you're interested in? How do you orient, for example, its attention to a certain portion of the world?

Devin Liddell: One quick story about that as an exemplar of that, that I think it was the first time we had, I won't say which one it was, but the first time we had a conversation-based AI in our home, and when it would lose its internet connection, just even for a moment, it would announce that it had lost its connection, which sounds completely benign. And to your point about how an engineer might say, "Well, yeah, of course, just when it loses its internet connection, it should just tell the user that it's lost its connection." But in practice, and to your point about context, this happened to me a number of times where I was near asleep on the couch, late at night, might be 12:30 AM, and all of a sudden this disembodied voice comes out of the darkness and says, "I've lost my internet connection," which, of course, I don't hear the words. All I hear is someone is speaking who should not be there.

Leonardo Giusti: This is a great example. I'm going to reuse it in some of my future talks because it's perfect. This idea that we are really putting a lot of effort to make devices smarter and smarter and smarter, but can we make them more polite too? Can we make them more considerate of what's happening around them? Is it too much to ask? Why they interact me with a stupid notification when I'm about to fall asleep on the couch? Or why they just announce something weird where I'm having dinner with my friends?

What if devices around us have some social grace as well, that they understand the world around them and social relationships and use that as an information to make decisions when to interrupt us? It's really, really hard because, as you know, social interactions are very complex and fuzzy and hard to define. And traditional machine learning was really struggling to make sense of the social world. I think we have a chance to actually build this intelligence that is not just intelligence, but actually is considerate of what's happening around it. And so we can really create devices that live with us, that are a little bit more polite, considerate, they're more harmonious with our everyday life.

We can create devices that live with us, that are a little bit more polite, considerate, they're more harmonious with our everyday life.

Devin Liddell: This idea of making AI technologies polite or considerate points to a more fundamental principle that I think a lot about in my work, and that is design needs to be a key consideration with all technology, especially technology that is so embedded in our everyday lives. It's the difference between something that integrates intuitively into our interactions with the world and something that gets discarded after a few uses. I'll always be an advocate for building systems that are not only intelligent, but also respectful, responsive, and aware of the social context around them.

Leonardo Giusti: One of the future project that we are thinking about is when the AI, for example, detects someone in a wheelchair or someone that is struggling to go across the street, maybe the lens can actually inform the light traffic itself and increase the time to go across the road. So in this case, the AI is here with me in the physical world and is helping me to do something in real time.

Devin Liddell: It's one of the reasons I'm in love with your approach to it is that in comparison, there are times when we envision AI powered futures, particularly through the lens of automation, I'll say, that we imagine future worlds that, to your point, work very well, but if you breathe them in a bit, they are very efficient with a coldness to them, meaning that it's great if you are the person who is not suffering an anomaly, if you are the user who is not outside the norm. But what you're describing is actually very, very different, which is this extraordinary superpower of seeing patterns, but also stepping in and being, actually, this might be a strange way to describe it, but almost warm in its embrace of everyone around it.

Leonardo Giusti: Yeah, if we really want to create a world where people in AI live together, they need to understand each other and understanding each other is not just anticipating your needs or everything, because humans are not just needs. We have emotions, we have feelings, and we need AI to be participating in our everyday life in a way that is considerate and meaningful. So if we really want to succeed, I think we have to do it.

Devin Liddell: I love that. That's fantastic. Okay, so time for a speed round. What is your take on the most over-hyped technology?

Leonardo Giusti: I just moved into a new house and I am putting blinds everywhere. And the most over-hyped technology are motorized blinds. Everyone that comes to my house tries to sell me motorized blinds. And I say, "I don't need it. I don't need to press a button to operate my blinds." First of all, I'm not that lazy, but also, I love the idea to go there and open the windows by myself in the morning because that gesture, it is beautiful. In the morning when I wake up, I open the window and the sun comes in. So I think I'm in this world now where I'm trying to install blinds in my house, and the most over-hyped technology, for me, is motorized blinds. And I think we should move beyond it.

Devin Liddell: It is perfect in its analogness. We just need to keep it. All right. Other end of the spectrum, what's your take on the most under-hyped technology?

Leonardo Giusti: During COVID, we developed a vaccine in one year and we accelerated development of this technology, the mRNA technology, which allow us to deliver specific instruction to cell to make proteins that can be used to fight viruses and possibly develop individualized cancer treatment. And I don't have a background in biology or anything, but it seems to be such an incredible thing that I don't understand why we don't talk more about it because this can dramatically change our life. It seems such a powerful thing and I feel like we are talking too much about AI.

Devin Liddell: I love that. What book or show has influenced how you're thinking about your role as an innovator right now?

Leonardo Giusti: I think one of the books that has influenced my career the most is the work of a Russian psychologist that I studied when I was in college. His name is Vygotsky, and he has this theory about how human intelligence doesn't only exist in our brain, but is actually the result between the interaction between us and the environment around us. And it is always mediated by the tools we use, whether this tool are materials or immaterials, like mathematics, for example. And then as a designer, this really clicked with me and say, "Well, if I can change this tool and make them better, I can make people smarter." That book for me was really important in the way I grew as a designer.

Devin Liddell: The book is called Mind in Society by L.S. Vygotsky. All right, last one. What's an innovation in another industry that is particularly compelling or exciting to you and that might've helped you think differently about what you're doing?

Leonardo Giusti: I found myself always in these weird places as a designer where I question myself, "Am I a designer? What am I doing here?" And so I often think about architects where a lot of the work they do is start from the material itself. They take a material and take inspiration from this material to create a building, and they try to build something that is unique and authentic to this material. So I'm really trying to study AI as a material so that I can actually build something that is authentic to this material, but also eventually meaningful for people. Because if you don't appreciate it, understanding that you're going to build something that it doesn't really make much sense.

Devin Liddell: That's very unexpected. I did not expect that's where you were going, but it's beautiful. You are for sure a designer, so you should put aside that self-doubt. But maybe all designers worth their salt are actually wracked with self-doubt. But it reminds me, there's an old dumb joke, of course, about designers, and it goes like this. It says, "How many designers does it take to change a light bulb?" The punchline, of course, is does it need to be a light bulb, or can it be something else? But you actually took it even farther than that, even further. The joke that you inadvertently wrote just now was, how many designers does it take to change a light bulb? And then the punchline is, do they even know they're designers?

Leonardo Giusti: That's fantastic. That's my life. Thank you.

Devin Liddell: Thank you for listening to Design This Day, a podcast by Teague. We will be back soon with more episodes. Subscribe on your favorite podcast app so you don't miss the next one. And if you have a complex problem that needs solving, we'd love to hear from you. Visit us at teague.com or send us an email at hello@teague.com.