AI Leadership Insights: AI at the Network Edge

Mike Vizard: Hello and welcome to Techstrong.ai. We’re talking with Jim White today who’s CTO for IOTech, and we’re talking about the challenges of deploying and maintaining AI at the network edge. Jim, welcome to the show.

Jim White: Thanks for having me, Mike. It’s a pleasure to be on.

Mike Vizard: At the risk of a gross oversimplification, we have AI models that are trained up in the cloud somewhere, and then there’s an inference engine created, and then we want to kind of move that inference engine as close to the point where data is being created and consumed, and that sounds simple enough, but we have a lot of data scientists involved, there’s DevOps teams involved. So from your perspective, what are the challenges that we’re seeing as we try to move AI close to the edge?

Jim White: Yeah, I think you’ve hit upon several already. I think first it’s developing that model, like you said, bringing expertise in place. Some of us, I’ll include myself in that category, can barely spell AI. So the challenges of knowing how to collect the data, sort the data, filter out what you need, what you don’t need, generating that initial model, that is a challenge. And that’s a challenge that I leave to those AI experts.
I think once you get beyond that, what at least we’ve seen at IOTech and in my work with open source projects like EdgeX Foundry at the edge is all well and good. Now you’ve developed this wonderful AI model. You could do some really smart things. That’s all good, but now what happens when you want to actually use that at the edge? How do you get it to the edge? How does it run at the edge? What happens when the model needs to be updated and refreshed? Those are not problems that typically the AI communities really know how to wrestle with. Just like we have a hard time spelling AI, they have a hard time spelling edge. So we have to work together and in concert, and that’s really where some of the work we’ve been doing at IOTech recently with groups like Intel have been bearing some fruit.

Mike Vizard: To your point on not seeing MLOps from the edge, it seems like MLOps begins and ends with the creation of the AI model and the inference engine, and then after that they’re kind of like, here’s this lovely AI baby. You guys figure out where to drop it in. So who’s in charge of actually implementing the AI model and working with the folks who are managing the edge and the guys who built the applications and the APIs that call the model? It seems like the whole thing’s a little loose at the moment.

Jim White: I think like any technology, it starts off with some great ideas and slowly evolves to standardization and best practices, and I think we’re going to see that. I mean, we’re already seeing, for example, in the AI model creation, what I’ll call a simplification, a great simplification of how those models get generated. Tools like Lenny AI and Getty from Intel are helping to do that. We’ll see the same types of things I think as we look at bringing AI ML down the edge.
So for our part, what we see as an instrumental part of that is you have to start to think about edge management as part of the solution. Edge management is already out there in terms of doing things like getting applications out to the edge, doing things like updating your OS, and just knowing what things are running out there and whether they’re connected or not connected. So edge management, edge monitoring is part of the solution. That just now has to incorporate some of the ideas around AI. Where is that model? Where does it have to be? When does it have to be updated? Oh, by the way, AI groups, when do you need information about the fact that the model is maybe not working out according to what you’d hoped, and how do you need information back from the edge that says we’ve got a problem out here?
So it’s what I call the feedback loop, right? The model generation, model feeding down to the edge, and then the feedback loop of corrective information coming back to the AI at the cloud or server edge, server level, that is slowly being worked out, but it requires some edge management. It requires some thoughts about how do you correlate the data that you’ve got at the edge with what the model is doing so you can start to say, oh, that’s right, that’s wrong. Just building that pipeline takes a bit of thought. All that is going into starting to think about these patterns and this feedback loop that we’re all going to start to work on.

Mike Vizard: Drift is, to your point, a serious issue, and as one wise person said, it’s one thing to be wrong, it’s another thing to be wrong at scale, and that’s what we’re looking at.

Jim White: Yeah.

Mike Vizard: How do I approach that? Do I just take all the people who manage the edge and the DevOps guys managing the applications and the data scientists and throw them all in one room and lock the door and hope reason prevails, or is there some way to think about this?

Jim White: Yeah, I think there are some ways. Yes, there are ways to think about it. Yes, there are also separations and concerns where you don’t have to think about everything. For example, I don’t need an AI person to help run the edge for me. I don’t need them in edge DevOps, for example. What I do need to have them start to realize is, hey, scale from our perspective is not here’s your USB stick with the AI model, go take care of it. It’s like now you don’t understand, the edge is now displacement of systems across hundreds of thousands of miles and hundreds of units. So a USB stick for deployment is not going to work. We have to start thinking together about how do you give me what I need? How do I give you what you need?
So like we see in a lot of computing, it’s the development of interfaces, it’s the development, the exchange of schemas and objects, and figuring out those touch points. That’s what we need to work on. So we don’t need everybody in the same room room all working on the exact same problems. What we need to start to know is an awareness of each other’s spaces and what are the touch points that bring us together and how can we start to formalize those touchpoints so it makes it much easier to integrate and bring other pieces and solutions into the mix.

Mike Vizard: We tend to view AI as something that a lot of high priest data scientists are involved in, but ultimately soon will those models just be yet another software artifact that we’re managing along in the process along with everything else, and we won’t think it’s magic anymore?

Jim White: Yeah, I’ve said that about edge computing in general, and I think the same thing in terms of AI. I mean, today I consider myself an edge computing IoT expert. I see the day when there will be no such thing. It’s just part of the extended network. And same thing I think will happen in terms of AI. We’re already seeing, as I mentioned, the simplification of that. I have gotten onto some of these systems and developed the first level of model with some simple data. Now, those models may not have had the best algorithm in place. They weren’t always that accurate coming out of the gate. That didn’t matter, because if we do have an appropriate feedback loop, they will improve over time with some of the automation we’re putting in place.
So I do see that it’s important to have some expertise there, but like all things, specialties tend to become more generic and become simpler over time. I think we will see that in the AI world as well, at least in terms of the AI that we’re working with and edge kind of use cases that we’re working with. I don’t proclaim to be an expert and say, oh, ChatGPT is going to be so easy for everybody to understand, and all the social moral problems are all going to go away. That’s a space for somebody else. But in terms of simplifications of the types of AI that are needed at the edge today, the generation of those models, I don’t think you’re going to need to hire a team of a dozen $500,000 a year employees to get it done. You can do things even today with much less resources and have some fantastic results.

Mike Vizard: Do you think we need to figure out some way to test all these inference engines at the edge? Because back in the day, a developer would build something on their laptop and then it would be deployed somewhere and crash, and the guy would be like, well, it worked on my laptop. I feel like we’re going to have the same thing going on here with inference engines, where it worked fine in the cloud, but when it got to the edge, something was amiss.

Jim White: Yeah. I think what we have advised a lot of our clients on is if you have a need for AI, ML at the edge, if you have some sort of inference you’re trying to do at the edge, let’s say let’s talk about something specific like visual inference. You’ve got a camera and that camera’s out in the field and it’s detecting things and giving you results. If you’re doing that kind of thing and you’re relying simply on those inferences, I think you’re going to be left wanting because of the situation you just suggested. Some of the models I’ve seen, even ones built by some fantastic AI engineers, well, they didn’t quite take into account the fact the sun comes sets and rises over the image in the lab. Oh, they didn’t quite think about the fact that in a stadium, that the camera’s going to jiggle quite a bit because of the noise and things that are going on. So there are things that are going to happen in the field that are going to impact that model’s ability to infer.
So what we tell people is think about how you can bring sensor fusion into the picture because that does a couple things. First of all, if you do have a model out there that is running at the edge and giving you some sort of inference, you corroborate that against other sensing devices to say, is it right or wrong? Or is it all wrong? You start to get some appreciation about where some of the differences are, and you can queue back to the back end humans in the loop to say, we got differences are showing up. If it’s something like people count off a camera, hey, the camera and the visual inference is saying we’ve got a hundred people. These other sensors are saying we’ve got a thousand people, somebody’s wrong, and you start to corroborate on that.
And that also, not only does that give you better situational awareness at the edge and tips you off to potential problems, that information itself can be used in these AI model generating tools to help you better generate the next model and again, provide that feedback loop. So corroboration and sensor fusion at the edge today is important so that you can build more trust into what you do at the edge with regard to things like AI and ML. Someday, maybe everything will operate in such a way we don’t need that, but I suspect there’s always going to be things that happen out there in the hot, smelly, stinky places that we put IoT, that things that are done in the lab just didn’t really conceive until it’s too late and things are deployed.

Mike Vizard: Also do you think that the inference engine will need to be optimized for a particular class of processors because, well, there’s five, six definitions of what the edge is, and they all have different types of processors running in different use cases, and I don’t know if I’ve seen that right, where it’s deploy anywhere inference.

Jim White: We’re seeing approaches to that. I won’t say we’re there yet. You’re absolutely right, and that happens across the entire edge IoT spectrum. Everything we do has to be taken into account with regard to, well, how big is your edge? How thick is your edge? Where is your edge? And there’s resource constraints on the edge. Otherwise, we wouldn’t call it the edge. It’d be enterprise computing. So there are constraints, there are things you have to consider, and yep, AI, ML could be a big impact on that, because typically some of these engines require a bit of power.
But we are seeing organizations start to address that. As an example, Intel and OpenVINO has built their engine such that it has awareness of GPUs, the CPU. Allows you to start to build things that operate, maybe not optimally, but at least operate on whatever you have. So there are some things in that realm that I see promise to, but as a matter of fact, today, you do have to make some of those considerations. And so that’s again, some of those touch points that I think we have to have with AI, ML people to say, hey, what you have in your lab and in the cloud in terms of resources ain’t what you’re going to find out there on that telephone pole. So can we think about that a little bit, and how do we either optimize things or reduce things to give us what we need, and not think of the world as this big huge resource pool that’s infinite?

Mike Vizard: So maybe the best thing anybody could do is take the data science team on a field trip, is that what I’m hearing?

Jim White: Well, I would like to do that. In fact, I encourage anybody who’s building an edge type of solution, it’s always a really good idea if you could start to give appreciation, get appreciation around just what does edge mean? I think people have some concepts there, and as you mentioned, there are different types of edges, right? So that’s very helpful. Again, don’t have to be an expert. There are a lot of people out there who have expertise in these fields. Just have an appreciation of some of the things you’re going to encounter, and the fact you may have to think a little bit differently in order to have things operate in these spaces.

Mike Vizard: All right, folks. You heard it here. Get out of the lab and into the field. That’s probably the best thing you can do. Jim, thanks for being on the show.

Jim White: Mike. It’s been my pleasure. Thanks for having us.

Mike Vizard: And thank you all for watching the latest episode of Techstrong.ai. You can find this episode and others on our website along with some show notes. We’ll see you all next time.