Synopsis
Mike Vizard: Hello, and welcome to Techstrong.ai. We’re here with Alex Spinelli, who’s vice president of product management for machine learning at Google. And we’re talking about Open XLA, a new project, that’s gonna help maybe smooth out a lot of the inconsistencies we see with various frameworks that are out there for machine learning. Alex, welcome the show.
Alex Spinelli: Thanks. I’m excited to talk about this project.
Mike Vizard: So walk us through it; it seems like there are interoperability challenges. And every time we seem to embark on some sort of new product category we come up with this issue seems to always arise. And now we seem to be moving down the path to address that. But where are we on the adventure?
Alex Spinelli: Yeah, I mean, I like any new technology and new set of capabilities. Lots of organizations and people and developers go and often innovate. And where we ended up is this pretty fragmented environment where, you know, different frameworks run better or targeted on different hardware platforms. When you build an ML program, ML really is a programming activity, right, you define the model, and then that actually ends up looking a lot like a traditional program. So it needs to be compiled, and it needs to be able to run somewhere in an optimized way. And, you know, for many, many years, different frameworks built, you know, different underlying programming structures, and there was no kind of lingua franca that allowed one framework to build kind of a almost, this is not a great analogy, but it’s a workable one kind of the Java runtime or Java bytecode version of the ML program that can then run well on different types of hardware and infrastructures. So one of the things that, you know, we started looking at the the ecosystem at Google. And we’re committed to enabling developers at large to innovate in ML; we just think that this is kind of a rising tide that will lift all ships. So we really want to see innovation at large, this idea of portability, being able to use any framework as a developer so that you can have choice, whatever your favorite framework is, if it’s TensorFlow, or Pytorch. Or now Jackson is evolving out of Google research. And you wanted to run it on your GPUs or GPUs at Google. Or, you know, we think there’s going to be an explosion of Ml running on all kinds of devices, and with a variety of hardware architectures. So we took a step back and Open XLA was born to say, how can we create kind of a common compiler framework and a common representation of an ML program that allows that portability. And so that’s the journey we’re on now.
Mike Vizard: So to your point, this almost sounds like the initial steps of an attempt to write once and run anywhere – is that what we’re after?
Alex Spinelli: Yeah, I think that’s fair as like a top level goal. I think we’re a little ways away from that. There’s some kind of interesting specifics from machine learning around how it’s optimized, and the mathematical operations that different hardware and frame and infrastructure support. So yes, ideally, you’d like to write once and run anywhere. Absolutely. I think there is a degree of optimization that’s required. So I think this is a big step in that direction, right, be able to write from anywhere, which is actually kind of important piece. It’s a little bit less, right anyway, right once but like right, from anywhere, your your framework of choice, have a representation of that ML program that you can reliably run on a compiler and runtime that, you know, supports that representation in an effective optimized way. So yeah, long term. That’s the goal. I think, you know, it’s gonna take a little while to get there. And there’s some steps. But but but that is definitely what, you know, what we’re hoping for.
Mike Vizard: How big is the community right now? And just how large do you think it will get?
Alex Spinelli: Yeah, so we’ve really started reaching out to the different hardware platforms. So now we have folks from Amazon, AWS, AMD, Apple, companies like Hugging Face, obviously Nvidia and Meta – the hardware and framework builders. And you know, we’re looking to grow that developer community. Now that we’re kind of out in the wild and starting to foster that engagement. So I think we have a really good footprint in terms of organizations that are participating. And now it really is sort of the bring on board the developers at large.
Mike Vizard: Do you think as we go along that just about every application or use case is going to be infused to a certain degree by an AI model, and it’s just going to become a standard artifact of everything we do?
Alex Spinelli: Yeah, I think, I mean, obviously a lot going on right now. And it’s kind of the Zeitgeist in terms of how AI can impact a lot of what we do. And I think we would agree, we think that ML will be a core component of many, many things, many bits of technology devices and applications. And I think one of the things that we’re really kind of obsessed with is this idea of it needs to be high performance need to be safe and responsible. Right, and it needs to kind of enable high quality experiences. And you know, for us, one of the best ways of doing that is two things really creating a standard, right that we can rally around. And then getting broad support from, you know, a lot of diverse groups and the wider developer community. So that’s, that’s kind of like you’re hitting on the hitting the nail on the head in terms of why we think Open XLA has some real legs here.
Mike Vizard: I think a lot of people are kind of scratching their heads these days. We all hear about MLOps, and we’ve had DevOps and we’re building applications, and then we have these AI models. Are there going to be a set of best practices, from your perspective about integrating those two things to kind of bring those motions together in a way that makes it easier to deploy all this stuff?
Alex Spinelli: Yeah, so again, I think that once you can create kind of a standard idea of like, this is the program, right? And you could build that program from your the your, your framework of choice. And then hey, here’s a standard for building a runtime, a compiler and a runtime for that type of program. I actually think that opens up the possibilities for other innovators to actually build all the management frameworks that can help, you know, run the run the run the framework build, take that artifact, which is really a bunch of operations and some data, right, that’s really what represents an ML program, a whole bunch of big graph, and then a bunch of operations on that graph. And then how do I deploy that and run it and have all the kind of telemetry and metrics out of that. So I think this is where your MLOps and DevOps do start to kind of blend together a bit. And then you see this kind of, we think this kind of next explosion of of operational, and infrastructure management tooling specific, specifically around some of the ML capabilities. So this should be kind of a key to start unlocking some of that opportunity for sure.
Mike Vizard: Do you think we’ve thought through enough about the process of continuously updating these AI models? I think we get so excited about building them and deploying them in the first place, but they’re subject to drift and things can happen. So do we need to kind of think through the best practices for not just building them but maintaining them?
Alex Spinelli: Yeah, I think that’s a really important point. So a couple of things that we were targeting, at least, you know, we’re down in the infrastructure here. But a key advancement of of Open XLA is something called Stable HLO, which allows versioning, and serialization and things like forward and backward compatibility, so that this idea of updating something, you can kind of again, have the toolkit to do that, right? You know, the version of it, you know, how the opposite it can support where it can want run, where it can’t run, you know, that you can guarantee it can run on something, it can’t run on something, and then you can update it and track those versions over time, in terms of the kind of ml and like what it does. This is a kind of a big, big area of research is you know, how quickly models are getting bigger? How quickly can we provide feedback loops to those models, so that we can update them, increase their quality, increase their safety, you know, take away challenges or issues that are get surfaced. So this is something that we’re investing quite a bit in, I think open XR isn’t isn’t the place for that outside of enabling kind of underlying versioning capabilities and deployment capabilities. So that might be a different conversation to have in terms of how to update models and the data in those models. But it’s definitely something we’re spending a lot of time and attention on.
Mike Vizard: In terms of making it possible to run the models anywhere, it seems like some models may be really large and run in the cloud and others are going to run at the edge and some of them will be built in the cloud and move to the edge. So the whole notion of a model is going to be much more fluid in terms of where it gets deployed and how it gets deployed. And that’s why we need to address these issues. It seems like it’s core to the thing.
Alex Spinelli: Yeah, I mean, I think, I think the short answer is I agree, yes. We’re gonna have to figure a lot of those things out. I think, again, we are but a model. I know there’s they’re special in some ways, right? Because they’re statistical based programs, right? They’re, they’re non deterministic, right? So that’s pretty different than a traditional program where you say, Do this, do that do this? A model is you know, at this likelihood you’re gonna have this kind of outcome, but in effect, there are programs, right? So some are really big and need a lot more processing, and some can be miniaturized, and run on lighter weight hardware. So I think that’s where you’re gonna have them as a business, you know, as a developer or product builder, or as business, looking at the performance parameters of different models will help you understand where you can run this and where you can put it. And I can tell you, for sure, the researchers at Google and across the industry are definitely, definitely running pretty hard and understanding how can we make these models more efficient? How can we, you know, take get the same quality and run them at lower cost? So I think that’s a big, big push across, not just Google, but across the whole industry is how can we make these things more efficient and more cost effective, and more able to run on, you know, lighter hardware.
Mike Vizard: You think at the end of the day, part of this whole effort is to make the whole building in the models more transparent and open to everybody. So we’re not so intimidated; sometimes we are less inclined to engage, because we don’t know how things work. And we’re really just not an entirely joint community education exercise.
Alex Spinelli: Yeah, this whole idea of like, programmability, debug ability, usability explainability, all these abilities are something that are really important, I think, for for ML, both for safety, responsibility and basic quality two, right? Did it do what you wanted it to do as a business, right? So those are all tied together, you’re not going to get higher quality models, without safer models, you’re not going to get safer models without higher quality models. So I think there’s a, you know, real virtuous cycle there. And I would say the first big step is creating a standard that the the industry can rally around on the formats of these things, right. So, again, I think one of the things we’re excited about open axillae, is if there’s a standard set of ops, a standard way to represent that model, that it’s transportable and portable, that you know, it can be generated from one framework and run on a different set of hardware. You basically open the door also for tools that can go and look at, well, what are the ops? What is the data? How do we actually provide that degree of explainability and debug ability? So I think a big first step is this idea of standardization. And we really hope the industry, you know, joins us in looking at, you know, building those sets of standards.
Mike Vizard: So what’s your best advice to folks about how to get started going down the path? It seems like a lot of folks have spun up a data science team, but they’re not well integrated with the rest of the business? It seems like, you know, we’re kind of stumbling along a little bit. Is there a more structured way about doing all this?
Alex Spinelli: Yeah, you know, I, I wish I had an answer for that. I think I think this is really early days. I do think, you know, the important for me, I think educating your business leaders, executives, on the power of ML and AI, and the downsides and dangers, and they go hand in hand, right, you’re not going to get one without the other. So you actually have to manage them both. I do believe in bringing on your roles. So ML engineers, Chief scientists, or data leaders who really understand the nuance around thinking about, you know, non deterministic programming, disco program, machine learning, programming. You know, do your research, join the communities, I think lean into the standards. I’m a big believer in standards. And you know, again, in something like open SLA, we would love you know, partnerships. It’s a it’s an open source project. And, you know, we encourage folks to to contribute.
Mike Vizard: Alright, folks, you heard it here – everybody needs to be part of this thing. As you may well find yourself being left behind. But Alex, thanks for being on the show and sharing your knowledge and insights.
Alex Spinelli: Oh, my pleasure. Absolutely.
Mike Vizard: All right. Thank you all for watching the latest episode of Techstrong.ai. You’ll find this and others on our website and we invite you to check them all out, and we’ll see you all next time.