AI Leadership Insights: Prompt Engineering

Mike Vizard: Hello, and welcome to the latest edition of the Techstrong.ai video series. I’m your host, Mike Vizard, and today I’m with Satish Jayanthi, who is CTO for Coalesce. And we’re talking about prompt engineers and just how many of them we might need or if we need them at all because well, there’s certainly a lot of hype going on. Satish, welcome to show.

Satish Jayanthi: Thanks, Mike. Thanks for having me here.

Mike Vizard: What is the job function of a prompt engineer? Is this just somebody who does something interesting that can get some interesting results out of a large language model, or is this a real job?

Satish Jayanthi: Yeah, good question. So we’ve been trying to talk to the systems. If you go back to the machine level programming where basically the idea was how do you communicate with the computer? We have started from the low level languages, high level languages. Now we are here abstracted all the way out where we can say in our own language to a system, speak to that system and make it do what we are looking to do. So in this case, specifically, we’re talking to AI models. So these large language models, that’s the advent of these models. Now you can communicate with them in a natural language, which is what prompt is. And prompt engineering has raised to importance because of the LLM’s nature.
So today, if you really want an LLM to get a desired result, you need to understand a little bit of the internal workings as well. And prompt engineering is a way to, hey, how do I say this prompt? How do I pass this prompt in the right way to get the desired results? And if you think, go back when Google came out, when we used to search in Google, we used to… I mean, I remember doing double codes around adding a plus at the end or something like that to make it search the right things. But as Google advanced, there was no need to do that anymore. You don’t do those kind of things. You just search what you’re looking for, it’ll tell you. And I believe that’s something similar that’s going to happen in this area as well. As the AI systems get better and better, the prompt engineering requirement may not be that significant, is how I see it down the road. Now, what is not going to change is the underlying data and the quality of the data that is needed to train these models.

Mike Vizard: So I may need more data engineers than I need prompt engineers per se. And it seems like the issue to me is, as you well pointed out, a lot of the core prompt engineering magic is going to be inside a platform somewhere that somebody else has already done for me because they’ll be able to infer from my initial prompt what it is I’m looking for from there. So how soon do we get that? Because right now I’ve talked to some folks and they’re showed me some results of something they did and it took them 40 prompts to do it, and I was like, “I can do that on my own without that 40 prompts thing in about five minutes.”

Satish Jayanthi: Yeah. It is a rapidly changing field. So while it is still in its… It’s a first generation LLMs, I would say, but it’s changing so fast. I mean, even the features, I was talking to somebody this morning, they were trying something on ChatGPT. The last time I met that person was six months ago, and he started that back then and now we were talking this morning and he said, “The system that I started six months ago working with that now, it’s so different. It’s so advanced now.” Is what he’s been telling me. So things are changing very, very fast. I don’t have a magic number when this is going to go away, but I would say they’re changing so fast that you may not be surprised a year or two years from now, our LLMs might be even way better than what they are today.

Mike Vizard: To your point, by the time I finish a project that I start today, the entire landscape will be completely different. Right?

Satish Jayanthi: It’s possible.

Mike Vizard: Is the issue at the end of the day though, how are we going to operationalize these LLMs? I mean, today it is great that I can interact with them to answer a question or whatever, but ultimately they need to become part of a larger process. They need to be embedded in something that’s automated. So maybe that’s where our focus should be is not so much how to make them dance on a particular tune on a particular day, as much as it is going to be about how do we use these things for some larger goal?

Satish Jayanthi: Yeah, absolutely. So in order for us to be able to take advantage of these systems, you need to really understand what are the main pieces that we need. For example, compute power. You need certain compute to run these things. You need to be able to also operate this under a certain cost. I mean, if it’s cost prohibitive, it doesn’t make sense. So those are technical things, but the big elephant in the room is the data. The data that you feed these models and how you train them, what is going into these models and the transparency. Once you train these models, are the users are able to trust these systems. Even before AI, we already have a trust issue with the data. Anytime the data is put in front of business, you get different answers. People are questioning these and all of that. Now with AI systems, it’s almost like a black box or magic for some people. We better be very, very careful what we put into these systems.

Mike Vizard: Do we need to vet the LLMs better? Because some of the things that we’re looking at, or I would call them general purpose systems, they collected data from everywhere in the world, and a lot of that data conflicts, and we’re surprised when there’s a “hallucination.” Of course, when you and I are wrong, we’re just flat out wrong and a machine is wrong and it hallucinates, but that’s okay. The difference though that seems to me is we can create LLMs on a narrower set of data that are trained for a specific purpose and then connect all these things together. Is that the ultimate way to go? Or is that just too much trouble?

Satish Jayanthi: That’s where the world is going. I mean, that seems practical to me. I mean, you don’t want to take this massive LLM into your production and work with it because it’s expensive. It’s going to hallucinate because there’s just so much in there. It’s general purpose. Performance may be a challenge, cost may be a problem. So where we are going with this is we are saying, how about a small foundational model that, for example, LLaMA or something like that, 7 billion parameters. How about we take something like that and then train it on a particular domain for a particular function? And that while it may be a little bit harder to train it, and it all depends on what you want from it, but it seems like that’s more manageable and practical in production identities.

Mike Vizard: Well, a lot of the use cases that people are looking at, I may not have to have any real LLM expertise, right? I can just put a vector database in front of this thing, load that database up and show it to the LLM and start getting some results without necessarily having to have all that data get sucked into the LLM, which has all kinds of interesting governance issues with it. But well, most people just wind up working with a vector database in front of these things.

Satish Jayanthi: Yeah. I mean, vector database is a big piece of this. I mean, I would say you can do some things without a vector database also. To keep it simple, it all comes down to the use case. I mean, if you’re new, I would start small, and then as the need grows and as your understanding becomes better, then you can put a vector database and use that to improve the responses.

Mike Vizard: What is your best advice to IT leaders about all this? Because I know that they are being confronted with everybody who works with them suddenly wants to work on an AI project. And of course to make time for that, they’re automating all kinds of things that they previously ignored, but that’s great. But do I really need that? And what is the right approach if you’re an IT leader? And I mean, how do I do this in a economically responsible fashion?

Satish Jayanthi: Yeah, absolutely. So this is not something new. I mean, we humans generally get excited about shiny objects and you get carried away and make some decisions impulsively and emotionally. So that will still happen. However, my advice is take a step back, understand the business use case first, understand what are we trying to solve. Not all use cases are applicable for an AI technology. I mean, you got to understand that first of all, anything that has to be 100% accurate, it’s a no for AI. In my opinion, where we are today. And down the road, I’m not sure. Maybe it’ll get there, maybe not. But today the use cases are much more like, Hey, I can live with 80% accuracy or 80+. It’s not going to kill anybody. It’s fine. It’s going to help. That’s the type of use case. So picking up the use case is the number one thing you want to understand. This is what…
The next important thing is, do we have the foundation to support an AI system? Do we have the data that we need? Is this data the quality of the data? What is the quality of the data? Because the quality of the output from these AI systems is directly proportional to the quality of the input data that we put feed this. So you got to be careful in what you’re feeding these systems and you got to consider the speed, the performance, but also maybe there’s ethics that play into this. The compliance maybe in another thing. If you mess up, are you going to be fined by, I don’t know, Security Exchange? Maybe they’ll fine you millions of dollars because you’ve screwed it up. So it’s important for companies to really understand that they have a strong foundation and they’re picking the right use case and they’re starting small.

Mike Vizard: Ultimately, do we need to clean up our data management act as a result of all this? Because historically I think we’ve been a little sloppy in the way we manage data, and is this finally going to force the issue?

Satish Jayanthi: I hope so. I mean, definitely it’s been a while. I’ve been in this space for more than 20 years now. We’ve been talking about data governance, data quality forever, and on one side you can say things have improved, but also at the same time you can say it’s still the same. That’s because the amount of data has grown. Maybe we understand the importance of quality, but the target is a moving target. Now, the amount of data that we collect is much larger than what it was before. So the net effect is probably same thing. We still have the same issues. But AI could raise the importance of that because of the nature of how the systems behave. There’s no transparency in a way. I mean, at least we are not there yet. Nobody’s talking about AI transparency yet. They are starting to, but not at the level. I think once we get some more maturity, people will start talking about that, but I guess that’s where we are today.

Mike Vizard: All right, folks, you heard it here. There’s going to be a lot of new jobs and there’s going to be a lot of old jobs that are going to be in higher demand. But you might want to think twice about that prompt engineering thing because as we just pointed out, this thing’s moving faster than you realize. Hey, Satish, thanks for being on the show.

Satish Jayanthi: Thank you. Bye.

Mike Vizard: And thank you for all watching the latest episode of the Techstrong.ai video series. You can find this episode and others on our website. By all means, spend as much time as you like with that. And until then, we’ll see you next time.