TL;DR — Key Takeaways
- Cribl has launched StreamAI, an AI gateway that routes prompts to the model best suited to handle each request while taking cost into account.
- The platform adds security and compliance controls to the routing layer, including bidirectional sensitive-data redaction and audit-ready telemetry for model calls and routing decisions.
- StreamAI can automatically fall back to alternative models when cost limits are reached, giving IT teams a centralized control point for managing AI usage and reducing dependence on individual model providers.
Cribl today launched an artificial intelligence (AI) gateway that includes a built-in router to enable IT teams to redirect prompts to the model best suited to respond at the lowest cost possible.
Announced at a CriblCon event, StreamAI also makes it possible to apply security and compliance controls directly into the routing layer using Cribl-Privacy capability to protect sensitive data and block malicious requests.
That capability makes it possible to enable bidirectional sensitive data redaction in both outgoing prompts and incoming responses. Additionally, it automatically records every model call and routing decision as normalized, audit-ready telemetry to provide visibility and auditable evidence of governance.
Finally, Cribl also announces it will provide free inference for IT teams using its automatic data routing to access benchmarked AI models to eliminate surprise token bills.
The overall goal is to enable organizations to enforce policies in a way that simultaneously enables them to better keep AI budgets under control, says Nikhil Mungel, head of AI research and development for Cribl. When an enterprise application reaches a cost limit, StreamAI automatically falls back to an alternative model. IT teams now have a built-in circuit breaker that gives them control over token consumption, he adds. “It provides a single control point,” says Mungel.
It’s not clear at what rate IT teams are deploying multiple AI models, but as costs rise, many of them are looking at relying more on smaller open-weight or open-source models that are less expensive to invoke. The vast majority of workflows in the average enterprise do not require the capabilities of the latest AI model, so routing requests to a smaller model presents a significant opportunity to rein in costs. That challenge, as always, is finding a way to centralize the management of prompt requests via a single pane of glass in a way that allows them to be dynamically routed to multiple models based on the nature of the request and the cost of fulfilling it.
Ultimately, it’s only a matter of time before AI applications, in general, become more distributed. That capability will, in effect, make it simpler to bring the AI model to wherever data resides versus always having to share data externally with a service provided by, for example, Anthropic or OpenAI. At the same time, being able to route requests over AI models should provide organizations with more leverage over contract negotiations with any provider of an AI model. In effect, a routing capability goes a long way toward turning AI models into a backend commodity service.
Most organizations, eventually, will be deploying AI models that are deployed everywhere from the network edge and mobile devices to the cloud. The opportunity now is to lay the foundation for enabling the deployment of those models using a platform that makes it possible to route prompt requests anywhere as needed. Otherwise, organizations will find themselves held hostage to a specific provider of an AI model that may, at some point, be likely to increase licensing fees to cover the rising costs of training next-generation AI models.
