TL;DR — Key Takeaways
– Jevstiller can classify routine requests locally in about 15 milliseconds, compared with roughly 300 milliseconds for Jev.
– The open-source system sits in front of Jev API calls and skips calls when it can statistically guarantee its local answer will match Jev at a preset rate.
– Jevstiller is designed for high-volume, repetitive workloads, including log parsing, security alert testing and multi-step agentic tasks.
TypeSafe’s Jev quickly rose to prominence for its rapid classification—returning labels that classify data in roughly 300 milliseconds per request. This is an order of magnitude faster than the frontier labs with their bulky LLMs. But that still wasn’t speedy enough for developer Tomer Glick.
Glick just released an open-source system called Jevstiller that offers a high-speed shortcut to Jev itself. By sitting in front of Jev API calls, Jevstiller classifies routine requests locally in about 15 milliseconds, backed by a hard statistical guarantee that its answers will match Jev’s at a preset rate. When it does, a call to Jev is eliminated.
Jevstiller is best used with high-volume recurring tasks, such as parsing log files and security alert testing. It can also be of help in multi-step agentic tasks, when the agent must make a series of rapid-fire decisions, and can’t wait for a Jev response for each query. Tasks that require very strict SLAs would also benefit from the technology.
A Highly Calculated Guess
Instead of relying on generic point estimates from validation samples, Jevstiller calibrates directly against a “test slice” of responses from Jev. The software tests candidate thresholds in a strict sequence and stops at the first failure, preventing the system from cherry-picking thresholds from favorable statistical noise.
As the user runs Jevstiller, it periodically refreshes itself, ensuring that it stays in harmony with what Jev would say. The model is only a few hundred kilobytes in size on the user’s machine, and is initially trained using a few thousand Jev prompts. After this initialization, the model is retrained after every 2,000 Jev answers, and then the update is versioned, shadow-tested and promoted to production.
The model does best on a relatively limited variation in source data. It suffers from unpredictable shifts in live data–though Glick notes it can recover quickly. In the middle of a 24 hour “soak test,” a mock Jev changed its answer pattern at hour 12.
Predictably, Jevstiller’s local routing plummeted from 90% to 9% within four minutes. But Jevstiller rapidly bounced back. Jevstiller retrained itself and was back at a 90% success rate within 49 minutes.
It’s worth keeping in mind that Jevstiller makes no assertions about the model’s accuracy against the truth. “If Jev is wrong, the local model is wrong the same way,” Glick wrote.
Faster, Cheaper, Less Expressive
Last month, the release of Jev took the AI world by storm. Diogo Almeida, a former researcher for OpenAI, led the work. Rather than traditional LLMs which prioritize human interactivity, Jev is calibrated towards decision-making.
Jev has a limited set of predefined outputs, along with a probability score of how certain it is of the answer. With Jev, you enter an unstructured set of data, and then either ask a question, issue a command or run a query against the data. You then get back one of a number of predefined answers.
Jev is quicker and cheaper because it doesn’t burn output tokens structuring a verbose answer, and because the answer types are defined ahead of time.
Jev itself is pretty inexpensive – $0.042 per million input tokens. Installing Jevstiller on your local machine could cut these costs even more.
