Some AI jobs are small decisions rather than conversations. A shop might need to sort a message into “billing,” “shipping,” or “technical support.” On October 2, the llama.cpp team announced support for several open decision models that can make choices like these on your own computer.
What changed?
llama.cpp, software used to run AI models locally, now has a /v1/systemone endpoint. An app sends the situation and the options it wants scored. A supported model returns a choice or score with probabilities, instead of writing a paragraph. The change is in a merged llama.cpp update.
The new endpoint follows the format introduced with TypeSafe AI’s Jev. The team lists five supported open models at launch, including smaller text-only models and OpenJev, which can also examine images. OpenJev is a separate open model; this announcement does not say TypeSafe released Jev’s own weights.
Why might someone use this?
Imagine sorting saved receipts. You give the model each receipt and the categories “groceries,” “travel,” and “other.” Your app can file clear cases and send uncertain ones to you. Running an open model locally gives developers another way to build that workflow without sending each receipt to a hosted decision-model service.
A probability is a model output, not proof that the category is right. The team recommends testing examples from your own task and choosing when a person should review a result. The article’s published speed numbers were measured on one NVIDIA RTX PRO 6000; they do not tell us how fast any model will run on a Mac or an ordinary PC.
If you are new to the idea, start with what a decision model does and its local-versus-hosted explanation.