A normal LLM call asks the model to write an answer. It might return a sentence, a label, or a JSON object. The application waits for that text, parses it, and extracts the selected team. (View Highlight)
That is unnecessary if every valid answer is already known. (View Highlight)
In fact, a better and more efficient way to do this is to treat the same request as a decision (what Jev does). The application provides the ticket/query and the three allowed answers to Jev. The model then returns a score for each answer in one go (we’ll discuss shortly how exactly you can do that): (View Highlight)
We will provide the input query and allowed answers. In one scoring request, the model will return a decision and a probability distribution without generating a sentence or JSON object. (View Highlight)
While Jev is closed-source, this inference pattern is already available in several open language models. (View Highlight)
More specifically, we will implement it using SGLang (through /v1/score), test it with Qwen and DeepSeek models, and compare it against structured output and ordinary text generation. (View Highlight)
To set expectations upfront, this article recreates the inference path, not the complete Jev system. Jev also includes training and calibration work that a scoring endpoint does not provide. (View Highlight)
Fixed-answer scoring is different from structured output (View Highlight)
It is easy to confuse Jev’s mechanism with structured output since both approaches restrict what the application receives, but they do different work inside the inference server. (View Highlight)
The specified schema prevents an invalid object, so it does not select the team on its own. (View Highlight)
However, this can be further improved since, under the hood, the model still generates the opening brace, the field name, the value, and the closing brace one token at a time. Once generation finishes, the application reads the team field. This video depicts this process: (View Highlight)
With scoring (which Jev does), the application can supply the three teams as the complete list of valid outcomes. The server can read one model score for each outcome and return the distribution shown earlier in this article. It does not generate a JSON object. (View Highlight)
A value of 0.91 means billing received 91 percent of the probability mass to these three choices. It does not prove that the model is correct 91 percent of the time. We need labeled examples to measure that. This is the calibration problem we will discuss later. (View Highlight)
But in the meantime, remember that structured output generates a valid object. Fixed-answer scoring returns a distribution over answers the application already knows. (View Highlight)
How an LLM generates the first output token
Before we discuss how a causal LLM can be turned into a Jev-style model, it would be better to first understand a regular generation step in LLMs.
• A tokenizer first converts the prompt into token IDs.
• The model processes that sequence and produces a vector for the next position.
• The vector has one number for every token in the model’s vocabulary. For instance, Qwen’s vocabulary contains tens of thousands of tokens, so the vector contains tens of thousands of numbers. (View Highlight)
Those raw numbers are logits. A larger logit means the model prefers that token as the next continuation. The values are not probabilities yet. (View Highlight)
During normal generation, the server applies the model’s decoding rules (temperature, etc.) to this vector, selects one token, and appends it to the prompt. (View Highlight)
“Label:” is the final text in the prompt.
So the next position is therefore where the model would normally generate either A, B, or C.
After processing this prompt, the model will produce its usual vocabulary-sized vector for that position. That vector will contain the logit for token A, the logits for B and C, and logits for every other token in the vocabulary. (View Highlight)
Read the three logits at those positions in the vocabulary vector.
Ignore every other logit.
Apply softmax across the three selected values.
If the selected logits are 8.2, 5.5, and 4.8, the restricted softmax produces approximately 0.91, 0.06, and 0.03. We can map those positions back to billing, technical support, and account access.
The normalization is restricted to the declared choices. We are not asking whether A has 91 percent probability across the entire vocabulary.
Instead, we are asking how the model divides its preference among A, B, and C after the application has ruled out every other response. (View Highlight)
This is an operation that SGLang already implements in /v1/score.
It runs the prompt through the model, reads the requested token positions, and returns their scores. It saves us from modifying the Qwen implementation and extracting the final tensor ourselves.
Btw, the reason why the answers use A, B, and C and not score the words “billing”, “technical support”, and “account access” directly is because a visible word is not necessarily one token. (View Highlight)
Comparing those phrases requires sequence scoring. The model must score the first token, append it, score the next token, and combine the values for the complete phrase. Length also becomes part of the comparison. (View Highlight)
The model reads those descriptions when it processes the prompt. The label is only the token whose logit we inspect afterward. (View Highlight)
For instance, tokenizers often encode a leading space as part of the token. The strings “A” and “ A” can therefore have different token IDs. (View Highlight)
To avoid this, render the complete prompt using the model’s chat template. Determine the exact continuation expected at the answer position. Send that continuation to /tokenize. Reject the label if it produces anything other than one token. (View Highlight)
This label mapping stays inside the scoring client. The application sends semantic choices such as billing and technical_support. It never sends token IDs and never receives A, B, or C. That is what it means for the public API to remain independent of the model labels. (View Highlight)
local example. SGLang loads Qwen into GPU memory and exposes its native HTTP endpoints. Our Python script sends requests straight to that server.
This is the complete process:
Start SGLang with a Qwen model.
Write the decision as a prompt with letter labels.
Ask SGLang to tokenize those labels.
Send one request to /v1/score.
Map the returned probabilities back to the choices. (View Highlight)
(View Highlight)
(View Highlight)
(View Highlight)
(View Highlight)
(View Highlight)
(View Highlight)
(View Highlight)