In Nish Tahir’s Qwen-based classification demonstration, predictions in the highest-confidence bin averaged 98.55% probability but were correct only 70.09% of the time. That gap is the most useful finding in his new decision-model guide: restricting an LLM to valid answers is straightforward; making its probabilities trustworthy requires additional work.
The October 10, 2026 tutorial uses Qwen3-1.7B to score five possible answers from a single forward pass. It evaluates the classifier on CommonsenseQA, reports a modest improvement after fine-tuning, and applies temperature scaling to adjust its probability distribution.
As a third-party implementation guide, it offers a concrete workflow developers can inspect and adapt, including the evaluation steps needed to assess a constrained answer generator as a decision system. It is not a new Qwen or Jev model release.
One Forward Pass, Five Allowed Answers
Tahir’s implementation starts with Qwen3-1.7B, a tokenizer and a multiple-choice prompt. Each answer receives a label, A through E, and the question and answer text appear together in the input.
The prompt uses Qwen’s chat template with enable_thinking=False, so the implementation can score the answer label immediately without first generating a reasoning sequence.
Instead of calling generate and waiting for a textual response, the code runs the model directly. It takes the logits at the final input position, where the model predicts the next token, and keeps only those corresponding to the five answer labels.
A logit is an unnormalized score. Applying softmax to those five selected scores produces a probability distribution over the allowed answers. The highest-scoring label becomes the prediction.
The central operation looks like this adapted fragment, assuming the model, tokenizer and tokenized prompt are already prepared:
options = ["A", "B", "C", "D", "E"]
option_token_ids = []
for option in options:
ids = tokenizer.encode(option, add_special_tokens=False)
assert len(ids) == 1, "Each answer label must be one token"
option_token_ids.append(ids[0])
temperature = 1.0
model.eval()
with torch.no_grad():
logits = model(**model_inputs).logits[0, -1]
scores = logits[option_token_ids].float()
probabilities = torch.softmax(scores / temperature, dim=-1)
prediction = options[probabilities.argmax().item()]
The single-token check is an implementation safeguard. Tahir’s example takes the first token ID returned for each label; developers adapting the method should verify that their labels really are single tokens for the tokenizer they use. Taking only the first token of a multi-token label would score something different from the intended answer.
Here, “single-pass” means the model processes the prompt and supplies all five answer scores without an autoregressive output-generation loop. Computation still depends on prompt length, and the guide does not establish a measured latency advantage on specified hardware.
For a fixed-choice decision, though, the efficiency opportunity is clear: the application need not generate a paragraph or JSON object just to recover one label.
A Valid Answer Is Not a Reliable Confidence Score
Restricting the output vocabulary solves a formatting problem. Correctness remains a separate problem.
The probabilities are normalized over the five selected answer tokens, not over the model’s entire vocabulary. They describe the relative preference among those allowed labels after every other token has been excluded. That distribution always sums to one, so even when none of the answers fits the input well, the classifier must allocate all its probability mass among them.
Tahir illustrates the problem with an ambiguous question: “Where would you most likely find a bat?” The choices include a cave, a baseball game, an attic, a zoo and a sporting goods store. His example assigns 99.78% probability to “Cave,” despite the ambiguity between the animal and sporting equipment.
This is an illustration, not a formal calibration test. The stronger evidence comes from grouping predictions by confidence and comparing each group’s mean probability with its observed accuracy.
For a well-calibrated classifier, predictions assigned roughly 80% confidence should be correct roughly 80% of the time across comparable examples. Calibration is a statistical property of a collection of predictions, not a guarantee about any individual answer.
A sharply peaked next-token distribution does not automatically meet that requirement. Training a model to predict text and restricting its possible outputs do not, by themselves, make the resulting scores empirical probabilities of correctness.
CommonsenseQA Shows a Modest Fine-Tuning Gain
Tahir reports testing the classifier on a random held-out sample of 1,221 CommonsenseQA questions. The tutorial’s evaluation results show:
| Configuration | Correct answers | Accuracy | Macro F1 |
|---|---|---|---|
| Before fine-tuning | 725 / 1,221 | 59.38% | 0.5844 |
| After fine-tuning | 762 / 1,221 | 62.41% | 0.6234 |
Fine-tuning adds 37 correct answers, an improvement of approximately 3.03 percentage points. These are the author’s reported results, not independently reproduced measurements.
The class-level results warrant a closer look. Before fine-tuning, recall for answer positions A and B is much higher than for D and E. After fine-tuning, the recall values are more evenly distributed.
That pattern gives developers a reason to investigate sensitivity to answer position, although it does not establish the cause. A useful follow-up experiment would shuffle the same answer choices and check whether predictions remain stable when the correct answer moves to a different label.
The evaluation demonstrates a particular prompt, answer-label scheme and multiple-choice task. It does not support the broader conclusion that “Qwen becomes a general decision model.” Developers adapting the workflow to another classification problem still need representative evaluation data from that problem.
Accuracy and macro F1 leave the calibration question open, too. A classifier can improve its label predictions while continuing to attach excessive confidence to its mistakes.
Temperature Scaling Adjusts Probabilities, Not Answers
Of the 1,221 predictions in the tutorial’s raw confidence table, 809 fall in the 90–100% confidence bin. Their average confidence is 98.55%, compared with 70.09% accuracy.
The mismatch extends beyond that group: the 80–90% bin averages 85.55% confidence but achieves only 47.11% accuracy.
Tahir addresses this overconfidence with temperature scaling: divide the selected logits by a positive scalar, T, before applying softmax. At , the distribution is unchanged. A temperature above one flattens it, reducing the dominance of the highest-scoring answer; a temperature below one makes it sharper.
Sources
- decision-model guidenishtahir.com
- Qwen3-1.7Bhuggingface.co
- build-your-own-jev repositorygithub.com
