Fixing Choice-Order Sensitivity in Decision Models: Strands Gemma E2B
“Which device directly converts chemical energy to electrical energy?”
The options were a solar-powered calculator, a wood-burning stove, a battery-powered flashlight, and a wind-powered sailboat. In that order, the checkpoint chose the battery-powered flashlight—the correct answer—with an API-returned probability of 0.5589.
Then I reversed the same four options. The checkpoint chose the solar-powered calculator, now at 0.5615.
| Semantic option | Original order | Reversed order |
|---|---|---|
| solar-powered calculator | 0.1224 | 0.5615 |
| wood-burning stove | 0.1229 | 0.0967 |
| battery-powered flashlight — correct | 0.5589 | 0.2873 |
| wind-powered sailboat | 0.1958 | 0.0545 |
Both columns map scores back to the same option meanings, regardless of where each option appeared. The values were rounded to four decimals by the original API and renormalized by the evaluation harness; they are not raw logits or calibrated confidence scores. Even with that boundary, the selected meaning changed while the question and available answers remained the same.
This was not an isolated anecdote. In a controlled original-versus-reversed comparison, the winning semantic option differed on 851 of 3,900 questions: 21.82%. The treatment, semantic alignment, task-level counts, and limitations are documented in the immutable order-sensitivity evidence. That led to a practical research question: Can we retrofit a released checkpoint so display order stops driving decisions, without retraining the Gemma torso or inherited adapter and without giving up useful quality?
The result is a two-mode scorer: one mode isolates options, while the other restores controlled comparison without adding display position. Under the supported execution path, both produced zero observed winner changes across all 3,900 original-versus-reversed cases.
Get the release: source, runtime, tests, and evidence on GitHub (v0.1.0) · model components on Hugging Face (v0.1.0)