Probabilities and confidence
Keep the model’s distribution separate from your application’s decision policy.
Read the whole distribution
Choice and score return probabilities for every option or level. Values must be finite and fall in [0, 1]. The draft allows a sum within 0.0001 of one to accommodate rounding.
A close second option can matter even when the selected answer is unchanged. Store the distribution when evaluating a model or adjusting an application policy.
Confidence is a summary
Confidence summarizes how concentrated a distribution is. Draft 0.1 preserves the field but leaves its formula provider-defined. Equal values from different implementations may have different meanings.
Measure calibration
To assess a reported probability, compare predictions with outcomes on held-out examples. Group similar probabilities and inspect how often the chosen answer is correct. Report the dataset, sample size, and evaluation procedure.
Recheck after changing the model, prompt, criteria, or input distribution. Protocol conformance tests the shape and semantics of an exchange; predictive quality requires a separate evaluation.