Confidence and probabilities

What probabilities and confidence mean in a Krun answer. Confidence is the margin between the two best options, not the probability of being correct.

Every answer has two related signals: probabilities, a distribution over your options, and confidence, the margin between the two best options.

Probabilities

probabilities maps each of your option ids to a probability:

"probabilities": {
  "shipping": 0.005228,
  "returns": 0.98943,
  "billing": 0.005342
}

• The keys are exactly the option ids you sent, in request order. Krun never renames, drops or adds options.
• The values form a distribution over the options you provided, and add up to about 1. Rounding can make the sum differ slightly.
• They are relative to your option set. Adding, removing or rewording options changes every probability.
• probabilities is always present, also when the model abstains.

Krun One’s probabilities are calibrated: on the benchmarks Krun One was evaluated on, options predicted at 0.9 are correct about 90% of the time. Calibration is measured on that data. It is not a guarantee for every domain, language or option set. Check it on your own traffic before you rely on specific values. See Benchmarks.

Confidence

confidence is the difference between the highest and the second-highest probability:

confidence = top1_probability - top2_probability

For the answer above:

returns  = 0.98943    (top 1)
billing  = 0.005342   (top 2)

confidence = 0.98943 - 0.005342 ≈ 0.984

confidence ranges from 0 to 1:

• Close to 1: one option clearly wins.
• Close to 0: the two best options are nearly tied.

confidence is not the probability that the answer is correct. A confidence of 0.6 does not mean the model is right 60% of the time. It means the best option is 0.6 ahead of the runner-up. If you need the model’s estimate for the chosen option, read probabilities[choice].

Why a margin

Krun uses the margin because, in Krun One’s evaluation, it separated answerable requests from requests where no option applies as well as the alternatives tested, with the lowest error rate on accepted answers. The abstention threshold is applied to this value: when confidence is below the threshold, the model abstains.

A margin also catches cases a single probability misses. With three options at 0.48, 0.47 and 0.05, the top probability looks moderate, but the margin of 0.01 shows that the model can’t tell the first two apart.

Using the signals

Python

answer = result.answers["department"]

# Probability of the chosen option
if answer.choice is not None:
    p = answer.probabilities[answer.choice]

# Runner-up, for example to show a second suggestion
ranked = sorted(answer.probabilities.items(), key=lambda kv: kv[1], reverse=True)
best, runner_up = ranked[0], ranked[1]

Recommendations:

• Branch on choice and abstain first. They already apply the model’s threshold.
• If you need a stricter policy, for example before an irreversible action, add your own threshold on confidence, tuned on your traffic.
• Log confidence and send feedback so you can see how it relates to errors in your data.