Uncertainty and models
Calibration / reliability plot
Compares predicted probabilities with observed outcome rates.
- Plot
- Model evaluation
- Probability agreement
What it answers
Do predicted probabilities match observed event rates?
- Shape
- Observed-rate points versus probabilities
- What it shows
- Overconfidence, underconfidence and calibration gaps.
Example
The question
When the model says 30%, do about 30% of customers really churn?
Predicted churn probability against observed churn rate (calibration plot)
Illustrative data. It belongs to no company.
When to use it
Use it when
Evaluating probability forecasts.
Avoid it when
Using arbitrary ranking scores or tiny bins without uncertainty.
How to read it
Compare observed rates with the diagonal at each predicted-probability level.
Note
Use probabilities rather than arbitrary ranking scores.
The data you need
- Layout
- Long table
- Fields
- CaseID: text; Actual: 0/1; Score: number
- One row is
- One case with its true binary outcome and prediction score. What is the grain?
- Rules
- Define the positive class and score direction. Use held-out predictions for evaluation. Calibration needs probabilities in [0,1].
- Optional
- Weight. ROC/PR alternatives: precomputed Threshold,TP,FP,TN,FN with complete threshold coverage.
- Preparation
- Use predicted probabilities and outcomes; choose documented bins or a calibration smoother and retain group sizes.
- Units
- Predicted probabilities and observed positive fractions both use[0,1]; observed fractions divide bycases perbin/group.
Worked example
A bin of 100 cases hasmean prediction 0.8 and 60 actual positives. Observedrate 0.6 lies 0.2 below the diagonal; predictions are too high in that bin.
| CaseID | Actual | Score |
|---|---|---|
| A | 1 | 0.90 |
| B | 0 | 0.20 |
| C | 1 | 0.65 |
| D | 0 | 0.55 |