Statistics for Evidence-led Decisions

Slide 1 of 4

The question first

A population is everyone Esmee wants her question to describe.

Here the decision concerns the contacts on Raket Parts marketing records.

Slide 2 of 4

Who can be reached

A frame is the list you can invite, not everyone who may hold an opinion.

The campaign snapshot records contact consent at that point.

Slide 3 of 4

One row, one contact

A sample is the smaller group you study; a respondent is someone who actually answers.

Contact references identify records without exposing personal details.

Slide 4 of 4

Quick check

First define who belongs in the question.

Slide 1 of 4

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 4

Commit before counting

Even a large email list can miss a particular group.

Before seeing the figures, commit to your answer.

Slide 3 of 4

Reveal the missing group

The coverage gap is the part of the frame you cannot reach through this method.

The January campaign includes unknown and opted-out consent states.

Slide 4 of 4

Quick check

Find who was excluded before computing a survey percentage.

Slide 1 of 4

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 4

Different groups, different reach

A segment is a meaningful group, such as universities or industrial buyers.

Compare how many contacts from each group were eligible to receive the January campaign.

Slide 3 of 4

Bias needs an assumption

Bias is a systematic push away from the answer for the full group.

If excluded contacts respond differently, the emailed result could lean high or low.

Slide 4 of 4

Quick check

An exclusion tells you who is absent, not their unrecorded opinion.

Slide 1 of 4

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 4

Which lots are counted

An inspection kind names the stage at which a part or lot was checked.

Receiving and final checks serve different questions.

Slide 3 of 4

Define the failure base

A failure share needs failures and checks with known pass or fail outcomes.

Do not merge these populations without saying what changed.

Slide 4 of 4

Quick check

Use each sample for the work it actually covers.

Slide 1 of 4

An uncertain estimate

An estimate is a measured share; it is not an exact answer for every possible lot.

Mira asks how uncertain the receiving-inspection failure share may be.

Slide 2 of 4

Read the range

An interval shows a range of values compatible with an estimate and its stated method.

This is an approximate 95% interval for independent inspections.

Slide 3 of 4

Not a guarantee

A 95% interval is a method that captures the fixed underlying rate in about 95 of 100 comparable repetitions.

It does not guarantee the group is representative, or put 95% probability on this particular fixed rate.

Slide 4 of 4

Quick check

An interval quantifies sampling uncertainty, not a missing population.

Slide 1 of 4

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 4

More observations, less noise

An interval width is the distance between its lower and upper ends.

Practice illustration: keep the observed receiving rate fixed while changing the sample size.

Slide 3 of 4

Count the right unit

The independent unit is what was sampled without treating linked observations as unrelated.

Repeated checks on the same lot may be related, so an ordinary interval can look too narrow.

Slide 4 of 4

Quick check

More independent observations generally sharpen the interval.

Slide 1 of 4

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 4

Commit before reveal

Two group intervals can overlap while the interval for their difference excludes zero.

Predict before seeing the June delivery data.

Slide 3 of 4

See the June figures

Compare on-time shares for delivery lines with known outcomes promised in June.

The difference uses CAT minus BTO, with independent-line approximation; lines within orders may be related.

Slide 4 of 4

Quick check

Overlap of group ranges is not a test for the difference.

Slide 1 of 4

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 4

The mean gets pulled

An average can be dragged upward by a few very large invoices.

Each sales_extract row describes one invoice.

Slide 3 of 4

Name a typical invoice

The median is the middle value after sorting invoices from small to large.

In this right-skewed sample the median is far below the mean.

Slide 4 of 4

Quick check

A mean interval and a median answer different questions.

Slide 1 of 4

Ask the decision question

A test asks whether a measured difference would be surprising if there were no underlying difference.

Jelmer asks about catalogue delivery lines in two separate promise-date periods.

Slide 2 of 4

Specify the groups

A comparison group is a clearly defined set of observations measured the same way.

Only on-time or late CAT lines enter this example; unresolved lines are excluded.

Slide 3 of 4

A possible chance pattern

The no-difference starting point assumes both periods have the same underlying on-time share.

The test asks how unusual the observed contrast is under that starting point.

Slide 4 of 4

Quick check

A test compares evidence with a specified starting point.

Slide 1 of 4

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 4

Commit before p-value

A small p-value means this result would be unusual under the no-difference model.

Predict whether a small p-value must mean a large effect.

Slide 3 of 4

See the real comparison

The effect size tells you how big the observed difference actually is.

Approximate independent-line test; line dependence and time changes limit interpretation.

Slide 4 of 4

Quick check

Statistical detectability and usefulness are two questions.

Slide 1 of 4

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 4

More comparisons, more chances

A multiple comparison problem appears when you search many groups for unusual differences.

Even if no true differences existed, some tests could look surprising by chance.

Slide 3 of 4

Count country groups

Raket Parts invoice records cover different country groups with differing average values.

These observed differences also reflect customer mix; they are not automatically country effects.

Slide 4 of 4

Quick check

Do not cherry-pick one surprising group after inspecting many.

Slide 1 of 4

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 4

A wide possible difference

A non-significant result does not prove no difference: its range may include meaningful changes.

Compare receiving checks with re-inspections as a descriptive example.

Slide 3 of 4

The interval crosses zero

A decision threshold is the smallest change that would matter to the decision.

Practice rule: use three points, not a real company policy.

Slide 4 of 4

Quick check

A wide interval can include zero and a useful effect.

Slide 1 of 5

Mira's note

Mira's note: BTO lines were no more often on time in September, "so nothing changed".

Jelmer asks what size of change these two months could have seen.

Slide 2 of 5

Commit first

What is the smallest change this comparison could reliably detect?

Reliably means found 80% of the time. Commit before the reveal.

Slide 3 of 5

What the range says

September was 8.62 points higher, but the range runs from -3.07 to +20.31.

Simple model with lines treated as independent, p about 0.15. The lines share 96 and 89 orders, so the real range is wider.

Slide 4 of 5

The reveal

These two months could reliably see only a change of about 16.7 points.

A 5-point change, a practice rule for what is worth acting on, was out of reach.

Slide 5 of 5

What does the range support?

A null from a small study leaves both stories open.

Slide 1 of 6

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 6

Power is planning

Power is the chance a study finds a change that is really there.

Set before data. The formulas count units chosen at random: if whole orders are assigned, count orders.

Slide 3 of 6

Smaller change, more data

BTO on-time base: 36.83%. Smaller rises need far more lines.

435 on time of 1,181 BTO lines, January to October 2025.

Slide 4 of 6

Spread matters too

CAT days late vary with a spread of 4.85 days.

Spread is the standard deviation: how far lines usually sit from the average.

Slide 5 of 6

Power by numbers

A real 5-point rise, from 36.83%: how often is it found?

Simulation of 1,000 studies (seed 19): a planning model, not a Raket Parts trial.

Slide 6 of 6

Halve the change

Halve the change you want to see, and the data needed grows fourfold.

Slide 1 of 5

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 5

Start from the decision

Judge a null by the change the decision needs.

August to September: could see 16.7 points; the practice rule needs 5.

Slide 3 of 5

Observed power adds nothing

Observed power uses the change you happened to see as if it were true.

It is the p-value restated: any non-significant result gives under about 50%.

Slide 4 of 5

One month is always weak

No single month of 2025 comes near 1,495 lines.

BTO lines with a known outcome per promised month: 34 to 170.

Slide 5 of 5

What does the null tell Mira?

A null says nothing about changes the study could not see.

Slide 1 of 6

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 6

Plan before asking

Esmee plans two survey waves. No answers exist yet.

Planning assumption: 50% satisfied, the most cautious value when nothing is known.

Slide 3 of 6

The frame is the limit

Even if every eligible contact answered, a 5-point change is out of reach.

Frame: the October 2025 catalogue mailing (X-CMP-2025-07). The response rate is unknown.

Slide 4 of 6

Choose the trade-off

When data is fixed, choose the trade-off openly.

Slide 5 of 6

A wider spread

April to June CAT lines had a spread of 7.36 days, against 4.85.

Variance 54.16 against 23.57: 2.3 times. Data needed grows with the variance.

Slide 6 of 6

Which move holds up?

Name the trade-off; never pretend the frame is bigger.

Slide 1 of 5

Esmee's headline

Esmee's September report: the webshop promotion lifted webshop orders by 53%.

Promotion X-CMP-2025-05 ran from 1 to 30th September 2025.

Slide 2 of 5

Commit first

Before anything else: does this before-and-after figure answer Esmee's question?

Commit before you see the other channels.

Slide 3 of 5

The other channels

Email orders, which the promotion did not target, rose just as much.

Webshop share of all orders moved only from 26.5% to 27.9%.

Slide 4 of 5

Season and timing

August was the lowest month of 2025, and October still had 226 webshop orders.

A causal question needs a design that rules these out (the Experimentation Path).

Slide 5 of 5

What kind of question?

"Did it work?" is a causal question.

Slide 1 of 5

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 5

Describe, compare, generalise

Three kinds of question use the data you have.

Each needs something different.

Slide 3 of 5

Cause, predict, explain

Three kinds need their own design and Path.

Here you name them and route them.

Slide 4 of 5

Counts need a fair bottom

CAT has more late lines, yet BTO is late more often.

Compare rates: late lines over all resolved lines of that kind.

Slide 5 of 5

Rates or counts?

Groups of different size need rates, not counts.

Slide 1 of 5

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 5

The same people twice

A paired design measures the same contacts twice.

The February and October 2025 catalogue mailings (X-CMP-2025-01 and X-CMP-2025-07) hold the same 1,179 contacts.

Slide 3 of 5

Lines inside orders

Lines sit inside orders, and orders inside customers: they are clustered.

The order or the customer is the unit; a grouped model is a job for regression modelling.

Slide 4 of 5

Months in a row

13 months of orders follow each other: they are time-ordered.

A test that treats months as separate draws fails. Name a time-series family and route to regression modelling.

Slide 5 of 5

What is the unit?

Count the unit that makes rows independent.

Slide 1 of 5

Warm-up

Two quick questions from earlier.

Answer both to continue.

Slide 2 of 5

Sends are actions

Esmee's summary counts what the team did, step by step.

Each step is activity. None shows what the promotion changed.

Slide 3 of 5

Attribution is a rule

Attribution shares enquiries out by a rule; it does not show cause.

Rule: the last campaign touch within 30 days.

Slide 4 of 5

What "did it work?" needs

A count answers how many; "did it work?" needs a comparison group.

Slide 5 of 5

What can 706 answer?

Match the count to the question it can answer.