Experimentation and Causal Impact Evaluation
Learn to plan a fair test before any data arrives and to judge whether a change really caused a result. No coding: you work on the data of Raket Parts, a fictional maker of rocket parts.
You will be able to
- write an experiment charter with one main outcome and a decision rule
- choose who gets which version and for how long
- set a stopping rule before the test starts
- spot a test that is no longer sound
Plan the test before the data
Draft an ethical experiment charter with a falsifiable hypothesis, one main outcome, safety checks and a decision rule.
Who gets which version, and for how long
Choose an appropriate assignment unit, sketch a sample requirement and register a whole-week stopping rule.
Is the test still sound?
Spot suspicious allocation, incomplete event recording and early novelty spikes, and record a defensible integrity action.
Read the result as planned
Read a finished test exactly as registered: the primary with its interval, guardrails against thresholds, subgroups as exploratory and no pooling across definitions.
Judge a test that broke its plan
Rebuild what happened in another team’s test, sort each breach as fatal, limiting or cosmetic, keep only the conclusion the valid data support and escalate with evidence.
Causal questions and assumptions
State exactly what effect you want and what must hold to estimate it.
Difference-in-differences for staged rollouts
Estimate an effect when a change reaches sites at different times.
One treated region: time series and synthetic control
Estimate an effect when only one region changed.
Thresholds and weighting
Use cut-offs and weighting when no test was run.
Attribution claims and incrementality tests
Check how much of a result a campaign or change really added.
Weighing conflicting evidence
Rank studies by the strength of their design.
Mid-Path project: did it work?
Judge one real change from plan to conclusion.
Final assessment
A causal evaluation on an unseen case.
Experimentation and Causal Impact / Module 1
Plan the test before the data
Esmee wants to know whether an email promotion added enquiries. The campaign tables credit enquiries, but they do not show what happened without the promotion. Plan a test that can answer the real question.
Sections
Experimentation and Causal Impact / Module 1 / Free practice
Try it on your own question
Run it in the practice editor
The practice editors have the Raket Parts tables loaded. Type a query or a formula, run it and change it as often as you like. Nothing is marked.
Experimentation and Causal Impact / Module 2
Who gets which version, and for how long
An email test needs fair assignment, enough participants and a fixed end date. Customer colleagues may share an offer, and the company has uneven weekday activity. Keep the versions separate and resist stopping at the first apparent win.
This module builds on module 1. If you are new to this, start with module 1.
Sections
Experimentation and Causal Impact / Module 2 / Free practice
Try it on your own question
Run it in the practice editor
The practice editors have the Raket Parts tables loaded. Type a query or a formula, run it and change it as often as you like. Nothing is marked.
Experimentation and Causal Impact / Module 3
Is the test still sound?
Before interpreting an experiment, check who was assigned and whether every event arrived. Raket Parts audience and interface records let us practice some checks, but they contain no randomised treatment and control groups. Make an honest continue, fix or stop decision from what the records can show.
This module builds on module 2. If you are new to this, start with module 2.
Sections
Experimentation and Causal Impact / Module 3 / Free practice
Try it on your own question
Run it in the practice editor
The practice editors have the Raket Parts tables loaded. Type a query or a formula, run it and change it as often as you like. Nothing is marked.
Experimentation and Causal Impact / Module 4
Read the result as planned
Esmee reads two real Raket Parts email sends against a practice experiment plan. These sends were not randomised groups, so their differences do not demonstrate cause.
This module builds on module 3. If you are new to this, start with module 3.
Sections
Experimentation and Causal Impact / Module 4 / Free practice
Try it on your own question
Run it in the practice editor
The practice editors have the Raket Parts tables loaded. Type a query or a formula, run it and change it as often as you like. Nothing is marked.
Experimentation and Causal Impact / Module 5
Judge a test that broke its plan
A practice email subject-line test has already been reported to leaders, but its plan was broken. Jelmer reads the record against the plan before Esmee repeats the number, then checks real Raket Parts consent records for emails sent after an opt-out.
This module builds on module 4. If you are new to this, start with module 4.
Sections
Experimentation and Causal Impact / Module 5 / Free practice
Try it on your own question
Run it in the practice editor
The practice editors have the Raket Parts tables loaded. Type a query or a formula, run it and change it as often as you like. Nothing is marked.
Experimentation and Causal Impact / Module 1 / 1.1
1.1 Credit is not cause
Section 1.1 complete
Next: 1.2 A hypothesis that can fail
Experimentation and Causal Impact / Module 1 / 1.2
1.2 A hypothesis that can fail
Section 1.2 complete
Next: 1.3 One primary, a few guardrails
Experimentation and Causal Impact / Module 1 / 1.3
1.3 One primary, a few guardrails
Section 1.3 complete
Next: 1.4 Decide the rule first
Experimentation and Causal Impact / Module 1 / 1.4
1.4 Decide the rule first
Section 1.4 complete
Next: Module 1 check
Continue to the module check Recap this section Back to module 1
Experimentation and Causal Impact / Module 1 / Check
Module 1 check
Module 1 complete
Next: Module 2, Who gets which version, and for how long
Continue to module 2 Try it on your own question Recap module 1 Back to module 1
Experimentation and Causal Impact / Module 2 / 2.1
2.1 The unit of randomisation
Experimentation and Causal Impact / Module 2 / 2.3
2.3 How big a test
Section 2.3 complete
Next: 2.4 Do not peek and stop
Experimentation and Causal Impact / Module 2 / 2.4
2.4 Do not peek and stop
Section 2.4 complete
Next: Module 2 check
Continue to the module check Recap this section Back to module 2
Experimentation and Causal Impact / Module 2 / Check
Module 2 check
Module 2 complete
Next: Module 3, Is the test still sound?
Continue to module 3 Try it on your own question Recap module 2 Back to module 2
Experimentation and Causal Impact / Module 3 / 3.1
3.1 Are groups the right size?
Section 3.1 complete
Next: 3.2 Is everything recorded?
Experimentation and Causal Impact / Module 3 / 3.2
3.2 Is everything recorded?
Experimentation and Causal Impact / Module 3 / 3.3
3.3 Early spikes
Section 3.3 complete
Next: 3.4 Continue, fix or stop
Experimentation and Causal Impact / Module 3 / 3.4
3.4 Continue, fix or stop
Section 3.4 complete
Next: Module 3 check
Continue to the module check Recap this section Back to module 3
Experimentation and Causal Impact / Module 3 / Check
Module 3 check
Module 3 complete
Next: Module 4, Read the result as planned
Continue to module 4 Try it on your own question Recap module 3 Back to module 3
Experimentation and Causal Impact / Module 4 / 4.1
4.1 Where did it work?
Section 4.1 complete
Next: 4.2 The primary, as registered
Experimentation and Causal Impact / Module 4 / 4.2
4.2 The primary, as registered
Section 4.2 complete
Next: 4.3 Guardrails can say no
Experimentation and Causal Impact / Module 4 / 4.3
4.3 Guardrails can say no
Section 4.3 complete
Next: 4.4 Subgroups and changed definitions
Experimentation and Causal Impact / Module 4 / 4.4
4.4 Subgroups and changed definitions
Section 4.4 complete
Next: Module 4 check
Continue to the module check Recap this section Back to module 4
Experimentation and Causal Impact / Module 4 / Check
Module 4 check
Module 4 complete
Next: Module 5, Judge a test that broke its plan
Continue to module 5 Try it on your own question Recap module 4 Back to module 4
Experimentation and Causal Impact / Module 5 / 5.1
5.1 Defend, qualify or withdraw?
Section 5.1 complete
Next: 5.2 Rebuild the timeline
Experimentation and Causal Impact / Module 5 / 5.2
5.2 Rebuild the timeline
Section 5.2 complete
Next: 5.3 Fatal, limiting or cosmetic
Experimentation and Causal Impact / Module 5 / 5.3
5.3 Fatal, limiting or cosmetic
Section 5.3 complete
Next: 5.4 Report it and raise it
Experimentation and Causal Impact / Module 5 / 5.4
5.4 Report it and raise it
Section 5.4 complete
Next: Module 5 check
Continue to the module check Recap this section Back to module 5
Experimentation and Causal Impact / Module 5 / Check
Module 5 check
Module 5 complete
Next: Module 6, coming later
Back to the Path Try it on your own question Recap module 5 Back to module 5
Experimentation and Causal Impact / Module 1 / Recap
Recap card
1.1 Credit is not cause
Credit is not cause
- Attribution credits a touch using a stated rule.
- A causal effect needs a fair alternative.
Common mistake
Calling credit an effect. The last touch could have received credit for an enquiry that would have happened anyway.
Experimentation and Causal Impact / Module 1 / Recap
Recap card
1.2 A hypothesis that can fail
A hypothesis that can fail
- Start with a decision to change or keep a message.
- Name the eligible group, primary outcome, direction and threshold.
Common mistake
Vague hopes. A goal without a group and measure cannot be disproved.
Experimentation and Causal Impact / Module 1 / Recap
Recap card
1.3 One primary, a few guardrails
One primary, a few guardrails
- Use a single pre-chosen main outcome.
- Add safety checks for consent and opt-outs.
Common mistake
Many ways to win. More measures make selective success stories easier.
Experimentation and Causal Impact / Module 1 / Recap
Recap card
1.4 Decide the rule first
Decide the rule first
- Write ship, change or stop criteria before launch.
- Consent exclusions apply before randomisation.
Common mistake
Moving the goalposts. Changing the threshold after seeing results is not a pre-registered test.
Experimentation and Causal Impact / Module 1 / Recap
Recap card
Module 1: Plan the test before the data
Credit is not cause
- Attribution credits a touch using a stated rule.
- A causal effect needs a fair alternative.
Common mistake
Calling credit an effect. The last touch could have received credit for an enquiry that would have happened anyway.
A hypothesis that can fail
- Start with a decision to change or keep a message.
- Name the eligible group, primary outcome, direction and threshold.
Common mistake
Vague hopes. A goal without a group and measure cannot be disproved.
One primary, a few guardrails
- Use a single pre-chosen main outcome.
- Add safety checks for consent and opt-outs.
Common mistake
Many ways to win. More measures make selective success stories easier.
Decide the rule first
- Write ship, change or stop criteria before launch.
- Consent exclusions apply before randomisation.
Common mistake
Moving the goalposts. Changing the threshold after seeing results is not a pre-registered test.
Experimentation and Causal Impact / Module 2 / Recap
Recap card
2.1 The unit of randomisation
The unit of randomisation
- Assign the unit that can share or influence the treatment.
- Customer accounts protect a shared-offer test from spillover.
Common mistake
Splitting colleagues. Different email versions within the same account undermine the comparison.
Experimentation and Causal Impact / Module 2 / Recap
Recap card
2.2 Cover whole weeks
Cover whole weeks
- Weekday activity is not uniform.
- Fix a full-week schedule before seeing results.
Common mistake
Stopping when ahead. A Wednesday win might be a weekday effect or chance.
Experimentation and Causal Impact / Module 2 / Recap
Recap card
2.3 How big a test
How big a test
- A baseline rate and smallest useful effect drive rough sizing.
- Related contacts in one customer need an adjusted plan.
Common mistake
Treating a proxy as truth. Credited enquiries are not a measured untreated enquiry rate.
Experimentation and Causal Impact / Module 2 / Recap
Recap card
2.4 Do not peek and stop
Do not peek and stop
- Fix the end date or valid sequential method before launch.
- Safety checks can run while outcome claims wait.
Common mistake
Stopping at the first win. Repeated looks create false winners.
Experimentation and Causal Impact / Module 2 / Recap
Recap card
Module 2: Who gets which version, and for how long
The unit of randomisation
- Assign the unit that can share or influence the treatment.
- Customer accounts protect a shared-offer test from spillover.
Common mistake
Splitting colleagues. Different email versions within the same account undermine the comparison.
Cover whole weeks
- Weekday activity is not uniform.
- Fix a full-week schedule before seeing results.
Common mistake
Stopping when ahead. A Wednesday win might be a weekday effect or chance.
How big a test
- A baseline rate and smallest useful effect drive rough sizing.
- Related contacts in one customer need an adjusted plan.
Common mistake
Treating a proxy as truth. Credited enquiries are not a measured untreated enquiry rate.
Do not peek and stop
- Fix the end date or valid sequential method before launch.
- Safety checks can run while outcome claims wait.
Common mistake
Stopping at the first win. Repeated looks create false winners.
Experimentation and Causal Impact / Module 3 / Recap
Recap card
3.1 Are groups the right size?
Are groups the right size?
- Compare the planned split at the correct assignment unit.
- Matching audience and send totals is useful but not arm-level balance.
Common mistake
Dismissing a tiny difference. A small percentage gap can matter with many assignments.
Experimentation and Causal Impact / Module 3 / Recap
Recap card
3.2 Is everything recorded?
Is everything recorded?
- Track batch acceptance and delivery timing.
- Compare event capture by assigned arm and day when testing live.
Common mistake
Trusting an accepted batch. Acceptance alone does not prove each arm has complete events.
Experimentation and Causal Impact / Module 3 / Recap
Recap card
3.3 Early spikes
Early spikes
- Temporary attention can make an effect look large at first.
- Wait for the registered duration before outcome claims.
Common mistake
Choosing the peak. The most flattering day does not define the effect.
Experimentation and Causal Impact / Module 3 / Recap
Recap card
3.4 Continue, fix or stop
Continue, fix or stop
- Record each check, evidence and decision.
- Restart or stop when biased collection cannot be repaired.
Common mistake
Declaring a winner anyway. Missing arm data prevents an experiment sign-off.
Experimentation and Causal Impact / Module 3 / Recap
Recap card
Module 3: Is the test still sound?
Are groups the right size?
- Compare the planned split at the correct assignment unit.
- Matching audience and send totals is useful but not arm-level balance.
Common mistake
Dismissing a tiny difference. A small percentage gap can matter with many assignments.
Is everything recorded?
- Track batch acceptance and delivery timing.
- Compare event capture by assigned arm and day when testing live.
Common mistake
Trusting an accepted batch. Acceptance alone does not prove each arm has complete events.
Early spikes
- Temporary attention can make an effect look large at first.
- Wait for the registered duration before outcome claims.
Common mistake
Choosing the peak. The most flattering day does not define the effect.
Continue, fix or stop
- Record each check, evidence and decision.
- Restart or stop when biased collection cannot be repaired.
Common mistake
Declaring a winner anyway. Missing arm data prevents an experiment sign-off.
Experimentation and Causal Impact / Module 4 / Recap
Recap card
4.1 Where did it work?
Where did it work?
- Country patterns are leads: 17 countries together give p = 0.48.
- With 17 looks and no real difference, at least one looks significant 58% of the time.
Common mistake
Promoting the top subgroup. One of 17 countries has to come top.
Experimentation and Causal Impact / Module 4 / Recap
Recap card
4.2 The primary, as registered
The primary, as registered
- Read the registered primary with its interval.
- An interval crossing zero supports neither worse nor better.
Common mistake
Changing the measure after the results. Switching to qualified only is an unregistered adjustment.
Experimentation and Causal Impact / Module 4 / Recap
Recap card
4.3 Guardrails can say no
Guardrails can say no
- A practice guardrail line can block a win.
- Overlapping windows stop you tying an opt-out to one send.
Common mistake
Raising the line after the results. The harm line is fixed before launch.
Experimentation and Causal Impact / Module 4 / Recap
Recap card
4.4 Subgroups and changed definitions
Subgroups and changed definitions
- Keep one definition for every comparison.
- After Holm adjustment no country survives; at most a lead.
Common mistake
Pooling two definitions. All credited against qualified only makes a false 4.19 drop.
Experimentation and Causal Impact / Module 4 / Recap
Recap card
Module 4: Read the result as planned
Where did it work?
- Country patterns are leads: 17 countries together give p = 0.48.
- With 17 looks and no real difference, at least one looks significant 58% of the time.
Common mistake
Promoting the top subgroup. One of 17 countries has to come top.
The primary, as registered
- Read the registered primary with its interval.
- An interval crossing zero supports neither worse nor better.
Common mistake
Changing the measure after the results. Switching to qualified only is an unregistered adjustment.
Guardrails can say no
- A practice guardrail line can block a win.
- Overlapping windows stop you tying an opt-out to one send.
Common mistake
Raising the line after the results. The harm line is fixed before launch.
Subgroups and changed definitions
- Keep one definition for every comparison.
- After Holm adjustment no country survives; at most a lead.
Common mistake
Pooling two definitions. All credited against qualified only makes a false 4.19 drop.
Experimentation and Causal Impact / Module 5 / Recap
Recap card
5.1 Defend, qualify or withdraw?
Defend, qualify or withdraw?
- Read the record against the registered plan before repeating a number.
- Being circulated adds no evidence.
Common mistake
Defending the headline. A circulated figure is not evidence of a valid causal effect.
Experimentation and Causal Impact / Module 5 / Recap
Recap card
5.2 Rebuild the timeline
Rebuild the timeline
- Put every breach on a timeline with its day and owning team.
- No snapshot date means "not recorded", not a guess.
Common mistake
Guessing a date. Filling in when the snapshot was taken from the send or consent date.
Experimentation and Causal Impact / Module 5 / Recap
Recap card
5.3 Fatal, limiting or cosmetic
Fatal, limiting or cosmetic
- Classify each breach: fatal, limiting or cosmetic.
- Days 1 and 2 survive as description: +0.58 points, -0.93 to +2.08.
Common mistake
Throwing it all away. A compromised test can still describe its clean days.
Experimentation and Causal Impact / Module 5 / Recap
Recap card
5.4 Report it and raise it
Report it and raise it
- 4 emails went after an earlier opt-out; 4 same-day cases are unresolved.
- Escalate counts, campaigns and dates to the privacy owner, no contact references.
Common mistake
Naming people. Putting contact references in a note that goes round.
Experimentation and Causal Impact / Module 5 / Recap
Recap card
Module 5: Judge a test that broke its plan
Defend, qualify or withdraw?
- Read the record against the registered plan before repeating a number.
- Being circulated adds no evidence.
Common mistake
Defending the headline. A circulated figure is not evidence of a valid causal effect.
Rebuild the timeline
- Put every breach on a timeline with its day and owning team.
- No snapshot date means "not recorded", not a guess.
Common mistake
Guessing a date. Filling in when the snapshot was taken from the send or consent date.
Fatal, limiting or cosmetic
- Classify each breach: fatal, limiting or cosmetic.
- Days 1 and 2 survive as description: +0.58 points, -0.93 to +2.08.
Common mistake
Throwing it all away. A compromised test can still describe its clean days.
Report it and raise it
- 4 emails went after an earlier opt-out; 4 same-day cases are unresolved.
- Escalate counts, campaigns and dates to the privacy owner, no contact references.
Common mistake
Naming people. Putting contact references in a note that goes round.