What a holdout measures
Some people would have bought, upgraded or finished the ID check without the email. A holdout group shows how many.
Opens and clicks cannot show this. A person who would have bought anyway also opens and clicks.
The gap between the two groups' conversion rates is the lift.
Run one
- Pick the outcome. Use the event the email is for, such as order.placed, and write it down before the test starts.
- Split at random. Assign each person once, when they first enter the journey, and keep them in that group.
- Send nothing to the holdout group. They get no version of the email, so the gap is the whole effect.
- Compare the rates. Count the outcome in each group over the same window, then divide by the number of people in that group.
Hold people back in Maqetin
A journey will hold back a random share of the people who enter it and report the lift. This is planned.
Today you make the groups yourself. Set a trait with identify() when a person signs up, and start the journey for a saved segment that leaves out the holdout group.
The outcome event lands on each profile, next to the emails the person got.
Check a result
Holdout result
Count the conversions over the same window for both groups.
Enter the four numbers to see the result.
z is the pooled two-proportion statistic. Compare its size with 1.96, the normal table value for a two-sided test at the five per cent level.
The test assumes reasonably large groups. For small groups an exact test is the better choice.
A small holdout makes a gap harder to read, because both group sizes are in the denominator of z.
Write the basis beside every rate: people who entered the journey, or people who were sent the email.
Sources
- NIST/SEMATECH e-Handbook of Statistical Methods, section 7.3.3, checked 27 Oct 2026