How to Run a Holdout Test on a Klaviyo Flow

TL;DR

Klaviyo's attributed revenue tells you which orders a flow touched. A holdout test tells you how many of them the flow caused. The difference can be large: an abandoned-checkout flow is sent to people who were already at the checkout, and some of them were coming back anyway. Klaviyo has a built-in holdout, but it needs 400,000 profiles and it holds people out of your marketing as a whole, not one flow [2]. For everyone else, the test lives inside the flow: a multi-branch split with a random-sample condition [3], a property update on each path [4], and two segments that count orders by group. Then a single formula, the same one Klaviyo uses on its own holdout card [2], turns the two rates into lift.

Key Takeaways

• Klaviyo credits a flow with any order inside its attribution window; for new accounts that's 5 days after an email open or click [1]. A holdout compares people who got the flow with people who didn't.
• Global holdout groups need at least 400,000 profiles, run one at a time and apply to campaigns and flows together [2].
• In a single flow, a split's random-sample condition applies to the profiles evaluated for that path, not to everyone entering the split [3]. Keep one random path and let "Everyone else" be the other group.
• Tag both groups with a Property update action [4], so you can build a segment for each and count orders by group.
• Lift is (treatment rate − holdout rate) ÷ holdout rate [2]. A flow lifting conversion from 8% to 10% has a 25% lift, and 8 of every 10 of its buyers would have bought anyway.
• In the illustrative numbers below, a 20% holdout reaches a readable result in about half the time a 10% holdout needs, for about four extra lost orders.

Why Klaviyo's attributed revenue can't answer this

Klaviyo counts a conversion for a message when the order lands inside the attribution window after an interaction. For new accounts the defaults are 5 days for email opens and clicks, 5 days for SMS clicks and 12 hours for SMS deliveries, and an open counts on its own [1]. That's the right number for comparing one flow with another, which is how Klaviyo's attributed revenue should be read.

It can't separate the buyer the email brought back from the buyer who opened it on the way to a purchase already planned. Both orders land in the window, so both count.

A holdout sidesteps attribution entirely. Klaviyo's own holdout reporting says its results "include conversions whether or not they are attributed to a Klaviyo message" [2]. You compare two groups of people, one that got the flow and one that didn't, and the difference in how often they bought is what the flow did.

Which holdout does your account qualify for?

Klaviyo has two routes, and your profile count picks one for you.

Global holdout groups. They live under Analytics > Experiments, on the Global Experiments tab [2]. The rules, from Klaviyo's page:

A global holdout answers "what is my whole email and SMS program worth", which is a different question from "is this one flow worth it". In principle you could aim it at one flow by overriding every other flow (in each trigger's Settings) and every campaign (at the Recipients step) [2], so the holdout misses only the flow under test. That's a lot of overrides to keep straight for three months.

A random split inside the flow. No profile minimum, one flow at a time, and you choose the share. This is the route the rest of this guide builds.

Build the split at the top of the flow

Klaviyo's multi-branch split (its help page was updated on October 7, 2026) can branch on a random sample, which Klaviyo defines as a percentage of the profiles evaluated for that path [3]. It's being released in phases, so check that your flow builder shows it. If yours still has the older conditional split, look for the same random-sample condition there before you plan around it.

The order of the steps below uses one documented behaviour on purpose: when you add a split in the middle of an existing flow, "the actions that were below that point move onto the split's first path (Path #1)" [3]. So Path #1 becomes the group that keeps getting your messages.

1. Open the flow and drag Split from the Logic section of the sidebar to the spot right under the trigger [3]. Your existing emails move onto Path #1.
2. On Path #1, select Add path criteria and add a random-sample condition at 80% [3]. Rename it "Treatment".
3. At the top of Path #1, drag in Property update from the Actions sidebar, click Add step, choose Create new property, and set a text property such as abandoned_test to treatment [4].
4. On the "Everyone else" path, which Klaviyo always keeps last and won't let you delete [3], add one Property update that sets abandoned_test to holdout. Nothing else goes on this path. A profile that enters the flow again is split again, and the property update overwrites its earlier tag.
5. Run flow preview on a few profiles to see which path each would take [3], then set both property updates to Live and save [4].

The percentage trap is the one to remember. With one random path, 80% means 80% of everyone entering. Add a second random path below it and its percentage applies only to the profiles left over: Klaviyo's example sends 50 of 100 profiles down a 50% first path and then evaluates the remaining 50 for the next one [3]. A 50% second path there reaches 25 profiles, not 50.

Once the split is live, its configuration panel shows how many profiles went down each path over the last 30 days [3]. That's your first sanity check: the two counts should sit near 80 and 20.

What should count as a conversion?

An order, counted once per profile, since the day the test went live.

Use the same filter builder the split uses [3] to build two segments per group. One holds the property alone, abandoned_test equals holdout, and gives you the group's size. The other adds Placed Order at least once in a time frame that starts on the day the test went live, and gives you its buyers. Do the same for treatment, then divide buyers by size for each. Counting profiles also protects you from one known mess. If Analytics > Metrics shows two metrics named Placed Order, two senders are writing orders, which is why Klaviyo shows double conversions. Pick one of them and use it for both groups.

Then apply Klaviyo's formula: lift = (treatment rate − holdout rate) ÷ holdout rate [2]. Run the arithmetic on an illustrative flow. The holdout group orders at 8% and the treatment group at 10%. Lift is (10 − 8) ÷ 8 = 25%. Of every 10 buyers in the treatment group, 8 would have bought without the flow, and Klaviyo's attribution can credit all 10 to it if each opened an email first.

Revenue per profile is the stronger readout for a flow that carries a discount, because a discount can raise the order rate while lowering what each order is worth. Compute it the same way, per group.

How long does a holdout test need to run?

Months, for most flows. The rough math for comparing two order rates says how many profiles each group needs, and the holdout group is the bottleneck. An incrementality test for Meta ads walks through sample size for ad lift.

Run the arithmetic on an illustrative abandoned-checkout flow with 3,000 entries a month, a true holdout rate of 8% and a treatment rate of 10%, at 95% confidence and 80% power:

The lost orders are the holdout profiles times the 2-point difference. That's the counterintuitive part: a 10% holdout feels cheaper, and it saves about four orders while taking nearly twice as long. A six-month test started in summer runs straight through Black Friday, and Klaviyo itself flags seasonal events as a timing question for its holdout groups [2].

If your flow sees a few hundred entries a month, no share gets you there in a reasonable time. Test the bigger flow first.

Reading a flat result

Sometimes the two groups buy at nearly the same rate. Before you switch the flow off, check three things.

Did the arms split near 80 and 20? If not, another condition is probably acting before the random sample, so recheck the path criteria.

Did the holdout really get nothing? Holdout profiles still receive your campaigns, which is fine, since the test measures the flow on top of everything else. But a second flow with the same trigger can send them a near copy.

Is the flow doing something the order rate can't see? A welcome flow that collects preferences or an SMS opt-in may pay off in later campaigns rather than in its own window.

If all three hold and the test reached the sizes in the table above, the flow's attributed revenue was mostly orders you were getting anyway. Shorten it, cut the discount, and test again.

FAQ

What is a holdout test in Klaviyo?

It withholds messages from a random group of profiles and compares their conversion rate with everyone who got them. Klaviyo's global version reports lift as (treatment rate − holdout rate) ÷ holdout rate [2].

Can I use a global holdout group with fewer than 400,000 profiles?

No. Klaviyo requires at least 400,000 profiles in total [2]. Use a random-sample split inside the flow instead.

How big should the holdout group be?

Pick a share big enough for the test to finish. In the illustrative example above, 20% read out in a little over 3 months where 10% needed about 6, for about four extra lost orders.

Does a holdout test change Klaviyo's attributed revenue?

Holdout profiles receive no flow messages, so they can't generate attributed revenue for that flow. Treatment profiles are attributed exactly as before, under your account's attribution settings [1].

A holdout reads orders. WeltPixel Conversion Tracking sends each Shopify order server-side from the order itself. The free plan is GA4 only, up to 100 storefront orders a month; Plus ($39/month, no order limit) adds Klaviyo, Meta, Google Ads and the rest, and also sends admin, POS and draft orders [5]. If it runs next to Klaviyo's own Shopify integration, its Placed Order is the second metric described above, so pick one for both groups.

Sources

1. Klaviyo Help Center, "Understanding message conversion tracking" (default windows for new accounts: 5 days for email opens and clicks, 5 days for SMS clicks, 12 hours for SMS deliveries; Settings > Attribution), help.klaviyo.com/hc/en-us/articles/115005248128, updated March 26, 2026, accessed October 27, 2026
2. Klaviyo Help Center, "Getting started with global holdout groups" (400,000-profile minimum; scope, overrides and one active group; about 5% recommended; percentage fixed once active; 3 months recommended; marketing lift formula; conversions counted whether or not attributed), help.klaviyo.com/hc/en-us/articles/18138290642971, updated September 23, 2025, accessed October 27, 2026
3. Klaviyo Help Center, "Multi-Branch Splits" (random-sample condition; percentage applies to the profiles evaluated for that path; "Everyone else" always last; actions move onto Path #1; flow preview; 30-day path counts; phased release), help.klaviyo.com/hc/en-us/articles/52369094030235, updated October 7, 2026, accessed October 27, 2026
4. Klaviyo Help Center, "How to add an update profile property action to a flow" (Property update action; create new property; text, number, list, boolean and date types), help.klaviyo.com/hc/en-us/articles/360001768432, updated October 7, 2026, accessed October 27, 2026
5. WeltPixel Conversion Tracking, Shopify App Store listing (free plan GA4 up to 100 orders a month; Plus $39/month), apps.shopify.com/weltpixel-conversion-tracking, accessed October 27, 2026