Holdout feasibility

Can you measure it?

Every upsell app reports revenue it says it earned you. A holdout test — showing the widgets to some visitors and not others — is the only way to check. This works out whether your store has the traffic to run one. For most stores it does not, including for us.

Your store

The offer

That implies a revenue lift of 1.34%. It is the number a test has to detect, and it is much smaller than the acceptance rate — which is the figure vendors quote.

Detecting the 1.34% lift your offer implies would take 7.4 years. In practice this store cannot measure it.

About 100,000 visitors a month. Running the test also withholds the widgets from part of them while it runs.

Traffic needed at 80% power on an even split, measured on revenue per visitor.
If it lifts revenue byVisitorsOrdersTime to an answer
0.5%63.9M1.6M53 years
1%16.0M399,43313 years
1.34%your offer8.9M222,6897.4 years
2%4.0M99,8583.3 years
3%1.8M44,38118 months
5%639,09315,9776.4 months
10%159,7733,9941.6 months

Acceptance is not lift. Three percent of orders adding $25 to an $80 basket is a 0.9% lift, not a 3% one. That order of magnitude is the difference between a test that finishes and one that never does.

Why revenue per visitor. Comparing order value among buyers only looks cheaper. An upsell changes who completes a purchase as well as what they spend, so measuring only the buyers turns a widget that merely deters small baskets into a measured win.

If the answer is no. That is not a reason to skip upsell apps. It is a reason to stop paying for reported increments nobody can check, and to judge these apps on price and on what they do to your pages instead.