Ecommerce A/B Testing: What to Test, How to Test & What to Ignore
Most ecommerce A/B tests fail to produce a usable answer, not because the idea was bad but because the test was set up to fail before it started: no clear hypothesis, an underpowered sample, or a primary metric picked after the results came in rather than before.
Short answer
Write the hypothesis, primary metric and required sample size before launching the test, not after. Most stores below roughly 10,000 monthly sessions per page do not have the traffic to run a statistically meaningful test on that page at all, and should prioritise other improvements over formal testing until they do.
A hypothesis framework
Every test needs a specific, falsifiable statement before it launches: “Because visitors currently see no shipping cost until checkout, showing it on the product page will reduce checkout abandonment.” Not “let’s test a new product page,” which is not a hypothesis, it is a redesign with no way to know afterward why it did or did not work.
A specific hypothesis also forces a specific primary metric before the test runs. If the hypothesis is about checkout abandonment, checkout completion rate is the primary metric, decided in advance, not selected afterward from whichever number happened to move.
Sample size and the primary metric
A test stopped early because the variant is “winning” is one of the most common ways ecommerce teams fool themselves. Early results on a small sample swing wildly and often reverse by the time enough traffic has passed through; use a sample size calculator before launching to know roughly how long the test needs to run, and commit to that duration rather than checking daily and stopping at the first favourable-looking result.
Pick one primary metric and treat everything else as secondary context. A test that improves add-to-cart rate but not completed purchases has not actually won; it has moved a step further up the funnel without moving the number that matters. Secondary metrics are useful for understanding why a test worked, not for deciding whether it did.
Real test ideas by page
| Page | Test idea | Primary metric |
|---|---|---|
| Product page | Shipping cost shown vs hidden until checkout | Add-to-cart rate |
| Product page | Review count/rating position: near title vs below fold | Add-to-cart rate |
| Cart | Cross-sell shown vs not shown | AOV and total revenue per session |
| Checkout | Guest checkout as default vs account-first | Checkout completion rate |
| Checkout | Number of steps: single-page vs multi-step | Checkout completion rate |
| Pricing | Charm pricing (€19.99) vs round number (€20) | Conversion rate at that price point |
| Merchandising | Category default sort: newest vs best-selling | Category-to-cart conversion |
If you are not sure where to start, the highest-value first test is usually whatever your conversion rate benchmarks against your own history suggest is your weakest funnel step, not the page that happens to be easiest to redesign.
When not to A/B test
Below roughly 10,000 monthly sessions on the page being tested, a formal test rarely reaches statistical significance in a reasonable timeframe, and the honest move is to make the change based on best judgment and established best practice rather than waiting months for an inconclusive test. Testing infrastructure and process also cost real time to run properly; below a certain traffic threshold, that time is better spent on changes with strong existing evidence (removing friction, adding trust signals) than on formal experimentation.
A test is also the wrong tool for a clear, low-risk fix. If a page has an obviously broken layout on mobile, that does not need a test, it needs fixing. Reserve formal testing for genuine uncertainty between two reasonable options, not for validating fixes to things that are simply broken.
Prioritising the experiment backlog
Score each test idea on expected impact and confidence, and run the ones with both high impact and reasonable confidence first. A test with high potential impact but low confidence (you are genuinely unsure which variant will win) is more valuable to run than a test with high confidence and low impact, since the latter barely moves anything even if it “wins.” Keep a running backlog rather than testing whatever idea came up most recently, which is how testing programs drift toward low-value work over time.