Analyzing Results

Once your test is up and running, the most crucial part is understanding what the results mean and what to do with them. Shogun A/B Testing provides a performance dashboard that highlights key metrics, allowing you to quickly identify which variant is winning and why.

Viewing your test dashboard

As your test collects data, the dashboard will update in real time. You'll be able to compare your original and variant side by side.

Each variant will show key metrics at a glance, along with indicators of which one is currently performing best. The dashboard also lets you check how traffic is being split and whether your audience segmentation is working as expected.

Interpreting test results and significance

A test’s performance isn’t just about who has the highest conversion rate, it’s about whether the difference between variants is statistically meaningful. Shogun helps you determine this by calculating statistical significance based on your selected goal, like purchases or clicks.

When one variant consistently outperforms the other and reaches significance, you'll see a clear indication that it’s the winning version. If the results are too close to call or if not enough data has been collected yet, the test will continue running until a clearer outcome is achieved.

Metrics: conversions, CTR, AOV

Shogun tracks the metrics that matter most for e-commerce, including:

• Conversions: The percentage of users who completed a purchase.
• Click-through rate (CTR): How many users clicked a specific button, link, or element you’re tracking.
• Average order value (AOV): The average spend per order from users who interacted with each variant.

You can choose which of these to use as your primary goal when setting up your test, and view all of them in the dashboard to get a fuller picture of performance.

This topic is covered in greater detail in the Test Goal Definitions article.

Troubleshooting data issues

If results aren’t showing as expected, there are a few things to check:

• Confirm that enough traffic has reached both variants; small sample sizes can skew early results.
• Double-check your goal setup; if the wrong metric was selected, you may not be seeing relevant data.
• Ensure that segmentation (if applied) hasn’t limited your test too much, especially on lower-traffic stores.

If your test has been live for several days and you’re still not seeing results, it may just need more time. Some tests may take longer to gather sufficient data, especially when traffic is low or your goal is based on purchases.

If you are still having trouble interpreting your data after checking these things, you can always reach out to us over email or live chat via the prompts located at the bottom of the dashboard in-app.