Why Analytics Look Different in Shogun A/B Testing vs. GA4 or Shopify

When you run an experiment in Shogun A/B Testing, you might notice that the numbers in your dashboard don’t match what you see in Google Analytics (GA4) or Shopify. This is entirely normal.

Shogun Measures Test Participants, Not Your Whole Store

Shopify and GA4 track every session and order across your entire store. Shogun is different: we only show data for shoppers who actually participated in your experiment.

For example, if you’re testing a product detail page, we’ll only track orders from shoppers who viewed that page while your test is running. If a customer purchases the product without visiting a page in your test, their order will appear in Shopify and GA4, but not in your Shogun results.

How Orders Get Counted in a Test

For an order to appear in your Shogun experiment results, three things need to happen:

  1. A shopper views a page that’s part of your test.
  2. Shogun assigns them to a test variant.
  3. They complete their purchase in the same session.

If any of those steps don’t happen, the order won’t be attributed to the experiment.

Why Some Orders Don’t Get Counted

There are a handful of common reasons you might see fewer orders in Shogun than in GA4 or Shopify:

  • Some shoppers check out without ever visiting the test page — for example, they add an item to their cart on a collection page or come through an abandoned cart email that takes them straight to checkout.
  • If you’re running multiple experiments, only one test can be tracked per session. A shopper assigned to a different test won’t be counted in this one.
  • Cookie blockers, GDPR/CCPA consent settings, or very heavy analytics scripts can prevent us from recording specific sessions.
  • Occasionally, site speed or competing scripts prevent our pixel from firing before a shopper leaves the page.

In all of these cases, the orders will appear in Shopify and GA4, but not in your Shogun results.

How Quickly Data Updates

Shogun data updates every 5 minutes. That includes sessions, clicks, and orders. Orders sometimes feel like they take longer to appear, but that’s only because they have to be tied back to the shopper’s test session before they show up in your dashboard.

So, if a customer makes a purchase, you can expect to see it in Shogun within a few minutes once it has been attributed to the correct test variant.

What’s Normal vs. When to Reach Out

It’s completely normal for Shogun order counts to be lower than what you see in Shopify or Google Analytics 4 (GA4). Minor differences, within a percent or two, are expected and just part of how attribution works.

If you’re running a brand-new test or only have a small handful of orders, you may not see much data right away. Allow the test more time and provide it with more traffic to obtain meaningful results.

That said, if you consistently see huge gaps between what’s in Shogun and what’s in your other analytics tools, let us know. We’re happy to take a closer look.