Test Concurrency & Traffic Distribution

Running multiple A/B tests simultaneously can accelerate learning and optimization across your storefront. Still, it is essential to understand how different test types interact, how traffic is assigned, and how your store’s caching and targeting systems impact test delivery.

This guide explains how Shogun handles concurrent testing, allowing you to run experiments confidently without compromising data quality.

Traffic Distribution Methods

When a visitor lands on a page included in a test, they’re assigned a variant based on your selected traffic split (e.g., 50/50 or 70/30). That assignment is stored in the browser to ensure consistent experiences across sessions.

Once assigned, visitors will always see their designated variant, even if they return days later.

How Theme Tests Interact with Other Tests

Shogun supports running Theme Tests alongside Page, Template, and Personalization tests. However, there are specific rules to maintain consistency:

Order of Assignments

• If a visitor hasn’t yet received a Theme Test assignment, that assignment will happen first.
• Once assigned, their theme determines what additional experiences they can receive.

Theme Variant Rules

• Main Theme (Original):

  Visitors assigned to the original theme can still participate in Page Tests, Template Tests, or Personalization, provided they meet the audience conditions.

• Theme Variant (B):

  Visitors using the alternate theme will not see any additional tests or personalization. This prevents layout conflicts and ensures clean data.

Analytics

Our reporting will reflect:

• The theme variant that a visitor saw
• Any additional test or personalization variant (if the visitor was using the Main Theme)

Cache Management

Shogun stores a visitor’s test variant assignment in local storage, which ensures they see the same version of a test each time they return.

Key details:

• Variant assignment is cached on the client side and refreshed every 24 hours.
• You can bypass the cache manually by using the ?cache=false URL parameter; this is particularly helpful during QA and testing.
• For theme and template tests, dynamic scripts ensure the correct variant is served regardless of Shopify’s default caching.

Audience Targeting in Concurrent Tests

Each test evaluates audience conditions independently, meaning a visitor might qualify for one test but not another.

Targeting rules include:

• Device type (e.g., mobile, desktop)
• Visitor status (new vs. returning)
• Geographic location
• Login status
• URL pattern matching
• Referrer sources (e.g., campaigns, social)

Best Practices for Concurrent Testing

Running multiple experiments requires strategic planning. Here are some guidelines:

1. Test Isolation

Avoid running multiple tests on the same elements (e.g., buttons, headers, hero sections) to prevent inconsistencies. Conflicting changes can skew results.

2. Traffic Management

Ensure each test receives enough traffic to reach statistical significance. If traffic is split too thinly across tests or variants, results may be inconclusive.

3. Duration Planning

Concurrent tests may require longer run times, especially if you’re distributing traffic unevenly or testing smaller audience segments.

4. Monitoring & Analysis

Use Shogun’s test dashboard to regularly monitor:

• Overlapping test traffic
• Timeline of variant interactions
• Any unexpected behavioral patterns

📘 For more help: See Best Practices for A/B Test Design