Quick A/B test

What is a quick A/B test?

The quick A/B test enables you to test different waterfalls without any additional setup. 

Traffic is automatically allocated 50/50 between the A and B groups. Performance KPIs can be broken down by A vs. B in the platform Performance, User Activity, and Bidding report pages to compare results.

What can you test with the quick A/B test?

You can create any kind of waterfall test with the quick a/b test. You can test different waterfall settings including country groups, new networks, different instance pricing, and cross promotion

How the quick A/B test works

Once activated, the quick A/B test divides traffic equally between groups A and B.  At this stage, the A and B groups are identical and no changes will be applied to either group.

  • Each group will include all of your active instances and mediation groups
  • The groups are identical in settings and waterfalls
  • No pre-test setup is required

How to start a quick A/B test

Step 1: Navigate to the A/B Testing page 

A/B Testing page

Click the Create Test button next to the Quick A/B Test option

Step 2: Name and activate the test

Give the test a name and description. Then click the Start Test button to start the test.

Start Quick A/B Test

Step 3: Go to your Mediation Management page and modify each group’s waterfall settings

You’ll see a toggle on the mediation management page that allows you to switch between groups A and B. Adjust the waterfall settings for the test group according to the variables that interest you. Each change you make will apply ONLY to the group you’ve selected.

Mediation Management page

Supported waterfall components include:

  • Capping 
  • Addition (or subtraction) of instances from the mediation group
  • New group 
  • Delete group
  • Duplicate group 
  • Export CSV 
  • Rate 

 Step 4: Terminate the test

Once you’ve analyzed your results and reached a conclusion about which group performs better, you can choose the winning group and terminate the test by clicking on the Continue with A or Continue with B buttons.

Terminate the test

Terminating a test means the winning group’s settings will apply to 100% of the traffic on your app. 

You've selected Group B

Note: Once you’ve terminated a test and approved the winning group, this action can’t be undone.  

Monitoring and reporting

To monitor an active bidding test, use the platform reporting pages—Performance, User Activity, and Bidding reports.

Breakdown by A/B is available on the Performance, User Activity, and Bidding reports, with all metrics supported. To view reports broken by A/B, select one app that completed or is currently running an A/B test.

Click Show Reports to view the Performance report with the app pre-filtered, test start/end dates selected, and the data broken down by A/B.

Show report

Here’s an example of a Performance Report that’s broken down into A and B groups:

Performance reports for A/B testing