Bidding test

What is a bidding test?

The Bidding Test enables publishers to compare the performance of a traditional waterfall (A) vs. a hybrid waterfall that includes in-app bidding + traditional instances (B).

The test requires minimal setup. Traffic allocation will be 50/50. Performance KPIs can be broken down by A vs. B in the platform Performance and User Activity pages to compare results.

How a bidding test works

Once activated, the Bidding Test will divide the traffic equally between groups A and B, comparing a traditional vs. bidding hybrid waterfall.

  • A group will include only traditional instances, according to Mediation Management setup
  • B group will include all active bidders + traditional instances. Any traditional instances that are of the same network as a bidder will be automatically filtered out. For example, if the waterfall includes an ironSource network instance, that instance will be removed because ironSource bidding network will be active

Pre-test setup

Step 1: Setup Waterfall

Set up your waterfall according to the instructions below.  This setup allows the AB test to run properly:

  1. Move all instances to Tier 2 (unless you’re running a deal)
  2. Set the tier to Sort by CPM
  3. Set rates for any instance that is less than 2 days old.  To learn how to set an instance rate,  see this article.

Step 2: Set up bidding networks

Activate the bidding networks you want to test:

  1. Activate each bidding network and include it in each waterfall group where you would like to test bidder vs. non-bidder. Run the test in all countries for the most conclusive results.  Setup should be done according to the instructions in this article.
  2. Make sure that each bidding network also has at least one traditional instance in the same waterfall.  This ensures that the traffic is split properly for the test, with the traditional instance of that network in A group, and the bidding instance of that network in B group.

    Note: Note: If a bidding network is set up without at least one traditional instance of the same network in the waterfall, the bidder will run in both A and B groups when the test is live.

Step 3: Navigate to the A/B Testing page and click the Create Bidding Test button

Step 4: Terminate the test

Once you’ve analyzed results and reached a conclusion about which group performs better, you can choose the winning group and terminate the test by clicking on the Continue with A or Continue with B buttons. Terminating a test means the winning group’s settings will apply to 100% of the traffic on your app.

  • If you choose group B (bidding group), the traditional instances of the bidding networks will automatically be deactivated
  • If you choose group A (non-bidding group), the bidding instances will be automatically deactivated

Monitoring and Reporting

To monitor an active bidding test, use the platform reporting pages (Performance and User Activity reports).

A breakdown by A/B is available on the Performance and User Activity reports, with all metrics supported. To view reports broken by A/B, select one app that completed or is currently running an A/B test.

Click Show Reports to view the Performance report with the app pre-filtered, test start/end dates selected, and the data broken down by A/B:

Performance Reports:

Best Practices

  1. Keep the test live for 14 days.
  2. Compare the ARPDAU of A vs. B group.  This KPI can be tracked in the User Activity page of the ironSource Mediation, by navigating to the ARPDAU KPI, then selecting AB Breakdown.