A/B Testing

What is AdTiming A/B Testing?

A/B testing isn’t difficult to understand. You start by creating a hypothesis about a certain element and then run a test to see if your theory was right. Well, AdTiming A/B testing is an experiment that splits the traffic (users) into two groups – Group A (Control Group) and Group B (Test Group) - with two different app configurations. This allows you comparing two monetization strategies each other through statistical analysis, thus helping you determine which group perform better for a given goal.


How AdTiming A/B Testing Works?

Firstly, you can define traffic allocation between A and B groups (E.g. 60% for Group A and 40% for Group B). Your traffic of Group A (Control Group) will be exposed to the original mediation configurations while your traffic of Group B will be exposed to the mediation configurations according to the variables you want to test.

Please Note: Each user is assigned to one of two groups and will remain in the same group until the test is terminated or the traffic allocation is changed.

Then, the user will be served ads either the control or variation, their engagement with each experience is collected and analyzed through a statistical engine. You can determine whether the variation configurations had a positive, negative, or no effect on users’ behavior.

Last, you can terminate the A/B testing by applying the configurations from the chosen group to all your users.

Why You Should A/B Test?

A/B testing allows publishers to construct hypotheses, and to learn better how their users engage with ads and ensure their monetization approach is maximizing revenue and growing their business. In another way, they can be proven wrong - their opinion about the best experience or for a given goal can be proven wrong through an A/B test. A/B testing helps you visualize the possibilities.

What Can You A/B Test?

There’re a number of key elements in Mediation Module can be A/B tested. E.g. different waterfall optimization strategies, different filtering of traffic, different mediation rules for one segment, etc. Reach out to your account manager or contact us to discuss which strategy is right for you.


A/B Testing Process

Step 1. Navigate to the Mediation → A/B testing

Click on 'Create New Test'.  


You can review your history A/B Testing on this app in 'Previous A/B Testing'.


Step 2. Define your A/B Testing

Set up the A/B Test name and identify your goals as well as define the traffic split between two groups.



Step 3. Finish Configurations Before Start A/B Test

When you finish defining the A/B Test group, you should follow the steps to start your A/B Test. Click on 'Ad Network Setup' in mceclip8.png .


As you can see you are already in the page 'Ad Network Setup' of Group B. At this page, AdTiming duplicates all existing Ad Network Settings from Group A (Control Group) to Group B (Test Group). You can also turn off a certain instance and this instance will not be included within Group B’s configurations any more. Please make sure that click on  to setup instances for Group B.


At 'Instance Setting' page, you are required to input the Mediation Parameters.


Please Note: App Key changes of same App in Test Group is NOT supported for Facebook Audience Network or AdMob.

Then, input new and unique instance IDs for each one of your app’s configured instances. AdTiming’s instance IDs are automatically generated for Group B (Test Group).

Please Note: This allows us to track unique revenue to compare performance between groups.


Step 4. Apply new mediation settings for Group B.

When you finish the setup of ad network for Test Group, you will see a mceclip7.png at the left side of Stepmceclip9.png.



Step 5. Start A/B Test

Now youcan see the 'Start A/B Test' button at the right side of step mceclip10.png. Click on this button and test will be started immediately. 


You can see that the system started to record the time since the A/B Test started and the A/B Test Report is available to review.


Step 6. Analyze Reports and Terminate A/B Testing.

AdTiming A/B testing Report will present the data from the test and show you the difference between the two groups. Once you’ve analyzed results and reached a conclusion about which group performs better, you can terminate the test by clicking on the 'Terminate → Select a Winner' option. Terminating a test means the winning group’s settings will apply to 100% of the traffic on your app.


You can also leave your comments for this A/B Test to make a record for what you've learned from the A/B Test carried by yourself. The record will be listed in the tab of 'PREVIOUS A/B TEST'.


Was this article helpful?
1 out of 2 found this helpful



Article is closed for comments.