1. What is A/B Test ?

The A/B testing is a function to test mediation rule strategies by dividing users equally into two groups.When the feature is turned on, the user is pinned to Group A or Group B and adopts the mediation rules of their grouping until the test stops. You can analyse the data based on the performance of groups A and B, and the difference in strategy, which has achieved the purpose of improving revenue.

2.How to use A/B Test

2.1 Create A/B test

Navigate to the [Mediation Rule] page of [Mediation], select the mediation rule for which you plan to do A/B testing, and click [ Set A/B Test ] in the operations column.

2.2 Configure A/B test basic information

In the A/B test configuration page, the original mediation rule name, targeting region, ATT settings, and advanced settings cannot be modified. After you are sure that the mediation rules you set are correct, enter the name of the A/B test.

Note: To test more convincingly, the ab group had a 50% : 50% ratio.

2.3 Configure the instances of the Group B

Group A, as the base group, cannot be adjusted during A/B testing. The initial configuration of Group B is consistent with Group A, and you can adjust the instance based on the strategy you want to test.

2.4 Turn on the AB test

When you have completed the A/B test configuration and clicked Save Changes, the A/B test will start running until you make a selection.

2.5 View the data

During A/B testing, you can view it through the mediation Rule List page or the mediation rule details page.

a. List of mediation rules

Click the pushdown triangle in [Set A/B Settings] in the operation column to view the overview data. You can also make a selection of AB groups on the current page to end the AB test.

b. Mediation rule details

After you enter the A/B test details page, you can view the data details of the mediation rules. You can use the overview card to get a summary of the data. Then Instance-level data of the mediation rules to understand the performance of the strategy and analyse the reasons for the performance.

Click [View more information], you can understand the performance of ads in the ab two groups, as well as user activity data.

2.6 Stop the A/B test

When you have accumulated enough data, you can stop the A/B test by selecting the group. You can select from the Mediation Rules List page or the A/B Test Details page.

3. Best practices

You can understand the interaction trends of users and ads by setting up different Mediation Rule strategies. After a confidence testing cycle, the winning strategy has been chosen to increase advertising revenue and business growth.

3.1 Placement traffic

To make the results more convincing, it is recommended that you request more than 10,000 requests during the test. Enough data accumulation can make the difference of the data between group A and group B more obvious and make your decision-making simpler.

3.2  The number of test variables

Only one strategy point is tested at a time. This can help you know more clearly what causes data discrepancies.

3.3 Test time cycle

Each test takes enough time to produce valid data, and it is recommended to take at least 1 week or so per test. If you want your strategy to be supported enough, it is recommended to make a decision after 2 weeks or more.

3.4 Testing also needs to be iterated

You can optimise the mediation rules based on the results of the A/B test, but this is not the end point. Mediation rules can be continuously optimised according to different time dimensions, the choice of ad networks, and the ordering of instances.