Send Docs Feedback

A/B testing

The A/B testing feature of app monitoring in App Services allows you to create an alternate set of monitoring and app-specific feature configurations for your mobile app that can then each be exposed to a percentage of your total user base. You can then leverage the monitoring data collected from this test group to determine whether the new configuration drives more user engagement. This is particularly useful when you are considering new features and layouts for your app, since it allows you to collect data on the user response to a new configuration without affecting your entire user base.

Starting an A/B Test

To apply monitoring and app-specific feature configurations to a percentage of users for the purpose of A/B testing, do the following:

  1. In the App Services admin portal, select Configure > A/B Testing Configs.
  2. Check the Enable A/B Testing checkbox.
  3. In the the Apply to field, enter the percentage of total devices running your app that you want the configurations applied to. Do not include a % sign.
  4. Configure the monitoring settings to be applied to your A/B test group.
  5. Configure the app-specific features to be applied to your A/B test group.
  6. Click Save Changes at the bottom of the menu.

Notifying users of configuration changes

If you modify any of the default configs or A/B testing configs during an A/B test, it is very important that you contact the members of your test group and ask them to restart the app. In some cases you may need to ask them to restart the app twice in the event that the updated settings do not load the first time.

The users whose devices have reverted back to the previous configuration will need to kill their app entirely (that is, not have it loaded in memory). In Android, they may need to use task killer. In iOS, they will need to close down their application.

This is necessary since in some cases, the new configuration may not have been downloaded before the initial screen of the app is displayed. As a result, the previous configuration is applied.

Help or comments?