A/B testing pages

A/B testing is an experiment that compares one or more alternative versions of a page (variants) against the original, gauging their efficiency at accomplishing selected goals (visitor conversions). The measured conversions are based on visitor actions that can range from simple – a banner click or a form submit, to complex – placing an item into the shopping cart, or performing a full purchase.

During a test, all page variants receive roughly the same visitor traffic and compete with each other for the highest conversion number or weighted value. After a test concludes, statistical analysis based on configured test parameters helps select a winning variant. When a variant is promoted as the test winner, it replaces the original content of the page.

Kentico supports the following forms of A/B testing:

• Content (page) A/B testing – allows you to test different variants of page content against one another.
• Email A/B testing – allows you to test different variants of marketing emails against one another.

A standard workflow when A/B testing pages usually entails:

1. Creating an A/B test together with one or more variants.
2. Configuring the test by setting conversion goals and other metrics.
3. Starting the test and gathering behavioral data and statistics based on its configuration.
4. Evaluating test results with the help of provided analytics.
5. Selecting a winning variant based on the collected insights.

Page builder requirements

Content A/B testing functionality is tightly connected with the page builder feature. A/B tests may only be created for pages that support the page builder functionality (i.e. page types that have the Page tab displayed in the Pages application).

Moreover, individual test variants may only differ in page builder content – this includes widgets and their configuration and assigned page templates. You cannot set different content of page fields for specific A/B test variants.

How A/B testing works

When a live-site user navigates to a page with a running A/B test, the system assigns them a randomly selected variant from the given A/B test variant pool. Over the course of the test’s duration, each page variant should receive roughly the same amount of visitor traffic.

Moreover, to maintain a consistent appearance of the tested page for repeat visitors, Kentico stores a CMSAB<A/B test code name> cookie containing the identifier of the selected variant in the visitor’s browser. This cookie also ensures that all conversions performed by A/B tested users get logged under the assigned variant. The cookie expires within 2 months after the last visit on the tested page.

Creating A/B tests

Create a new A/B test by following these steps:

1. Open the Pages application.
2. Select a page you wish to A/B test in the content tree.
3. Switch to the Page tab.
4. Click Create A/B variant.
   

The system automatically creates a new A/B test, named after the page and the currently selected culture, and displays the first page variant.

The Create A/B variant button changes into a selector that allows you to switch between variants and further interact with the page’s A/B test.

• Manage A/B test – opens a modal dialog that allows you to configure, start, and evaluate an A/B test created for the corresponding page.
• Add new variant – creates a new A/B variant by directly copying all saved content from the currently selected A/B variant.

You have created an A/B test together with variants. Continue by creating additional variants and editing variant content. You can define any number of A/B variants for the test.

Defining additional variants

To create additional test variants:

1. Open an A/B tested page in the Pages application.
2. Click the A/B testing selector.
3. Click Add new variant.
   

The system creates a new variant and makes it active on the Page tab, allowing for immediate editing. The variant is created as a direct copy of the previously selected variant.

Editing test variants

After you have created an A/B test and its variants, make the desired changes to the variant’s content. 

1. Open an A/B tested page in the Pages application.
2. Choose the variant you want to edit using the selector in the upper right.
   • (Optional) You can rename the variant by clicking the edit () icon. Variant names are limited to 100 characters.
3. Edit the variant’s content (e.g., by swapping or re-configuring widgets, or choosing a different page template).
4. Save your changes.

Deleting test variants

If you wish to permanently delete a test variant:

1. Open an A/B tested page in the Pages application.
2. Click on the A/B testing selector in the upper right.
3. Hover your mouse cursor over the variant you want to delete.
4. Click the delete () icon.
   

You have deleted the variant and its content from the system.

Configuring and starting A/B tests

Before starting an A/B test, configure its behavior on the Settings tab:

1. Edit an existing A/B test:
   1. Open the  Pages  application.
   2. Select a page with an A/B test.
   3. Click on the A/B testing selector.
   4. Click Manage A/B test.
   – OR –
   1. Open the A/B tests application.
   2. Edit () an A/B test.
2. Switch to the Settings tab.
3. Configure general test parameters.
4. Assign conversions you wish to track to the test.
5. Save your changes.

The A/B test is now configured and ready to start on the selected Start date.

Configuring general test parameters

General test parameters, located under the General section, allow you to set the test’s Start and End dates and provide internal metadata about the test:

After you have configured general test parameters, continue by assigning tracked conversions.

Assigning conversions

Visitor conversions are the metrics that you use to evaluate the test’s variants (using the reports on the Overview tab). Newly created A/B tests do not track any conversions by default. You need to specify the types of conversions you wish to track for each A/B test under the Conversions and Traffic section:

1. Click Add conversion next to the Conversions field.
2. Select a conversion from the Visitor activity drop-down list.
3. Enter the desired conversion value into the Value field.
   • For some conversion types, such as Purchase of a specific product or Page visit, you may need to input additional information.
4. Click Apply.
5. Click Save after you are done adding the required conversions.

You can also optionally configure the Visitor targeting and Included traffic fields:

• Visitor targeting – a macro condition that filters which visitors are included in the test.
• Included traffic – sets the percentage of page visitors included in the test.

Excluded visitors always see the original variant of the tested page and their actions on the website do not trigger any conversion logging.

A/B test conversions

By default, you can assign the following visitor actions as conversions. Your site’s developers may have implemented additional conversion types not listed here:

Limiting tests based on visitor consent

By default, the system serves A/B testing variants and logs subsequent conversions for all visitors. The A/B testing functionality does not collect or store any personal data and all statistics are anonymous. It is NOT legally necessary to add a consent condition in typical scenarios. However, if your website’s data protection policy requires such a condition for any reason, you can use the following approach:

1. Make sure that contact tracking is enabled for your website.
2. Prepare the consent that you wish to use for A/B testing.
   • To simplify consents for visitors, you can reuse your site’s main contact tracking consent for the purposes of A/B testing.
   • If you wish to use an additional consent for A/B testing, create a new one and then provide a way for visitors to give agreements (e.g. using a form with a field based on the Consent agreement component). Note that the system associates consents with contacts, so visitors need to give both contact tracking consent and the additional A/B testing consent in this scenario.
3. Edit the given A/B test and select the Settings tab.
4. Click Edit next to the Visitor targeting setting and perform the following:
   1. Add the Contact has agreed with consent rule.
   2. Click select consent in the rule designer, choose the related consent, and click Set parameter.
   3. Click Save & Close.
5. Save the test settings.

With this configuration, the A/B test only affects visitors who consent to being tracked as contacts and have given an agreement with the additional A/B testing consent if one is specified. Other visitors always see the original variant of the tested page and their actions on the website do not trigger any conversion logging.

Evaluating A/B tests

Each test contains a filterable performance summary of individual variants on its Overview tab:

1. Open the  Pages  application.

2. Select a page with an A/B test.

3. Click on the A/B testing selector.

4. Click Manage A/B test.

   – OR –

5. Open the  A/B tests application.

6. Edit () a running A/B test.

The overview tab is horizontally divided into two sections:

• A filterable results graph allowing you to inspect the performance of individual conversions.
• An overview of all page variants and their performance.

Conversion goals

In the Conversion goals drop-down list, you can select the conversion you want to evaluate. The displayed graph shows how each variant performs for the selected conversion. 

Success metric

Using the Success metric drop-down list, you can change the conversion data visualized by the graph:

• Conversion rate – expresses what percentage of total visitors performed the conversion selected in the Conversion goals drop-down list.
• Conversion value – displays conversion values for the conversion selected in the Conversion goals drop-down list.
• Average conversion value – displays the average value of the logged conversions for the conversion selected in the Conversion goals drop-down list.
• Conversion count – displays the number of conversions for the conversion selected in the Conversion goals drop-down list.

Counting methodology

In the Counting methodology drop-down list, you can change the kind of data visualized by the graph:

• Visitor conversion – the number of visitors that performed at least one conversion on the page during any of their visits. A single visitor with multiple conversions is counted once.
• Visit conversion – the number of visits during which visitors performed at least one conversion. A single visit with multiple conversions is counted once.
• Every conversion – the total number of conversions performed by visitors across all visits.

Chance to beat the original

Chance to beat original is a percentage that expresses the confidence that you can have in a variant outperforming the original. For example, a 97% Chance to beat original means that the variant outperforms the original in 97% of cases. This number is based on data the test has collected to date and may change during repeated observations. We strongly recommend you let each A/B test run for at least 14 days and let it gather 100 or more conversions and then decide on the outcome of the test. Otherwise, you risk selecting a sub-par variant as a winner.

Each variant’s significance is set to 95% (p-values of 5% or lower). That is, once the system determines the variant outperforms the original in 95% or more cases, the variant is marked as significant and its background turns green.

The following picture shows the original variant at the top and two variants, each with a different Chance to beat original.

The Conversion rate column displays the calculated mean conversion rates together with their corresponding confidence intervals. The adjacent box plots visualize the calculated intervals. Confidence intervals of treatment variants overlapping with the original are rendered grey. Overlapping confidence interval generally signal a lack of statistical significance – the test cannot conclusively prove the treatment variant performs better than the original (the higher the overlap, the higher the uncertainty).

The Chance to beat original column displays the percentage confidence, based on collected test data, the system has in the variant outperforming the original (as described above). 

Selecting a winner

After a test has finished, you can select one of the test’s variants as the winner of that particular experiment. The selected variant then replaces the original page.

1. Open the test’s overview tab.
2. Click Select variant as winner.
   

After you select a winning variant, the test automatically concludes and becomes archived. You can view the test data (conversion rates, variant performance, test configuration, etc.) of archived tests by editing () the test in the A/B tests application. However, all defined variants are permanently deleted.

On the A/B tested page, the testing environment is reset, enabling the start of another experiment.