A/B Testing

Before implementing any changes to your website or newsletters, you must conduct appropriate A/B testing. With its help, you can:

• test the validity of marketing hypotheses;
• evaluate the effectiveness of the suggested changes;
• choose the best version of pages/messages;
• run updates supported by objective data, not assumptions and intuition, etc.

Why do you need an A/B test and how to conduct reliable testing — let's figure it out.

What is Website A/B Testing

This method is also called split testing for websites and landing pages. For example, you have page A that you want to optimize to improve some indicators. Create a copy of this page (page B) and change an element you think should positively impact the desired parameter.

Now, you must compare pages A and B by testing them on your website visitors. The page that shows the best result in terms of the studied indicator will be considered the most effective for your website. The results will be reliable if you follow all the testing rules and achieve high statistical significance.

Let's say you have a company that sells online English courses. Over the past few months, the number of applications submitted on your website has dropped significantly. Your marketer has suggested that the old CTA is no longer working and needs to be changed.

During the meetings and communication with clients, several new CTAs were formed. But how do you know which will appeal to your website visitors the most? What if conversions drop even further after adding a new call-to-action to your landing page?

In this case, A/B testing comes in. It will help you choose the best version of the call to action and track how the changes affected the number of applications from the site.

 A/B Testing in Email Campaigns

You need to conduct experiments and choose the best option not only for websites and landing pages but also for email campaigns. This will help you significantly improve customer communication efficiency and increase the conversion rate of your emails, Viber messages, In-App, App Inbox, SMS, mobile, and web push campaigns.

You will no longer have to argue about which option is better, which will attract customers' attention and encourage them to purchase, as an objective test will show everything.

You can test the following in your newsletters:

• the subject line and the main text of the newsletter;
• design — fonts, images, arrangement of elements, etc;
• buttons and CTAs;
• links;
• banners.

In Yespo CDP, you can conduct the necessary A/B testing of your email campaigns. Testing your email subject line is the easiest — completing this experiment will take a few minutes. In the email, specify several subject lines, and then see which one worked better and showed a higher open rate in the reports. 

To test buttons, banners, links, design, and content, you must create separate email campaigns, divide your contact database into several groups, and send them different email variants. In automatic reports, you'll see which variant was the most effective based on email opens and clicks.

If you are using another service to analyze your email testing, where you don't have access to detailed analytics, be sure to check the reliability of the results using an online statistical significance calculator for A/B tests. This way, you will understand whether the experiment can be trusted.

In addition to email A/B testing, Yespo CDP allows you to run tests for widgets, product recommendations, and other website features – all in one account without switching between different services.

If you want to learn about all the Yespo CDP features, leave a request. Our managers will contact you and explain everything in more detail.

Why Conduct Website and Newsletters A/B Testing

Conducting A/B test helps to:

• optimize website pages or newsletters based on objective data and user interests, not theory or intuition;
• minimize risks and financial losses from changes made to the website and newsletters;
• find a constructive solution to the problems and difficulties that visitors encounter while using the website or reading messages;
• ensure ROMI increases, in particular in website promotion and direct marketing;
• reduce the bounce rate on the website by improving the webpage and its usability;
• increase the messages' open rate and customer engagement with newsletter content.

What You Can Check With A/B Testing

Any of the hypotheses can be tested. However, most often, A/B testing is carried out whenever it`s needed to:

• change the website page or newsletter content, add new keywords to the text, change the semantic load or style of communication with potential customers;
• optimize the design of a page, email, or message using modern layout and techniques to capture the attention of visitors;
• change the website/newsletter color palette;
• add new images or improve old ones, including reducing their weight;
• change the call to action, size, color, shape, and text of the button below it;
• add new or change old widgets and product recommendations on the website; you can test product recommendations and dynamic content in emails;
• change the location of elements and content blocks on the resource page or in the newsletter.

How to Conduct a Website A/B Testing: Instructions

Even a beginner in marketing can handle A/B testing of email campaigns in Yespo, especially with the support of our experts. However, experiments with website pages require specific skills and knowledge. To ensure that your web page A/B testing is effective and error-free, follow the instructions:

1. Determine the website changes purpose

You need to clearly understand why changes are required. For example, the number of orders from the site has dropped, and this parameter needs to be improved. Another option is that the abandoned views and abandoned carts percentage have increased, etc.

2. Select an indicator to evaluate the test results

Determine what will be the leading indicator of successful changes for you. This can be an increase in the number of applications received from the website, an increase in the click-through rate of buttons and form completion, increased chatting with the manager, a change in the frequency of registrations, etc.

The most popular metric for evaluating test results is the growth of sales revenue on the website (for this purpose, A/B test revenue is conducted).

3. Formulate hypotheses

Assume what website changes will lead to the goal achievement. For example, you have a hypothesis that the website has poor conversion rates due to ineffective CTA or inconvenient navigation, which makes it difficult for users to search for products and place orders.

Write down all the ideas that, if implemented, can lead to the desired goal. Each of these hypotheses needs to be tested using A/B testing.

Don't try to implement all the changes in one updated test version. Otherwise, when you analyze the results, you won't know what worked and what didn't. The fewer changes you test at a time, the easier it is to understand their effectiveness.

4. Determine the sample of visitors needed for successful A/B testing

The sample size for A/B testing is important: you will understand that the experiment results have sufficient statistical significance and are not random.

A special online calculator can calculate the minimum sample size for A/B testing. The number obtained after such calculations is the number of people who should view each test version of the page for the results to be considered reliable.

In addition, you need to define who will be included in this number of users: all website users or only new ones. Remember to exclude your company's employees from the test — enter their device IP addresses into the analytics system in advance so that these visits are not considered when evaluating the results.

5. Specify the test duration

Usually, a website A/B test lasts at least 10-14 days. During this period, you can reach the required number of visitors and see the reaction of users who visit the website on different weekdays at different times.

If the test lasts more than a month, there is a risk that users will clear the cache on their devices and be perceived as new users when they visit again, distorting the experiment results.

6. Consider external factors

Conduct testing among a homogeneous audience, evenly distribute traffic between the tested page versions and analyze the results separately for each traffic channel.

Try to conduct A/B testing during your business's standard sales period. If you launch an experiment during the hot season, the increase in the number of orders on the website may be due to seasonal demand, not to the changes you've implemented. The same goes for the downtime period — a drop in sales at this point is a logical consequence, not an indicator of your optimization ideas' ineffectiveness.

7. Run the test

Run the test for a specified period and on a set sample of visitors. Use special testing services for the experiment.

8. Analyze the results and assess statistical significance

If the significance of the statistical results is sufficient, the testing was successful, and you can use the winning page version. Now, your task is to implement the appropriate changes and, after some time (usually a few months), analyze whether you have achieved your goal.

Website A/B Testing Services

Use special services for A/B testing, such as:

• Google Analytics — free testing right in your account. It has minimal functionality for simple experiments and no visual editor, so you need to know HTML code.
• Vwo.com — a service for testing mobile and desktop versions of a website. It has a visual editor, a library of ideas for conducting experiments, advanced targeting, and integration with many services and platforms, including Google Analytics.
• Optimizely — a paid tool for conducting A/B tests, split tests, multichannel, and multivariate tests in a visual interface. It offers results segmentation and advanced targeting.
• Changeagain.me — there is a visual editor that targets by country and device. You can test both desktop and mobile versions of the website, and there is a full and automatic integration with your Google Analytics account.
• Convert.com — there is a possibility to test simultaneously for several metrics and revenues; diverse targeting, a visual editor, and advanced integration with other services.

Now you know what A/B testing is for and how it works, what you can test in email campaigns, how to run split tests on your website, and what services to use. Good luck with your experiments!