A/B Testing Guide

A/B testing (also commonly referred to as split testing or bucket testing), put simply, is a way to compare two different versions of something – be it a webpage, app or email – in order to determine which performs better.

The way A/B testing is executed is thus; half your website visitors (or if you are split testing an email, half your recipients) will be shown version A and half will be shown version B. Following the test, you can review the performance of both versions using the data/analytics the split test generated to ascertain which performed better. These results allow you to make informed decisions on design/content changes with a view to improving your overall conversion rates. 

A/B testing such things as websites and emails can be carried out for different purposes. You may be thinking of redesigning a particular page of your website or considering an email template change. In such a case, if the split test produces the desired results, you can replace the incumbent page design or email template with the new one going forward.

It is important to make informed decisions regarding split testing, so utilise the data/analytics you currently possess as this will enable you to spot opportunities for optimisation. Areas/pages of your website or app, or particular emails that get sent to large volumes of recipients will allow you to gather a large amount of data quickly. It could be good to begin by looking at pages/screens/emails with a low conversion rate or high bounce rate.

Of course, full webpage/email redesigns aren’t the only things you can split test, you might also consider testing:

• Calls-to-action - e.g. “Buy Now”/“Read More”/”Checkout” buttons on your webpage/email.
• Subject lines - i.e. testing 2 email subject lines to see which results in a higher open or click-through rate (CTR).
• Page/copy/body layout – e.g. 1 column vs. 2 column text or alternate placing of different page/body elements such as images/buttons etc.
• Headline text
• Body text / PPC Ad copy
• Closing text
• Images
• Offers – e.g. “free delivery” vs. “10% off everything”

Whilst you may wish to test more than one thing, you should only test one at a time in order to get the most accurate results possible. Similarly, you should avoid dragging the tests out over too long a period of time. Econsultancy recommends no longer than a period of two months as, otherwise, tests could risk losing their validity.

Crucially, what you test should tie-in with what you wish to get out of the test you are performing, so ensure your end goal is clear to avoid wasting time and resources carrying out a test which will not yield the results you desire. You therefore, at this stage, essentially need to define what success means to you. Defining goals is vital to be able to review the performance of the split test.

Once you feel the test period is over, you can analyse the results. If your results are statistically significant and shows a clear winner, then you can go ahead and implement the change/proceed with that particular design/CTA/subject line or whatever it is you have tested. If your results do not display what you had hoped, or even if they show no clear result (are not statistically significant) then do not worry, you can certainly learn from the experience and look to generating new testable hypotheses. The results of the inconclusive A/B test will almost certainly be able to inform these future tests and be used to continually optimise the end-user experience of your site/app or email marketing materials.

So you need more convincing to implement some A/B testing do you? Allow me…

A/B testing allows you to make data-driven decisions rather than relying on guesses and hunches. No longer will you have to commit to changes and “hope for the best”, you will be acting on proven hypotheses.

It also allows you to optimise your site/app/email experience in order to improve conversion rates in a very methodical way. Improving your conversion rate will mean you get more value from your current users/customers, which can mean less money spent on customer acquisition. It generally costs around 5x more to acquire a new customer than it does to keep an existing one.  It has been found that 80% of your company’s future revenue will come from 20% of your existing customers, so investing the time and resources into improving the customer experience can really pay off. Finally and aside from the business benefits, A/B testing can help to transform workplace culture; creating a more data-driven environment where data and facts determine the direction of your product.

Ultimately, whether you are a marketer, developer or designer, A/B split testing is an effective way of utilising the power of the data and statistics you already possess to reduce risks, improve results and become more data-driven in the work that you carry out.