A quick guide to email testing frameworks

According to the DMA’s Marketer Email Tracker 2021, 72% of brands continue to use email as their primary channel to engage with customers and prospects. However over 50% of companies are either doing email testing but aren’t excelling at it, or are simply not doing any testing at all.

Email testing allows you to continually refine and optimise your approach to email marketing to ensure that you are doing everything you can to engage with your customers. “Testing allows you to take a data-led approach to email optimisation” and even though it may seem like a lot of work to test every single variable within an email, once it’s complete and you have all the insights into your customers preferences, you should see a big shift in results from campaigns.

What are the key principles of email testing?

When coming up with an email testing strategy, you should initially follow a set of basic principles:

• A/B testing using random selection
• Utilise control groups and/or fallow cells
• Ensure a minimum viable test cell
• Allow enough time to increase reliability of results
• Create obvious and adequate differentiation between variants
• Don’t test too much, too quickly

Let’s explain a few of these principles and terms:

What is a minimum viable test cell?

Whenever you are conducting email testing, you need to ensure that the group you are testing on is big enough to gain a level of certainty in the results it produces. This is also often referred to as sample size.

Using a simple statistical calculation we can determine the minimum % of consumers that need to be included in the A/B test groups before the winning version is sent to the remaining members. The industry standard is around 95% confidence level and there are many free tools on the internet that can be used to work out the right cell size for your campaigns.

What are control groups and fallow cells?

A control group is a selection of people who are suppressed from the specific email campaign, whilst a fallow cell is a group of customers who are withheld from receiving any communications. By utilising these, you are able to benchmark responses of your campaigns and determine their effectiveness at driving the desired behaviour/call to action.

It is important to ensure customers are selected at random to ensure fairness. When working with a fallow cell we recommend refreshing the customers selected periodically (e.g. every 3-4 months).

With commercial stretch goals it may be unpalatable to leave some customers out of a campaign – yet without the discipline of doing so you will never know if your marketing is helping to deliver your goals.

Using an email testing framework

To ensure that email testing becomes part of your regular marketing, it is key to come up with a solid framework that you can follow for each campaign. A framework is made up of the variable components or ‘key engagement levers’ that are available to you in email campaign design:

Key engagement lever #1 – Target audience

We all know the best practice of serving your audience content that is relevant and personalised to them, rather than mass marketing with messages that they can’t relate to; this applies strongly to email marketing.

When you’re deploying a campaign, the target audience should be adjusted depending on the content to ensure it is relevant. The best place to start would be to segment by demographics, such as age, gender and location to start to understand what your consumer preferences are.

By segmenting, you ensure relevance of content to you customers, helping to drive engagement and reduce the likelihood of loss of interest in your communications.

Key engagement lever #2 – Email mechanics

Email mechanics refers to the entire design and structure of an email and is where the most variation and testing can take place. Below are some of the testing levers you can vary, and different ways to do so:

• Subject line – try including sales or promotional information, emojis or personalisation eg. a person’s first name
• Call to Action – try a button vs in-text links, placement within the email, language of the CTA eg. ‘Buy Now’ vs ‘Click Here’, shape and colour
• Images – include people or just products? Include prices? Try imagery that is inspirational vs informative or even movement with the use of GIFs
• Email structure – here you can play around with elements such as tone, length of email, placement of images, copy type eg. sentences or bullet points, personalisation in the copy, personal product recommendations and more

Exploring each of these different mechanics, and identifying the optimal combination of each, should have a great effect on your open and click-through rates.

Key engagement lever #3 – Time of send

We all know that there will never be a perfect day and time to send an email, if this were the case then everyone would be doing it and inboxes would be flooded. However, through some simple A/B testing you can start to gain an insight into when your customers are most likely to open your emails.

Recent research from Campaign Monitor suggests that there isn’t too much difference between open rates on any given day, however during the working week tends to perform better, especially for the B2B sector. Monday to Wednesday had the best metrics so these would be a good place to start, however remember to always think of your audience personas and check this ties in with their lifestyle.

According to Hubspot, “in general, the highest click-to-open rates are 10am, 1pm and 6pm…which reflects when most audiences begin or conclude their day and have the most time to check their emails” however they also comment that if it is a sales or promotional email it would be good to test sending during lunch hours. Implementing a testing framework that looks into both time and day will be the best way to go to find the best time of send for your customers.

We would always recommend that engagement levers focused on increasing open rates should be prioritised over click-through rates and in-email engagement, as maximising the number of people who open will give you a larger base of customers to then test the CTR email elements described in email mechanics.

Email marketing will continue to be a large part of each companies lead generation and CRM strategies, and by partnering the correct content with the most relevant audience, finding the optimal email structure and design, and the day and time your audience are most open to receiving and interacting with communications, your business will begin to see some great results and ROI.