A/B testing is one of my favorite disciplines in email marketing. It blends scientific testing structures, analytical skills, creativity, and even education (when clients have misconceptions we need to clear up).

With smart strategy and precise execution, A/B testing can be the difference between good email sends and great email campaigns that drive revenue over the long term. Getting from point A to point B is a journey we’ve undertaken with many clients over the years, and I’m going to lay out an overview to help you chart your own course.

In this blog, I’ll cover:

• How to set expectations for your A/B testing efforts

• The elements in focus for comprehensive A/B email testing

• How to store and reference insights to build on your A/B tests

• Common mistakes to avoid in A/B testing initiatives

Let’s jump in.

The right expectations for A/B email tests

You might have a killer idea you want to test, but it’s rare that A/B testing produces one-off blockbusters (remember, even if one option is great, that means the other half of the audience will under-perform in comparison). What you should expect from consistent A/B testing is a solid lift over time – if you’re testing effectively and gathering information to optimize future sends.

The other thing about expectations is that A/B testing tends to defy them. For one client, we tested email versions where one block had the hero above the copy as normal, and another had body copy above the hero. We tested, and surprisingly, the version with the hero below the copy drove higher engagement.

Whatever the testing outcome, remember that that the biggest impact is the learnings that you can apply in future.

The elements in focus for comprehensive A/B email testing

Everyone knows you can A/B test subject lines, but that’s the beginning of a long list of options.

If you’re looking to increase open rate, I recommend testing (along with subject lines) different days of the week and times of day to figure out when your segment or list will be the most inclined to open.

If you’re looking to test impact on click rate or conversion, you can and should test copy elements: body copy headline copy, and CTA to start. I also like to test background colors and the order of the email’s building blocks – and even whether adding or omitting blocks moves the needle.

Image type is another one – if you have products, do you want a lifestyle shot or a product shot? Or maybe a gif?

Personalization is another element we like to test. It seems intuitive that personalization will improve engagement, but for one client, we used personalization in the CTA (calling out the user’s location), and the non-personalized CTA drove far more engagement.

How to store and reference insights to build on your A/B tests

There are two elements in play here: reporting and archiving.

In terms of reporting, I personally like Looker or just a good old Excel pivot table. The key is to break the insights out by variables to get specific insights. Choose a tool you’re comfortable with and can manipulate to see metrics by variation.

The archiving system is where you store the good stuff: all the variables you’ve tested or want to test; winners and losers; insights, etc. My preferred way is to do a testing repository in combination with a reporting dashboard. The repository can be a Word doc, Excel sheet, or LucidChart where you can add visuals and notes and collaborate with team members.

Depending on your ESP, you may have the ability to add campaign tags, which I highly recommend for segmentation of initiatives like a newsletter or promotion or product announcement. This allows more precision with the test findings and will help you recognize trends more quickly – e.g. every time you send a promo, you get more clicks on block X.

Common mistakes to avoid in A/B email testing initiatives

There are some frequent mistakes we work with clients to correct when they dig into A/B testing.

The first is testing two things at once –essentially, trying to do too much at once. Instead, stick to one variable, and really understand what you’re trying to learn. For instance, subject line tests should be super-focused: does a question work better than a statement? Does adding an emoji help?

The second, which I hinted at above, is tunnel vision on subject line tests. That’s actually the last test I recommend because, for one thing, privacy laws make it hard to measure the real impact on open rate.

Another common mistake: not splitting the audience 50/50 for a true test (it’ll take longer to find statistical significance if you employ holdouts or other measures that skew the data). Some clients want to test the audience and then hold out the remaining and do iterative tests (e.g. test 10%, test another 10%, and send the winner to the last 80% in a few hours). The biggest issue with that is that you’re getting data on only 20% of the audience, and you’re only letting the test mature for a few hours, which isn’t enough time – sometimes the results develop over the course of the week. When you mix up variables (like time of day), it also gets a little more difficult to know what works and why.

The last and most consequential mistake: not taking notes when things fail. A failure is a win if it provides knowledge and a reference point for what not to do in future sends.

Wrapping up

The brands that build the steadiest, most predictably effective email marketing campaigns are those that approach A/B testing not as a silver bullet but as an important source of insights and performance gains that become clear over time.

As I said at the beginning, we’ve spent 20 years guiding brands through this process. If you sense you’re leaving performance on the table with your email campaigns but can’t put a finger on why, A/B testing should be a top priority.

Interested in learning more about our process and getting a peek under the hood of your ESP? Drop us a line at info@dmipartners.com.