Testing Email Subject Lines

A powerful subject line is what determines the success of the whole email marketing funnel. It generates opens and eventual purchases. Comparing two subject lines with an A/B test (split test) is the best way to ensure your campaigns are headed with the most effective subject line.

A/B testing can be done using two methods:

• within one email;
• with several emails on different segments.

The choice of the testing method depends on your goal:

• understand the behavior of the subscribers;
• determine effective wording for a specific audience;
• identify elements that increase the open rate (e.g. personalization, emoji, etc.).

How to Test Subject Lines Within One Email

1. Go to Messages → Messages and click the ready email or create a new one.

2. Click Message Subject Line, and enter the first subject line that will be the main one.

3. Сlick the "plus" icon at the right of the box to enter the second subject line.

You can add any number of alternative subject lines. At the time of sending, the system will randomly split the recipients into the required number of equal groups. 

How to Assess Test Results

Go to Campaigns → Reports → Email → Completed, and click the necessary email.

The test results is provided in Experiment.

The statistics includes the following rates:

• sent;
• read;
• clicked;
• click-to-open;
• unsubscribed;
• reported spam.

The system calculates the results based on special formulas and accesses their statistical validity. If the results are valid, the table shows which subject line is more or less effective.

The more contacts are included in the segments for testing, the more reliable the result is. To be able to apply the results to the whole contact base, the number of contacts for testing is to be not less than 100.

Important!

1) The number of contacts in the report may vary slightly, because some contacts are blacklisted or inactive, and are not available for sending as the time of the campaign launch.

2) Google Analytics collects statistics on the email within one campaign, with no details on separate subject lines.

To determine how much sales each subject line generated, use the second method and split contacts into groups.

How to Test Subject Lines with Several Emails on Different Segments

This A/B testing is more relevant if you need to test a certain segment. You can configure each segment and add there any number of contacts.

1. Go to Contacts → Contact Segments, and click the necessary segment. You can choose both dynamic or list segments.

2. Click Split segment.

3. In Segment splitting, specify the number of groups and percentage.

After splitting, you can see new segments in the general list.

4. Go to Messages → Messages and click the ready email or create a new one.

Important! For this type of testing, create the necessary number of emails with a duplicate copy but different subject lines. Give emails different message names or attribute with different marks for more convenient work in Google Analytics.

5. Click Create campaign, and choose the corresponding segment for each email.

How to Assess Test Results

Go to Campaigns → Reports → Email → Completed, and click the necessary email.

The report contains statistics for each subject line. 

Christmas Is Coming_test_1

Christmas Is Coming_test_2

Apart from subject lines, this method can be used to test preheaders, CTAs, etc.

Important Notes

To ensure the correct result, check the following before the start:

• Set a hypothesis - an assumption you want to prove or disprove with the test. For example, emojis in the subject line increase the open rate.
• Every test should have a purpose. For example, build an effective communication strategy with subscribers to increase conversion.
• The preheader of each email must thematically fit the corresponding subject line.
• Every detail matters. Any word, symbol, or letter can affect the results.
• One testing isn’t enough to provide valid results. What works for one segment may fail for another. Moreover, with time customer preferences may change depending on many factors.
• The lower the percentage of test participants, the higher the error rate of the results.
• Split testing is conducted to test one parameter only. Testing several elements of a single letter (e.g. themes and headers) violates the purity of the experiment.