Testing frequency modelling

See also: Frequency modelling overview

It's common for retailers to maintain a consistent weekly email frequency, e.g. four emails per week.

Because Ometria only has your current email frequency, our first predictions will be less accurate than if we had data on different email frequencies for your account, e.g. four emails per week vs. six emails per week. 

This is why we strongly recommend testing different frequencies based on our initial predictions. 

Over time, Ometria's frequency modelling tool will learn and develop even better frequency recommendations for you.

How to test email frequency

To achieve incrementally better results, we recommend conducting A/B testing to evaluate the effectiveness of different email frequencies. 

The goal is to determine whether the frequency that Ometria is proposing will result in increased revenue and user engagement compared to your existing frequency. 

We will be using metrics such as click-through rates and unsubscribe rates to evaluate engagement.

Testing plan

Campaign type: Broadcast

Length of test: Ideally eight weeks, but four weeks at a minimum.

There is no ideal time to begin testing, but you might find it simpler to begin your test at the beginning or end of a month. 

Create your control groups 

Split your contact base by the percentage recommended below:

See also: Split segments

Give your control group a name that will be easy to remember and identify later. 

In this example, the active contacts segment has 1 million contacts, so we've applied a 5% split, giving us a control group of 50,000 contacts.

Set up your broadcast campaign

Create a broadcast campaign and create your segments.

See also: Segmenting your broadcast recipients

In our example, the campaign segments are based on average order value (AOV).

Here's an example of a control group filter, which includes high AOV contacts who are also in the control group segment:

When setting up segments, alternate your control groups with your non-control groups. 

Your first segment should be a control group so that the segment 'catches' all of the high AOV contacts who are in your test segment, and they're excluded from the non-control group segment:

Assign templates to your variants

Each segment creates a variant which you can assign a different template to in the Content tab of the broadcast builder.

In this example we've assigned the same template to the 'High value - control group' and 'High value' segments, the same template to the 'Medium value - control group' and 'Medium value' segments, and so on.

Send your campaign

Schedule and send your broadcast campaign and create further campaigns with the same segmentation.

Schedule these to send at your usual weekly frequency.

In this example, we're testing a recommended frequency of three emails per week against a current frequency of four emails per week. Adjust your testing plan accordingly.

On the day of the fourth send (in this example), come back to the campaign builder and exclude the control group from receiving the final email:

Review your results

Using the broadcast campaign performance dashboard, track and compare the following metrics for each of your control groups:

• Click-through rate
• Conversion rate
• Unsubscribe rate
• Revenue per email 
• AOV 

We've created a spreadsheet with all the formulas you need to monitor your test results: