pilot-icon – This feature is currently on Pilot release. Please note that it does not currently support:

  • Email campaigns sent via Automation Center programs.
  • Email campaigns that use A/B Testing versions.

Emarsys Value Control is a control group testing feature that measures the effectiveness of your email campaigns. It does this by preventing a percentage of the launch list from receiving the campaign (the control group) and comparing their behavior over a predefined time period against the rest of the launch list who did receive the campaign (the test group).

Value Control presents the revenue and engagement figures of the test groups side-by-side with those of the control group, so you can see exactly what influence your campaign has had on their behavior. In this way you can see instantly if your strategy is successful or if you should look for ways to improve it.

Note: This feature is only available for Smart Insight users (at no extra cost), and can be used for template-based or custom-HTML batch email campaigns.

Contents

The results of the trial period for each campaign can be found in the Analysis menu:

How does Value Control Work?

Value Control takes as its baseline what the contacts in the launch list would have done anyway during the trial period. This is the behavior of the control group, and is measured in four metrics:

  • Purchase rate (% of contacts who made a purchase)
  • Average revenue per purchase
  • Average revenue per person
  • Engagement rate (% of contacts who visited the website)

It separates these figures for the test and control groups and compares them side by side:

reporting-value-control-metrics

Working with Value Control

If you are a Smart Insight customer, Value Control is automatically available for you in the email campaign creation pages.

For template-based emails it is in the toolbar:

cms-value-control

For custom-HTML emails it is in the wizard:

html-editor-value-control

Turn on Value Control for a campaign by simply activating the check box. You are then presented with these controls:

cms-value control-active

  • Select how much of the launch list should be used for the Control group. These will be contacts selected at random from the email launch list and will not receive the email.
  • Select how many days you want the evaluation period to run.
  • Decide whether or not the control group should also receive a copy of the campaign at the end of the trial period. This copy will be created immediately after the original campaign is scheduled and and will be scheduled for launch at the end of the evaluation period.

Note: If you do schedule a copy of the campaign for the control group, but decide that you want to edit or cancel it (for example if the results are not as expected), you can find this campaign in the Emails list and unschedule it at any time during the evaluation period, and make your desired changes.

Once you have saved your settings, Value Control is active and will randomly select the test and control groups when the campaign is launched. Please note that this means the Original target figure on the Scheduling will not take Value Control into account and will still show the full launch list. However, the Original launch list figure on the Results Summary page for this email will show the true number of contacts who received the email (i.e. only the test group).

You can then measure the added value the campaign has brought you on the Value Control reporting page.

When & how to use Value Control

Value Control can be used on any batch email campaign. We recommend using for as many campaigns as you can, as it not only measures the additional value of that campaign, but the control groups also give you continuous insights in to the regular behavior of your different contact segments.

Note: The contacts in a control group will be omitted only from that particular campaign and not from any other campaign unless specified. If you would like to exclude control group contacts from other campaigns during the test period, you will have to use the Save as contact list button to save them as a contact list, and then exclude that list manually from the relevant campaigns.

Here are a few other points to help you make the most of Value Control.

  • This is not A/B Testing
    Value Control is not about responses, it is about value. You can use it to validate even the most basic assumptions, such as whether discounts actually trigger additional purchases or whether the same number of purchases would have happened anyway.
  • Keep the control group large enough to be significant
    It is tempting to make the control group as small as possible, but this reduces the validity of the information you receive from the trial. We recommend to start with a 50/50 split and move from that only for fine-tuning campaigns.
  • Don’t confuse this with revenue attribution
    Revenue attribution links purchases to email campaigns if the purchases happened x days after the click (according to your attribution window). While this is important, it does not show you how many people received the campaign and made a purchase afterwords without clicking – by going directly to the website, for example. Furthermore, some people would have made a purchase anyway and only clicked the campaign because it was there. Value Control is designed specifically to exclude these scenarios.