Mark Everard

Hello, I'm Mark – a PhD physicist turned technologist / architect.

Archive for February, 2013

Multivariate content testing using EPiServer Visitor Groups

with one comment

I’ve been working closely with our performance marketing experts at POSSIBLE recently. One of the key questions is the decision about which is the right technology to use to perform multivariate content tests. There are two technical approaches:

  1. Client side – from free tools such as Google Content Experiments to paid-for services such as Optimizely. These work by injecting javascript into your pages which alter the content seen by a user. This impacts browser performance and the effective site speed as seen by the user. In the case of Google Content Experiments users are redirected to page variants defined by editors as urls within the tool, where as Optimizely and the more advanced tools manipulate the DOM post page-load to set up the specifics of your test. The paid-for services offer a lot more flexibility for editors to set up their own tests through online interfaces, however from a content management perspective this means that your testing content lives outside of your CMS system, and so becomes more difficult to manage, maintain and govern.
  2. Server side – you can of course push the logic for AB testing back to your server. There are a number of open source .NET AB testing packages available, such as FairlyCertain which allow you to set up AB tests relatively simply as a developer, but offer no control over each test to site editors. More mature CMS platforms also offer these features.

Multivariate testing in EPiServer

For EPiServer 7 (Sparrowhawk UI) – Multi-variant testing is now built into the core product. This is a great improvement and hooks into the ‘Block’s feature to allow editors to define different variations of blocks and then test and measure their performance.

Previously EPiServer offered multivariate testing through its Campaign Monitor and Optimization product which amongst other features allowed for basic A/B testing of content on a page by page basis only (i.e. you must test a whole page against a variant – these are actually set up as different versions of the same page in EPiServer).

An A/B testing Visitor Group

One of the limitations of CMO (beyond the ability to only run tests against full pages) is that all of your visitors are subjected to your test. Often (and certainly on high-traffic sites), there is already a clear segmentation and knowledge of visitor behaviour and you may only want to test particular user segments. I think the new Multi-variant testing features in EPiServer 7 should also overcome this limitation (but I haven’t played around enough yet).

So I set myself a challenge to see whether you could use the existing Personalization framework in CMS6 R2 to allow AB tests to be set only for specific visitor segments.  The answer is – yessort of. Specifically I wanted to enable the following situation:

  • Display an A/B content test on a single page to only to a single EPiServer Visitor Group

The first piece of this was to build a new Visitor Group criterion which matched for an editor set percentage of the time. The idea being that  should be used in conjunction with an existing Visitor Group.


So here – we are going to enable an AB content test only against users of the ‘Test Querystring’ visitor group. I’ve used the ‘Visitor Group Membership’ criteria to specify that this new group should be composed of the ‘Test Querystring’ group and the new randomly matching A/B testing criteria.

The second piece was the content setup. This was slightly more complicated (though not from a coding perspective). Below you will see three bits of content – all personalized and linked together in a fallback group (all standard CMS6 R2 functionality).

  1. Test content B – personalized only to match when the random A/B testing criterion matches and the user is in the ‘Test Querystring’ group
  2. Test content A – personalized only to match when the user is in the ‘Test Querystring’ group
  3. Fallback content – content displayed to a user not in the ‘Test Querystring’ group

The order the content is listed in is extremely important. The EPiServer logic to work out which content piece from the WYSIWYG editor is displayed works on a first match basis. So if you put your content in a different order (with content A above B) you would short-circuit your test.


Great – so we have a content test – how do we see the results?

Measuring any conversion should be achieved in the same way as CMO – by setting up a separate conversion page and measuring the number of hits on that page – which is outside of this scope. Limited I know 🙂

You can  use the Visitor Group statistics gadget – though all this will tell you is whether the test is set up correctly and whether the A/B testing criteria works!


The source code is available as part of the Criteria Pack on Codeplex and of course there is a Nuget package on – (as soon as its been approved!)

Written by mark

February 18th, 2013 at 10:00 am