‘What works in e-commerce - a meta-analysis...' reviewed

experiment

E-commerce has not been a dull place for a quant or for anyone who takes above average interest in performance and measurable outcomes. The variety of number of levers to pull or buttons to move is almost infinite and is going to keep every graduate who is numerate employed for years to come. On top of testing conversion optimization for various design issues, technology opens up possibilities for segmentation and personalization of content and interactions. So, in which areas can you expect changes to result in substantial uplift of Revenue per visitor (RPV)? 

Will Browne and Mike Swarbrick Jones recently published their research in which they categorized thousands of online experiments into 29 categories and their effect on RPV. This review summarizes their key findings, discusses implications and adds practical perspectives on what you can do as e-commerce manager to improve on key indicators. For more background on RPV in relationship to other online KPIs, there is this blog. 

Scarcity works, call-to-action does not 

So let’s start with what works. The test categories that have most impact on RPV are scarcity (stock pointers) +2.9% uplift, social proof (informing users of others’ behaviour) +2.3% uplift and urgency (countdown timers) +1.5% uplift. These outcomes should not come as a surprise as the psychology behind people’s reaction to stimuli in these categories are well described in Cialdini’s bestseller Influence: The Psychology of Persuasion (1981). 

Noteworthy is that UI changes that conversion specialist often focus on, hardly move the needle or are even negative: colour (changing the colour of elements on a website) +0.0% uplift, buttons (modifying website buttons) -0.2% uplift, calls to action (changing the wording on a website to be more suggestive) -0.3% uplift. Perhaps the research gives cause to cast a critical eye on what items to prioritize for A/B testing….. 

Now for the real gems (and the reward for people who read this far). In this 21-page publication there are two important limitations to consider: 1. The researchers only looked at the immediate uplift of the experiment, possible site-wide uplift in revenue as a result of an overall better user experience has not been measured. 2. The experiments only considered the ‘what’, not the ‘how’. Researchers only looked at the categories of experiments and their results, not at how well experiments have been designed and executed. Why is this important? Consider table A.3. Revenue per converter, or more commonly known as average order value (AOV). 

x

The experiments that are most likely to increase order value and have the highest impact on average order value are upselling, social proof and product recommendations. The challenge is to preserve these powerful stimuli without deteriorating conversion, as that would reduce overall impact on RPV. Indeed, we have seen cases in which recommendations that are personalized and context sensitive, e.g. made to fit the visitor’s behavior in the current session or to fit the items that are already in the shopping basket to work very well. As RPV = AOV x CVR (conversion), ensuring good process design keeps conversion constant so that the higher average order value can result in higher revenue per visitor. 

Furthermore, personalization and context information, such as weather (forecast), time of day, type of device, can also be deployed to automate the display of information such as scarcity. When weather is expected to be warm and sunny, which relates to high beachwear sales, you may want to display ‘last two items in stock’ for bikinis but not for woolen sweaters. Both cases underscore how important it is to get the ‘how’ right. Below is the complete picture of how individual KPIs and ROI are impacted by the various categories of experiments. 

roi

SPARQUE provides companies with the technology to reach out to each user at various touchpoints in the customer journey. There are many other examples how mobilizing data can help you to anticipate needs of visitors and interact with them meaningfully, as if your platform was a person.