To analyze and act upon the results from the reports page, you will first need to understand the following:
A Winning Variation: Variation which has a statistically significant improvement (%) over the original page.
Losing Variation(s): Variation(s) with a statistically significant negative performance than the original page.
Inconclusive results: When there is statistically no significant improvement (%) compared to the original page, there are inconclusive results. In other words, the variation(s) have not performed very differently from the original page.
|
Winning Variation(s)
Once you have the Winning Variation, the next step you should focus on is implementing the winning elements on your original page. However, you should perform this after carefully studying the results. There may be situations where a particular component of a Variation may have performed exceptionally well (concerning a goal), but the overall Variation may not have won. In such cases, you should verify if changes in the respective element caused a performance uplift. If so, you should include these changes in the winning variation to improve its performance further.
Start planning for new tests on the same page. We have a few parameters on any web page that can be considered ‘the low hanging fruits’ from our experience. If you have not experimented on these items already, this would be a good time.
Call to Action
Imagery
Content and messaging
Note: If you have performed a considerable number of tests on a page where you have already tested the significant parameters, it is best to focus on other pages. This way, you are giving the page considerable time to gather data and move your focus from less impactful elements to other pages requiring more fine-tuning.
Losing Variation(s)
We hate to call this a ‘losing’ variation. Instead, let’s call these variations as ‘Learning’ variations since they provide ample opportunities to learn. From a learning Variation, you can:
Avoid future negative results: Let’s say you were thinking about introducing a new segment in your current product line and want to test it out to your customers. In this case, a negative result points out that your customers do not favor such an introduction. Hence you can avoid costly investments on this introduction.
Gain insights on your customer: This is something you will be able to gather after multiple tests. By analyzing such losing variations across numerous tests, you can build a list of changes that your customers do not prefer. For example, if you notice your customer does not like big images on the pages through multiple tests, you can ensure you create the next set of pages avoiding an image.
If you have learning Variations in your experiment, continue testing till you land on the perfect combination.
Inconclusive Results
If your experiment results ‘inconclusive’, make a few changes to obtain a statistically significant result, such as:
Test on more elements: Sometimes, the variations might not be entirely different. For eg., if the difference between the Original and variation was a minor one, such as button color, it might not lead to a statistically significant result. Hence, it is better to test additional elements on the page so that the difference between the Original and the Variations is evident.
Track more goals: Instead of tracking a single purpose such as ‘Engagements’ or ‘Clicks on an element,’ adding more goals adds more depth to the test and provides an opportunity to collect more data, increasing the chances of a statistically significant result.
If you have losing Variations in your experiment, fret not! It is commendable that the result was statistically significant. It is only a matter of time before negative Variations turn positive. Continue testing till you land on the perfect combination.
Analyze, implement, and continue testing on Winning Variations till you reach a saturation point. Learn from Learning Variations and improvise on them to get positive results.
Expand Inconclusive tests to achieve statistically significant results.