Now that your experiment is over and you got the results in the Freshmarketer Reports page.What next?


To analyze and act upon the results from the reports page, a basic question has to be answered - What should I understand first?


The answer is quite simple. From the experiment, the results will either contain

  • A Winning Variation: Variation which has a statistically significant positive improvement (%) over the Original page.
  • Losing Variation(s): Variation(s) which have a statistically significant negative performance compared to the Original page.
  • Inconclusive results: When there is no statistically significant improvement (%) compared to the Original page. Or in other words, the variation(s) have not performed very differently from the Original page.


Winning Variation - What next?


Once you have the Winning Variation, the next step should be focused on implementing the winning elements on your Original page. However, this should be done after carefully studying the results. There might be situations where a particular element in a variation might have performed extremely well (with respect to a goal) but the overall variation might not have won. In such cases, care should be taken to verify if changes in the respective element caused the performance uplift. If so, these changes could also be included in the winning variation to further improve its performance.


A wise man once said, “Website optimization is like a Stairway to Heaven. You don’t reach Heaven in one jump!” Hence, start planning for new tests in the same page. From our experience, we have a few parameters in any webpage which can be considered as’ the low hanging fruits’. If you have not experimented on these items already, this would be a good time.

  • Call to Action
  • Imagery
  • Content and messaging


NOTE: If you have done the considerable number of tests on a page where you have already tested the major parameters, then it is best to focus on other pages. This would give the page considerable time to gather data and also move your focus from less impactful elements on this page to other pages requiring more ‘fine-tuning’.


Losing Variation(s) - What Next?


We hate to call this a ‘losing’ variation. Rather these variations should be called ‘Learning’ variations since they provide ample opportunities to learn.
  • Avoid future losses: Let’s say you were thinking about introducing a new segment in your current product line and want to test it out to your customers. A negative result, in this case, points out that such an introduction is not favoured by your customers. Hence you can avoid costly investments on this introduction.
  • Insights on your customer: This is something you will be able to gather after multiple tests. By analyzing such losing variations across multiple tests, you can build a list of changes which your customers do not prefer. For eg. If through multiple tests, you notice your customer do not prefer big images on the pages then you can ensure you create the next set of pages avoiding an image.
    If you have losing variations in your experiment, fret not! It is commendable that the result was statistically significant. It is only a matter of time before negative turns positive. We urge you to continue testing till you land on the perfect combination.


Inconclusive Results - What next?


If your experiment resulted ‘inconclusive’, then we recommend you make a few changes to obtain a statistically significant result. To achieve this you need two important things - Expansion of the current testing frame and Hope. There are multiple ways to ‘expand’ the testing frame


  • Test on more elements: Sometimes the variations might not be entirely different. For eg., if the difference between the Original and variation was a minor one such as button colour, it might not lead to a statistically significant result. Hence, it is better to make test additional elements on the page so that the difference between the Original and the variations are evident.
  • Track more goals: Expansion can also be achieved by tracking more goals. Instead of just tracking one goal such as ‘Engagements’ or ‘Clicks on an element’, adding more goals adds more depth to the test and provides an opportunity to collect more data which in turn increases the chances of a statistically significant result.


If you have losing variations in your experiment, fret not! It is commendable that the result was statistically significant. It is only a matter of time before negative turns positive. We urge you to continue testing till you land on the perfect combination.


Concluding Notes


Analyze, implement and continue testing on Winning variations till you reach a saturation point.
Learn from Losing variations and improvise on them to get positive results.
Expand Inconclusive tests to achieve statistically significant results.



Technical Support

Don't see what you're looking for? Just drop us a mail