At Freshmarketer we believe that there should be no limitations while testing. However, we do recommend some best practices while the experiment is running to ensure there are no glitches and the results are error-free.
To avoid mishaps, we have made sure changes cannot be made while the experiment is running. You will have to pause the experiment, make the changes and un-pause the experiment to continue. If you realize you have made any minor errors while creating the variations such as spelling mistakes or minor color changes you can always pause the experiment and correct them. This way we avoid any accidental changes while the experiment is running.
However, we do not recommend making any major changes while the experiment is running, such as
- Changing any influential factor in the current variation like buttons, images, headings etc.
- Deleting existing variations
- Adding new variations
- Changing traffic allocation settings
- Changing URL targeting settings
- Making changes to the audience targeting settings
Why should you not make major changes while the experiment is running?
The answer is simple and obvious - the resultant data may be inaccurate. Though there is a small gap in data between the ‘pause’ and the ‘un-pause’ duration, data is consolidated at a variation level and hence contains both the ‘before’ and ‘after’ data aggregated.
The aggregated results can either be an increase in the conversion rate or a decrease. However, it is not possible to say decisively that the change (made while the experiment was running) caused this positive/negative impact. In fact, changes in traffic allocation or audience targeting will have a much greater impact on the data than other minor ones.
Hence, if you need any major changes incorporated in the variation, as a best practice we recommend duplicating the experiment, making the necessary changes and running it as a new one. This will ensure the data is perfect and accurate.