Establish a single hypothesis
Instead of "new page is better", write a testable proposal such as "explaining delivery coverage in front of the form can reduce unqualified application." Define the measure of success and adverse effect together. More forms should not be considered success if the entire application is unrelated.
Maintain comparison
If you send different users from different campaigns to variants, it becomes difficult to separate the design effect. User assignment, repeat visits and measurement events should be checked in the experiment installation. The duration and amount of data required depends on the expected effect by traffic; there is no valid fixed day or number of visitors for all sites.
Do not interpret the result in haste
Stopping the experiment at the first positive surge increases the risk of misjudgment. Follow the pre-determined assessment plan and report the uncertain result as unclear. User interview, task test and sales team feedback on low-traffic B2B sites can provide more useful tips before the experiment. Do not present them as an A/B test result.
Check before you start
- Does the hypothesis depend on a single change?
- Is the measurement confirmed?
- Is there a decision plan for the uncertain outcome?