The G-Test is considered to be the best statistical significance test to determine if you have a statistically significant uplift in your conversion rate. Learn more about the G-Test here. You can use this free online tool to generate charts and summary reports (2nd tab) to include into your own AB test reports.
For online AB tests, a general rule of thumb is to achieve 90%+ confidence level. In other industries such as the medical field, a 95%+ threshold is more common. To ensure you account for daily seasonality, it is recommended you run your AB tests for at least a full 7-day cycle. It's also best practice to make a decision on the outcome of your AB test on a 7-day basis i.e. after the AB test has run for 7, 14, 21, 28, ... days.
If you are planning to use the revenue impact estimations, I encourage you to click here to learn more about how estimated revenue impact is calculated.
Definition
The estimated revenue impact metric is the incremental revenue we can expect from setting the winning variation at 100% of your traffic. It's important to note that this estimate is only accurate once the variation in question is set to 100%.
You should only add conversion numbers for a KPI that correlates directly with a purchase such as a Visited Checkout Thank You Page KPI. The reason for this is that the revenue is calculated by measuring Average Order Value (AOV) = revenue / conversions. Here's an example to illustrate the reason why.
Suppose we have thousands of conversions for a Click Add to Cart KPI and hundreds of actual purchases. It's obvious that clicks to that button don't always translate into purchases. This means that when we're calculating AOV and using the Click Add to Cart KPI, we'll be counting thousands of purchases instead of hundreds. There are drop-off rates at each step of the funnel, which we could potentially estimate. However, as long as your purchase KPI didn't reach statistical significance, that estimation would not be accurate. That is why it's easiest and best to measure revenue impact using a purchase-related KPI which has reached statistical significance.
Calculation
We first estimate how many conversions we expect if the winning variation was running at 100% during the experiment duration:
Estimated Winner Conversions = Total Visitors * Winning Variation's Conversion Rate
* Note, Total Visitors is the sum of all visitors across all variations.
We then estimate the number assuming the control variation was running at 100%:
Estimated Control Conversions = Total Visitors * Control Variation's Conversion Rate
We're interested in the incremental number of conversions when comparing these 2 scenarios (the winner variation running at 100%) vs (the control variation running at 100%):
Estimated Incremental Conversions = Estimated Winner Conversions - Estimated Control Conversions
Then, we want to find out what's the dollar value of these incremental conversions. To do that we estimate the Average Order Value. We calculate the one for the winning variation since we're assuming the results from this variation will carry through when set to 100%
Test AOV = Winning Variation's Revenue / Winning Variation's Conversions
* Note, Test Revenue and Test Conversions are the revenue and conversion numbers for the test variation.
So the revenue impact is measured as follows:
Revenue Impact = Estimated Incremental Conversions * Test AOV
We then normalize this figure to get daily and annual figures:
Daily Revenue Impact = Revenue Impact / Experiment Duration
Annual Revenue Impact = Daily Revenue Impact * 365
Here's the calculation for the daily number from the raw initial inputs:
Daily Revenue Impact = {Total Visitors * (Winner Conversion Rate - Control Conversion Rate) * (Test Revenue / Test Conversions)} / Experiment Duration
Placeholder Example
In the placeholders above, you'll notice that Base Visitors + Test Visitors does not equal to Total Visitors. That's because this example has 3 variations. To avoid having you to input all the numbers for each and every variation, this tool only requires you to input the total visitor number.
Here are the details behind the numbers for the placeholders. Assume Base is control, Test1 is the winning variation and Test2 is another variation.
Total Visitors = 30,000
Base Visitors = Test1 Visitors = Test2 Visitors = 10,000
Total Conversions = 1,630
Base Conversions = 500; Test1 Conversions = 570; Test2 Conversions = 560
Total Revenue = $40650
Base Revenue = $13,500; Test1 Revenue = $13,600; Test2 Revenue = $13,550
Estimated Winner Conversions = 30,000 * (570 / 10000) = 1,710
Estimated Base Conversions = 30,000 * (500 / 10,000) = 1,500
Estimated Incremental Conversions = 210
Test AOV = $13,600 / 570 = $23.86
Revenue Impact = 210 * $23.86 = $5,010.53
Experiment Duration = 14 days
Daily Revenue Impact = $5,010.53 / 14 = $358
Annual Revenue Impact = $358 * 365 = $130,632