Plan power first
Sample-size planning happened before generating the simulated experiment data.
Data Analytics / BI · Experimentation · Case study
A promising lift.
A promising lift.
A narrower decision.
A simulated checkout A/B test where a positive aggregate result did not survive correction across multiple tests. The robust signal was on mobile, with a page-speed guardrail to monitor.
RLRevenueLift — A/B Test Decision Platform cover
+35.6%Mobile relative lift
0.003Mobile adjusted p-value
10 testsMultiple-testing correction
From experiment to rollout decision
The analysis moves from test design to a scoped recommendation.
1
Design
Define the test and the metric
2
Power
Check the sample can detect the effect
3
Test
Significance on each comparison
4
Correct
Adjust for multiple testing
5
Decide
Roll out to mobile, test desktop apart
The problem
A raw p-value can push the wrong rollout.
A simulated e-commerce checkout redesign appeared to improve aggregate conversion with a raw p-value of 0.015. But the project evaluated ten significance tests. After Benjamini–Hochberg correction, the aggregate adjusted p-value was 0.050, while a mobile segment still showed a robust effect. The decision had to account for both statistical uncertainty and guardrail regressions.
Architecture
From experiment design to a scoped rollout.
The diagram shows how data moves through the project.
Power analysisPlan sample size
DuckDBSessions + orders
Python testsSegments + guardrails
BH correctionTen tests
Power BIBusiness decision
Python uses pandas, SciPy and statsmodels for analysis; DuckDB stores simulated sessions and orders. Power BI presents the resulting decision and guardrails.
Key decisions
Why I built it this way.
Correct for multiple tests
Benjamini–Hochberg adjustment changed the interpretation of the aggregate result.
Check guardrails and segments
Refunds, tickets and page load time are assessed alongside conversion, with explicit device and user-type segmentation.
Results
Ship with caveats: mobile only.
Scope and measurement matter as much as the headline numbers.
+35.6% mobile
Relative conversion lift in the mobile segment; adjusted p=0.003.
Aggregate near-miss
Raw p=0.015, adjusted p=0.050 across the ten tests in this analysis.
+2.9% load time
The new flow significantly worsened page load time, p < 0.0001, and needs monitoring.
Projected impact: The analysis estimates approximately $857,401 in additional annual revenue for a mobile-only rollout. It is an assumption-based projection from a simulated experiment, not realized revenue. The recommendation is to roll out to mobile, test desktop separately, and monitor speed.
What broke
What I learned while building it.
Aggregate result changed
Multiple-testing correction turned a tempting headline win into a statistical near-miss. The analysis changed the recommendation rather than hiding the correction.
Guardrail regression
Page load time worsened under treatment; the decision report calls for monitoring instead of focusing only on conversion.
Desktop uncertainty
The mobile effect should not be applied to desktop. A separate desktop test is the next measurement step.
Limitations
What this project does not claim.
This is a simulated experiment. The revenue
projection depends on documented assumptions, and no live customer
behavior or realized revenue is claimed. Adjusted p=0.050 is a
boundary result whose interpretation depends on the pre-registered
criterion. Desktop needs an independent test, and mobile page-speed
regression needs monitoring.
Explore the work
Code, report and next steps.
The repository includes analysis scripts, methodology documents, tests, a Power BI report and screenshots. The Power BI report is shared as screenshots and a .pbix file in the repository.