Every monetization team knows the feeling: you are confident that a waterfall change will improve revenue, but the moment you push it live, you hold your breath. What if it backfires? What if fill rates drop? What if you just cost your company thousands of dollars in the time it takes to notice and revert?
This fear is not irrational — it is the reason most publishers leave their waterfall configurations untouched for months at a time, leaving significant revenue on the table. The solution is not to stop making changes. It is to test them properly before committing.
Why Waterfall Testing Is Different
A/B testing a waterfall is not like testing a button color or onboarding flow. Ad revenue is inherently noisy — it fluctuates by hour, day of week, season, and dozens of other factors. A change that looks like a 10 percent improvement on Monday might be entirely explained by normal weekly variation. And unlike product A/B tests where a bad variant causes a slightly worse user experience, a bad waterfall variant can mean thousands of dollars in lost revenue per day.
The Traffic Split Approach
The safest way to test waterfall changes is to split your traffic between the current configuration (control) and the proposed change (variant). Most mediation platforms — including AppLovin MAX and Unity LevelPlay — support traffic segmentation that lets you route a percentage of users to a different waterfall configuration.
How to Set Up a Clean Test
Start with a 90/10 split: 90 percent of traffic continues on your current waterfall, and 10 percent gets the new configuration. This limits your downside risk to 10 percent of traffic while giving you enough data to detect meaningful differences. Run the test for at least seven days to capture weekly cyclicality in ad demand.
What to Measure
Do not just measure eCPM. Track these metrics for both groups: total revenue per thousand daily active users (revenue per mille DAU), fill rate, average eCPM, impressions per session, and — critically — user retention. A waterfall change that lifts eCPM by 15 percent but increases ad load time and drops 7-day retention by 2 percent is a net negative.
The Holdout Group Method
For more significant changes — like adding or removing a demand source, or restructuring your entire waterfall — use a holdout group. Keep 20 percent of your traffic on the old configuration permanently (or for the duration of the test) and roll the new configuration to the remaining 80 percent. This gives you a persistent baseline to compare against, which is especially valuable for changes whose impact may take weeks to fully materialize.
Incremental Rollouts
Once a test shows positive results at 10 percent, do not immediately push to 100 percent. Increase to 25 percent for another few days, then 50, then 75, then 100. Each step gives you a checkpoint to verify that the improvement holds at higher traffic volumes and to catch any issues that only surface at scale — like a demand partner that performs well at low volume but cannot maintain fill rate when given more traffic.
The publishers who consistently grow their ad revenue are not the ones who make the boldest changes — they are the ones who test every change methodically and only commit to winners. Small, validated improvements compound into massive gains over time.
Common Testing Mistakes
Testing Too Many Variables at Once
Change one thing per test. If you simultaneously adjust floor prices, add a new demand source, and rearrange waterfall priority, you cannot attribute the result to any single change. Isolate variables.
Ending Tests Too Early
Ad revenue has significant day-of-week effects. A test that runs Monday through Wednesday will give you a different picture than one that includes a full weekend. Always run tests for at least seven full days, ideally fourteen.
Ignoring Statistical Significance
A 5 percent revenue improvement on a small traffic segment might be noise. Before declaring a winner, ensure the difference is statistically significant — most mediation platforms provide confidence intervals, or you can use standard statistical tools to verify.
Automating the Process
A managed monetization partner can run continuous waterfall tests on your behalf, using automated systems that split traffic, measure results, and promote winning configurations — all without requiring your engineering team to set up and manage test infrastructure. This turns waterfall optimization from an occasional manual process into a continuous improvement engine.