long-term-growth-impact-assessment
The Long-Term Growth Impact Assessment is a framework for ensuring growth experiments drive actual business value (like GMV or Gross Profit) rather than just "pulling forward" results or optimizing local conversion ratios. By using long-term holdouts and absolute metrics, you can identify which "wins" are illusory and which "neutral" experiments are actually long-term powerhouses.
Core Principles
1. Optimize for Absolute Volume, Not Ratios
Avoid using conversion rates as a primary North Star. To increase a conversion rate, a team can implicitly make the previous step harder (e.g., adding friction to sign-up to ensure only "high-intent" users reach the next step). This makes the rate look better but reduces the total number of successful users.
- The Rule: Measure the absolute number of users who reach a "success" milestone.
- The Mindset: It is often better to have a lower conversion rate on a much larger pool of users than a high rate on a tiny, restricted pool.
2. Implement the "Long-Term Ping" Holdout
Don't stop measuring an experiment once it reaches statistical significance.
- The Mechanism: When an experiment concludes, ship the winner to 100% of users, but maintain the metadata tags for the original treatment and control cohorts.
- The Cadence: Set automated reporting "pings" to re-evaluate the cohort's performance at 3, 6, 12, and 24 months.
- The Metric: Measure downstream business value (e.g., Total Revenue, GMV, or Net Profit) rather than the local metric the experiment was designed to move.
3. Account for the "Pull-Forward" Effect
Recognize that 30-40% of short-term growth wins are neutral after one year. Often, an experiment simply accelerates an action a user would have taken anyway. If the long-term holdout shows the control group eventually "catches up" to the treatment group, the experiment had no incremental value.