---
name: long-horizon-holdout
description: Use when running experiments on platforms where user value compounds over time, when growth teams claim credit for revenue that would have happened anyway, or when short-term wins don't translate to long-term value
---

# The Long-Horizon Holdout Protocol

## Overview

A measurement framework that challenges short-term growth wins by maintaining **long-term control groups (holdouts)** to verify if immediate lifts translate to sustainable value over 1-3 years.

**Core principle:** 30-40% of short-term "wins" show neutral or zero impact long-term.

## The Process

```
┌─────────────────────────────────────────────────────────────────┐
│  T-0 (Launch)                                                   │
│  Split Traffic: 90% Treatment / 10% Long-term Holdout           │
├─────────────────────────────────────────────────────────────────┤
│  T+3 Weeks                                                      │
│  Initial Decision: Ship if short-term positive/neutral          │
│  Keep holdout running                                           │
├─────────────────────────────────────────────────────────────────┤
│  T+6 Months                                                     │
│  Automated Ping: Check retention and emerging GMV patterns      │
├─────────────────────────────────────────────────────────────────┤
│  T+12-18 Months                                                 │
│  Final Reckoning: Compare Cohort GMV                            │
│  If neutral/negative: deprecate feature or pivot strategy       │
└─────────────────────────────────────────────────────────────────┘
```

## Key Principles

| Principle | Description |
|-----------|-------------|
| **Long holdout** | Keep control group for 12-18 months |
| **Automated pings** | System reminds team to check at 6, 12 months |
| **Cohort GMV focus** | Long-term value, not short-term conversion |
| **Accept reversals** | Be willing to deprecate "successful" features |

## Common Mistakes

- Declaring victory after 2 weeks of significance
- Assuming neutral short-term = failure (might compound later)
- Not setting up infrastructure for long-term tracking

---

*Source: Archie Abrams (Shopify VP Product & Growth) via Lenny's Podcast*