Sample size calculator for your soft launch cohorts

As I keep on playing with Gemini (my personal favorite) and my posts on soft launch I wanted to put together a quick cohort size calculator. This will work for any binary metric (that measures something that happens or not). It won’t work for measures of intensity like LTV or playtime

https://g.co/gemini/share/17bf44994b8d

You’ll find below the math that goes with the calculator

The Math Behind the Magic: How Sample Size Calculators Work

Ever wondered if your new feature really improved your conversion rate, or if the result you’re seeing is just random noise? The key to answering that question with confidence lies in having a large enough sample size. A sample size calculator tells you exactly how many users you need to observe to trust your metrics. But how does it work?

It’s not magic; it’s math. The calculator is built on a cornerstone of statistics: Cochran’s formula for determining the sample size of a proportion.

The Core Formula

The formula can be represented like this:

      Z² * p * (1-p)
n = ------------------
            E²

Let’s deconstruct it piece by piece.

The Ingredients of the Formula

• n — Sample Size: This is the number we’re solving for. It represents the total size of the initial user group (or “cohort”) you need to measure.
• Z — The Z-score (Your Confidence): The Z-score represents how confident you want to be in your results. It’s a value from the standard normal distribution (the “bell curve”) that corresponds to a specific confidence level. It tells you how many standard deviations from the mean you need to go to capture a certain percentage of all possible outcomes. For the most common confidence levels:
  • 90% Confidence: Z = 1.645
  • 95% Confidence: Z = 1.96
  • 99% Confidence: Z = 2.576A 95% confidence level, the most common choice, means that if you were to run the same experiment 100 times, you would expect the true metric of the entire population to fall within your margin of error 95 of those times.
• p — The Estimated Proportion (Your Best Guess): This is your expected metric, expressed as a decimal. For example, a 20% retention rate becomes p = 0.20. The part of the formula p * (1-p) represents the variance of your data. This value is highest when p = 0.5 (a 50/50 chance, like a coin flip), which requires the largest sample size. This is why using p = 0.5 is the most “conservative” or “safest” estimate if you have no idea what to expect.
• E — The Margin of Error (Your Precision): This is how precise you need your measurement to be—the “plus or minus” value. A margin of error of ±3% means you’re okay with your true metric being 3 percentage points higher or lower than what you measured. This value is also expressed as a decimal (e.g., E = 0.03). Because it’s squared in the denominator, a small decrease in your margin of error (demanding higher precision) will dramatically increase the required sample size.

A Worked Example

Let’s put it all together. Imagine you want to validate an expected Day 7 retention rate of 20%. You want to be 95% confident in your result, and you’re willing to accept a margin of error of ±3%.

1. Identify your variables:
   • p (Expected Metric) = 20% = 0.20
   • Z (Confidence Level) = 95% = 1.96
   • E (Margin of Error) = ±3% = 0.03
2. Plug them into the formula: n = (1.96² * 0.20 * (1-0.20))/0.03²
3. Solve the equation:
   • Numerator: 3.8416 * 0.20 * 0.80 = 0.614656
   • Denominator: 0.03² = 0.0009
   • Result: n = 0.614656 / 0.0009 ≈ 682.95
4. Round Up:Since you can’t have 0.95 of a user, you always round up to the nearest whole number.

Therefore, you would need an initial cohort of 683 users to be 95% confident that your Day 7 retention is 20% ±3%.