Sample Ratio Mismatch

Math of Sample Ratio Mismatch (SRM)

Sample Ratio Mismatch (SRM) is a diagnostic test used in Randomized Controlled Trials (RCTs) to detect if the actual distribution of participants between variants (e.g., Control and Treatment) significantly deviates from the intended design. It uses a Pearson’s Chi-square Goodness-of-Fit test.

1. Setup

• Variants: $k$ experimental groups (e.g., $k=2$ for Control and Treatment).
• Observed Counts ($O_i$): The actual number of users assigned to variant $i$.
• Total Sample Size ($N$): The sum of all observed counts, $N = \sum_{i=1}^k O_i$.
• Target Allocation ($p_i$): The intended probability of assignment for variant $i$ (e.g., $0.5$ for a 50/50 split).

2. Expected Counts ($E_i$)

The number of users we expected to see in each variant if the randomization worked perfectly: $E_i = N \times p_i$

3. Chi-square Statistic ($\chi^2$)

We calculate the cumulative squared deviation between observed and expected counts, normalized by the expected counts: $\chi^2 = \sum_{i=1}^k \frac{(O_i - E_i)^2}{E_i}$

4. Hypothesis Testing

• Null Hypothesis ($H_0$): The observed counts follow the target distribution (no mismatch).
• Degrees of Freedom ($df$): $df = k - 1$.
• P-value: The probability of observing a $\chi^2$ statistic as extreme as the one calculated, assuming $H_0$ is true. It is derived from the Chi-square distribution: $p\text{-value} = P(\chi^2_{df} > \chi^2_{calculated})$
• Decision: If $p\text{-value} < \alpha$ (where $\alpha$ is a conservative threshold like $0.001$), we reject $H_0$ and flag an SRM.

Example (50/50 split)

If you intended to split 1,000 users evenly but observed 450 in Control and 550 in Treatment:

1. $E_{control} = 1000 \times 0.5 = 500$
2. $E_{treatment} = 1000 \times 0.5 = 500$
3. $\chi^2 = \frac{(450-500)^2}{500} + \frac{(550-500)^2}{500} = \frac{2500}{500} + \frac{2500}{500} = 5 + 5 = 10$
4. With $df=1$, a $\chi^2$ of 10 gives $p\text{-value} \approx 0.0015$, indicating a likely mismatch.

Real Example

Using without CausalData