CUPED

We call 'Controlled-experiment Using Pre-Experiment Data' (CUPED) a scenario where a treatment is randomly assigned to participants, and we have pre-experiment data of participants like pre-treatment outcome.

Treatment - new product category for users.

We will test hypothesis:

$H_o$ - There is no difference in LTV between treatment and control groups.

$H_a$ - There is a difference in LTV between treatment and control groups.

Data

We will use DGP from Causalis. Read more at https://causalis.causalcraft.com/articles/make_cuped_tweedie_26

EDA

We see heavy tale distribution

Monitoring

Some system is randomly splitting users. Half must have new onboarding, other half has not. We should monitor the split with SRM test. Read more at https://causalis.causalcraft.com/articles/srm

Now let's check the balace of confounders

• SRM is good
• SMD < 0.1 and ks_pvalue > 0.05

Split is random. Uncofoundedness is true

Inference

Math of CUPEDModel

The CUPEDModel implements the Lin (2013) "interacted adjustment" for ATE (Average Treatment Effect) estimation in randomized controlled trials (RCTs). This method is a robust version of ANCOVA that remains valid even when the treatment effect is heterogeneous with respect to the covariates.

1. Specification

The model fits an Ordinary Least Squares (OLS) regression of the outcome $Y$ on the treatment indicator $D$ and centered pre-treatment covariates $X^c$. The specification includes full interactions between the treatment and the centered covariates:

$Y_i = \alpha + \tau D_i + \beta^T X_i^c + \gamma^T (D_i \cdot X_i^c) + \epsilon_i$

Where:

• $Y_i$: Outcome for individual $i$.
• $D_i$: Binary treatment indicator ($D_i \in \{0, 1\}$).
• $X_i$: Vector of pre-treatment covariates.
• $X_i^c = X_i - \bar{X}$: Centered covariates (where $\bar{X}$ is the sample mean).
• $\alpha$: Intercept (represents the mean outcome of the control group when $X = \bar{X}$).
• $\tau$: Average Treatment Effect (ATE) or Intent-to-Treat (ITT) effect.
• $\beta$: Vector of coefficients for the main effects of the covariates.
• $\gamma$: Vector of coefficients for the interaction terms between treatment and covariates.
• $\epsilon_i$: Residual error term.

2. Why Centering and Interaction?

• Centering ($X^c$): By centering the covariates, the coefficient $\tau$ directly represents the ATE at the average value of the covariates.
• Interactions ($D \cdot X^c$): Including interactions (as proposed by Lin, 2013) ensures that $\tau$ is a consistent estimator of the population ATE even if the true treatment effect varies with $X$ (heterogeneity). In traditional ANCOVA without interactions, the estimator for $\tau$ can be biased or less efficient under heterogeneity.

3. Inference and Variance Reduction

The model uses robust covariance estimators (defaulting to HC3) to calculate standard errors, which accounts for potential heteroscedasticity:

• Standard Error ($SE$): Derived from the robust covariance matrix of the OLS fit.
• Variance Reduction: The main goal of CUPED is to reduce the variance of the ATE estimate by "soaking up" explainable variation in $Y$ using pre-treatment data $X$. The variance reduction percentage is calculated by comparing the variance of the adjusted model to a "naive" model ($Y \sim 1 + D$): $\text{Variance Reduction \%} = 1 - \frac{Var(\hat{\tau}_{adjusted})}{Var(\hat{\tau}_{naive})}$

4. Absolute and Relative Confidence Intervals

Absolute Confidence Interval The $1-\alpha$ confidence interval for the ATE $\hat{\tau}$ is: $CI_{abs} = [\hat{\tau} - z_{crit} \cdot SE(\hat{\tau}), \hat{\tau} + z_{crit} \cdot SE(\hat{\tau})]$ Where $z_{crit}$ is the critical value for the significance level $\alpha$.

Relative Effect The relative effect is the ATE expressed as a percentage of the control group mean: $\tau_{rel} = 100\% \cdot \frac{\hat{\tau}}{\bar{Y}_C}$ where $\bar{Y}_C$ is the sample mean of the control group.

Relative Confidence Interval (Delta Method) The variance of the relative effect is estimated using the Delta Method. Assuming $\hat{\tau}$ and $\bar{Y}_C$ are approximately independent, the variance is: $Var(\tau_{rel}) \approx \left( \frac{100}{\bar{Y}_C} \right)^2 Var(\hat{\tau}) + \left( \frac{-100 \cdot \hat{\tau}}{\bar{Y}_C^2} \right)^2 Var(\bar{Y}_C)$ The confidence interval for the relative effect is then: $CI_{rel} = [\tau_{rel} - z_{crit} \cdot \sqrt{Var(\tau_{rel})}, \tau_{rel} + z_{crit} \cdot \sqrt{Var(\tau_{rel})}]$

Refutation

Unconfoundedness

True by design. Proxy tested with SRM and balance check in Monitoring section

SUTVA

Overlap

Overlap is true by design

Regression specification