Uncofoundedness

We call 'Uncofoundedness' a scenario where a treatment is not randomly assigned to participants, so confounders effect on treatment assignment and outcome. We have client - level data. Confounders were measured before treatment and outcome after

Data

Let's look at the example:

In our ecosystem we have a product, which effect on LTV we want to estimate

Treatment - first purchase in product.

Outcome - LTV after first purchase.

We will test hypothesis:

$H_o$ - There is no difference in LTV between treatment and control groups.

$H_a$ - There is a difference in LTV between treatment and control groups.

We will use DGP from Causalis. Read more at https://causalis.causalcraft.com/articles/generate_obs_hte_26_rich

Our data has strong treatment class disbalance. Only 5% of sample activated in treatment.

Treatment group has lower mean LTV. It's too early to draw conclusions.

We see large right tale

We see many outliers. It's common situation for LTV metric. Dropping them will lead to a biased conclusion

As we see clients are differ on this confounders. We need to controll them to make causal inference

Inference

ATTE is right estimand here. We will estimate effect on clients that were treated, had first purchase in our product

Math Explanation of the IRM Model and ATTE Estimand

The Interactive Regression Model (IRM) is a flexible framework used in Double Machine Learning (DML) to estimate treatment effects. Unlike linear models, it allows the treatment effect to vary with confounders $X$ (interaction) and makes no parametric assumptions about the functional forms of the outcomes.

We write $W=(Y,D,X)$ for an observation, where $D\in\{0,1\}$ is treatment and $Y$ is the observed outcome.

1. Nuisance Functions

The IRM framework relies on three "nuisance" components estimated from the data:

• Outcome Regression (Control): $g_0(X) = \mathbb{E}[Y | X, D=0]$
• Outcome Regression (Treated): $g_1(X) = \mathbb{E}[Y | X, D=1]$
• Propensity Score: $m(X) = \mathbb{P}(D=1 | X)$

Let $p = \mathbb{P}(D=1) = \mathbb{E}[D]$ denote the overall treatment rate (estimated by the sample mean of $D$).

In the provided implementation (irm.py), these are estimated using cross-fitting (splitting data into folds) to avoid overfitting bias.

2. ATTE (Average Treatment Effect on the Treated)

The Average Treatment Effect on the Treated (ATTE) measures the impact of the treatment specifically on those individuals who received it: $\theta_{ATTE} = \mathbb{E}[Y(1) - Y(0) \mid D=1]$

Under unconfoundedness, $(Y(1),Y(0)) \perp D \mid X$, and overlap $0 < m(X) < 1$, this is identified from observed data.

3. The Orthogonal Score

DML uses a Neyman-orthogonal score $\psi$ to ensure the estimator is robust to small errors in the nuisance function estimates. The score for ATTE is defined as: $\psi(W; \theta, \eta) = \psi_b(W; \eta) + \psi_a(W; \eta)\theta$

To match the implementation in irm.py, define:

• Residuals: $u_0 = Y - g_0(X)$, $u_1 = Y - g_1(X)$
• IPW terms: $h_1 = \frac{D}{m(X)}$, $h_0 = \frac{1-D}{1-m(X)}$
• Weights (ATTE): $w = \frac{D}{p}$ and $\bar{w} = \frac{m(X)}{p}$ (the normalized form with $\mathbb{E}[w]=1$)

Then:

(If normalize_ipw=True, the code rescales $h_1$ and $h_0$ to have mean 1.)

4. Final Estimation (Step-by-step simplification)

For brevity, write $m = m(X)$, $g_0 = g_0(X)$, and $g_1 = g_1(X)$. Plug in $w, \bar{w}, h_1, h_0$:

So the $g_1(X)$ terms cancel, and the ATTE score depends only on $g_0(X)$ and $m(X)$. The estimator solves $\mathbb{E}[\psi(W;\theta,\eta)]=0$:

Equivalently, $\hat{\theta}_{ATTE} = \mathbb{E}\left[\frac{D}{p}(Y-g_0(X)) - \frac{m(X)}{p}\frac{1-D}{1-m(X)}(Y-g_0(X))\right].$

Our estimate is 12.1542 dollars (ci_abs: 7.7933, 16.5152). Mean in treatment group is 58.5062 dollars, so without our product it would be 46.3520 dollars.

Refutation

Unconfoundedness

balance_max_smd is 0.011635 so dml specification dealt with controlling confounders

Sensitivity

Even if we have latent confounder as strong as 'tenure_months' our estimate will be > 0 with bias_aware_ci': (4.646164852659804, 18.5265291001404)

SUTVA

SUTVA is untestable from data alone, so we call it true by design

Score

DML is specified correctly. There are many outliers in data that effect the score

Overlap

Customers are not inclined to activate the our product

if calib_ECE is YELLOW/RED look at plot_propensity_reliability(dml_result)

Logistic recalibration trend is a fitted correction curve for your predicted propensities.

The plot takes your original propensity p and fits this model:

corrected_p = sigmoid(alpha + beta * logit(p)) So the orange line answers:

“If I recalibrate these predicted probabilities to better match observed treatment frequencies, what corrected probability would I use?”

How to read it:

• If orange line matches the diagonal: predictions are already well calibrated
• If orange line is below the diagonal: model tends to overpredict treatment
• If orange line is above the diagonal: model tends to underpredict treatment

Rule of thumb:

• ECE near 0 and slope near 1: great
• ECE near 0 but slope far from 1: average calibration is fine, but probabilities are distorted in shape
• Many tiny bad bins + one huge good bin: visually noisy, but often not a major practical issue

Conclusion

First purchase in our product is increasing LTV 11.7542 (ci_abs: 7.1434, 16.3651) dollars. Model is specified correctly and there is no evidence that assumptions are false