RCT Benchmark: t-test vs Linear Regression vs DML ATE

Generate RCT data

We'll generate a balanced (50/50) RCT with a continuous outcome where the treated group's mean exceeds control by 0.5 units.

	y	t	x1	x2	x3	x4	x5
0	0.302852	0.0	0.304717	-1.039984	0.750451	0.940565	-1.951035
1	-0.942206	1.0	-1.302180	0.127840	-0.316243	-0.016801	-0.853044
2	0.910524	1.0	0.879398	0.777792	0.066031	1.127241	0.467509
3	2.332408	1.0	-0.859292	0.368751	-0.958883	0.878450	-0.049926
4	-1.012732	0.0	-0.184862	-0.680930	1.222541	-0.154529	-0.428328
...	...	...	...	...	...	...	...
95	0.305495	0.0	-0.339258	1.063852	-1.141938	0.006339	2.597674
96	-0.865824	1.0	0.223080	1.433215	0.091520	0.580777	-0.056783
97	-0.027818	1.0	-0.170408	-0.779482	0.430301	-0.851537	0.665585
98	-1.263472	0.0	1.085287	0.366531	-0.286249	0.453966	-0.308673
99	-0.421367	0.0	0.935547	-1.831406	-0.335607	-1.990812	-1.495061

100 rows × 7 columns

Wrap in CausalData

We provide a few covariates as confounders for DML (although the data is truly randomized).

	count	mean	std	min	p10	p25	median	p75	p90	max
treatment
0.0	5010	-0.009797	1.005403	-3.482370	-1.290654	-0.684156	-0.014663	0.657298	1.286337	3.627004
1.0	4990	0.519886	1.007209	-3.021571	-0.746023	-0.165270	0.503424	1.206379	1.790902	3.885919

	mean_t_0	mean_t_1	abs_diff	smd
confounders
x2	-0.014765	0.018033	0.032798	0.032770
x1	-0.022735	-0.008229	0.014505	0.014486
x5	0.007285	-0.006229	0.013514	-0.013459
x3	0.014590	0.007731	0.006859	-0.006917
x4	-0.000891	-0.000040	0.000850	0.000838

1) t-test (difference in means)

{'p_value': 1.176719325899924e-147, 'absolute_difference': 0.5296827782315828, 'absolute_ci': (0.49023151024069744, 0.5691340462224682), 'relative_difference': 5406.753548476556, 'relative_ci': (5004.053494844908, 5809.453602108204)}

2) Linear regression

The coefficient on treatment equals the difference in group means in an RCT.

(0.5294584904981685, (np.float64(0.4899759986996098), np.float64(0.5689409822967272)), 0.02014449862854194, np.float64(0.0))

3) Double Machine Learning (ATE)

We estimate ATE using DoubleML with default learners.

{'coefficient': 0.5438121651196453, 'std_error': 0.021106462512239948, 'p_value': 2.1780238928091667e-146, 'confidence_interval': (0.5024442587546102, 0.5851800714846803), 'model': <doubleml.irm.irm.DoubleMLIRM at 0x1683801a0>}

Compare estimates

	method	estimate	ci_lower	ci_upper
0	t-test	0.529683	0.490232	0.569134
1	linear_regression	0.529458	0.489976	0.568941
2	dml_ate	0.543812	0.502444	0.585180

Ground truth theta = 0.5