Synthetic Control Method

Data

Result

Ground-truth ATTE is 1.919269

Result

	unit_id	calendar_time	y	y_cf	mu_cf	mu_treated
0	donor_1	2000-01	9.971611	9.971611	10.059081	10.059081
1	donor_1	2000-02	10.344379	10.344379	10.097467	10.097467
2	donor_1	2000-03	10.998498	10.998498	11.354098	11.354098
3	donor_1	2000-04	11.508717	11.508717	11.277716	11.277716
4	donor_1	2000-05	11.125281	11.125281	10.098791	10.098791

EDA

Result

PanelDataSCM(df=(1763, 4), y='y', unit_col='unit_id', time_col='calendar_time', treated_time='treated_time', time_freq='M', treated_unit='treated', treatment_start=Period('2003-02', 'M'), last_post_period=Period('2003-07', 'M'), n_pre_periods=37, n_post_periods=6, donor_units=['donor_1', 'donor_10', 'donor_11', 'donor_12', 'donor_13', 'donor_14', 'donor_15', 'donor_16', 'donor_17', 'donor_18', 'donor_19', 'donor_2', 'donor_20', 'donor_21', 'donor_22', 'donor_23', 'donor_24', 'donor_25', 'donor_26', 'donor_27', 'donor_28', 'donor_29', 'donor_3', 'donor_30', 'donor_31', 'donor_32', 'donor_33', 'donor_34', 'donor_35', 'donor_36', 'donor_37', 'donor_38', 'donor_39', 'donor_4', 'donor_40', 'donor_5', 'donor_6', 'donor_7', 'donor_8', 'donor_9'])

Result

	donor	pre_mean	pre_std	pre_slope	corr_with_treated_pre	rmse_to_treated_pre	rmse_to_treated_pre_standardized	mean_diff_pre	slope_diff_pre	max_abs_gap_pre	is_never_treated	corr_rank	std_rmse_rank	slope_rank	composite_similarity_score	rank_by_similarity	notes
0	donor_20	20.756877	4.235256	0.065672	0.699298	3.050574	1.177242	0.055800	0.001045	7.515230	True	6	2	1	0.950000	1	high_std_rmse
1	donor_23	18.859982	3.459160	0.109325	0.792879	2.802849	1.081642	-1.841096	0.044698	6.193337	True	1	1	19	0.850000	2	high_std_rmse
2	donor_17	27.374399	5.004251	0.062159	0.694945	7.633294	2.945751	6.673322	-0.002467	15.022205	True	7	23	3	0.750000	3	high_std_rmse
3	donor_35	12.890196	2.478484	0.054896	0.774341	7.995072	3.085364	-7.810882	-0.009730	11.244939	True	2	25	7	0.741667	4	high_std_rmse
4	donor_27	12.138052	2.561773	0.065732	0.710107	8.784934	3.390178	-8.563025	0.001106	12.226100	True	4	29	2	0.733333	5	high_std_rmse
5	donor_39	15.933235	3.493479	0.101601	0.756099	5.287926	2.040654	-4.767843	0.036974	9.398086	True	3	16	16	0.733333	6	high_std_rmse
6	donor_2	22.023910	3.752940	0.025593	0.599773	3.299044	1.273128	1.322833	-0.039033	8.913091	True	19	4	17	0.691667	7	high_std_rmse
7	donor_34	23.225266	4.258988	0.116799	0.685302	4.012347	1.548398	2.524189	0.052173	8.782578	True	8	11	21	0.691667	8	high_std_rmse
8	donor_26	15.531548	2.367240	0.058569	0.584382	5.645636	2.178697	-5.169529	-0.006057	10.132082	True	21	18	5	0.658333	9	high_std_rmse
9	donor_9	20.198346	3.894696	0.185173	0.609819	3.134843	1.209762	-0.502731	0.120546	10.776096	True	15	3	31	0.616667	10	high_std_rmse
10	donor_31	15.974505	3.809154	0.046519	0.573221	5.678805	2.191497	-4.726572	-0.018107	12.461670	True	22	19	9	0.608333	11	high_std_rmse
11	donor_3	22.984470	4.678813	0.162331	0.639562	4.279210	1.651382	2.283392	0.097704	12.714594	True	12	13	26	0.600000	12	high_std_rmse
12	donor_7	19.888000	3.565032	0.048846	0.434870	3.471405	1.339644	-0.813077	-0.015780	8.749577	True	37	6	8	0.600000	13	high_std_rmse
13	donor_33	22.467401	3.890376	0.061634	0.437267	4.019136	1.551018	1.766323	-0.002993	9.071825	True	36	12	4	0.591667	14	high_std_rmse
14	donor_6	10.817057	1.619471	0.030223	0.705904	10.055180	3.880377	-9.884020	-0.034404	15.388984	True	5	33	14	0.591667	15	high_std_rmse
15	donor_21	21.001540	4.037106	-0.018281	0.542038	3.429640	1.323526	0.300463	-0.082908	10.637405	True	27	5	25	0.550000	16	high_std_rmse
16	donor_13	28.400664	5.292688	0.037227	0.605290	8.798181	3.395290	7.699587	-0.027399	17.004455	True	18	30	12	0.525000	17	high_std_rmse
17	donor_15	21.647789	4.210829	0.182164	0.554289	3.639493	1.404510	0.946712	0.117537	13.105415	True	24	7	29	0.525000	18	high_std_rmse
18	donor_12	9.964996	1.894391	0.090093	0.607253	10.936409	4.220450	-10.736082	0.025466	14.965497	True	17	34	10	0.516667	19	high_std_rmse
19	donor_8	8.772232	2.098724	0.104214	0.647465	12.098529	4.668922	-11.928845	0.039588	17.063234	True	10	35	18	0.500000	20	high_std_rmse
20	donor_30	23.684935	4.356977	0.116176	0.497646	4.833674	1.865355	2.983858	0.051549	10.057305	True	30	15	20	0.483333	21	high_std_rmse
21	donor_4	21.801573	4.207243	0.179567	0.524523	3.766945	1.453695	1.100496	0.114941	8.288093	True	28	9	28	0.483333	22	high_std_rmse
22	donor_10	22.140468	4.508805	0.224646	0.558779	4.007346	1.546468	1.439391	0.160019	9.740145	True	23	10	33	0.475000	23	high_std_rmse
23	donor_5	25.412361	4.374273	0.057271	0.323830	6.379970	2.462083	4.711284	-0.007355	14.186181	True	40	21	6	0.466667	24	high_std_rmse
24	donor_18	15.375511	2.894278	0.142564	0.551353	5.931533	2.289027	-5.325566	0.077938	11.432539	True	25	20	23	0.458333	25	high_std_rmse
25	donor_25	25.638145	4.719678	0.100007	0.491059	6.430663	2.481646	4.937068	0.035381	13.366836	True	33	22	15	0.441667	26	high_std_rmse
26	donor_36	17.556815	3.510483	0.132070	0.490282	4.472671	1.726041	-3.144262	0.067444	10.183903	True	34	14	22	0.441667	27	high_std_rmse
27	donor_1	11.442375	2.525462	0.146053	0.608511	9.531624	3.678332	-9.258702	0.081427	14.575727	True	16	31	24	0.433333	28	high_std_rmse
28	donor_40	11.363776	1.958071	0.090476	0.506775	9.622439	3.713378	-9.337301	0.025850	14.358223	True	29	32	11	0.425000	29	high_std_rmse
29	donor_32	18.875009	3.180241	-0.049527	0.404701	3.673285	1.417551	-1.826069	-0.114153	9.321772	True	38	8	27	0.416667	30	high_std_rmse
30	donor_28	27.821562	5.599506	0.336526	0.645032	8.369695	3.229934	7.120484	0.271899	16.331011	True	11	26	37	0.408333	31	high_std_rmse
31	donor_16	26.230599	5.760959	0.095907	0.370933	7.707416	2.974355	5.529522	0.031281	18.330394	True	39	24	13	0.391667	32	high_std_rmse
32	donor_11	31.898384	5.748397	0.278022	0.627211	12.101775	4.670174	11.197307	0.213395	24.414887	True	13	36	35	0.325000	33	high_std_rmse
33	donor_38	23.341229	5.607884	0.327023	0.473125	5.602050	2.161877	2.640152	0.262396	13.541965	True	35	17	36	0.291667	34	high_std_rmse
34	donor_19	64.642074	13.285880	0.416100	0.666003	45.477279	17.550056	43.940997	0.351474	87.274045	True	9	40	40	0.283333	35	high_std_rmse
35	donor_29	32.069172	6.680389	0.392642	0.623576	12.608994	4.865914	11.368095	0.328015	20.594296	True	14	37	38	0.283333	36	high_std_rmse
36	donor_24	12.604643	3.169244	0.199055	0.493436	8.614321	3.324337	-8.096434	0.134428	14.907594	True	31	28	32	0.266667	37	high_std_rmse
37	donor_14	27.342267	6.286205	0.393100	0.547435	8.514798	3.285931	6.641190	0.328473	15.190467	True	26	27	39	0.258333	38	high_std_rmse
38	donor_37	34.405731	7.663463	0.231876	0.590564	15.159349	5.850117	13.704654	0.167249	33.571787	True	20	39	34	0.250000	39	high_std_rmse
39	donor_22	34.353351	5.235590	0.183591	0.492258	14.392843	5.554316	13.652274	0.118964	27.762663	True	32	38	30	0.191667	40	high_std_rmse

Result

png

Inference

So the avarage effect per period is 3.9456 with (ci_abs: 0.3664, 7.5249)

Refutation

Result

png

Result

png

As we see synthetic prediction of treated unit is close to real behavior before the intervention and has a gap after it

Placebo

Result

	placebo_treatment_start	n_pre_before_placebo	n_post_after_placebo	average_att_placebo	ci_lower	ci_upper	p_value	rejects_zero	pre_fit_metric
0	2000-03	2	6	1.882234	-16.331756	20.096224	0.414358	False	3.892686e-09
1	2000-04	3	6	1.269647	-2.922810	5.462105	0.322397	False	4.675731e-08
2	2000-05	4	6	1.558211	-2.782112	5.898534	0.262430	False	7.742026e-09
3	2000-06	5	6	1.314722	-2.945265	5.574709	0.315493	False	8.664846e-09
4	2000-07	6	6	1.415188	-1.584680	4.415056	0.179511	False	8.109636e-09
5	2000-08	7	6	1.303851	-3.103501	5.711204	0.331010	False	6.429904e-04
6	2000-09	8	6	0.780444	-2.048430	3.609318	0.357094	False	1.040816e-03
7	2000-10	9	6	0.096426	-5.840251	6.033103	0.950644	False	9.916453e-04
8	2000-11	10	6	-0.153322	-5.298898	4.992254	0.909715	False	9.340734e-04
9	2000-12	11	6	-0.446809	-6.734040	5.840422	0.788669	False	8.344978e-04
10	2001-01	12	6	-0.533170	-8.201066	7.134727	0.793032	False	8.445828e-04
11	2001-02	13	6	-1.000895	-6.324216	4.322425	0.503460	False	1.652109e-03
12	2001-03	14	6	-1.029470	-4.574049	2.515109	0.337841	False	1.603717e-03
13	2001-04	15	6	-2.038412	-5.329455	1.252630	0.116670	False	2.599095e-03
14	2001-05	16	6	-2.044195	-6.030791	1.942400	0.158111	False	2.932379e-03
15	2001-06	17	6	-1.036771	-5.847229	3.773686	0.451654	False	3.083937e-03
16	2001-07	18	6	0.083509	-4.860537	5.027555	0.948678	False	1.392011e-02
17	2001-08	19	6	-0.009216	-4.401569	4.383136	0.993616	False	1.370059e-02
18	2001-09	20	6	-0.293440	-1.412307	0.825426	0.376294	False	1.695760e-02
19	2001-10	21	6	-0.446168	-3.057748	2.165412	0.538804	False	1.588254e-02
20	2001-11	22	6	-0.856900	-4.022197	2.308397	0.364242	False	1.628198e-02
21	2001-12	23	6	-0.674271	-3.930313	2.581771	0.466940	False	1.626443e-02
22	2002-01	24	6	-0.742675	-1.665118	0.179767	0.074178	False	1.561534e-02
23	2002-02	25	6	-0.703350	-2.420842	1.014142	0.220123	False	1.549205e-02
24	2002-03	26	6	-1.016981	-5.886223	3.852261	0.463681	False	1.542201e-02
25	2002-04	27	6	-0.813863	-3.768090	2.140364	0.357634	False	1.574490e-02
26	2002-05	28	6	-0.321249	-2.782331	2.139834	0.630907	False	1.451867e-02
27	2002-06	29	6	-0.522668	-4.026442	2.981106	0.586723	False	2.973807e-02
28	2002-07	30	6	-0.305589	-5.261126	4.649948	0.815602	False	3.467259e-02
29	2002-08	31	6	-0.049685	-4.870095	4.770726	0.968657	False	3.722549e-02

We pretend treatment has happened earlier.

placebo_treatment_start - where treatment starts
n_pre_before_placebo - how many periods were fitted before the treatment
n_post_after_placebo - how many periods were fitted after the treatment
average_att_placebo - average_post_effect
rejects_zero - p_value < alpha

So weak robustness when we have many rows have rejects_zero = false and average_att_placebo is small

Result

	unit_id	pre_rmse	post_rmse	post_pre_rmspe_ratio	average_post_gap	max_abs_post_gap	rank_post_pre_rmspe_ratio	is_actual_treated
0	donor_36	0.126184	5.361687	42.491162	-3.645154	8.686006	1	False
1	donor_1	0.090722	3.685575	40.625069	-1.702321	5.149012	2	False
2	donor_9	0.270737	9.590485	35.423589	-4.579774	17.029518	3	False
3	treated	0.159791	5.312488	33.246386	1.536003	9.708561	4	True
4	donor_11	0.465310	14.624469	31.429528	10.645438	20.023547	5	False
5	donor_10	0.401070	12.221049	30.471131	-7.281389	19.769855	6	False
6	donor_23	0.303599	9.205654	30.321735	3.990906	17.303789	7	False
7	donor_35	0.327089	9.915500	30.314367	-5.193231	16.693273	8	False
8	donor_24	0.135587	4.098194	30.225576	3.095115	6.144010	9	False
9	donor_26	0.293840	8.657327	29.462709	-6.539752	16.299358	10	False
10	donor_12	0.201532	5.932315	29.436027	4.107740	10.408219	11	False
11	donor_8	0.059911	1.723919	28.774858	0.063256	3.233991	12	False
12	donor_3	0.612248	16.813999	27.462716	11.532806	32.197769	13	False
13	donor_40	0.295235	7.774020	26.331673	-1.473769	12.059195	14	False
14	donor_20	0.239533	6.236275	26.035093	-4.725626	8.661925	15	False
15	donor_18	0.287376	7.392529	25.724274	1.918772	12.702399	16	False
16	donor_31	0.296797	7.443741	25.080284	-0.281308	8.855945	17	False
17	donor_29	0.631821	15.834072	25.061005	-10.901977	29.844629	18	False
18	donor_38	0.506852	12.092285	23.857624	1.288799	17.046681	19	False
19	donor_19	1.341580	30.767777	22.933994	16.591205	60.760771	20	False
20	donor_27	0.211628	4.643916	21.943816	3.575389	6.708018	21	False
21	donor_25	0.349672	7.589038	21.703296	3.170792	13.480264	22	False
22	donor_13	0.348416	6.629126	19.026465	1.004351	12.018635	23	False
23	donor_6	0.281206	5.104271	18.151348	-2.365311	8.380162	24	False
24	donor_2	0.854790	15.308159	17.908687	9.916793	25.063513	25	False
25	donor_17	0.699948	11.928231	17.041598	-8.293538	20.483329	26	False
26	donor_4	0.604784	9.932731	16.423596	5.895927	15.130913	27	False
27	donor_33	0.946598	15.176927	16.033132	9.925254	25.677443	28	False
28	donor_21	0.515542	8.108760	15.728618	2.162652	14.017763	29	False
29	donor_34	0.371591	5.546140	14.925369	1.751542	9.663275	30	False
30	donor_5	0.578614	8.314714	14.370061	-2.724464	12.569952	31	False
31	donor_14	0.540823	7.762695	14.353496	-2.024107	13.245248	32	False
32	donor_22	0.782195	10.477241	13.394660	-2.216629	19.119793	33	False
33	donor_15	0.504406	6.702065	13.287035	3.532679	11.589464	34	False
34	donor_16	0.914931	11.640657	12.722994	-0.767309	18.665231	35	False
35	donor_30	0.593733	7.448935	12.545926	2.302397	11.135762	36	False
36	donor_37	0.990143	12.321512	12.444172	-3.335595	21.458012	37	False
37	donor_32	0.588609	6.899649	11.721950	2.440457	12.725852	38	False
38	donor_28	0.690999	7.849498	11.359643	-0.109926	8.851531	39	False
39	donor_39	0.393544	3.680572	9.352370	0.691549	5.536206	40	False
40	donor_7	0.820626	7.101691	8.653990	4.797007	11.593821	41	False

How to read columns:

unit_id: unit used as treated in that placebo run.
pre_rmse: pre-treatment fit error (lower is better).
post_rmse: post-treatment fit error.
post_pre_rmspe_ratio: post_rmse / pre_rmse (main Abadie-style statistic).
average_post_gap: mean post-treatment gap (observed - synthetic).
max_abs_post_gap: largest absolute post gap.
rank_post_pre_rmspe_ratio: rank of ratio, descending (1 = largest ratio).
is_actual_treated: True for your real treated unit.

Interpretation: You want the real treated unit (is_actual_treated=True) to have one of the largest post_pre_rmspe_ratio values (ideally rank near 1). If many placebo units have similar/larger ratios, treatment effect is less convincing. Also check pre_rmse: if pre-fit is poor for some placebos, their high ratios are less informative.

Sensitivity

Result

	dropped_donor	average_att_reestimated	delta_vs_full_model	pre_rmse_reestimated	max_weight_after_refit	effective_n_donors_after_refit
0	donor_1	4.207858	0.262224	0.171405	0.303030	0.946472
1	donor_10	4.587799	0.642165	0.160802	0.349188	0.909246
2	donor_11	4.189137	0.243503	0.160120	0.353247	0.909497
3	donor_12	4.916257	0.970623	0.189167	0.336427	0.978294
4	donor_13	4.468131	0.522497	0.162525	0.341735	0.927950
5	donor_14	4.725074	0.779440	0.160180	0.358832	0.910182
6	donor_15	4.513058	0.567424	0.169711	0.391910	0.908753
7	donor_16	4.382018	0.436384	0.159357	0.352835	0.903757
8	donor_17	3.622174	-0.323460	0.158809	0.356443	0.899311
9	donor_18	4.436393	0.490759	0.198705	0.368077	1.149358
10	donor_19	5.047382	1.101748	0.219917	0.510889	0.814392
11	donor_2	5.294567	1.348934	0.202215	0.518748	0.700821
12	donor_20	4.766594	0.820960	0.160017	0.354066	0.910676
13	donor_21	4.881931	0.936297	0.174764	0.391185	0.830214
14	donor_22	4.553513	0.607879	0.189970	0.397334	1.012348
15	donor_23	5.031115	1.085481	0.205574	0.410620	0.976600
16	donor_24	4.702066	0.756432	0.160943	0.347964	0.912077
17	donor_25	4.580630	0.634996	0.179322	0.431234	0.862914
18	donor_26	4.600980	0.655346	0.157975	0.341059	0.851219
19	donor_27	4.574977	0.629343	0.160845	0.357505	0.843273
20	donor_28	4.451933	0.506300	0.174201	0.343802	1.016529
21	donor_29	5.115016	1.169383	0.176526	0.376909	0.909911
22	donor_3	5.589095	1.643461	0.290984	0.469115	1.061081
23	donor_30	4.635528	0.689894	0.198909	0.355522	0.836912
24	donor_31	5.194494	1.248860	0.253320	0.315068	1.168234
25	donor_32	5.594067	1.648433	0.255886	0.901456	0.421563
26	donor_33	4.637100	0.691466	0.190820	0.342409	1.021246
27	donor_34	5.507941	1.562307	0.245173	0.772210	0.663935
28	donor_35	4.582141	0.636508	0.226884	0.319520	1.592468
29	donor_36	4.943702	0.998068	0.200368	0.321539	0.956472
30	donor_37	4.626041	0.680407	0.152758	0.341441	0.787436
31	donor_38	3.669948	-0.275686	0.183982	0.356237	1.093042
32	donor_39	5.128140	1.182506	0.171480	0.422500	0.764932
33	donor_4	4.809473	0.863839	0.167348	0.361395	0.942293
34	donor_40	4.783345	0.837711	0.178037	0.347251	1.081478
35	donor_5	5.201186	1.255552	0.166645	0.454421	0.706770
36	donor_6	4.564400	0.618766	0.158656	0.355788	0.894602
37	donor_7	4.718838	0.773204	0.184685	0.329164	1.003480
38	donor_8	4.517386	0.571752	0.162223	0.397854	0.872781
39	donor_9	4.765666	0.820032	0.211321	0.395626	1.133550

dropped_donor: donor unit removed in that refit.
average_att_reestimated: ATT after removing that donor.
delta_vs_full_model: reestimated_ATT - original_full_model_ATT. Near 0 → estimate is stable to that donor. Large |deltal → that donor is influential.
pre_rmse_reestimated: pre-treatment fit error after refit. Higher than baseline → worse pre-fit when donor is removed.
max_weight_afterrefit: largest donor weight in refit.Very high (close to 1) → dependence on one donor.
effective_n_donors_after_refit: effective donor count (1 / sum (w^2)). Lower → concentration on fewer donors.

Model

Result

	test	flag	value	threshold	message
0	pre_rmse_augmented	GREEN	0.366248	<= 0.5254	Pre-treatment RMSE is within tolerance.
1	max_abs_pre_gap_augmented	GREEN	0.988290	<= 1.314	Largest pre-treatment gap is within tolerance.
2	mean_gap_last_k_pre_augmented	GREEN	0.207285	<= 0.6568	Average gap in the last 3 pre periods is cente...
3	max_abs_weight_augmented	GREEN	0.201942	<= 2	No extreme augmented donor weight detected (mo...
4	l1_norm_weights_augmented	GREEN	2.245170	<= 5	Total absolute augmented weight mass is contro...
5	negative_weight_share	GREEN	0.277300	<= 0.3	Negative-weight share is moderate.
6	slsqp_fallback_count	GREEN	0.000000	== 0	No optimizer fallback events recorded.
7	suppressed_fit_warning_count	GREEN	0.000000	== 0	No suppressed fit warnings were captured.

We can call our estimate robust