Estimation of CX with MutliTreatment DLM

This notebook presents the estimation of cx case with MultitreatmentIRM model

The problem

When analyzing customer experience, we encounter the problem that there may be multiple impacts that overlap. We can no longer encode treatments binary, so we create groups of unique clients that do not overlap. Let's look at an example from a large fintech company.

We think the effect depends on how users were exposed to the product.

We segmented users and now want to estimate the effect for each group.

• Target

  • y — binary target variable representing product utilization.

• Treatments (problems)

  • neg_contact_flg — negative contact to support
  • error_flg — repeat application / error flag of application.
  • neg_contact_flg_error_flg — joint treatment where both problem flags are present.
  • control — baseline group with no problem flag.

• Strong confounders (past applications/utilization history)

  • prev_apps — past applications.
  • prev_util — past utilization events.

• Features (covariates)

  • age — age.
  • risk_latent — latent risk factor.
  • income — income.
  • sessions_30d — sessions in the last 30 days.
  • clicks_7d — clicks in the last 7 days.
  • n_products — number of products.
  • has_debt — debt flag.
  • csat_prev — previous CSAT score.
  • prev_contact — past contact history.
  • prev_repeat — past repeat-application history.
  • channel — application channel, represented in the generated dataset as one-hot columns: channel_callcenter, channel_partner, channel_web with app as the reference level.
  • region — region, represented in the generated dataset as one-hot columns: region_B, region_C, region_D with A as the reference level.

• Noise features

  • product_emb_1, product_emb_2 — no-signal features.

Data

EDA

Training the Model and Reviewing the Results

"Error" or "Negative Contact + Error" decreases probability to utilization in product by 11 p.p

Refutation

Because “bad overlap” and “accurate point estimate” are different things here.

• Your overlap is genuinely bad, not a plotting bug. The notebook diagnostics are extreme: KS = 0.716 / 0.951 / 0.9997, AUC = 0.945 / 0.998 / 1.0, and clip_m_total = 0.43 / 0.85 / 0.999 for the three pairwise comparisons. That means the arms are almost perfectly separable in propensity space.
• This is expected from the CX DGP. Treatment assignment is intentionally very predictable from observed history: prev_contact enters the contact logit with coefficient 2.20, prev_repeat enters the repeat logit with coefficient 1.30, and the joint arm literally uses contact_logit + repeat_logit as its score in dgp.py, dgp.py, dgp.py. So near-deterministic assignment is baked into the benchmark.
• The outcome side is much easier than the treatment side. In this DGP the treatment effects are constant on the link scale: theta=[0.0, 0.0, -0.65, -0.65] and tau=[None, None, None, None] in dgp.py. So neg_contact_flg truly has zero effect, and error_flg and neg_contact_flg_error_flg have the same effect by construction.
• There is also no hidden confounding in the generator defaults: u_strength_y=0.0 and u_strength_d=0.0 in base.py and base.py. So all the variables driving treatment are observed in X, which makes the synthetic problem much friendlier than real data.
• MultiTreatmentIRM is not pure IPW. Its score is orthogonal / AIPW-style, combining outcome regression g_hat with residual weighting by 1 / m_hat in model.py. If g_hat is learned well, the ATE can still be accurate even when overlap is weak.
• You also have a huge sample. Even with poor overlap, the ESS ratios still leave thousands of effective observations: roughly 9.6k, 5.3k, and 3.8k treated ESS, with about 11.3k baseline ESS in each comparison. That is plenty for a simple constant-effect synthetic DGP.

One subtlety: the overlap plot and overlap diagnostics prefer m_hat_raw when available, while estimation uses trimmed m_hat with threshold 0.01 in overlap_plot.py, overlap_validation.py, and model.py. So the diagnostic picture is intentionally harsher than the exact weights used in the score.

The takeaway is: this overlap is bad, but the estimate is still close to oracle because the benchmark is very favorable to outcome-model-based recovery. In real data, overlap this bad would make me treat the result as fragile and model-dependent, not reassuring. Your later sensitivity section in estimation_of_cx.ipynb is consistent with that: the bias-aware intervals get much wider even though the sampling CI is tight.

If you want, I can next help you answer one of these precisely:

1. whether the estimate is mostly coming from g_hat rather than IPW,
2. how much trimming is rescuing the estimate,
3. how to soften this DGP so overlap becomes realistic.

Unconfoundedness

rep_uc Meaning run_unconfoundedness_diagnostics() is checking weighted covariate balance after reweighting by the estimated multiclass propensities d_k / m_hat_k, then computing standardized mean differences (SMDs) between each treatment arm and control in unconfoundedness_validation.py. It uses the trimmed m_hat from the estimate, not m_hat_raw, so this is already judging the “safer” propensities actually used by the score in unconfoundedness_validation.py and model.py.

Your numbers are a hard fail:

• control vs neg_contact_flg: balance_max_smd = 0.75095, balance_frac_violations = 0.30
• control vs error_flg: balance_max_smd = 1.655942, balance_frac_violations = 0.45
• control vs neg_contact_flg_error_flg: balance_max_smd = 1.924802, balance_frac_violations = 0.65
• Overall: 46.7% of feature-comparison cells are still above the default SMD threshold 0.10, and anything above 0.20 is already RED by the implementation in unconfoundedness_validation.py and unconfoundedness_validation.py.

Why This Happens This does not mean the DGP violates unconfoundedness. In this synthetic CX setup, unconfoundedness is true by construction because treatment and outcome depend only on observed X, with no hidden U effect by default in the shared generator at base.py and base.py. What fails is the practical balancing step induced by estimated propensities.

The reason is the treatment assignment is intentionally very sharp:

• neg_contact_flg is driven strongly by prev_contact with coefficient 2.20 in _cx_contact_logit
• error_flg is driven by prev_repeat and related history in _cx_repeat_logit
• the joint arm uses the sum of both logits in generate_multi_dml_cx_26

So the arms live in different parts of covariate space. Poor overlap means IPW cannot create balanced pseudo-populations because there just are not enough comparable controls and treated units to reweight into each other. That’s exactly why overlap and balance both fail.

Why ATE Is Still Close The estimate can still be near oracle because MultiTreatmentIRM is doubly robust / outcome-augmented, not pure IPW. Its score combines outcome regression g_hat with weighted residual terms in model.py. In this benchmark the treatment effect structure is very simple on the link scale, with theta=[0.0, 0.0, -0.65, -0.65] and no extra tau(X) in dgp.py. So g_hat can recover a lot even when weighting is weak.

That’s also why the pattern of errors makes sense:

• neg_contact_flg has the least-bad overlap/balance and smallest absolute miss from oracle
• neg_contact_flg_error_flg has the worst overlap/balance and the biggest miss

So the clean interpretation is:

• overlap diagnostics: bad
• balance diagnostics: bad even after trimming
• oracle vs estimate: still close because this synthetic problem is easy for the outcome model, not because the design is well supported

Sensetivity analysis

Summary

A regular contact/support request is not a problem, while submitting the application multiple times leads to a decrease in product utilization.