Skip to content
Case Study2 min read

Rct Design

Because your hash key is your “random universe,” and those three fields control which universe you’re in.

Because your hash key is your “random universe,” and those three fields control which universe you’re in.

You effectively hash:

code.text

Each piece solves a different problem.


1. experiment_id: isolate experiments + support reruns

Why include it?

  • Independence between experiments. If you didn’t use experiment_id, the same user would always map to the same bucket across all experiments. That means:

    • If user is “treatment” once, they’d be “treatment” everywhere → correlated assignments → biased results when you compare across tests.
  • Stable within a single experiment. Given fixed experiment_id, a user keeps their bucket forever (idempotent).

  • Versioning / reruns. Want a fresh randomization? Use experiment_id="exp_checkout_v2" and you get a new random split without touching code.

Rule of thumb: new logical test or new randomization → new experiment_id.


2. salt: prevent gaming + cross-system collisions

Why include it?

  • Security / anti-gaming. Without a secret salt, savvy users/devs could guess:

    • “If my user_id ends with X I get treatment, so I’ll farm accounts with that pattern.”
  • Isolation from other uses of hashing. You might use SHA256 elsewhere; salt ensures the A/B assignment space is unique.

  • Safe to share IDs. You can expose experiment names & variant labels in logs/BI without making it trivial to reverse or shape assignments.

Rule of thumb: keep salt server-side, same across platform; rotate only with a migration plan (rotation changes assignments).


3. layer_id: control mutual exclusivity & traffic layers

Think of layer_id as “which slot of the product UI this experiment lives in”.

Why include it?

  • Mutual exclusivity. Suppose you have many experiments on the home screen. You don’t want one user in 5 conflicting treatments.

    • Put them in the same layer_id="home_feed", then use the same hash stream to decide which single experiment (or variant) a user sees.
  • Independent surfaces. Experiments on unrelated areas (e.g. "search_page" vs "pricing_page") should not compete:

    • Different layer_id → independent assignments, even for same user.

Rule of thumb:

  • Use one layer per logical surface or exclusivity group.
  • Default "default" is fine if you don’t yet run overlapping tests.

So:

  • experiment_id → “which experiment”.
  • layer_id → “which exclusivity sandbox”.
  • salt → “make the whole thing safe & non-predictable”.

Together they give you: deterministic, debuggable, replayable, non-correlated, and non-gameable assignments — i.e. what modern A/B platforms try to guarantee.

How layer_id is intended to be used in a full platform:

You define a “layer router”:

Example: layer_id="home_feed" decides which one of several experiments (or none) a user enters.

Those experiments share the same layer to enforce mutual exclusivity.

SRM

Result

SRMResult(status=SRM DETECTED, p_value=3.744e-19, chi2=80.0000, df=1) Observed: {'control': 1200, 'treatment': 800} Expected: {'control': 1000.0, 'treatment': 1000.0} is_srm: True warning: None