Experimental Design & Causal Inference

• Correlation: variables move together.
• Causation: changing \(X\) changes \(Y\).
• Randomization helps isolate causal effects by breaking confounding.

• Each unit has potential outcomes \(Y(1)\) and \(Y(0)\).
• Fundamental problem: we never observe both for the same unit.
• Causal effect for a unit: \(Y(1) - Y(0)\).

\[ ATE = \mathbb{E}[Y(1) - Y(0)] \]

• Often estimated in experiments via difference in group means.

“A third variable affects both treatment assignment and the outcome.”

• Leads to biased estimates if not controlled via randomization/design.

“Treatment and control groups differ systematically because assignment is not random.”

• Example: self-selection into treatment.

• Compare means across \(k\) groups (one factor with \(k\) levels).
• Tests whether at least one group mean differs.

• F-statistic:

\[ F = \frac{MS_{between}}{MS_{within}} \]

• Under \(H_0\):

\[ F \sim F_{k-1,\; N-k} \]

\[ Y_{ijk} = \mu + \alpha_i + \beta_j + (\alpha\beta)_{ij} + \varepsilon_{ijk} \]

• \(\alpha_i\): main effect of factor A level \(i\)
• \(\beta_j\): main effect of factor B level \(j\)
• \((\alpha\beta)_{ij}\): interaction effect

• ANOVA is regression with indicator (dummy) variables + interactions.
• Example structure:

\[ Y = \beta_0 + \beta_1 D_{A2} + \beta_2 D_{A3} + \beta_3 D_{B2} + \beta_4(D_{A2}D_{B2}) + \beta_5(D_{A3}D_{B2}) + \varepsilon \]

• Dummies represent factor levels relative to a reference (base) level.

• Three joint F-tests:
  1. Main effect of A
  2. Main effect of B
  3. Interaction effect \(A\times B\)

If factor A has \(a\) levels, factor B has \(b\) levels, total \(N\) observations:

\[ F_A \sim F_{a-1,\; N-ab} \]

\[ F_B \sim F_{b-1,\; N-ab} \]

\[ F_{AB} \sim F_{(a-1)(b-1),\; N-ab} \]

• Each regression coefficient has a t-test:

\[ t = \frac{\hat{\beta}}{SE(\hat{\beta})} \]

• For a single coefficient test:

\[ F = t^2 \]

• Main effects in ANOVA are usually joint tests → use F.

• Independence:

  “Observations are independent across units.”

• Homoskedasticity:

  “Residual variance is equal across groups.”

  \[ Var(\varepsilon) = \sigma^2 \]

• Normality (exact finite-sample inference):

  “Residuals are approximately normal.”

  \[ \varepsilon \sim N(0,\sigma^2) \]

• Full factorial tests all combinations of factor levels.
• If \(k\) factors each have \(L\) levels:

\[ L^k \]

• If levels differ across factors (A, B, C):

\[ a \times b \times c \]

• 3 factors, 2 levels each:

\[ 2^3 = 8 \]

• Can estimate main effects and interactions (including the 3-way interaction).

A full factorial design tests all combinations of factor levels so that you can estimate main effects and interaction effects independently.

For 3 factors (A, B, C) at 2 levels:

\[ 2^3 = 8 \]

So you run 8 experiments:

From the same runs you compute interaction columns:

\[ AB = A\times B,\quad AC = A\times C,\quad BC = B\times C,\quad ABC = A\times B\times C \]

These allow estimation of:

• 3 main effects (A, B, C)
• 3 two-way interactions (AB, AC, BC)
• 1 three-way interaction (ABC)
• 1 intercept

Because all columns are independent, no aliasing exists in the full factorial design.

A fractional factorial design uses only a fraction of the full factorial runs to save time/cost. The reduction is achieved with a generator.

For 3 factors (A, B, C), a half fraction is:

\[ 2^{3-1} = 4 \text{ runs} \]

A commonly used generator is:

\[ C = AB \]

This means factor C is defined by the interaction of A and B, so C is not independently set — its level is derived.

Using this generator, the 4 runs become:

Only 4 combinations are tested instead of 8.

Aliasing occurs when two effects are indistinguishable — they share the same column in the design matrix.

In the \(2^{3-1}\) fractional design with generator:

\[ C = AB \]

the columns for C and AB are identical. This means:

\[ C \equiv AB \]

They are confounded — the experiment cannot tell them apart.

From the generator, you derive the alias structure:

Multiply both sides of the generator by each factor:

\[ C = AB \]

Multiply by A \(\rightarrow AC = B\)
Multiply by B \(\rightarrow BC = A\)
Multiply by C \(\rightarrow ABC = I\)

So the key aliases are:

In fractional factorials, effects are aliased due to fewer runs than parameters. This is why generators are chosen carefully — to control which effects are confounded.

• 3 factors, 3 levels each:

\[ 3^3 = 27 \]

• Tests all 27 combinations; can estimate main effects + interactions.

• More factors/levels → more cells → fewer observations per cell (if \(N\) fixed) → lower power per effect.
• Higher-order interactions are often small in practice.

• Random assignment:

  “Units are randomly assigned to treatment combinations (cells).”

• Independence:

  “Observations are independent across units.”

• SUTVA:

  “Each unit’s outcome depends only on its own treatment assignment, and the treatment is consistently defined.”

• Homoskedasticity:

  “Residual variance is equal across cells/groups.”

• Normality (exact finite-sample inference):

  “Residuals are approximately normal.”

• Block on a nuisance factor, randomize treatment within each block.
• Example: pricing experiment where revenue varies by time of day (Morning, Afternoon, Evening).
• Regression:

\[ Y = \beta_0 + \beta_1\,Discount + \gamma_1\,Afternoon + \gamma_2\,Evening + \varepsilon \]

Where:

• \(Y\): outcome (e.g., revenue)
• \(Discount\): treatment dummy (1=discount, 0=standard)
• \(Afternoon\), \(Evening\): block dummies (Morning is reference)
• \(\beta_1\): treatment effect holding time-of-day constant
• Block coefficients absorb nuisance variation to reduce residual variance and increase power.

• Randomly assign units to Control (A) and Treatment (B).
• Difference-in-means estimator:

\[ \hat{\tau} = \bar{Y}_T - \bar{Y}_C \]

\[ SE(\hat{\tau}) = \sqrt{\frac{s_T^2}{n_T} + \frac{s_C^2}{n_C}} \]

• Test statistic:

\[ t = \frac{\hat{\tau}}{SE(\hat{\tau})} \]

• Small samples → use \(t\)-distribution; large samples → approx normal.