Business Frameworks Interactive

Incrementality Testing

Measure true causal impact of marketing. How many conversions happened BECAUSE of the promo vs would have happened anyway?

📊 The Attribution Problem

❌ Last-Touch Attribution

"User saw ad, then converted. Ad gets credit."

Problem: They might have converted anyway!

✓ Incrementality

"How many MORE conversions vs no ad?"

Solution: Randomized control group

Experiment Setup

5 25

2 20

1000 10000

5 50

📊 Incrementality Results

Incremental Lift

+4.0pp

50% relative

Incremental Customers

200

new from promo

Cost per Incremental

$250

iCAC

p-value

0.000

✓ Significant

Conversion Breakdown

Treatment Group

Conversions

Organic (400)

Incremental (200)

Control Group

Conversions

Organic only (400)

⚠️ Cannibalization Warning

Of your 600 treatment conversions, only 200 (33%) are truly incremental. The rest would have converted anyway!

🧪 Incrementality Test Types

Holdout

Random control group receives no promo

✓ Clean causality

Geo Test

Different regions get different treatments

✓ Real-world conditions

Ghost Bids

Track "would have converted" in control

✓ No lost revenue

Pre/Post

Compare before vs after treatment

✓ Simple setup

R Code Equivalent

# Incrementality analysis
calculate_incrementality <- function(treat_conv, control_conv, 
                                      treat_n, control_n, promo_cost) { 
  # Lift
  lift_pp <- treat_conv - control_conv
  lift_rel <- (treat_conv / control_conv - 1)
  
  # Incremental conversions
  incremental <- treat_n * lift_pp / 100
  
  # Cost per incremental
  total_cost <- treat_n * promo_cost
  icac <- total_cost / incremental
  
  # Significance test
  test <- prop.test(
    x = c(treat_conv * treat_n / 100, control_conv * control_n / 100),
    n = c(treat_n, control_n)
  )
  
  list(
    lift_pp = lift_pp,
    lift_rel = lift_rel,
    incremental = incremental,
    icac = icac,
    p_value = test$p.value,
    significant = test$p.value < 0.05
  )
}

result <- calculate_incrementality(12, 8, 
                                   5000, 5000, 10)
cat(sprintf("Incremental: %.0f, iCAC: $%.0f, p=%.3f\n",
    result$incremental, result$icac, result$p_value))

✅ Key Takeaways

• Incrementality = causal impact of marketing
• Most "attributed" conversions are cannibalized
• Need randomized holdout for true measurement

• iCAC often 3-10x higher than standard CAC
• Statistical significance required
• Geo tests for expensive campaigns