Sensitivity analysis for the generalization of experimental results

Abstract

Randomized controlled trials (RCT’s) allow researchers to estimate causal effects in an experimental sample with minimal identifying assumptions. However, to generalize or transport a causal effect from an RCT to a target population, researchers must adjust for a set of treatment effect moderators. In practice, it is impossible to know whether the set of moderators has been properly accounted for. I propose a two parameter sensitivity analysis for generalizing or transporting experimental results using weighted estimators. The contributions in the article are threefold. First, I show that the sensitivity parameters are scale-invariant and standardized, and introduce an estimation approach for researchers to account for both bias in their estimates from omitting a moderator, as well as potential changes to their inference. Second, I propose several tools researchers can use to perform sensitivity analysis: (1) numerical measures to summarize the uncertainty in an estimated effect to omitted moderators; (2) graphical summary tools to visualize the sensitivity in estimated effects; and (3) a formal benchmarking approach for researchers to estimate potential sensitivity parameter values using existing data. Finally, I demonstrate that the proposed framework can be easily extended to the class of doubly robust, augmented weighted estimators.