This book is in Open Review. We want your feedback to make the book better for you and other students. You may annotate some text by selecting it with the cursor and then click "Annotate" in the pop-up menu. You can also see the annotations of others: click the arrow in the upper right hand corner of the page

13.2 Threats to Validity of Experiments

The concepts of internal and external validity discussed in Key Concept 9.1 are also applicable for studies based on experimental and quasi-experimental data. Chapter 13.2 of the book provides a thorough explanation of the particular threats to internal and external validity of experiments including examples. We limit ourselves to a short repetition of the threats listed there. Consult the book for a more detailed explanation.

Threats to Internal Validity

  1. Failure to Randomize

    If the subjects are not randomly assigned to the treatment group, then the outcomes will be contaminated with the effect of the subjects’ individual characteristics or preferences and it is not possible to obtain an unbiased estimate of the treatment effect. One can test for nonrandom assignment using a significance test (\(F\)-Test) on the coefficients in the regression model \[X_i = \beta_0 + \beta_1 W_{1i} + \dots +\beta_2 W_{ri} + u_i \ \ , \ \ i=1,\dots,n.\]

  2. Failure to Follow the Treatment Protocol

    If subjects do not follow the treatment protocol, i.e., some subjects in the treatment group manage to avoid receiving the treatment and/or some subjects in the control group manage to receive the treatment (partial compliance), there is correlation between \(X_i\) und \(u_i\) such that the OLS estimator of the average treatment effect will be biased. If there are data on both treatment actually received (\(X_i\)) and initial random assignment (\(Z_i\)), IV regression of the models (13.1) and (13.2) is a remedy.

  3. Attrition

    Attrition may result in a nonrandomly selected sample. If subjects systematically drop out of the study after being assigned to the control or the treatment group (systematic means that the reason of the dropout is related to the treatment) there will be correlation between \(X_i\) and \(u_i\) and hence bias in the OLS estimator of the treatment effect.

  4. Experimental Effects

    If human subjects in treatment group and/or control group know that they are in an experiment, they might adapt their behaviour in a way that prevents unbiased estimation of the treatment effect.

  5. Small Sample Sizes

    As we know from the theory of linear regression, small sample sizes lead to imprecise estimation of the coefficients and thus imply imprecise estimation of the causal effect. Furthermore, confidence intervals and hypothesis test may produce wrong inference when the sample size is small.

Threats to External Validity

  1. Nonrepresentative Sample

    If the population studied and the population of interest are not sufficiently similar, there is no justification in generalizing the results.

  2. Nonrepresentative Program or Policy

    If the program or policy for the population studied differs considerably from the program (to be) applied to population(s) of interest, the results cannot be generalized. For example, a small-scale program with low funding might have different effects than a widely available scaled-up program that is actually implemented. There are other factors like duration and the extent of monitoring that should be considered here.

  3. General Equilibrium Effects

    If market and/or environmental conditions cannot be kept constant when an internally valid program is implemented broadly, external validity may be doubtful.