Learning from bad experiments

Thinking about bad experiments is a good device helping reinforce the concepts of experimental design. No one should deliberately perform an experiment with a defective design. However, thinking about such designs is a great learning device. For example, why does an experiment have a control group (a comparison group)? Why not just apply the intervention to all experimental units? What is the problem with this approach?

Here are more questions.

  • Why does an experiment have a control group (a comparison group)?
  • Why do the researchers use random chance to assign experimental units or subjects to the different treatment groups?
  • In clinical trials, why is freedom of choice taken away from the subjects? Why not just let each subject pick his or her own treatment?
  • To the extent that is possible, why are clinical trials blinded (the person administering the treatment or the subject taking the treatment does not know what pills a given subject is taking, or both)?
  • Why is it not a good idea to perform an experiment on just one or only a handful of subjects?

The above questions are related to the three basic principles of experimental designs, namely control, randomization and repetition. Any design that is lacking in any one of these principles is a defective design. Some of these questions are derived from the idea: what happens when one or more of these principles is violated?

Some questions about specific experimental designs.

  • Why are completely randomized designs often inferior to block designs and matched pairs designs?
  • What is the advantage of block designs over completely randomized designs?
  • A controlled experiment manipulates the factor being studied while controlling other factors that are not being studied but may affect the outcome of interest. How do the three major experimental designs (completely randomized designs, block designs and matched pairs designs) control the other factors that are not being studied?

Some questions about the contrast between experiments and observational studies.

  • Why observational studies generally provide less convincing proof for cause and effect than controlled experiments?
  • Why are controlled experiments not a feasible method to resolve all research questions?
  • What are some examples of research questions that cannot be answered using controlled experiments?


  1. Moore. D. S., McCabe G. P., Craig B. A., Introduction to the Practice of Statistics, 6th ed., W. H. Freeman and Company, New York, 2009
This entry was posted in Observational Studies, Randomized Experiments, Statistical studies, Statistics and tagged , , , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s