Experimental Methods

I think there are only very few people that can structure this course as well as Professor Morton. The feedback she provided during consultations and class presentations was extremely useful. — participant from Switzerland

Social scientists are increasingly interested in the conditions under which causal relationships can be identified and demonstrated to be robust within and beyond a given dataset. This course explores how to establish causality by using experimental reasoning in the analysis of non-experimental data as well as how to conduct experiments in the field or lab. Participants will learn the conditions and assumptions necessary for causality and the problems of generalization from a given dataset, whether experimental or not.


Dates

This course was offered in 2012, 2013, and 2014.


Instructor

Rebecca B. Morton (picture), New York University


Detailed Description

Experimental methodology has become 'hot' in recent years, as illustrated by the fact that a lab and field experimenter in political science, the late Elinor Ostrom, was awarded the Economics Nobel Prize in 2009.

There are several reasons why experiments are now a commonly used tool in the social sciences. First, researchers turned to experiments in order to establish causal inferences, using the techniques of control and random assignment. In this course we spend substantial time discussing how these two aspects of experimental design allow for the establishment of causal inferences in ways not available when working with observational data alone. Many researchers are turning to experiments as a cost effective and rigorous way to study natural political phenomena in developing countries for which observational data is of particularly poor quality and causal inferences are exceedingly difficult to establish. We discuss the advantages of experimentation as a method of generating data for the study of comparative politics.

Second, researchers have found that experiments allow for the evaluation of nuanced aspects of formal modeling that is often not possible with observational data and the investigation of processes largely unobservable and unmeasurable (such as mental processes, free-form communication between individuals, and physical and time responses to political stimuli). We explore how these methods have been used as well.

The course starts out with the study the theory of causal inference. This first part has four subparts. First, we begin with a discussion of the nature of causal inference, the Rubin Causal Model (RCM), measures of causality, and the importance of the two aspects of control and random assignment in establishing causal inferences. Second, we examine in detail how control is implemented in experimentation and discuss how control works in the laboratory and the field.

Third, we turn to an in depth consideration of random assignment in experimentation and pseudo random assignment methods in observational data (such as instrumental variables) and the establishment of causality. Special attention will be paid to methods of injecting randomization in the lab (e.g., using a common homogeneous subject pool and within-subject designs), in the field (e.g., information campaigns, spatially clustered treatments, multilevel modeling in randomization) and in survey experiments (e.g., list experiments, informational cues). We discuss dealing with the important problems in randomization in the field such as noncompliance, nonresponse, and violations of stable unit treatment value assumption (SUTVA).

Fourth, we address the formal theory approach to establishing causality, which allows researchers to investigate causal predictions drawn from formal models as well as the assumptions underlying causal theoretical propositions. In experiments using a RCM based approach, researchers simply assume that there is coherence between the theoretical predictions investigated and the empirical evaluation. In the formal theory approach, the connection between the two is explicitly stated and the disconnects that exist are explicit as well. The formal theory approach also allows for a relaxation of the notorious SUTVA assumption of RCM. Although mainly known as an approach used in the lab, we discuss how formal theory is used to establish causal inferences in field and survey experiments as well. Finally, we discuss approaches such as quantal response equilibrium and k-level cognitive hierarchical models that have been used in behavioral game theoretic experiments.

We turn to applied issues in experimentation in the second part of the course. This part has two sub-parts. First, we consider the issue of validity of experimentation. We discuss the three aspects of internal validity: statistical, causal, and construct. As the previous part explores causal and construct validity, we spend most of our time on statistical validity. We then consider the external validity of experimental results and how external validity can be established through scientific replication. We consider nonrandom holdout procedures as well as meta-analyses.

Second, we explore many of the nuanced issues in experimentation, such as choosing an experimental location, an experimental environment within that location, subjects, and a method of motivating subjects. We explore the issue of artificiality in experimentation and the experimental environment and discuss choosing and recruiting subjects, motivating subjects (either using financial incentives or not), dealing with risk aversion, and writing instructions.

The last part of the course deals with ethical considerations in experiments. We consider the risks and benefits from experimentation for subjects and third parties in the lab and field, the practical issues in dealing with Institutional Review Boards (IRB), the role of informed consent and the difficulties of informed consent in field experimentation. Participants learn how to secure IRB approval for their own experimental designs. Finally, we explore the debate over deception in experimentation, discussing the types of deception in experimentation and the methodological and ethical debate over deception.


Prerequisites

There are no formal prerequisites. However, familiarity with some basic statistical concepts would be beneficial.


Requirements

Participants are expected to bring a WiFi-enabled laptop computer. Access to data, temporary licenses for the course software, and installation support will be provided by the Methods School.


Core Readings

Morton, Rebecca B., and Kenneth C. Williams. 2010. From Nature to the Lab. Experimental Political Science and the Study of Causality. New York, NY: Cambridge University Press.