Experiment

In probability and statistics, an experiment typically refers to a study in which the experimenter is trying to determine whether there is a relationship between two or more variables. In an experiment, the subjects are randomly assigned to either a treatment group or a control group (there can be more than one of either group).

Generally, the control group in an experiment receives a placebo (substance that has no effect) or no treatment at all. The treatment group receives the experimental treatment. The goal of the experiment is to determine whether or not the treatment has the desired/any effect that differs from the control group to a degree that the difference can be attributed to the treatment rather than to random chance or variability. Well-designed experiments can yield informative and unambiguous conclusions about cause and effect relationships.

As an example, if a scientist wants to test whether a new medication they developed has any effect, they would select subjects from a common population and randomly assign them to either a treatment group or a control group. They would then administer the treatment to the treatment group, and either a placebo or no treatment to the control group, and study the effects of each using statistical measures to determine whether the medication had any effect beyond chance or variability.

Note that an experiment does not necessarily need to have a physical treatment. The term "treatment" is used fairly loosely. Another experiment could look at the effects of getting advice from a college counselor on admission rates compared to not getting advice from a college counselor. In this case, the "treatment" would be getting advice from a college counselor. The control group would get no advice from a college counselor.

Importance of experimental design

Like survey methodology, experimental design is essential to the validity of the results of the experiment. A poorly designed experiment can result in false or incorrect conclusions. Proper statistical experiment design generally involves the following:

• Identification of the explanatory variable, also referred to as the independent variable. The explanatory variable is the "treatment," or the thing that causes the change, and can be anything that causes a change in the response variable.
• Identification of the response variable, also referred to as the dependent variable. It is the variable that may be affected by the explanatory/independent variable.
• Defining the population of interest and taking a random sample from the population. Generally the larger the random sample, the less potential for sample error, since the larger sample will likely be more representative of the population.
• Random assignment of the subjects in the sample to either the treatment group or the control group.
• Administration of the treatment to the treatment group, and placebo (or nothing) to the control group), possibly using a blind experiment (the subject doesn't know whether they are receiving the treatment or the placebo) or double blind experiment (neither experimenter nor subject knows which treatment they are getting).
• Measurement of the response over a chosen period of time.
• Statistical analysis of the supposed response to determine whether there is an actual response, or the response can be attributed to chance, to determine whether there is a causal relationship between the treatment and the response.
• Replication of the experiment by peers, assuming there is a causal relationship between the treatment and the response.

Experiments vs surveys

Experiments and surveys are both techniques used as part of inferential statistics. A survey involves the use of a random sample of the population, rather than the whole, with the goal that all subjects in the population have an equal chance of being selected. The random sample of the population is then used to draw conclusions or make inferences about the population as a whole.

In contrast, an experiment typically involves the use of random assignment such that all subjects have an equal chance of being assigned to the groups (treatment and control) in the study, which minimizes potential biases, as well as allows the experimenters to evaluate the role of variability in the experiment. This in turn allows them to determine whether any observed differences between the groups merit further study or not based on whether or not the differences can be attributed to variability or chance.