In probability and statistics, an experiment typically refers to a study in which the experimenter is trying to determine whether there is a relationship between two or more variables. In an experiment, the subjects are randomly assigned to either a treatment group or a control group (there can be more than one of either group).

Generally, the control group in an experiment receives a placebo (substance that has no effect) or no treatment at all. The treatment group receives the experimental treatment. The goal of the experiment is to determine whether or not the treatment has the desired/any effect that differs from the control group to a degree that the difference can be attributed to the treatment rather than to random chance or variability. Well-designed experiments can yield informative and unambiguous conclusions about cause and effect relationships.

As an example, if a scientist wants to test whether a new medication they developed has any effect, they would select subjects from a common population and randomly assign them to either a treatment group or a control group. They would then administer the treatment to the treatment group, and either a placebo or no treatment to the control group, and study the effects of each using statistical measures to determine whether the medication had any effect beyond chance or variability.

Note that an experiment does not necessarily need to have a physical treatment. The term "treatment" is used fairly loosely. Another experiment could look at the effects of getting advice from a college counselor on admission rates compared to not getting advice from a college counselor. In this case, the "treatment" would be getting advice from a college counselor. The control group would get no advice from a college counselor.

Importance of experimental design

Like survey methodology, experimental design is essential to the validity of the results of the experiment. A poorly designed experiment can result in false or incorrect conclusions. Proper statistical experiment design generally involves the following:

Experiments vs surveys

Experiments and surveys are both techniques used as part of inferential statistics. A survey involves the use of a random sample of the population, rather than the whole, with the goal that all subjects in the population have an equal chance of being selected. The random sample of the population is then used to draw conclusions or make inferences about the population as a whole.

In contrast, an experiment typically involves the use of random assignment such that all subjects have an equal chance of being assigned to the groups (treatment and control) in the study, which minimizes potential biases, as well as allows the experimenters to evaluate the role of variability in the experiment. This in turn allows them to determine whether any observed differences between the groups merit further study or not based on whether or not the differences can be attributed to variability or chance.