Evaluating Statistical Claims: Observational Studies and Experiments
Distinguish observational studies from experiments, random sampling from assignment, and association from causation in SAT statistical claims.
Evaluating Statistical Claims: Observational Studies and Experiments
SAT Math — Problem-Solving and Data Analysis
Learning Objectives
By the end of this lesson, you should be able to:
Distinguish observational studies from experiments.
Identify populations, samples, and treatment groups.
Explain the purpose of random sampling.
Explain the purpose of random assignment.
Identify appropriate populations for generalizing results.
Distinguish association from causation.
Understand treatment groups and control groups.
Identify confounding variables and sources of bias.
Recognize when an experiment provides evidence of a causal relationship.
Evaluate statistical claims using the information provided.
1. Understanding Statistical Claims
A statistical claim is a statement about a population, relationship, or effect based on collected data.
Examples:
Students who exercise regularly tend to have higher fitness scores.
A new teaching method improves examination performance.
Approximately 65% of residents support a new public park.
Students who sleep longer tend to report less stress.
Some claims describe an association between variables.
Other claims suggest that one variable causes a change in another.
These are different conclusions and require different kinds of evidence.
SAT Tip: A statistical question often asks which conclusion is justified, not simply whether two groups have different numerical results.
2. Observational Studies
An observational study collects information about people or objects without assigning treatments or deliberately changing the conditions being studied.
Researchers observe existing characteristics, choices, or behaviors.
Example
A researcher records students' daily study hours and examination scores.
The researcher discovers that students who study longer generally have higher scores.
This is an observational study because the researcher did not randomly assign students to different study schedules.
The results show an association.
They do not, by themselves, prove that additional study hours caused the higher scores.
Other factors may influence performance, including prior knowledge, study techniques, motivation, and access to resources.
Characteristics of Observational Studies
Researchers observe existing differences.
Participants are not randomly assigned to treatments by the researchers.
Relationships or associations can be investigated.
Confounding variables may affect the results.
Causal conclusions are generally not justified from association alone.
3. Experiments
An experiment investigates the effects of a treatment or intervention.
Researchers assign treatments or conditions and compare outcomes.
Example
A school wants to determine whether a new mathematics learning application improves student performance.
Researchers recruit 200 students.
Students are randomly assigned to two groups.
Group A uses the new learning application.
Group B uses the school's existing study resources.
Both groups complete the same final assessment.
This is a randomized experiment.
Random assignment helps create comparable groups, providing stronger evidence that any observed difference in outcomes was caused by the assigned treatment.
Characteristics of Experiments
Researchers assign a treatment or condition.
Outcomes are measured and compared.
Random assignment can reduce systematic differences between groups.
Well-designed randomized experiments can provide evidence of causation.
Important: Not every experiment uses random assignment. Experiments without random assignment may provide weaker evidence for causal conclusions.
4. Observational Study vs. Experiment
Feature | Observational Study | Experiment |
|---|---|---|
Researcher assigns treatment | No | Yes |
Existing behaviors may be observed | Yes | Yes |
Can identify associations | Yes | Yes |
Random assignment possible | Not for the observed exposure | Yes |
Strong evidence of causation | Usually not from association alone | Possible with appropriate randomized design |
SAT Comparison
Observational study:
Researchers compare students who already choose to use an SAT preparation application with those who do not.
Randomized experiment:
Researchers randomly assign some students to use the application and others to use a comparison method.
The second design provides stronger evidence about the application's causal effect.
5. Association vs. Causation
An association occurs when two variables are statistically related.
Causation means a change in one variable produces a change in another.
An association does not automatically establish causation.
Example
A study finds that people who buy more sunscreen also spend more time outdoors.
Buying sunscreen and time spent outdoors may be associated.
However, purchasing sunscreen does not necessarily cause people to spend more time outdoors.
A third factor, such as sunny weather, may influence both variables.
This is an example of a potential confounding variable.
Important SAT Rule
Correlation or association alone does not prove causation.
Before accepting a causal claim, examine how the study was designed.
6. Random Sampling
Random sampling is a method of selecting participants using chance.
Its main purpose is to obtain a sample that can support inferences about the population from which it was selected.
Example
A school has 2,000 students.
Researchers randomly select 200 students from the school's complete enrollment list.
These 200 students form a random sample of the school's student population.
If the sampling method is appropriate, the results can help estimate characteristics of students at that school.
However, the results cannot automatically be generalized to all students in the country.
Key Principle
Random sampling supports population generalization.
The population to which results can reasonably be generalized depends on the population from which the sample was selected.
7. Random Assignment
Random assignment is different from random sampling.
Random assignment means using chance to place participants into treatment or comparison groups.
Its main purpose is to make groups comparable and reduce systematic differences that could explain the treatment effect.
Example
A researcher recruits 160 volunteers.
The volunteers are randomly assigned:
80 receive a new study method.
80 receive the existing study method.
Both groups take the same assessment afterward.
Because participants were randomly assigned, differences between the groups provide evidence about the effect of the assigned study method.
Key Principle
Random assignment supports causal inference.
However, random assignment alone does not guarantee that results apply to everyone outside the participants in the experiment.
8. Random Sampling vs. Random Assignment
This is one of the most important distinctions in SAT statistical reasoning.
Method | Main purpose |
|---|---|
Random sampling | Support generalization to a population |
Random assignment | Support conclusions about cause and effect |
Four Important Study Designs
Case 1: Random Sample, No Random Assignment
A survey randomly selects residents and records their existing exercise habits.
Population generalization may be supported.
Causation is not established.
Case 2: Random Assignment, No Random Sample
Researchers recruit volunteers and randomly assign them to two teaching methods.
A causal conclusion about the treatment may be supported.
Generalization beyond the volunteer participants is limited.
Case 3: Random Sample and Random Assignment
Researchers randomly sample students from a school and randomly assign them to different learning methods.
Both causal inference and generalization to the sampled school population may be supported, assuming the study is appropriately conducted.
Case 4: Neither Random Sampling nor Random Assignment
Researchers survey volunteers from a single online group about their existing habits.
Strong causal claims are not justified, and generalization may be limited.
SAT Tip: When a question asks whether results can be extended to a population, look for random sampling. When it asks whether a treatment caused an outcome, look for random assignment.
9. Treatment Groups and Control Groups
A treatment group receives the intervention being tested.
A control or comparison group provides a basis for evaluating the treatment.
Example
A company tests a new training program.
Treatment group:
Employees receive the new training.
Control group:
Employees continue using the existing training.
Afterward, both groups complete the same skills assessment.
The researcher compares the group results to investigate the effect of the new training.
A control group does not always receive nothing. It may receive an existing treatment, a placebo, or another appropriate comparison condition.
10. Confounding Variables
A confounding variable is an outside factor associated with the explanatory variable that may also influence the response.
Example
Students who attend optional tutoring obtain higher average examination scores.
Can we conclude that tutoring caused the higher scores?
Not necessarily.
Students choosing tutoring may differ in motivation, prior knowledge, family support, or study habits.
These factors could influence performance independently of tutoring.
This makes a causal conclusion from the observational comparison uncertain.
Random assignment can help reduce the influence of these preexisting differences.
11. Sampling Bias
Sampling bias occurs when the method used to select participants systematically favors certain groups.
Example 1 — Convenience Sampling
A school wants to estimate how many students enjoy reading.
Researchers survey only students who are currently in the library.
Library users may be more interested in reading than other students.
The sample may not represent the entire school.
Example 2 — Voluntary Response
A website asks visitors to participate voluntarily in a survey.
People with stronger opinions may be more likely to respond.
The result may not represent all website users.
Example 3 — Incomplete Sampling Frame
A city wants to study all residents but samples only people who own cars.
Residents without cars are excluded.
The sample may not represent the full population.
SAT Tip: A large sample does not automatically eliminate sampling bias.
12. Population Generalization
Generalization means using a sample's results to draw conclusions about a broader population.
Example
A researcher randomly selects 300 students from one high school.
The students answer questions about study habits.
The results may reasonably be generalized to students at that high school.
However, the results do not automatically describe all teenagers nationwide.
SAT Rule
A sample's results should generally be extended only to the population the sampling method was designed to represent.
Random sampling does not automatically justify generalization to unrelated populations.
13. Understanding Causal Evidence
Suppose a randomized experiment compares two teaching methods.
Treatment group average score: 82
Control group average score: 75
Difference: 82 − 75 = 7 points
The treatment group scored 7 points higher on average.
Because the study used random assignment, this difference provides evidence of a treatment effect.
However, the strength of the conclusion also depends on factors such as sample size, variability, measurement quality, adherence to assigned treatments, and the possibility of chance differences.
Important: A difference in sample means is evidence to investigate, not a guarantee that every individual receiving the treatment will improve.
14. Misleading Statistical Claims
SAT questions may present conclusions that go beyond what a study supports.
Example A
Claim: A survey shows that students who drink more water score higher on tests. Therefore, drinking water causes higher scores.
Problem: The survey is observational. Other variables may influence scores.
Example B
Claim: An experiment involving 40 volunteers proves that a program works equally well for all students worldwide.
Problem: Even with random assignment, a small nonrandom volunteer group does not automatically represent all students worldwide.
Example C
Claim: A poll of 5,000 customers on a company's own website represents the opinions of all residents in a city.
Problem: The sample may exclude residents who do not use that website.
Example D
Claim: A randomized experiment demonstrates an average improvement of 6 points, so every participant must improve by 6 points.
Problem: An average difference does not describe every individual result.
15. Numerical Reasoning in Statistical Claims
SAT questions in this skill sometimes include group counts, percentages, and averages.
You may need to calculate an observed difference before interpreting the statistical conclusion.
Example
A study randomly assigns 120 students to two learning methods.
Group | Students | Mean score |
|---|---|---|
New method | 60 | 84 |
Existing method | 60 | 78 |
Observed difference:
84 − 78 = 6 points
Because the students were randomly assigned, the study provides evidence that the new method may have caused the higher average score.
However, this comparison alone does not establish that all students would achieve the same improvement.
16. Worked SAT-Style Examples
Example 1 — Identifying an Observational Study
Problem
Researchers collect information about the number of hours students sleep and their examination scores. No sleep schedules are assigned.
What type of study is this?
Solution
Step 1: Identify whether the researchers assigned a treatment.
They did not assign sleep schedules.
Step 2: Researchers only recorded existing behaviors and results.
Answer: Observational study
The study can identify an association but does not, by itself, establish causation.
Example 2 — Identifying an Experiment
Problem
A researcher randomly assigns 100 students to use one of two mathematics study methods and compares their results.
What type of study is this?
Solution
Step 1: The researcher assigns study methods.
Step 2: Participants are randomly assigned to groups.
Answer: Randomized experiment
This design can provide evidence about the causal effect of the study method.
Example 3 — Generalization
Problem
A random sample of 250 students is selected from a school containing 1,500 students.
The sample is surveyed about school lunches.
To which population can the results most reasonably be generalized?
Solution
Step 1: Identify the sampling frame.
Students were selected from one school.
Step 2: Identify the target population represented.
All 1,500 students at that school.
Answer: Students at the school
Example 4 — Random Assignment and Causation
Problem
A company recruits 200 employees and randomly assigns them to two training programs.
The employees using Program A achieve a higher average assessment score.
Which conclusion is best supported?
Solution
Step 1: The researchers assigned the training programs.
Step 2: Random assignment reduces systematic differences between the groups.
Step 3: The resulting comparison provides evidence about the effect of the training programs.
Answer: There is evidence that the assigned training method affected average scores.
Example 5 — Sampling Bias
Problem
A school surveys only basketball team members to estimate the percentage of all students who enjoy basketball.
Why is this problematic?
Solution
Step 1: The target population is all school students.
Step 2: The sample contains only basketball players.
Step 3: Basketball players are more likely than average to enjoy the sport.
Answer: The sample is likely biased and unrepresentative.
Example 6 — Difference in Group Means
Problem
A randomized experiment compares two teaching programs.
The treatment group has a mean test score of 86.
The control group has a mean score of 79.
What is the observed difference in mean scores?
Solution
Difference = 86 − 79
Difference = 7
Answer: 7 points
Example 7 — Random Assignment Without Random Sampling
Problem
A researcher recruits 120 volunteers and randomly assigns them to two exercise programs.
What is the most appropriate interpretation?
Solution
Step 1: Random assignment supports comparisons about causal effects.
Step 2: Participants were volunteers rather than a random sample of all adults.
Step 3: Therefore, generalization beyond the participants may be limited.
Answer: The design supports investigating causation, but broad population generalization is limited.
Example 8 — Observational Association
Problem
Researchers find that students who participate in extracurricular activities tend to have higher grades.
The researchers did not assign students to participate.
Can the study establish that extracurricular participation causes higher grades?
Solution
Step 1: The study observes existing participation.
Step 2: There was no random assignment.
Step 3: Other factors, such as motivation and prior achievement, may influence both participation and grades.
Answer: No. The study identifies an association, not sufficient evidence of causation.
17. SAT Quick Reference
Observational study: Researchers observe existing conditions without assigning treatments.
Experiment: Researchers assign a treatment or intervention.
Random sample: Supports generalization to the sampled population.
Random assignment: Supports causal inference through comparable treatment groups.
Treatment group: Receives the intervention.
Control group: Provides a comparison.
Confounding variable: Another factor that may influence the observed relationship.
Sampling bias: Systematic distortion caused by how a sample is selected.
Association: Two variables are related.
Causation: Changing one variable causes a change in another.
18. Common SAT Mistakes
Assuming an association proves causation.
Confusing random sampling with random assignment.
Thinking every experiment is randomized.
Assuming any large sample is representative.
Generalizing a school's results to every school in a country.
Treating volunteer participants as a random sample.
Assuming a control group must receive no treatment.
Ignoring confounding variables.
Claiming an average improvement applies to every participant.
Assuming random assignment guarantees identical groups.
Assuming random assignment alone allows worldwide generalization.
Confusing a statistical difference with proof of a universal effect.
Overlooking which population was actually sampled.
Assuming every reported numerical relationship is causal.
19. Final Revision Checklist
Before attempting SAT questions, make sure you can:
Identify observational studies and experiments.
Identify a treatment group and a control group.
Explain what random sampling accomplishes.
Explain what random assignment accomplishes.
Recognize when causal evidence is supported.
Determine which population can be represented by a sample.
Identify confounding variables.
Recognize biased sampling methods.
Evaluate whether a statistical conclusion is justified.
Interpret differences between experimental groups.
Final SAT Strategy: Ask two questions: "Were the participants randomly sampled?" and "Were they randomly assigned to treatments?" The first helps determine generalizability; the second helps determine evidence for causation.