One group posttest only design

Research Design1

1.Based upon your reading of the experimental study of Project CRISS, and the information provided by the authors of the study, provide a discussion board posting of 200 - 250 words in which you analyze the study in terms of threats to the validity of the study. Be sure to address both external and internal validity.

Single-Subject Experimental design

2. Follow the steps for application of single-subject experimental design as described above. Select one of these research studies and do the following: Explain how the study might be conducted and draw a representation of your proposed design.

Chapter 10: Experimental Research

Educational Research:

Competencies for Analysis and Application

11/E

Geoffrey Mills and Lorraine Gay

© 2016, 2012, 2009, 2006 Pearson Education, Inc. All Rights Reserved

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

After reading Chapter 10, you should be able to do the following:

Briefly define and state the purpose of experimental research.

Briefly explain the threats to validity in experimental research.

Define and provide examples of group experimental designs.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Experimental Research

Definition and purpose

Experimental research is the only type of research that can test hypotheses to establish cause-effect relations.

The researcher manipulates at least one independent variable and controls other relevant variables, and observes the effect on one or more dependent variables.

The researcher has control over selection and assignment.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Experimental Research

In experimental research studies the independent variable is also called the treatment, causal, or experimental variable.

In experimental research studies the dependent variable is also called the criterion, effect, or posttest variable.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Experimental Research

Experimental research is the most structured of all research.

When conducted well, experimental research studies produce the soundest evidence for cause-effect relations.

Replicating an experiment involving different contexts and participants often produces results than can be widely generalized.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Experimental Research

The experimental process

The steps in the experimental research process are the same as in other types of research.

Selecting and defining a problem

Selecting participants and measuring instruments

Preparing a research plan

Executing procedures

Analyzing the data

Formulating conclusions

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Experimental Research

The experimental process

In experimental studies, the researcher controls selection and assignment.

Experimental studies often examine comparisons between or among groups.

Comparison of two different approaches (A versus B)

Comparison of a new approach and the existing approach (A versus no A)

Comparison of different amounts of a single approach (A little of A versus a lot of A)

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Experimental Research

The experimental process

In experimental research studies the group that receives the treatment is the experimental group.

The group that does not receive the treatment is called the control group.

Sometimes groups are comparison groups that receive alternative treatments (e.g., two types of instruction in a content area).

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Experimental Research

Experimental studies in educational settings often have two problems:

A lack of sufficient exposure to treatments (i.e., treatments are too short or diffuse).

Failure to make treatments significantly different from one another (e.g., an experimental instructional program in math may not be different enough from the comparison math instructional program).

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Experimental Research

Manipulation and control

In experimental studies, researchers directly manipulate independent variables and control, or remove, the influence of extraneous variables.

It is challenging to control all the relevant extraneous variables.

Participant variables

Organismic (e.g., age)

Environmental variables (e.g., school or teacher effects)

Concentrate on controlling variables that affect or interact with the dependent variable.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to Experimental Validity

Statistical conclusion validity

Statistical conclusion validity refers to appropriate use of statistics to infer whether the the presumed independent and dependent variables co-vary in the experiment..

Internal validity

Internal validity refers to the degree to which observed differences in the dependent variable are a direct result of manipulation of the independent variable and not some other variable.

Internal validity is concerned with rival explanations for an effect.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to Experimental Validity

Construct validity

Construct validity refers to inferences about the variables or constructs in a study.

External validity

External validity, also called ecological validity, refers to the degree to which the results from a study are generalizable to other groups.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to Experimental Validity

Internal and external validity

When researchers increase the internal validity of their study, they decrease their external validity.

When researchers are concerned with external validity, their ability to control important extraneous variables suffers.

When there is a choice, researchers should err on the side of control and maximize internal validity.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to Internal Validity

History

Maturation

Testing

Instrumentation

Statistical regression

Differential selection of participants

Mortality

Selection-maturation and interaction effects

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to Internal Validity

History

Any event occurring during a study that is not part of the experimental treatment but that may effect the dependent variable represents a history threat.

Longer-lasting studies are more prone to history threats.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to Internal Validity

History threat example

In a study of the effects of instructional simulations in learning chemistry content, a history threat would be demonstrated if students in the study were exposed to simulations in a different setting, such as when learning geography, while the study was being conducted.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to Internal Validity

Maturation

Maturation refers to physical, intellectual, and emotional changes that naturally occur within participants over a period of time.

Maturation threat example

In studies of interventions that are designed to increase children’s theory of mind, if the interventions lasted more than a couple of weeks at critical time points, participants may gain significant theory of mind awareness simply due to cognitive development and not due to the treatment.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to Internal Validity

Testing

Testing as a threat to internal validity is demonstrated when taking a pretest alters the result of a posttest.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to Internal Validity

Instrumentation

Instrumentation is a threat to internal validity when the instrumentation is either unreliable or is changed between pre- and posttesting.

Instrumentation threat example

A researcher collects data through observing the classroom. Pretreatment observations adhere closely to the observation protocol. However, at posttreatment, the observers deviate and only record behavior supporting the hypotheses.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to Internal Validity

Statistical regression

Extremely high or low scores tend to regress to the mean on retesting.

Statistical regression example

If students perform poorly on a pretest it is difficult to determine if the gain in their scores is due to treatment effects.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to Internal Validity

Differential selection of participants

Participants in the control and experimental groups differ in ways that influence the dependent measure.

Mortality

Mortality refers to attrition or a reduction in the number of research participants over time as individuals drop out of a study.

Do not assume attrition is random.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to Internal Validity

Selection-maturation interaction and other interactive effects

Participants selected into the treatment and control conditions have different experiences or maturation rates

Selection-instrumentation occurs when instrumentation varies across conditions.

Selection can interact with other threats to internal validity (i.e., history, maturation, instrumentation).

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to External Validity

External validity threats can be divided into two categories:

‘Generalizing to whom’ threats

Threats affecting groups to which the study can be generalized

‘Generalizing to what’ threats

Threats affecting the settings, conditions, variables, and contexts to which the results can be generalized

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to External Validity

Pretest-treatment interaction

Multiple-treatment interference

Selection-treatment interaction

Specificity of variables

Treatment diffusion

Experimenter effects

Reactive arrangements

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to External Validity

Pretest-treatment interaction

This threat occurs when participants respond differently to a treatment because they have been exposed to a pretest.

Pretest may alert participants.

Self-report measures are often susceptible to pretest-treatment interaction effects.

At times, unobtrusive measures can be used as pretests, to limit this threat to validity (e.g., using previously administered standardized assessments to measure ability in science instead of using a pretest).

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to External Validity

Multiple-treatment interference

This threat occurs when previous treatments cross-over into a current experiment. This makes it challenging to determine the effectiveness of the later treatment.

This threat may occur in studies that access participants who have been exposed to other research studies (e.g., university participant pools).

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to External Validity

Selection-treatment interaction

When a study’s findings only apply to the groups selected and are not representative of other groups.

This may happen in non-randomly assigned studies where a treatment is less or more effective for certain demographics (e.g., ability levels).

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to External Validity

Specificity of variables

When researchers do not adequately define their variables, instruments, or population, it makes it difficult to determine how well the findings will generalize to an alternative population.

Related threats

Interaction of history and treatment

Interaction of time of measurement and treatment effect

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to External Validity

Treatment diffusion

When different treatment groups communicate with and learn from each other, the treatments are no longer distinctly different. The treatments overlap.

For example, two classes are randomly assigned to a different approach to math instruction. Students and teachers begin talking and one treatment diffuses into the other.

To reduce treatment diffusion, instruct teachers to not discuss the treatment until the study is completed.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to External Validity

Experimenter effects

Experimenter effects occur when characteristics or behaviors of the experimenter influence the participants’ responses.

These might be personal attributes effects

Research expectations effects: Experimenter bias

Reactive arrangements

These threats are also referred to as participant effects. These threats are associated with differences in participants’ behavior, feelings, and attitudes because they are in a study.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to External Validity

Reactive arrangements

Hawthorne effect: Any situation in which participants’ behavior is affected because they are in a study.

John Henry effect:(Compensatory rivalry): Members of the control group compete with the experimental group.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Threats to External Validity

Reactive arrangements

Placebo effect: To combat compensatory rivalry, researchers attempt to give control groups a placebo, not the experimental treatment, but something to decrease the perception that they are in the control group. Participants should perceive they are all getting the same thing.

Novelty effect: When participants are engaged in something different this may increase attention, interest, behavior, learning, etc., just because it is something new.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Control of Extraneous Variables

The validity of an experiment is a function of the degree to which extraneous variables are controlled.

Randomization is the best mechanism to control for extraneous variables.

Randomization distinguishes experimental designs.

Randomization should be used whenever possible.

If groups cannot be randomly formed, variables should be held constant when at all possible (e.g., time of day, which researcher is present).

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Control of Extraneous Variables

Confound:

A situation in which the effects of the independent variable are intertwined with extraneous variables and it is difficult to determine independent effects.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Control of Extraneous Variables

Randomization

Participant variables can be controlled and held constant

Matching can equate groups through random assignment of pairs.

Comparing homogeneous groups allows the researcher to control for extraneous variables.

Participant as their own controls involve a single group exposed to multiple treatments, one at a time.

Analysis of covariance (ANCOVA)can be used to control for participant variables.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Group Experimental Designs

There are two major classes of experimental designs: single-variable and factorial designs.

Single-variable designs are any design that involves one manipulated variable.

Pre-experimental designs do not adequately control for extraneous variables and should be avoided.

True-experimental designs offer a very high degree of control and are always preferred designs.

Quasi-experimental designs do not control as well as experimental designs but are preferable over pre-experimental designs.

Factorial designs are any design that involves two or more independent variables, at least one that is manipulated.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Pre-Experimental Designs

The designs do not do a good job of controlling extraneous variables that jeopardize validity.

These designs do not control for maturation.

There may be some control for history in this design.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Pre-Experimental Designs

The one-group posttest-only design involves a single group that is exposed to a treatment (X) and then tested (O).

X O

Threats to validity are not adequately controlled with this design.

Avoid this design if at all possible.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Pre-Experimental Designs

The one-group pretest-posttest design involves a single group that is pretested, exposed to treatment, and then tested again.

O X O

The success of the treatment is determined by comparing pretest and posttest scores.

This design does not control for history, testing, instrumentation, regression, or maturation.

Statistical regression is not controlled nor is pretest-treatment interaction.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Pre-Experimental Designs

The posttest-only design with nonequivalent groups involves at least two nonrandomly formed groups.

One group receives an experimental treatment and the other group receives the traditional treatment. Both groups are posttested.

X1 O

X2 O

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

True Experimental Designs

The pretest-posttest control group design requires at least two groups.

Groups are formed by random assignment.

Both groups are administered a pretest, each group receives a different treatment and both groups are posttested.

The design may be extended to include additional groups.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

True Experimental Designs

R O X1 O

R O X2 O

R O X3 O

The combination of random assignment and the presence of a pretest and a control group serve to control for all threats to internal validity.

The only potential weakness in this design is a possible interaction between the pretest and the treatment.

Researchers should report assess and report the probability of a pretest-treatment interaction

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

True Experimental Designs

One variation includes random assignment of matched pairs to the treatment groups.

There is little advantage to this variation.

Another variation of this design involves one or more additional posttests.

R O X1 O O

R O X2 O O

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

True Experimental Designs

The posttest-only control group design is the same as the pretest-posttest control group design except that it lacks a pretest.

R X1 O

R X2 O

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

True Experimental Designs

This design is often expanded to include more than two groups.

The posttest-only control group design is best used when there is likelihood of a pretest-treatment interaction threat.

As with the pretest-posttest control group design, the addition of a matched random assignment does not represent an increased advantage.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

True Experimental Designs

The Solomon Four-Group Design is a combination of the pretest-posttest control group design and the posttest-only control group design.

R O X1 O

R O X2 O

R X1 O

R X2 O

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

True Experimental Designs

The analysis of the Solomon four-group design is a 2 x 2 factorial analysis of variance.

This analysis tests whether those who received the treatment performed different than those who did not.

This analysis can assess for a testing effect.

This analysis assesses for pretest-interaction effects.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

True Experimental Designs

The Solomon four-group design requires a large number of participants.

The Solomon four-group design may not always be the best design.

The design selected should be based upon potential threats and the nature of the proposed study.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Quasi-Experimental Designs

When it is not possible to assign participants to groups randomly, researchers can use quasi-experimental studies.

In the nonequivalent control group design, two or more treatment groups are pretested, administered a treatment, and posttested.

O X1 O

O X2 O

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Quasi-Experimental Designs

The nonequivalent control group design involves the random assignment of groups not individuals.

The lack of random assignment introduces validity threats (e.g., regression, and selection interaction effects).

To reduce threats when using this design researchers often assure groups are as equivalent as possible (e.g., use ANCOVA).

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Quasi-Experimental Designs

The time-series design is an elaboration of the pretest-posttest design.

One group is repeatedly pretested until pretest scores are stable. The group is then exposed to a treatment and after treatment is repeatedly posttested.

O O O O X O O O O

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Quasi-Experimental Designs

A variation of the time-series design is the multiple time-series design that includes a control group.

This variation eliminates the history and instrumentation threats.

O O O O X1 O O O O

O O O O X2 O O O O

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Quasi-Experimental Designs

History is a threat with time-series designs.

Instrumentation may also be a threat if testing changes.

Pretest threats are problematic with time-series designs. However, it is relatively easy to establish the degree of the threat given data from repeated testing.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Quasi-Experimental Designs

In a counterbalanced design, all groups receive all treatments but in a different order, and all groups are posttested after each treatment.

Counterbalanced designs can include any number of groups.

The number of groups is equal to the number of treatments.

Treatment order is randomly assigned.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Quasi-Experimental Designs

A unique threat with a counter-balanced design is multiple treatment interaction.

X1 O X2 O X3 O

X3 O X1 O X2 O

X2 O X3 O X1 O

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Factorial Designs

Factorial Designs are elaborations on single-variable experimental designs to permit investigation of two or more variables, at least one of which is manipulated by the researcher.

Factorial designs are often employed after an independent variable has first been investigated individually.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Factorial Designs

The purpose of a factorial design is to determine whether the effects of an independent variable are generalizable across all levels.

One example of a factorial design is the 2 X 2 design.

Type of instruction (computer-based or paper and pencil) by gender.

Many factors (independent variables) studies are possible to address specific research questions.

Gay & Mills

Educational Research, 11e

© 2016 Pearson Education, Inc. All rights reserved.

10-‹#›

Factorial Designs

Many factorial designs are possible but in reality more than 3 factors are rarely considered because of sample size considerations.

Interactions are difficult to interpret with multiple factors in factorial designs.

Gay & Mills

Educational Research, 11e