Solomon four group design statistical analysis

Exercise 1 - Research Design Validity

Discuss the importance of validity and research design.

Next, choose one type of validity (internal, external, construct, or statistical conclusion) and discuss its relevance to experimental, quasi-experimental, and non-experimental research.

Exercise 2 - Comparison Groups

Comparison groups are one of the important elements to the scientific control of a research design.

Choose one type of comparison group from the list provided in the book and expand upon how the inclusion of this type of comparison group would improve the overall validity of the findings.

Exercise 3 - Control Techniques

Control is an important element in any type of research.

Considering experimental research, come up with a hypothetical research scenario and apply each of the five types of control to the scenario. Use specific examples to illustrate your point.

Exercise 4 - Establishing Cause and Effect

No unread replies.No replies.

What are the major differences between experimental, quasi experimental, and non-experimental research?

Discuss the three major conditions to meet cause and effect (be sure to review your text for further information). Provide a typical experimental "weakness" that wouldn't allow a researcher to determine cause and effect.

Chapter 1 A Primer of the Scientific Method and Relevant Components
The primary objective of this book is to help researchers understand and select appropriate designs for their investigations within the field, lab, or virtual environment. Lacking a proper conceptualization of a research design makes it difficult to apply an appropriate design based on the research question(s) or stated hypotheses. Implementing a flawed or inappropriate design will unequivocally lead to spurious, meaningless, or invalid results. Again, the concept of validity cannot be emphasized enough when conducting research. Validity maintains many facets (e.g., statistical validity or validity pertaining to psychometric properties of instrumentation), operates on a continuum, and deserves equal attention at each level of the research process. Aspects of validity are discussed later in this chapter. Nonetheless, the research question, hypothesis, objective, or aim is the primary step for the selection of a research design.

The purpose of a research design is to provide a conceptual framework that will allow the researcher to answer specific research questions while using sound principles of scientific inquiry. The concept behind research designs is intuitively straightforward, but applying these designs in real-life situations can be complex. More specifically, researchers face the challenge of (a) manipulating (or exploring) the social systems of interest, (b) using measurement tools (or data collection techniques) that maintain adequate levels of validity and reliability, and (c) controlling the interrelationship between multiple variables or indicating emerging themes that can lead to error in the form of confounding effects in the results. Therefore, utilizing and following the tenets of a sound research design is one of the most fundamental aspects of the scientific method. Put simply, the research design is the structure of investigation, conceived so as to obtain the “answer” to research questions or hypotheses.

The Scientific Method
All researchers who attempt to formulate conclusions from a particular path of inquiry use aspects of the scientific method. The presentation of the scientific method and how it is interpreted can vary from field to field and method (qualitative) to method (quantitative), but the general premise is not altered. Although there are many ways or avenues to “knowing,” such as sources from authorities or basic common sense, the sound application of the scientific method allows researchers to reveal valid findings based on a series of systematic steps. Within the social sciences, the general steps include the following: (a) state the problem, (b) formulate the hypothesis, (c) design the experiment, (d) make observations, (e) interpret data, (f) draw conclusions, and (g) accept or reject the hypothesis. All research in quantitative methods, from experimental to nonexperimental, should employ the steps of the scientific method in an attempt to produce reliable and valid results.

The scientific method can be likened to an association of techniques rather than an exact formula; therefore, we expand the steps as a means to be more specific and relevant for research in education and the social sciences. As seen in Figure 1.1, these steps include the following: (a) identify a research problem, (b) establish the theoretical framework, (c) indicate the purpose and research questions (or hypotheses), (d) develop the methodology, (e) collect the data, (f) analyze and interpret the data, and (g) report the results. This book targets the critical component of the scientific method, referred to in Figure 1.1 as Design the Study, which is the point in the process when the appropriate research design is selected. We do not focus on prior aspects of the scientific method or any steps that come after the Design the Study step, including procedures for conducting literature reviews, developing research questions, or discussions on the nature of knowledge, epistemology, ontology, and worldviews. Specifically, this book focuses on the conceptualization, selection, and application of common research designs in the field of education and the social and behavioral sciences.

Again, although the general premise is the same, the scientific method is known to slightly vary from each field of inquiry (and type of method). The technique presented here may not exactly follow the logic required for research using qualitative methods; however, the conceptualization of research designs remains the same. We refer the reader to Jaccard and Jacoby (2010) for a review on the various scientific approaches associated with qualitative methods, such as emergent- and discovery-oriented frameworks.

Figure 1.1 The Scientific Method

Figure 1

Validity and Research Designs
The overarching goal of research is to reach valid outcomes based upon the appropriate application of the scientific method. In reference to

Independent and Dependent Variables
In simple terms, the independent variable (IV) is the variable that is manipulated (i.e., controlled) by the researcher as a means to test its impact on the dependent variable, otherwise known as the treatment effect. In the classical experimental study, the IV is the treatment, program, or intervention. For example, in a psychology-based study, the IV can be a cognitive-behavioral intervention; the intervention is manipulated by the researcher, who controls the frequency and intensity of the therapy on the subject. In a pharmaceutical study, the IV would typically be a treatment pill, and in agriculture the treatment often is fertilizer. In regard to experimental research, the IVs are always manipulated (controlled) based on the appropriate theoretical tenets that posit the association between the IV and the dependent variable.

Statistical software packages (e.g., SPSS) refer to the IV differently. For instance, the IV for the analysis of variance (ANOVA) in SPSS is the “breakdown” variable and is called a factor. The IV is represented as levels in the analysis (i.e., the treatment group is Level 1, and the control group is Level 2). For nonexperimental research that uses regression analysis, the IV is referred to as the predictor variable. In research that applies control in the form of statistical procedures to variables that were not or cannot be manipulated, the IVs are sometimes referred to as quasi- or alternate independent variables. These variables are typically demographic variables, such as gender, ethnicity, or socioeconomic status. As a reminder, in nonexperimental research the IV (or predictor) is not manipulated whether it is a categorical variable such as hair color or a continuous variable such as intelligence. The only form of control that is exhibited on these types of variables is that of statistical procedures. Manipulation and elimination do not apply (see types of control later in the chapter).

The dependent variable (DV) is simply the outcome variable, and its variability is a function of IV and its impact on it (i.e., treatment effect). For example, what is the impact of the cognitive-behavioral intervention on psychological well-being? In this research question, the DV is psychological well-being. In regard to nonexperimental research, the IVs are not manipulated, and the IVs are referred to as predictors and the DVs are criterion variables. During the development of research questions, it is critical to first define the DV conceptually, then define it operationally.

A conceptual definition is a critical element to the research process and involves scientifically defining the construct so it can be systematically measured. The conceptual definition is considered to be the (scientific) textbook definition. The construct must then be operationally defined to model the conceptual definition.

An operational definition is the actual method, tool, or technique that indicates how the construct will be measured (see Figure 1.2).

Consider the following example research question: What is the relationship between Emotional Intelligence and conventional Academic Performance?

Figure 1.2 Conceptual and Operational Definitions

Figure 2

Internal Validity
Internal validity is the extent to which the outcome was based on the independent variable (i.e., the treatment), as opposed to extraneous or unaccounted-for variables. Specifically, internal validity has to do with causal inferences—hence, the reason why it does not apply to nonexperimental research. The goal of nonexperimental research is to describe phenomena or to explain or predict the relationship between variables, not to infer causation (although there are circumstances when cause and effect can be inferred from nonexperimental research, and this is discussed later in this book). The identification of any explanation that could be responsible for an outcome (effect) outside of the independent variable (cause) is considered to be a threat. The most common threats to internal validity seen in education and the social and behavioral sciences are detailed in Table 1.1. It should be noted that many texts do not indentify sequencing effects in the common lists of threats; however, it is placed here, as it is a primary threat in repeated-measures approaches.

Table 3

Construct Validity
Construct validity refers to the extent a generalization can be made from the operationalization (i.e., the scientific measurement) of the theoretical construct back to the conceptual basis responsible for the change in the outcome. Again, although the list of threats to construct validity seen in Table 1.3 are defined to imply issues regarding cause-effect relations, the premise of construct validity should apply to all types of research. Some authors categorize some of these threats as social threats to internal validity, and some authors simply categorize some of the threats listed in Table 1.3 as threats to internal validity. The categorization of these threats can be debated, but the premise of the threats to validity cannot be argued (i.e., a violation of construct validity affects the overall validity of the study in the same way as a violation of internal validity).

Statistical Conclusion Validity
Statistical conclusion validity is the extent to which the statistical covariation (relationship) between the treatment and the outcome is accurate. Specifically, the statistical inferences regarding statistical conclusion validity has to do with the ability with which one can detect the relationship between the treatment and outcome, as well as determine the strength of the relationship between the two. As seen in Table 1.4, the most notable threats to statistical conclusion validity are outlined. Violating a threat to statistical conclusion validity typically will result in the overestimation or underestimation of the relationship between the treatment and outcome in experimental research. A violation can also result in the overestimation or underestimation of the explained or predicted relationships between variables as seen in nonexperimental research.

Design Logic
The overarching objective of a research design is to provide a framework from which specific research questions or hypotheses can be answered while using the scientific method. The concept of a research design and its structure is, at face value, rather simplistic. However, complexities arise when researchers apply research designs within social science paradigms. These include, but are not limited to, logistical issues, lack of control over certain variables, psychometric issues, and theoretical frameworks that are not well developed. In addition, with regard to statistical conclusion validity, a researcher can apply sound principles of scientific inquiry while applying an appropriate research design but may compromise the findings with inappropriate data collection strategies, faulty or “bad” data, or misdirected statistical analyses. Shadish and colleagues (2002) emphasized the importance of structural design features and that researchers should focus on the theory of design logic as the most important feature in determining valid outcomes (or testing causal propositions). The logic of research designs is ultimately embedded within the scientific method, and applying the principles of sound scientific inquiry within this phase is of the utmost importance and the primary focus of this guide.

Control
Control is an important element to securing the validity of research designs within quantitative methods (i.e., experimental, quasi-experimental, and nonexperimental research). However, within qualitative methods, behavior is generally studied as it occurs naturally with no manipulation or control. Control refers to the concept of holding variables constant or systematically varying the conditions of variables based on theoretical considerations as a means to minimize the influence of unwanted variables (i.e., extraneous variables). Control can be applied actively within quantitative methods through (a) manipulation, (b) elimination, (c) inclusion, (d) group or condition assignment, or (e) statistical procedures.

Manipulation.
Manipulation is applied by manipulating (i.e., controlling) the independent variable(s). For example, a researcher can manipulate a behavioral intervention by systematically applying and removing the intervention or by controlling the frequency and duration of the application (see section on independent variables).

Elimination.
Elimination is conducted when a researcher holds a variable or converts it to a constant. If, for example, a researcher ensures the temperature in a lab is set exactly to 76° Fahrenheit for both conditions in a biofeedback study, then the variable of temperature is eliminated as a factor because it is held as a constant.

Inclusion.
Inclusion refers to the addition of an extraneous variable into the design to test its affect on the outcome (i.e., dependent variable). For example, a researcher can include both males and females into a factorial design to examine the independent effects gender has on the outcome. Inclusion can also refer to the addition of a control or comparison group within the research design.

Group assignment.
Group assignment is another major form of control (see more on group and condition assignments later). For the between-subjects approach, a researcher can exercise control through random assignment, using a matching technique, or applying a cutoff score as means to assign participants to conditions. For the repeated-measures approach, control is exhibited when the researcher employs the technique of counterbalancing to variably expose each group or individual to all the levels of the independent variable.

Statistical procedures.
Statistical procedures are exhibited on variables, for example, by systematically deleting, combining, or not including cases and/or variables (i.e., removing outliers) within the analysis. This is part of the data-screening process as well. As illustrated in Table 1.5, all of the major forms of control can be applied in the application of designs for experimental and quasi-experimental research. The only form of control that can be applied to nonexperimental research is statistical control.

Comparison and Control Groups

The group that does not receive the actual treatment, or intervention, is typically designated as the control group. Control groups fall under the group or condition assignment aspect of control. Control groups are comparison groups and are primarily used to address threats to internal validity such as history, maturation, selection, and testing. A comparison group refers to the group or groups that are not part of the primary focus of the investigation but allow the researcher to draw certain conclusions and strengthen aspects of internal validity. There are several distinctions and variations of the control group that should be clarified.

· Control group. The control group, also known as the no-contact control, receives no treatment and no interaction.

· Attention control group. The attention control group, also known as the attention-placebo, receives attention in the form of a pseudo-intervention to control for reactivity to assessment (i.e., the participant’s awareness of being studied may influence the outcome).

· Nonrandomly assigned control group. The nonrandomly assigned control is used when a no-treatment control group cannot be created through random assignment.

· Wait-list control group. The wait-list control group is withheld from the treatment for a certain period of time, then the treatment is provided. The time in which the treatment is provided is based on theoretical tenets and on the pretest and posttest assessment of the original treatment group.

· Historical control group. Historical control is a control group that is chosen from a group of participants who were observed at some time in the past or for whom data are available through archival records, sometimes referred to as cohort controls (i.e., a homogenous successive group) and useful in quasi-experimental research.

Sampling Strategies

A major element to the logic of design extends to sampling strategies. When developing quantitative, qualitative, and mixed methods studies, it is important to identify the individuals (or extant databases) from whom you plan to collect data. To start, the unit of analysismust be indicated. The unit of analysis is the level or distinction of an entity that will be the focus of the study. Most commonly, in social science research, the unit of analysis is at the individual or group level, but it can also be at the programmatic level (e.g., institution or state level).

There are instances when researchers identify units nested within an aggregated group (e.g., a portion of students within a classroom) and refer to this as nested designs or models. It should be noted that examining nested units is not a unique design, but rather a form of a sampling strategy, and the relevant aspects of statistical conclusion validity should be accounted for (e.g., independence assumptions). After identifying the unit, the next step is to identify the population (assuming the individual or group is the unit of analysis), which is the group of individuals who share similar characteristics (e.g., all astronauts). Logistically, it is impossible in most circumstances to collect data from an entire population; therefore, as illustrated in Figure 1.4, a sample (or subset) from the population is identified (e.g., astronauts who have completed a minimum of four human space-flight missions and work for NASA).

The goal often, but not always, is to eventually generalize the finding to the entire population. There are two major types of sampling strategies, probability and nonprobability sampling. In experimental, quasi-experimental, and nonexperimental (survey and observational) research, the focus should be on probability sampling (identifying and selecting individuals who are considered representative of the population). Many researchers also suggest that some form of probability sampling for observational (correlational) approaches (predictive designs) must be employed—otherwise the statistical outcomes cannot be generalizable. When it is not logistically possible to use probability sampling, or as seen in qualitative methods not necessary, some researchers use nonprobability sampling techniques (i.e., the researcher selects participants on a specific criterion and/or based on availability). The following list includes the major types of probability and nonprobability sampling techniques.

Probability Sampling Techniques
· Simple random sampling. Every individual within the population has an equal chance of being selected.

· Cluster sampling. Also known as area sampling, this allows the researcher to divide the population into clusters (based on regions) and then randomly select from the clusters.

· Stratified sampling. The researcher divides the population into homogeneous subgroups (e.g., based on age) and then randomly selects participants from each subgroup.

· Systematic sampling. Once the size of the sample is identified, the researcher selects every nth individual (e.g., every third person on the list of participants is selected) until the desired sample size is fulfilled.

· Multistage sampling. The researcher combines any of the probability sampling techniques as a means to randomly select individuals from the population.

Nonprobability Sampling Techniques
· Convenience sampling. Sometimes referred to as haphazard or accidental sampling, the investigator selects individuals because they are available and willing to participate.

· Purposive sampling. The researcher selects individuals to participate based on a specific need or purpose (i.e., based on the research objective, design, and target population); this is most commonly used for qualitative methods (see Patton, 2002). The most common form of purposeful sampling is criterion sampling (i.e., seeking participants who meet a specific criterion). Variations of purposive sampling include theory-guided, snowball, expert, and heterogeneity sampling. Theoretical sampling is a type of purposive sampling used in grounded-theory approaches. We refer the reader to Palinkas et al. (2014) for a review of recommendations on how to combine various sampling strategies for the qualitative and mixed methods.

The reader is referred to the following book for an in-depth review of a topic related to sampling strategies for quantitative and qualitative methods:

· Levy, P. S., & Lemeshow, S. (2009). Sampling of populations: Methods and applications (4th ed.). New York, NY: John Wiley & Sons.

Now that we covered a majority of the relevant aspects to research design, which is the “Design the Study” phase of the scientific method, we now present some steps that will help researchers select the most appropriate design. In the later chapters, we present a multitude of research designs used in quantitative, qualitative, and mixed methods. Therefore, it is important to review and understand the applications of these designs while regularly returning to this chapter to review the critical elements of design control and types of validity, for example. Let’s now examine the role of the research question.

Research Questions
Simply put, the primary research question sets the foundation and drives the decision of the application of the most appropriate research design. However, there are several terms related to research questions that should be distinguished. First, in general, studies will include an overarching observation deemed worthy of research. The “observation” is a general statement regarding the area of interest and identifies the area of need or concern.

Based on the initial observation, specific variables lead the researchers to the appropriate review of the literature and a theoretical framework is typically established. The purpose statement is then used to clarify the focus of the study, and finally, the primary research question ensues. Research studies can also include hypotheses or research objectives. Many qualitative studies include research aims as opposed to research questions. In quantitative methods (this includes mixed methods), the research question (hypotheses and objectives) determines (a) the population (and sample) to be investigated, (b) the context, (c) the variables to be operationalized, and (d) the research design to be employed.

Types of Inquiry
There are several ways to form a testable research inquiry. For qualitative methods, these can be posed as research questions, aims, or objectives

Part I Quantitative Methods for Experimental and Quasi-Experimental Research
Part I includes four popular approaches to the quantitative method (experimental and quasi-experimental only), followed by some of the associated basic designs (accompanied by brief descriptions of published studies that used the design). Visit the companion website at study.sagepub.com/edmonds2e to access valuable instructor and student resources. These resources include PowerPoint slides, discussion questions, class activities, SAGE journal articles, web resources, and online data sets.

Figure I.1 Quantitative Method Flowchart

Figure 11

Note: Quantitative methods for experimental and quasi-experimental research are shown here, followed by the approach and then the design.

Research in quantitative methods essentially refers to the application of the systematic steps of the scientific method, while using quantitative properties (i.e., numerical systems) to research the relationships or effects of specific variables. Measurement is the critical component of the quantitative method. Measurement reveals and illustrates the relationship between quantitatively derived variables. Variables within quantitative methods must be, first, conceptually defined (i.e., the scientific definition), then operationalized (i.e., determine the appropriate measurement tool based on the conceptual definition). Research in quantitative methods is typically referred to as a deductive process and iterative in nature. That is, based on the findings, a theory is supported (or not), expanded, or refined and further tested.

Researchers must employ the following steps when determining the appropriate quantitative research design. First, a measurable or testable research question (or hypothesis) must be formulated. The question must maintain the following qualities: (a) precision, (b) viability, and (c) relevance. The question must be precise and well formulated. The more precise, the easier it is to appropriately operationalize the variables of interest. The question must be viable in that it is logistically feasible or plausible to collect data on the variable(s) of interest. The question must also be relevant so that the result of the findings will maintain an appropriate level of practical and scientific meaning. The second step includes choosing the appropriate design based on the primary research question, the variables of interest, and logistical considerations. The researcher must also determine if randomization to conditions is possible or plausible. In addition, decisions must be made about how and where the data will be collected. The design will assist in determining when the data will be collected. The unit of analysis (i.e., individual, group, or program level), population, sample, and sampling procedures should be identified in this step. Third, the variables must be operationalized. And last, the data are collected following the format of the framework provided by the research design of choice.

Experimental Research

Experimental research (sometimes referred to as randomized experiments) is considered to be the most powerful type of research in determining causation among variables. Cook and Campbell (1979) presented three conditions that must be met in order to establish cause and effect:

1. Covariation (the change in the cause must be related to the effect)

2. Temporal precedence (the cause must precede the effect)

3. No plausible alternative explanations (the cause must be the only explanation for the effect)

The essential features of experimental research are the sound application of the elements of control: (a) manipulation, (b) elimination, (c) inclusion, (d) group or condition assignment, or (e) statistical procedures. Random assignment (not to be confused with random selection) of participants to conditions (or random assignment of conditions to participants [counterbalancing] as seen in repeated-measures approaches) is a critical step, which allows for increased control (improved internal validity) and limits the impact of the confounding effects of variables that are not being studied.

The random assignment to each group (condition) theoretically ensures that the groups are “probabilistically” equivalent (controlling for selection bias), and any differences observed in the pretests (if collected) are considered due to chance. Therefore, if all threats to internal, external, construct, and statistical conclusion validity were secured at “adequate” levels (i.e., all plausible alternative explanations are accounted for), the differences observed in the posttest measures can be attributed fully to the experimental treatment (i.e., cause and effect can be established). Conceptually, a causal effect is defined as a comparison of outcomes derived from treatment and control conditions on a common set of units (e.g., school, person).

The strength of experimental research rests in the reduction of threats to internal validity. Many threats are controlled for through the application of random assignment of participants to conditions. Random selection, on the other hand, is related to sampling procedures and is a major factor in establishing external validity (i.e., generalizability of results). Randomly selecting a sample from a population would be conducted so that the sample would better represent the population. However, Lee and Rubin (2015) presented a statistical approach that allows researchers to draw data from existing data sets from experimental research and examine subgroups (post hoc subgroup analysis). Nonetheless, random assignment is related to design, and random selection is related to sampling procedures. Shadish, Cook, and Campbell (2002) introduced the term generalized causal inference. They posit that if a researcher follows the appropriate tenets of experimental design logic (e.g., includes the appropriate number of subjects, uses random selection and random assignment) and controls for threats of all types of validity (including test validity), then valid causal inferences can be determined along with the ability to generalize the causal link. This is truly

Chapter 2 Between-Subjects Approach
The between-subjects approach, also known as a multiple-group approach, allows a researcher to compare the effects of two or more groups on single or multiple dependent variables (outcome variables). With a minimum of two groups, the participants in each group will only be exposed to one condition (one level of the independent variable), with no crossover between conditions. An advantage of having multiple groups is that it allows for the (a) random assignment to different conditions (experimental research) and (b) comparison of different treatments. If the design includes two or more dependent variables, it can be referred to as a multivariate approach, and when the design includes one dependent variable, it is classified as univariate.

Pretest and Posttest Designs
A common application to experimental and quasi-experimental research is the pretest and posttest between-subjects approach, also referred to as an analysis of covariance design (i.e., the pretest measure is used as the covariate in the analyses because the pretest should be highly correlated with the posttest). The 1-factor pretest and posttest control group design is one of the most common between-subjects approaches with many variations (one factor representing one independent variable and sometimes referred to as a single-factor randomized-group design). This basic multiple-group design can include a control group and is designed to have multiple measures between and within groups. Although there is a within-subject component, the emphasis is on the between-subject variance. The advantage of including pretest measures allows for the researcher to test for group equivalency (i.e., homogeneity between groups) and for providing a baseline against which to compare the treatment effects, which is the within-subject component of the design (i.e., the pretest is designated as the covariate in order to assess the variance [distance between each set of data points] between the pretest and posttest measures).

There is no set rule that determines the number of observations that should be made on the dependent variable. For example, in a basic pretest and posttest control group design, an observation is taken once prior to the treatment and once after the treatment. However, based on theoretical considerations, the investigator can take multiple posttest treatment measures by including a time-series component. Depending on the research logistics, groups can be randomly assigned or matched, then randomly assigned to meet the criteria for experimental research, or groups can be nonrandomly assigned to conditions (quasi-experimental research). With quasi-experimental research, the limitations of the study significantly increase as defined by the threats to internal validity discussed earlier.

k-Factor Designs
The between-subjects approach can include more than one treatment (factor) or intervention (i.e., the independent variable) and does not always have to include a control group. We designate this design as the k-factor design, with or without a control group. Shadish et al. (2002) refer to this design as an alternative- or multiple-treatment design. We prefer the k-factor design as a means to clearly distinguish exactly how many factors are present in the design (i.e., the k represents the number of factors [independent variables]). To clarify, the treatments in a 3-factor model (k = 3), for example, would be designated as XA, XB, and XC (each letter of the alphabet representing a factor) within the design structure. The within-subjects k-factor design is referred to as the crossover design and is discussed in more detail later in this book under repeated-measures approaches.

A between-subjects k-factor design should be used when a researcher wants to examine the effectiveness of more than one type of treatment and a true control is not feasible. Within educational settings, a control group is sometimes not accessible, or there are times when a university’s Institutional Review Board considers the withholding of treatment from specific populations as unethical. Furthermore, some psychologists and educators believe that using another treatment (intervention) as a comparison group will yield more meaningful results, particularly when the types of interventions being studied have a history of proven success; therefore, a k-factor design is the obvious choice. We present a variety of examples of 2-, 3-, and 4-factor pretest and posttest designs, as well as posttest-only designs with and without control groups.

Most common threats to internal validity are related, but not limited, to these designs:

· Experimental. Maturation, Testing, Attrition, History, and Instrumentation

· Quasi-Experimental. Maturation, Testing, Instrumentation, Attrition, History, and Selection Bias

We refer the reader to the following article and book for full explanations regarding threats to validity, grouping, and research designs:

· Shadish, W. R., & Cook, T. D. (2009). The renaissance of field experimentation in evaluating interventions. Annual Review of Psychology, 60, 607–629.

· Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.

Diagram 2.1 Pretest and Posttest Control Group Design

Figure 13

Note: In regard to design notations, a dashed line (- - -) would separate Groups 1 and 2 in the design structure if the participants were not randomly assigned to conditions, which indicates quasi-experimental research.

Example for Diagram 2.1
Chao, P., Bryan, T., Burstein, K., & Ergul, C. (2006). Family-centered intervention for young children at-risk for language and behavior problems. Early Childhood Education Journal, 34(2), 147–153.

Chapter 3 Regression-Discontinuity Approach
The regression-discontinuity (RD) approach is often referred to as an RD design. RD approaches maintain the same design structure as any basic between-subjects pretest and posttest design. The major differences for the RD approach are (a) the method by which research participants are assigned to conditions and (b) the statistical analyses used to test the effects. Specifically, the researcher applies the RD approach as a means of assigning participants to conditions within the design structure by using a cutoff score (criterion) on a predetermined quantitative measure (usually the dependent variable, but not always). Theoretical and logistical considerations are used to determine the cutoff criterion. The cutoff criterion is considered an advantage over typical random or nonrandom assignment approaches as a means to target “needy” participants and assign them to the actual program or treatment condition.

The most basic design used in RD approaches is the two-group pretest–posttest control group design. However, most designs designated as between-subject approaches can use an RD approach as a method of assignment to conditions and subsequent regression analysis. RD approaches can also be applied using data from extant databases (e.g., Luytena, Tymms, & Jones, 2009) as a means to infer causality without designing a true randomized experiment (see also Lesik, 2006, 2008). As seen in Figure 3.1, the cutoff criterion was 50 (based on a composite rating of 38 to 62). Those who scored below 50 were assigned to the control group, and those who scored above were assigned to the treatment group. As the figure shows, once the posttest scores were collected, a regression line was applied to the model to analyze the pre–post score relationship (i.e., a treatment effect is determined by assessing the degree of change in the regression line in observed and predicted pre–post scores for those who received treatment compared to those who did not).

Some researchers argue that the RD approach does not compromise internal validity to the extent the findings would not be robust to any violations of assumptions (statistically speaking). Typically, an RD approach requires much larger samples as a means to achieve acceptable levels of power (see statistical conclusion validity). We present two examples of studies that employed RD approaches: one that implemented an intervention, and one that used observational data. See Shadish, Cook, and Campbell (2002) for an in-depth discussion of issues related to internal validity for RD approaches, as well as methods for classifying RD approaches as experimental research, quasi-experimental research, and fuzzy regression discontinuity (i.e., assigning participants to conditions in violation of the designated cutoff score).

Figure 3.1 Sample of a Cutoff Score

Figure 23

Most common threats to internal validity are related, but not limited, to these designs:

· Experimental. History, Maturation, and Instrumentation

· Quasi-Experimental. History, Maturation, Instrumentation, and Selection Bias

We refer the reader to the following articles and book chapter for full explanations regarding RD approaches:

· Imbens, G. W., & Lemieux, T. (2008). Regression discontinuity designs: A guide to practice. Journal of Econometrics, 142, 615–635.

· Trochim, W. (2001). Regression-discontinuity design. In N. J. Smelser, J. D. Wright, & P. B. Baltes (Eds.), International encyclopedia of the social and behavioral sciences (Vol. 19, pp. 12940–12945). North-Holland, Amsterdam: Pergamon.