Mixed methods sampling strategies in social science research

KDiscussion: Designing Mixed Methods Research

Mixed methods research designs refer to a set of designs that purposively mix or integrate both qualitative data and quantitative data. As with quantitative research and qualitative research, the choice to use mixed methods research is influenced by the researcher’s philosophical orientation.
This week’s readings provide an overview of various types of mixed methods research designs. As with previous discussions on design, the selection of the most appropriate mixed design is guided by the study’s purpose and research questions and/or hypotheses. The choice of design links the research questions and/or hypotheses to the data that will be collected achieving alignment among research components.

In this Discussion, you will explore the basics of mixed methods research designs, calling upon your growing understanding of both quantitative and qualitative research.

With these thoughts in mind:

By Day 4
Post your response to the question, “To what extent is mixed methods research simply taking a quantitative design and a qualitative design and putting them together?” Next, explain the types of research questions best served by mixed methods research. Then, explain one strength and one limitation of mixed methods research. Finally, provide a rationale for or against the utility of mixed methods research in your discipline.

2 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 1, April 2009

Mixed methods research is a rapidly emergingresearch paradigm and, although various sources are available to assist the novice researcher in terms of books (e.g. Creswell & Plano Clark 2007; Greene 2007; Johnson & Christensen 2008; Onwuegbuzie, Collins, Leech & Slate [2009]; Ridenour & Newman 2008; Teddlie & Tashakkori 2009), methodological articles (e.g. Johnson & Onwuegbuzie 2004; Onwueg- buzie & Johnson 2006), and journal editorials (e.g. Tashakkori & Creswell 2007), it might be quite daunting for the novice researcher to stay abreast of the emerging trends in the field of mixed methods research. Therefore, our goal in writing this article is to present ten points that a novice researcher should be cognizant of when designing a mixed methods study in accordance to the following three phases: research formulation,

research planning, and research implementation. Additionally, we present rationales for why these points are important, and a brief description of selective typologies that novice researchers might access when conducting mixed methods research.

RESEARCH FORMULATION PHASE

1. Importance of a definition

Individuals who share a profession develop and use a professional language or lexicon. An impor- tant component of a lexicon is definitions. Shared definitions provide precision when researchers are communicating to an audience and collaborating with peers when designing a study or program of research. Because mixed methods research is an emerging paradigm, ‘new’ definitions also are emerging. In this article, we use the term ‘mixed

Copyright © eContent Management Pty Ltd. International Journal of Multiple Research Approaches (2009) 3: 2–7.

INTRODUCTION Ten points about mixed methods research to be considered by the

novice researcher

KATHLEEN MT COLLINS Associate Professor, Department of Curriculum & Studies, College of Education and Health Professions, University of Arkansas at Fayetteville AR, USA

ALICIA O’CATHAIN Senior Research Fellow, Medical Care Research Unit, ScHARR, University of Sheffield, UK

ABSTRACT Our goal in writing this article is to present ten points that a novice researcher should be cognizant of when formulating, planning, and implementing a mixed methods study. We provide rationales for why these points are important and a brief description of selective typologies that novice researchers might access to address these points when conducting mixed methods research.

Keywords: mixed methods design, research planning, research formulation

methods research’ to be consistent with the title of this special issue; however, other terms such as mixed research and integrative methods also are used by researchers when conducting this form of research (Onwuegbuzie & Johnson 2006; Teddlie & Tashakkori 2009). Mixed methods studies have been defined as studies that ‘combine qualitative and quantitative approaches into the research methodology of a single study or multi-phased study’ (Tashakkori & Teddlie 1998: 17-18) and ‘as a research design in which QUAL [i.e. qualitative] and QUAN [i.e. quantitative] approaches are used in types of questions, research methods, data col- lection and analysis procedures, and /or inferences’ (Tashakkori & Teddlie 2003: 711). Mixed meth- ods designs have been defined as designs which include at least one quantitative method (designed to collect numbers) and one qualitative method (designed to collect words), where neither type is linked to a particular inquiry paradigm (Greene, Caracelli & Graham 1989). Similarly, mixed methods have been defined as quantitative and qualitative data collection, data analysis and the mixing of quantitative and qualitative approaches within a single study, with data integrated at some stage (Creswell & Plano Clark 2007; Creswell, Plano Clark, Gutmann & Hanson 2003). A defi- nition is useful to the novice researcher because it can be used to facilitate his or her awareness and understanding of mixed methods research as a research paradigm distinct from other monomethod approaches (quantitative and quali- tative) and provide useful terminology for report- ing research findings across various venues (e.g. conference presentations, technical and govern- ment reports, and published articles).

2. Importance of a mental model for mixing A researcher approaches a mixed methods investi- gation by initiating and completing a series of steps focused on delineating the process of mixing in a study or a program of research. Collins, Onwuegbuzie and Sutton (2006) have conceptu- alized the mixed methods research process as com-

prising 13 distinct steps (cf. Onwuegbuzie and Leech, this issue, for an identification of each step). Although similar to the research process in general, the intent of this 13-step process is to facilitate novice researcher decisions pertaining to the process of mixing at each step and it represents a recursive process. Additionally, it is important that the novice researcher recognize that decisions made at each of these steps are shaped by the researcher’s mental model (Greene 2007). Greene (2007) conceptualizes a mental model ‘as a com- plex, multifaceted lens through which a social inquirer perceives and makes sense of the social world’ ….and it is the ‘inquirer’s’ mental models that importantly frame and guide social inquiry’ (p.13). A mental model consists of the researcher’s personal assumptions, experiences, values, and beliefs about what constitutes an effective mode of inquiry (Greene 2007). Therefore, it is important that the novice researcher is cognizant of his or her mental model and also aware of the degree that this model shapes his or her interpretation of what constitutes rigor within an investigation.

3. Utilizing typologies of designs Although we advocate that the novice researcher access typologies of designs when conducting mixed methods research, we also caution the novice researcher to be aware that typologies do not offer a panacea. Indeed, given the breath of mixed meth- ods studies, typologies have been criticized because they cannot address sufficiently the wide range of mixed methods designs implemented in various fields (Maxwell & Loomis 2003); in some cases, typologies delineate only minimally the informa- tion required by the researcher, or give inconsistent information, or present overly complex informa- tion (Leech & Onwuegbuzie 2009). However, we agree with Teddlie and Tashakkori (2006) who note that, although typologies are not exhaustive, they can provide to researchers distinct guidelines that serve to differentiate mixed methods as a research paradigm from other paradigms, namely quantitative and qualitative, thereby legitimating mixed methods research as a unique research para-

3

Introduction – Ten points about mixed methods research to be considered by the novice researcher

Volume 3, Issue 1, April 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES

digm (Johnson & Onwuegbuzie 2004). Typologies also provide to researchers an organizational struc- ture to design and to implement studies, and a lexi- con to utilize when interpreting and disseminating information (Teddlie & Tashakkori 2009). Lastly, typologies facilitate learning by providing to researchers opportunities to compare and to con- trast various typologies, consequently expanding their levels of understanding of the mixed methods research process (Teddlie & Tashakkori 2006). In our later points we recommend selective typologies that the novice researcher might find useful when conducting mixed methods research.

4. Selecting the reason, rationale, and purpose for mixing

The decisions pertaining to the reason, the ration- ale, and the purpose for mixing serve to differenti- ate the mixed methods research process from other research processes and it is these decisions that lead the novice researcher to develop the study’s research question(s) (Teddlie & Tashakkori 2009). Teddlie and Tashakkori (2009) have developed a typology comprising the following three general categories for identifying various reasons for con- ducting mixed methods research: (a) personal rea- sons for conducting the study, (b) reasons associated with advancing knowledge, and (c) soci- etal reasons associated with improving or empow- ering society, institutions, and oppressed groups. Adhering to this three-component process leads the novice researcher to develop research objectives followed by the development of research question(s) and hypotheses. It is the research ques- tion that drives the methods that will be imple- mented in the study (Tashakkori & Teddlie 1998). The typology developed by Greene et al. (1989) offers the novice researcher five options for deter- mining the purpose for mixing approaches: trian- gulation (i.e. comparison of findings derived from different methods to interpret the phenomenon); complementarity (different methods utilized to assess various dimensions of the phenomena); development (methods implemented sequentially, thereby allowing results of one method [e.g. quali-

tative] to inform development of the other method [e.g. quantitative]); expansion (different methods utilized to measure different phenomena); and ini- tiation (to address the goal of divergence, different methods used to assess various dimensions of the phenomena of interest). These five purposes relate to the data analysis step of the mixed methods research process. Collins, Onwuegbuzie and Sut- ton (2006) developed a typology that presents the novice researcher with four rationales for mixing and 65 purposes that are applicable to multiple steps of the mixed methods research process.

5. Determining the research question Teddlie and Tashakkori (2009) define mixed meth- ods research questions as questions ‘concerned with the unknown aspects of a phenomena and are answered with information that is presented in both narrative and numerical forms’ (p.129). They recommend developing one mixed methods ques- tion that serves as an overarching question and this question can be extended into qualitative and quantitative sub-questions. Formulating one over- arching question provides a justification for mixing and guides the novice researcher’s processes of mix- ing methods and integrating findings. Creswell and Plano Clark (2007) offer an alternative approach to developing research questions. They advocate separate quantitative and qualitative ques- tions, followed by development of a mixed meth- ods question framing integration of the findings from both phases of the study. Onwuegbuzie and Leech (2006) provide to novice researchers specific examples of how to write quantitative, qualitative, and mixed methods research questions and they also provide a framework connecting research questions to various data analytical techniques.

RESEARCH PLANNING PHASE

6. Selecting a mixed methods research design

The novice researcher can select a preexistent mixed methods design or develop a specific design to address the study’s particular research

4

Kathleen MT Collins and Alicia O’Cathain

INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 1, April 2009

objectives, purposes and research questions. Ted- dlie and Tashakkori (2006, 2009) provide to the novice researcher a typology called the Methods- Strands Matrix. This matrix facilitates a novice researcher’s decisions by presenting design options that are organized by: (a) choosing the type of approach that will be

utilized in the study (i.e. monomethod [qual- itative or quantitative approach used across all stages of the study] or mixed methods [qualitative and quantitative approaches mixed across the stages of the study); and

(b) selecting the number of strands or phases that will be implemented in the study (Ted- dlie & Tashakkori 2009).

The two types of mixed methods designs are designs with one strand (monostrand) and designs with more than one strand (multistrand). When utilizing mixed methods multistrand designs, the novice researcher can select from the following five families of mixed methods research designs: paral- lel, sequential, conversion, multilevel, and fully integrated (Teddlie & Tashakkori 2009). The Methods-Strands Matrix also can guide the novice researcher in the process of deciding the stage(s) that mixing will occur (i.e. conceptualization, experimental, inferential stages) within the study (Ridenour & Newman 2008; Teddlie & Tashakkori 2009). Additionally, the novice researcher also can access other typologies to guide the design process (e.g. Creswell et al. 2003; John- son & Onwuegbuzie 2004; Leech & Onwueg- buzie 2009; Maxwell & Loomis 2003).

7. Determining the sampling design The researcher’s choice of a sampling design impacts the legitimation of the researcher’s infer- ences and the appropriate generalization of results (Collins, Onwuegbuzie & Jiao 2006, 2007; Onwuegbuzie & Collins 2007; Teddlie & Yu 2007). A sampling scheme represents the strategies used by the novice researcher to select the unit of analysis (individuals, cases, groups, contexts in terms of settings and events) accompanied by a decision pertaining to the size of the sample (the

number of units chosen for the study). When con- ducting mixed methods research, a novice researcher’s sampling decisions must pertain to both the quantitative and qualitative phases of the study. The typology developed by Teddlie and Yu (2007) presents to the novice researcher sampling schemes that are categorized into four types: probability, purposive sampling, convenience sampling, and mixed methods sampling. Relevant to mixed meth- ods sampling, four schemes are introduced, namely: basic sampling strategies, sequential sampling, con- current sampling, and multilevel sampling. An alternative typology developed by Onwuegbuzie and Collins (2007) presents to the novice researchers sampling schemes and sample size guidelines appropriate for the quantitative and qualitative of the mixed methods study. Additional- ly, they present a matrix comprising two dimen- sions. Dimension one matches the time orientation of the quantitative and qualitative phases of the study (the novice researcher’s decision to administer the quantitative and qualitative phase at the same or approximately the same point in time) or sequen- tially (one phase is initiated and informs delivery of the second phase) to the purpose of mixing (cf. Greene et al. 1989). Dimension two provides guidelines to identify clearly the relationship of the study’s participants in the quantitative and qualita- tive samples, namely: identical (the same individu- als participate in both phases); parallel (different individuals participate in the phases but are drawn from the same population); nested (individuals for one phase represent a subset of individuals who par- ticipated in the other phase); multilevel (different individuals participate in the phases and represent different levels of the population as exemplified by selecting corporate personnel as one sample versus consumers of a product as the other sample).

RESEARCH IMPLEMENTATION PHASE

8. Collecting data

Johnson and Turner (2003) note that the investi- gator’s selection of data methods reflects the fun- damental principle of mixed methods research,

5

Introduction – Ten points about mixed methods research to be considered by the novice researcher

Volume 3, Issue 1, April 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES

such that the ‘methods should be mixed in a way that has complementary strengths and non overlap- ping weaknesses’ (Johnson & Turner 2003: 299, italics in original). Adherence to this principle enables the novice researcher to collect strategical- ly multiple forms of evidence, such that the com- bination of methods presents convergent and divergent evidence, subsequently strengthening the findings of the mixed methods study (Johnson & Turner 2003; Johnson & Christensen 2008). Johnson and Turner (2003) present a two-dimen- sional matrix outlining data collection techniques to allow the novice researcher to engage in two forms of mixing: intramethod mixing (employing a single method that includes quantitative and qualitative components [e.g. open-closed items on a single questionnaire] and intermethod mixing (mixing two or more methods [e.g. questionnaire, interview and observation]). The first dimension of the matrix is to select a research approach (i.e. pure quantitative, pure qualitative or mixed) and the second is to select a method of data collection (questionnaire, interviews, focus groups, tests, observations secondary data [archival data]).

9. Conducting data analysis A mixed methods analysis entails the use of quali- tative and quantitative analytical techniques that are implemented either concurrently (at the same time or in a relatively close time frame) or sequen- tially (one form of analysis is conducted first and it informs the other type of analysis) from which interpretations are made in a parallel or an integra- tive or an iterative manner (Onwuegbuzie & Ted- dlie 2003; Teddlie & Tashakkori 2009). When analyzing data, a novice researcher can utilize a combination of the following seven phases of the mixed methods analysis process: (a) data reduction (e.g. quantitative data are analyzed using descrip- tive statistics and qualitative data are categorized as descriptive themes; (b) data display (e.g. data per- taining to both strands are organized and presented visually in graphs and matrices); (c) data transfor- mation (quantitative data converted into narrative codes [qualitized] that can be analyzed using quali-

tative techniques and qualitative data converted into numerical codes [quantitized] and analyzed using quantitative techniques); (d) data correlation (correlating quantitative data with qualitized data or vice versa; (e) data consolidation (different data types merged into one data set); (f ) data compari- son (comparing data from two different sources); and (g) data integration (integrating quantitative and qualitative data into one coherent whole that will be analyzed and interpreted simultaneously as a single data set or two data sets [quantitative and qualitative] to be analyzed separately by the researcher) (Onwuegbuzie & Teddlie 2003).

10. Legitimating Inferences and formulating generalizations

Data validation refers to the implementation of appropriate steps or procedures to assure legiti- mation (Onwuegbuzie & Johnson 2006) by establishing a process to examine ‘inference qual- ity’ (Teddlie & Tashakkori 2003) in terms of the design quality and the interpretive rigor of the study’s outcomes, and, thereby leading the novice researcher to formulate appropriate generaliza- tions termed ‘inference transferability’ by Teddlie and Tashakkori (2003). Legitimation also has been defined as a recursive process in which the novice researcher evaluates the quality of the inferences drawn from the quantitative and qual- itative phases at each stage of the study and/or across a program of research (Onwuegbuzie & Johnson 2006). Subsequently, the decisions per- taining to both the quantitative and qualitative phases of the study impact the novice researcher’s ability to draw appropriate inferences and gener- alizations. Onwuegbuzie and Johnson’s (2006) legitimation model and Dellinger and Leech’s validation framework (2007) offer the novice researcher two alternatives to evaluate inferences on the basis of the study’s findings.

CONCLUSION In conclusion, we hope that consideration of the ten points above and the accompanying typolo- gies will facilitate the novice researcher’s efforts to

6

Kathleen MT Collins and Alicia O’Cathain

INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES Volume 3, Issue 1, April 2009

formulate, plan, and implement both successful and rigorous mixed methods research.

References Collins KMT, Onwuegbuzie AJ and Jiao QG

(2006) Prevalence of mixed methods sampling in social science research. Evaluation and Research in Education, 19(2): 267-291.

Collins KMT, Onwuegbuzie AJ and Jiao, QG (2007) A mixed methods investigation of mixed methods sampling designs in social and health science research. Journal of Mixed Methods Research, 1(3): 267-294.

Collins KMT, Onwuegbuzie AJ and Sutton IL (2006) A model incorporating the rationale and purpose for conducting mixed methods research in special education and beyond. Learning Disabilities: A Contemporary Journal, 4: 67-100.

Creswell JW and Plano Clark VL (2007) Designing and conducting mixed methods research. Thousand Oaks, CA: Sage.

Creswell JW, Plano Clark VL, Gutmann M and Hanson W (2003) Advanced mixed methods research designs. In A Tashakkori & C Teddlie (Eds) Handbook of mixed methods in social and behavioral research (pp. 209-240) Thousand Oaks, CA: Sage.

Dellinger AB and Leech NL (2007) Toward a unified validation framework in mixed methods research. Journal of Mixed Methods Research, 1: 359-375.

Greene JC (2007) Mixed methods in social inquiry. San Francisco, CA: Jossey Bass.

Greene JC, Caracelli VJ and Graham WF (1989) Toward a conceptual framework for mixed- method evaluation designs. Educational Evaluation and Policy Analysis, 11: 255– 274.

Johnson B and Christensen L (2008) Educational research quantitative, qualitative, and mixed approaches (3rd ed) Thousand Oaks, CA: Sage

Johnson RB and Onwuegbuzie AJ (2004) Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7): 14–26.

Johnson RB and Turner LA (2003) Data collection strategies in mixed methods research. In A Tashakkori & C Teddlie (Eds) Handbook of mixed methods in social and behavioral research (pp 297–319) Thousand Oaks, CA: Sage.

Leech NL and Onwuegbuzie AJ (2009) A typology of mixed methods research designs. Quality & Quantity International Journal of Methodology, 43: 265-275.

Maxwell JA and Loomis DM (2003) Mixed methods design: An alternative approach. In A

Tashakkori & C Teddlie (Eds) Handbook of mixed methods in social and behavioral research (pp. 241–272) Thousand Oaks, CA: Sage.

Onwuegbuzie AJ and Collins KMT (2007). A typol- ogy of mixed methods sampling designs in social science research. The Qualitative Report, 12(2). Retrieved February 21 2009, from http://www. nova.edu/ssss/QR/QR12-2/onwuegbuzie2.pdf

Onwuegbuzie AJ, Collins KMT, Leech NL and Slate JR (2009) Mixed research: A step-by-step guide. New York: Taylor & Francis, in press.

Onwuegbuzie AJ and Johnson RB (2006) The validity issues in mixed research. Research in Schools, 13(1): 48-63.

Onwuegbuzie AJ and Leech NL (2006) Linking research questions to mixed methods data analysis procedures. The Qualitative Report, 11(3). Retrieved February 23, 2009 from http://www. nova.edu/ssss/QR/QR11-3/onwuegbuzie.pdf

Onwuegbuzie AJ and Teddlie C (2003) A framework for analyzing data in mixed methods research. In A Tashakkori & C Teddlie (Eds) Handbook of mixed methods in social and behavioral research (pp. 351-383) Thousand Oaks, CA: Sage.

Ridenour CS and Newman I (2008) Mixed methods research: Exploring the interactive continuum. Carbondale: Southern Illinois University Press.

Tashakkori A and Creswell JW (2007) Exploring the nature of research questions in mixed methods research. Journal of Mixed Methods Research, 1: 207-211.

Tashakkori A and Teddlie C (1998) Mixed method- ology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: Sage

Tashakkori A and Teddlie C (2003) Handbook of mixed methods in social and behavioral research. Thousand Oaks, CA: Sage.

Teddlie C and Tashakkori A (2003) Major issues and controversies in the issue of mixed methods in the social and behavioral sciences. In A Tashakkori & C Teddlie (Eds) Handbook of mixed methods in social and behavioral research (pp.3-50) Thousand Oaks, CA: Sage.

Teddlie C and Tashakkori A (2006) A general typology of research designs featuring mixed methods. Research in the Schools, 13: 12-28.

Teddlie C and Tashakkori A (2009) Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks, CA: Sage.

Teddlie C and Yu F (2007) Mixed methods sampling: A typology with examples. Journal of Mixed Methods Research, 1: 77-100.

7

Introduction – Ten points about mixed methods research to be considered by the novice researcher

Volume 3, Issue 1, April 2009 INTERNATIONAL JOURNAL OF MULTIPLE RESEARCH APPROACHES