John hopkins research evidence appraisal tool

Johns Hopkins Nursing Evidence-Based Practice Appendix C: Evidence Level and Quality Guide

© The Johns Hopkins Hospital/Johns Hopkins University. May not be used or reprinted without permission. Page 1

Evidence Levels

Quality Guides

Level I Experimental study, randomized controlled trial (RCT) Systematic review of RCTs, with or without meta-analysis

A High quality: Consistent, generalizable results; sufficient sample size for the study design; adequate control; definitive conclusions; consistent recommendations based on comprehensive literature review that includes thorough reference to scientific evidence

B Good quality: Reasonably consistent results; sufficient sample size for the study

design; some control, fairly definitive conclusions; reasonably consistent recommendations based on fairly comprehensive literature review that includes some reference to scientific evidence

C Low quality or major flaws: Little evidence with inconsistent results; insufficient

sample size for the study design; conclusions cannot be drawn

Level II Quasi-experimental study Systematic review of a combination of RCTs and quasi- experimental, or quasi-experimental studies only, with or without meta-analysis Level III Non-experimental study Systematic review of a combination of RCTs, quasi-experimental and non-experimental studies, or non-experimental studies only, with or without meta-analysis Qualitative study or systematic review with or without a meta- synthesis

Johns Hopkins Nursing Evidence-Based Practice Appendix C: Evidence Level and Quality Guide

© The Johns Hopkins Hospital/Johns Hopkins University. May not be used or reprinted without permission. Page 2

Evidence Levels

Quality Guides

Level IV Opinion of respected authorities and/or nationally recognized expert committees/consensus panels based on scientific evidence Includes:  Clinical practice guidelines  Consensus panels

A High quality: Material officially sponsored by a professional, public, private organization, or government agency; documentation of a systematic literature search strategy; consistent results with sufficient numbers of well-designed studies; criteria-based evaluation of overall scientific strength and quality of included studies and definitive conclusions; national expertise is clearly evident; developed or revised within the last 5 years

B Good quality: Material officially sponsored by a professional, public, private

organization, or government agency; reasonably thorough and appropriate systematic literature search strategy; reasonably consistent results, sufficient numbers of well-designed studies; evaluation of strengths and limitations of included studies with fairly definitive conclusions; national expertise is clearly evident; developed or revised within the last 5 years

C Low quality or major flaws: Material not sponsored by an official organization or

agency; undefined, poorly defined, or limited literature search strategy; no evaluation of strengths and limitations of included studies, insufficient evidence with inconsistent results, conclusions cannot be drawn; not revised within the last 5 years

Johns Hopkins Nursing Evidence-Based Practice Appendix C: Evidence Level and Quality Guide

© The Johns Hopkins Hospital/Johns Hopkins University. May not be used or reprinted without permission. Page 3

Level V Based on experiential and non-research evidence Includes:  Literature reviews  Quality improvement, program or financial evaluation  Case reports  Opinion of nationally recognized experts(s) based on

experiential evidence

Organizational Experience:

A High quality: Clear aims and objectives; consistent results across multiple settings; formal quality improvement, financial or program evaluation methods used; definitive conclusions; consistent recommendations with thorough reference to scientific evidence

B Good quality: Clear aims and objectives; consistent results in a single setting;

formal quality improvement or financial or program evaluation methods used; reasonably consistent recommendations with some reference to scientific evidence

C Low quality or major flaws: Unclear or missing aims and objectives; inconsistent

results; poorly defined quality improvement, financial or program evaluation methods; recommendations cannot be made

Literature Review, Expert Opinion, Case Report, Community Standard, Clinician Experience, Consumer Preference:

A High quality: Expertise is clearly evident; draws definitive conclusions; provides scientific rationale; thought leader(s) in the field

B Good quality: Expertise appears to be credible; draws fairly definitive conclusions;

provides logical argument for opinions C Low quality or major flaws: Expertise is not discernable or is dubious; conclusions

cannot be drawn

Johns Hopkins Nursing Evidence-Based Practice Appendix E: Research Evidence Appraisal Tool

© The Johns Hopkins Hospital/Johns Hopkins University. May not be used or reprinted without permission. Page 1

Article Title:

Number:

Author(s):

Publication Date:

Journal: Setting:

Sample (Composition & size):

Does this evidence address my EBP question?

Yes

No Do not proceed with appraisal of this evidence

Level of Evidence (Study Design) A. Is this a report of a single research study? If No, go to B.

1. Was there manipulation of an independent variable? 2. Was there a control group? 3. Were study participants randomly assigned to the intervention and control

groups? If Yes to all three, this is a Randomized Controlled Trial (RCT) or Experimental Study If Yes to #1 and #2 and No to #3, OR Yes to #1 and No to #2 and #3, this is Quasi

Experimental (some degree of investigator control, some manipulation of an independent variable, lacks random assignment to groups, may have a control group)

If No to #1, #2, and #3, this is Non-Experimental (no manipulation of independent

variable, can be descriptive, comparative, or correlational, often uses secondary data) or Qualitative (exploratory in nature such as interviews or focus groups, a starting point for studies for which little research currently exists, has small sample sizes, may use results to design empirical studies)

NEXT, COMPLETE THE BOTTOM SECTION ON THE FOLLOWING PAGE, “STUDY FINDINGS THAT HELP YOU ANSWER THE EBP QUESTION”

 LEVEL I  LEVEL II  LEVEL III

Yes Yes Yes Yes

No No No No

Evidence Level and Quality:_______________________

Johns Hopkins Nursing Evidence-Based Practice Appendix E: Research Evidence Appraisal Tool

© The Johns Hopkins Hospital/Johns Hopkins University. May not be used or reprinted without permission. Page 2

B. Is this a summary of multiple research studies? If No, go to Non-Research

Evidence Appraisal Form.

1. Does it employ a comprehensive search strategy and rigorous appraisal method (Systematic Review)? If No, use Non-Research Evidence Appraisal Tool; if Yes:

a. Does it combine and analyze results from the studies to generate a new statistic (effect size)? (Systematic review with meta-analysis)

b. Does it analyze and synthesize concepts from qualitative studies? (Systematic review with meta-synthesis) If Yes to either a or b, go to #2B below.

2. For Systematic Reviews and Systematic Reviews with meta-analysis or meta- synthesis:

a. Are all studies included RCTs?

b. Are the studies a combination of RCTs and quasi-experimental or quasi-experimental only?

c. Are the studies a combination of RCTs, quasi-experimental and non-experimental or non-experimental only?

d. Are any or all of the included studies qualitative? COMPLETE THE NEXT SECTION, “STUDY FINDINGS THAT HELP YOU ANSWER THE EBP QUESTION”

 LEVEL I  LEVEL II  LEVEL IIl  LEVEL IIl

Yes Yes Yes Yes

No No No No

STUDY FINDINGS THAT HELP YOU ANSWER THE EBP QUESTION: NOW COMPLETE THE FOLLOWING PAGE, “QUALITY APPRAISAL OF RESEARCH STUDIES”, AND ASSIGN A QUALITY SCORE TO YOUR ARTICLE

Johns Hopkins Nursing Evidence-Based Practice Appendix E: Research Evidence Appraisal Tool

© The Johns Hopkins Hospital/Johns Hopkins University. May not be used or reprinted without permission. Page 3

Quality Appraisal of Research Studies

 Does the researcher identify what is known and not known about the problem and how the study will address any gaps in knowledge?

 Was the purpose of the study clearly presented?  Was the literature review current (most sources within last 5 years or classic)?  Was sample size sufficient based on study design and rationale?  If there is a control group:

o Were the characteristics and/or demographics similar in both the control and intervention groups?

o If multiple settings were used, were the settings similar? o Were all groups equally treated except for the intervention group(s)?

 Are data collection methods described clearly?  Were the instruments reliable (Cronbach's α [alpha] > 0.70)?  Was instrument validity discussed?  If surveys/questionnaires were used, was the response rate > 25%?  Were the results presented clearly?  If tables were presented, was the narrative consistent with the table content?  Were study limitations identified and addressed?  Were conclusions based on results?

Yes Yes Yes Yes Yes Yes Yes Yes

Yes Yes

Yes

Yes

Yes

Yes

Yes

No No No No No No No No

No No

No

No

No

No

No

NA NA NA

NA NA

NA

NA

Quality Appraisal of Systematic Review with or without Meta-Analysis or Meta-Synthesis

 Was the purpose of the systematic review clearly stated?  Were reports comprehensive, with reproducible search strategy?

o Key search terms stated o Multiple databases searched and identified o Inclusion and exclusion criteria stated

 Was there a flow diagram showing the number of studies eliminated at each level of review?

 Were details of included studies presented (design, sample, methods, results, outcomes, strengths and limitations)?

 Were methods for appraising the strength of evidence (level and quality) described?  Were conclusions based on results?

o Results were interpreted o Conclusions flowed logically from the interpretation and systematic review question

 Did the systematic review include both a section addressing limitations and how they were addressed?

Yes Yes Yes Yes Yes Yes Yes

Yes

Yes Yes Yes Yes

No No No No No No No

No

No No No No

QUALITY RATING BASED ON QUALITY APPRAISAL

A High quality: consistent, generalizable results; sufficient sample size for the study design; adequate control; definitive conclusions; consistent recommendations based on comprehensive literature review that includes thorough reference to scientific evidence

B Good quality: reasonably consistent results; sufficient sample size for the study design; some control, and fairly definitive conclusions; reasonably consistent recommendations based on fairly comprehensive literature review that includes some reference to scientific evidence

C Low quality or major flaws: little evidence with inconsistent results; insufficient sample size for the study design; conclusions cannot be drawn

Johns Hopkins Nursing Evidence-Based Practice Appendix F: Non-Research Evidence Appraisal Tool

© The Johns Hopkins Hospital/Johns Hopkins University. May not be used or reprinted without permission. Page 1

Article Title: Number:

Author(s):

Publication Date:

Journal: Does this evidence address the EBP question? Yes

No Do not proceed with appraisal of this evidence

Clinical Practice Guidelines: Systematically developed recommendations from nationally recognized experts based on research evidence or expert consensus panel. LEVEL IV

Consensus or Position Statement: Systematically developed recommendations based on research and nationally recognized expert opinion that guides members of a professional organization in decision-making for an issue of concern. LEVEL IV

 Are the types of evidence included identified?  Were appropriate stakeholders involved in the development of recommendations?  Are groups to which recommendations apply and do not apply clearly stated?  Have potential biases been eliminated?  Were recommendations valid (reproducible search, expert consensus,

independent review, current, and level of supporting evidence identified for each recommendation)?

 Were the recommendations supported by evidence?  Are recommendations clear?

Yes

Yes Yes Yes Yes Yes Yes

No

No No No No No No

Literature Review: Summary of published literature without systematic appraisal of evidence quality or strength. LEVEL V

 Is subject matter to be reviewed clearly stated?  Is relevant, up-to-date literature included in the review (most sources within

last 5 years or classic)?  Is there a meaningful analysis of the conclusions in the literature?  Are gaps in the literature identified?  Are recommendations made for future practice or study?

Yes

Yes Yes

Yes

Yes

No

No No

No

No

Expert Opinion: Opinion of one or more individuals based on clinical expertise. LEVEL V  Has the individual published or presented on the topic?  Is author’s opinion based on scientific evidence?  Is the author’s opinion clearly stated?  Are potential biases acknowledged?

Yes Yes Yes Yes

No No No No

Evidence Level & Quality:________________________

Johns Hopkins Nursing Evidence-Based Practice Appendix F: Non-Research Evidence Appraisal Tool

© The Johns Hopkins Hospital/Johns Hopkins University. May not be used or reprinted without permission. Page 2

Organizational Experience:

Quality Improvement: Cyclical method to examine organization-specific processes at the local level. LEVEL V

Financial Evaluation: Economic evaluation that applies analytic techniques to identify, measure, and compare the cost and outcomes of two or more alternative programs or interventions. LEVEL V

Program Evaluation: Systematic assessment of the processes and/or outcomes of a program and can involve both quantitative and qualitative methods. LEVEL V

Setting:

Sample (composition/size):  Was the aim of the project clearly stated?  Was the method adequately described?  Were process or outcome measures identified?  Were results adequately described?  Was interpretation clear and appropriate?  Are components of cost/benefit analysis described?

Yes Yes Yes Yes Yes Yes

No No No No No No N/A

Case Report: In-depth look at a person, group, or other social unit. LEVEL V

 Is the purpose of the case report clearly stated?  Is the case report clearly presented?  Are the findings of the case report supported by relevant theory or

research?  Are the recommendations clearly stated and linked to the findings?

Yes Yes Yes

Yes

No No No

No

Community Standard, Clinician Experience, or Consumer Preference

Community Standard: Current practice for comparable settings in the community LEVEL V

Clinician Experience: Knowledge gained through practice experience LEVEL V

Consumer Preference: Knowledge gained through life experience LEVEL V

Information Source(s): Number of Sources:

 Source of information has credible experience.  Opinions are clearly stated.  Identified practices are consistent.

Yes Yes Yes

No No N/A No N/A

Findings that help you answer the EBP question:

Johns Hopkins Nursing Evidence-Based Practice Appendix F: Non-Research Evidence Appraisal Tool

© The Johns Hopkins Hospital/Johns Hopkins University. May not be used or reprinted without permission. Page 3

QUALITY RATING FOR CLINICAL PRACTICE GUIDELINES, CONSENSUS OR POSITION STATEMENTS (LEVEL IV) A High quality: Material officially sponsored by a professional, public, private organization, or

government agency; documentation of a systematic literature search strategy; consistent results with sufficient numbers of well-designed studies; criteria-based evaluation of overall scientific strength and quality of included studies and definitive conclusions; national expertise is clearly evident; developed or revised within the last 5 years.

B Good quality: Material officially sponsored by a professional, public, private organization, or

government agency; reasonably thorough and appropriate systematic literature search strategy; reasonably consistent results, sufficient numbers of well-designed studies; evaluation of strengths and limitations of included studies with fairly definitive conclusions; national expertise is clearly evident; developed or revised within the last 5 years.

C Low quality or major flaws: Material not sponsored by an official organization or agency; undefined,

poorly defined, or limited literature search strategy; no evaluation of strengths and limitations of included studies, insufficient evidence with inconsistent results, conclusions cannot be drawn; not revised within the last 5 years.

QUALITY RATING FOR ORGANIZATIONAL EXPERIENCE (LEVEL V) A High quality: Clear aims and objectives; consistent results across multiple settings; formal quality

improvement or financial evaluation methods used; definitive conclusions; consistent recommendations with thorough reference to scientific evidence

B Good quality: Clear aims and objectives; formal quality improvement or financial evaluation methods

used; consistent results in a single setting; reasonably consistent recommendations with some reference to scientific evidence

C Low quality or major flaws: Unclear or missing aims and objectives; inconsistent results; poorly

defined quality improvement/financial analysis method; recommendations cannot be made QUALITY RATING FOR LITERATURE REVIEW, EXPERT OPINION, COMMUNITY STANDARD, CLINICIAN EXPERIENCE, CONSUMER PREFERENCE (LEVEL V) A High quality: Expertise is clearly evident; draws definitive conclusions; provides scientific rationale;

thought leader in the field B Good quality: Expertise appears to be credible; draws fairly definitive conclusions; provides logical

argument for opinions C Low quality or major flaws: Expertise is not discernable or is dubious; conclusions cannot be drawn

Appendix C final Evidence Level and Quality Guide
Appendix E
Appendix F