Health program planning and evaluation issel pdf

L. Michele Issel, PhD, RN Professor of PhD Program

University of North Carolina at Charlotte College of Health and Human Services

Charlotte, North Carolina

Rebecca Wells, PhD, MHSA Professor

The University of Texas School of Public Health

Houston, Texas

Health Program Planning and Evaluation A Practical, Systematic Approach for Community Health

FOURTH EDITION

World Headquarters Jones & Bartlett Learning 5 Wall Street Burlington, MA 01803 978-443-5000 info@jblearning.com www.jblearning.com

Jones & Bartlett Learning books and products are available through most bookstores and online booksellers. To contact Jones & Bartlett Learning directly, call 800-832-0034, fax 978-443-8000, or visit our website, www.jblearning.com.

Substantial discounts on bulk quantities of Jones & Bartlett Learning publications are available to corporations, professional associations, and other qualified organizations. For details and specific discount information, contact the special sales department at Jones & Bartlett Learning via the above contact information or send an email to specialsales@jblearning.com.

Copyright © 2018 by Jones & Bartlett Learning, LLC, an Ascend Learning Company

All rights reserved. No part of the material protected by this copyright may be reproduced or utilized in any form, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without written permission from the copyright owner.

The content, statements, views, and opinions herein are the sole expression of the respective authors and not that of Jones & Bartlett Learning, LLC. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not constitute or imply its endorsement or recommendation by Jones & Bartlett Learning, LLC and such reference shall not be used for advertising or product endorsement purposes. All trademarks displayed are the trademarks of the parties noted herein. Health Program Planning and Evaluation: A Practical, Systematic Approach for Community Health, Fourth Edition is an independent publication and has not been authorized, sponsored, or otherwise approved by the owners of the trademarks or service marks referenced in this product.

There may be images in this book that feature models; these models do not necessarily endorse, represent, or participate in the activities represented in the images. Any screenshots in this product are for educational and instructive purposes only. Any individuals and scenarios featured in the case studies throughout this product may be real or fictitious, but are used for instructional purposes only.

This publication is designed to provide accurate and authoritative information in regard to the Subject Matter covered. It is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional service. If legal advice or other expert assistance is required, the service of a competent professional person should be sought.

15842-7

Production Credits VP, Executive Publisher: David D. Cella Publisher: Michael Brown Associate Editor: Danielle Bessette Vendor Manager: Nora Menzi Senior Marketing Manager: Sophie Fleck Teague Manufacturing and Inventory Control Supervisor: Amy Bacus Composition and Project Management: S4Carlisle Publishing

Services

Cover Design: Scott Moden Director of Rights & Media: Joanna Gallant Rights & Media Specialist: Merideth Tumasz Media Development Editor: Shannon Sheehan Cover Image: © Lynne Nicholson/Shutterstock Printing and Binding: Edwards Brothers Malloy Cover Printing: Edwards Brothers Malloy

Library of Congress Cataloging-in-Publication Data Names: Issel, L. Michele, author. | Wells, Rebecca, 1966- author. Title: Health program planning and evaluation: a practical, systematic approach for community health/L. Michele Issel and Rebecca Wells. Description: Fourth edition. | Burlington, MA: Jones & Bartlett Learning, [2018] | Includes bibliographical references and index. Identifiers: LCCN 2017010386 | ISBN 9781284112115 (pbk.) Subjects: | MESH: Community Health Services—organization & administration | Program Development—methods | Health Planning—methods | Program Evaluation—methods | United States Classification: LCC RA394.9 | NLM WA 546 AA1 | DDC 362.12068—dc23 LC record available at https://lccn.loc.gov/2017010386

6048

Printed in the United States of America 21 20 19 18 17 10 9 8 7 6 5 4 3 2 1

iii

© Lynne Nicholson/Shutterstock

Contents List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi

List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

List of Exhibits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii

Preface to the Fourth Edition . . . . . . . . . . . . . . . . . . xix

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv

List of Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxvii

SECTION I The Context of Health Program Development 1

Chapter 1 Context of Health Program Development and Evaluation . . . . . . . . . . . . . . . 3

History and Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Concept of Health . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Health Programs, Projects, and Services . . . . . . 4

History of Health Program Planning and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Evaluation as a Profession . . . . . . . . . . . . . . . . . . . . . . . . . 8

Who Does Planning and Evaluations? . . . . . . .10

Roles of Evaluators . . . . . . . . . . . . . . . . . . . . . . . . . .10

Planning and Evaluation Cycle . . . . . . . . . . . . . . . . . . .11

Interdependent and Cyclic Nature of Planning and Evaluation . . . . . . . . . . . . . . .11

Using Evaluation Results as the Cyclical Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13

Program Life Cycle . . . . . . . . . . . . . . . . . . . . . . . . . .13

The Fuzzy Aspects of Planning . . . . . . . . . . . . . . . . . . .14

Paradoxes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14

Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

Uncertainty, Ambiguity, Risk, and Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17

Introduction to the Types of Evaluation . . . . . . . . . .19

Mandated and Voluntary Evaluations . . . . . . .20

When Not to Evaluate . . . . . . . . . . . . . . . . . . . . . .21

The Public Health Pyramid . . . . . . . . . . . . . . . . . . . . . . .21

Use of the Public Health Pyramid in Program Planning and Evaluation . . . . . . . .23

The Public Health Pyramid as an Ecological Model . . . . . . . . . . . . . . . . . . . . .23

The Town of Layetteville in Bowe County . . . . . . . . .25

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25

Discussion Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . .27

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .27

Chapter 2 Relevance of Diversity and Disparities to Health Programs . . . . . . . . . . . . . . . . . . 29

Health Disparities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .30

Diversity and Health Disparities . . . . . . . . . . . . .32

Diversity and Health Programs . . . . . . . . . . . . . .33

Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33

Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38

Influences of Sociocultural Diversity on Interventions . . . . . . . . . . . . . . . . . . . . . . . . .38

Influences of Biological Diversity on Interventions . . . . . . . . . . . . . . . . . . . . . . . . .39

Approaches to Developing Programs . . . . . . .39

Profession and Provider Diversity . . . . . . . . . . . .40

The Three Health Provider Sectors . . . . . . . . . .43

Diversity Within Healthcare Organizations and Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43

Organizational Culture . . . . . . . . . . . . . . . . . . . . . .44

Cultural Competency Continuum . . . . . . . . . . .44

Enhancing Cultural Competency . . . . . . . . . . .48

iv Contents

Types of Assessments . . . . . . . . . . . . . . . . . . . . . . . . . . . .75

Organizational Assessment . . . . . . . . . . . . . . . . .75

Marketing Assessment . . . . . . . . . . . . . . . . . . . . . .76

Needs Assessment . . . . . . . . . . . . . . . . . . . . . . . . .76

Community Health Assessment . . . . . . . . . . . . .77

Workforce Assessment . . . . . . . . . . . . . . . . . . . . . .77

Steps in Planning and Conducting the Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77

Form and Develop the Team . . . . . . . . . . . . . . . .78

Create a Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79

Involve Community Members . . . . . . . . . . . . . .79

Define the Population . . . . . . . . . . . . . . . . . . . . . .80

Define the Problem to Be Assessed . . . . . . . . .81

Investigate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81

Prioritize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82

Make a Decision . . . . . . . . . . . . . . . . . . . . . . . . . . . .82

Implement and Continue . . . . . . . . . . . . . . . . . . .83

Anticipate Data-Related and Methodological Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85

Discussion Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . .85

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87

Chapter 4 Characterizing and Defining the Health Problem . . . . . . . . . . . . . . . . . . . 91

Collecting Data From Multiple Sources . . . . . . . . . . .91

Public Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .91

Primary Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92

Observational Data . . . . . . . . . . . . . . . . . . . . . . . . .92

Archival Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93

Proprietary Data . . . . . . . . . . . . . . . . . . . . . . . . . . . .93

Published Literature . . . . . . . . . . . . . . . . . . . . . . . .93

Data Beyond Street Lamp . . . . . . . . . . . . . . . . . . .93

Collecting Descriptive Data . . . . . . . . . . . . . . . . . . . . . .94

Magnitude of the Problem . . . . . . . . . . . . . . . . . .94

Dynamics Leading to the Problem . . . . . . . . . .94

Population Characteristics . . . . . . . . . . . . . . . . . .96

Attitudes and Behaviors . . . . . . . . . . . . . . . . . . . .96

Years of Life and Quality of Life . . . . . . . . . . . . . .96

Stakeholders and Coalitions . . . . . . . . . . . . . . . . . . . . .50

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51

Discussion Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . .53

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54

SECTION II Defining the Health Problem 57

Chapter 3 Community Health Assessment for Program Planning . . . . . . . . 59

Defining Community . . . . . . . . . . . . . . . . . . . . . . . . . . . .59

Community as Context and Intended Recipient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60

Defining Terms: Based, Focused, and Driven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61

Types of Needs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62

Types of Strengths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63

Approaches to Planning . . . . . . . . . . . . . . . . . . . . . . . . .64

Incremental Approach . . . . . . . . . . . . . . . . . . . . . .64

Apolitical Approach . . . . . . . . . . . . . . . . . . . . . . . .66

Advocacy Approach . . . . . . . . . . . . . . . . . . . . . . . .66

Communication Action Approach . . . . . . . . . .67

Comprehensive Rational Approach . . . . . . . . .67

Strategic Planning Approach . . . . . . . . . . . . . . .68

Summary of Approaches . . . . . . . . . . . . . . . . . . .69

Models for Planning Public Health Programs . . . . .69

Mobilizing for Action through Planning and Partnership (MAPP) . . . . . . . . . . . . . . . . . .70

Community Health Improvement Process (CHIP) . . . . . . . . . . . . . . . . . . . . . . . . . . . .70

Protocol for Assessing Community Excellence in Environmental Health (PACE-EH) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70

In Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70

Perspectives on Assessment . . . . . . . . . . . . . . . . . . . . .71

Epidemiological Perspective . . . . . . . . . . . . . . . .72

Public Health Perspective . . . . . . . . . . . . . . . . . . .74

Social Perspective . . . . . . . . . . . . . . . . . . . . . . . . . .74

Asset Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . .74

Rapid Perspective . . . . . . . . . . . . . . . . . . . . . . . . . .75

v Contents

Path to Program Outcomes and Impacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Components of the Effect Theory . . . . . . . . . 135

Matching Levels: Audience, Cause, Intervention, and Effects . . . . . . . . . . . . . . . 137

Generating the Effect Theory . . . . . . . . . . . . . . . . . . 138

Involve Key Stakeholders . . . . . . . . . . . . . . . . . 138

Draw Upon the Scientific Literature . . . . . . . 138

Diagram the Causal Chain of Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

Check Against Assumptions . . . . . . . . . . . . . . 141

Functions of Program Theory . . . . . . . . . . . . . . . . . . 141

Provide Guidance . . . . . . . . . . . . . . . . . . . . . . . . . 141

Enable Explanations . . . . . . . . . . . . . . . . . . . . . . 142

Form a Basis for Communication . . . . . . . . . . 142

Make a Scientific Contribution . . . . . . . . . . . . 143

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

Discussion Questions and Activities . . . . . . . . . . . . 144

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Chapter 6 Program Objectives and Setting Targets . . . . . . . . 147

Program Goals and Objectives . . . . . . . . . . . . . . . . . 147

Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

Foci of Objectives . . . . . . . . . . . . . . . . . . . . . . . . . 148

Objectives and Indicators . . . . . . . . . . . . . . . . . 151

Good Goals and Objectives . . . . . . . . . . . . . . . 154

Using Data to Set Target Values . . . . . . . . . . . . . . . . 156

Decisional Framework for Setting Target Values . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

Stratification and Object Target Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

Use of Logic Statements to Develop Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

Options for Calculating Target Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

Caveats to the Goal-Oriented Approach . . . . . . . 170

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

Discussion Questions and Activities . . . . . . . . . . . . 171

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

Statistics for Describing Health Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .99

Descriptive Statistics . . . . . . . . . . . . . . . . . . . . . . 100

Geographic Information Systems: Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

Small Numbers and Small Areas . . . . . . . . . . 101

Epidemiology Rates . . . . . . . . . . . . . . . . . . . . . . 102

Stating the Health Problem . . . . . . . . . . . . . . . . . . . . 102

Diagramming the Health Problem . . . . . . . . 102

Writing a Causal Theory of the Health Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

Prioritizing Health Problems . . . . . . . . . . . . . . . . . . . 110

Nominal Group Technique . . . . . . . . . . . . . . . . 111

Basic Priority Rating System . . . . . . . . . . . . . . . 111

Propriety, Economics, Acceptability, Resources, and Legality (PEARL) Component . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

Prioritizing Based on Importance and Changeability . . . . . . . . . . . . . . . . . . . . . 114

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

Discussion Questions and Activities . . . . . . . . . . . . 117

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

SECTION III Health Program Development and Planning 121

Chapter 5 Program Theory and Interventions Revealed . . . . . . . . . . . . . . . . . . 123

Program Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

Process Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Effect Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

Finding and Identifying Interventions . . . . . 126

Types of Interventions . . . . . . . . . . . . . . . . . . . . 127

Specifying Intervention Administration and Dosage . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

Interventions and Program Components . . . . 130

Characteristics of Good Interventions . . . . . 131

vi Contents

Budgeting as Part of Planning . . . . . . . . . . . . . . . . . . 204

Monetize and Compute Program Costs . . . . . 204

Budget for Start-Up and Evaluation Costs . . . 205

Break-Even Analysis . . . . . . . . . . . . . . . . . . . . . . . 205

Budget Justification . . . . . . . . . . . . . . . . . . . . . . 207

Budget as a Monitoring Tool . . . . . . . . . . . . . . . . . . . 209

Budget Variance . . . . . . . . . . . . . . . . . . . . . . . . . . 209

Types of Cost Analyses . . . . . . . . . . . . . . . . . . . . 209

Information Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

Health Informatics Terminology . . . . . . . . . . . 214

Information Systems Considerations . . . . . . 214

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

Discussion Questions and Activities . . . . . . . . . . . . 217

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218

Chapter 9 Implementation Evaluation: Measuring Inputs and Outputs . . . . . . . . . . . . . . . 219

Assessing the Implementation . . . . . . . . . . . . . . . . . 219

Implementation Documentation . . . . . . . . . 220

Implementation Assessment . . . . . . . . . . . . . 221

Implementation Evaluation . . . . . . . . . . . . . . . 221

Efficacy, Effectiveness, and Efficiency . . . . . . . . . . . 222

Data Collection Methods . . . . . . . . . . . . . . . . . . . . . . 223

Quantifying Inputs to the Organizational Plan . . . . . . . . . . . . . . . . . . . 223

Human Resources . . . . . . . . . . . . . . . . . . . . . . . . 228

Physical Resources . . . . . . . . . . . . . . . . . . . . . . . . 229

Quantifying Outputs of the Organizational Plan . . . . . . . . . . . . . . . . . . . 230

Information Systems . . . . . . . . . . . . . . . . . . . . . . 230

Monetary Resources . . . . . . . . . . . . . . . . . . . . . . 230

Quantifying Inputs to the Services Utilization Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230

Participants and Recipients . . . . . . . . . . . . . . . 230

Intervention Delivery and Fidelity . . . . . . . . . 231

Quantifying Outputs of the Services Utilization Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234

Coverage as Program Reach . . . . . . . . . . . . . . 234

Participant-Related Issues . . . . . . . . . . . . . . . . . 238

Program Logistics . . . . . . . . . . . . . . . . . . . . . . . . . 240

SECTION IV Implementing and Monitoring the Health Program 173

Chapter 7 Process Theory for Program Implementation . . . . . . . . . . . 175

Organizational Plan Inputs . . . . . . . . . . . . . . . . . . . . . 175

Human Resources . . . . . . . . . . . . . . . . . . . . . . . . 177

Physical Resources . . . . . . . . . . . . . . . . . . . . . . . . 179

Transportation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180

Informational Resources . . . . . . . . . . . . . . . . . . 180

Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180

Managerial Resources . . . . . . . . . . . . . . . . . . . . 180

Fiscal Resources . . . . . . . . . . . . . . . . . . . . . . . . . . 182

Organizational Plan Outputs . . . . . . . . . . . . . . . . . . . 182

Time Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

Operations Manual . . . . . . . . . . . . . . . . . . . . . . . 182

Organizational Chart . . . . . . . . . . . . . . . . . . . . . . 184

Information System . . . . . . . . . . . . . . . . . . . . . . . 185

Inputs to Service Utilization Plan . . . . . . . . . . . . . . . 185

Social Marketing . . . . . . . . . . . . . . . . . . . . . . . . . . 185

Eligibility Screening . . . . . . . . . . . . . . . . . . . . . . . 185

Queuing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

Intervention Delivery . . . . . . . . . . . . . . . . . . . . . 189

Services Utilization Plan Outputs . . . . . . . . . . . . . . . 191

Summary: Elements of Organizational and Services Utilization Plans . . . . . . . . . . . 192

Alternative Plan Formats . . . . . . . . . . . . . . . . . . . . . . . 192

Logic Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

Business Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

Discussion Questions and Activities . . . . . . . . . . . . 197

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

Chapter 8 Monitoring Implementation Through Budgets and Information Systems . . . . . . . 201

Budgets and Budgeting . . . . . . . . . . . . . . . . . . . . . . . 201

Budgeting Terminology . . . . . . . . . . . . . . . . . . . 202

vii Contents

Evaluation and Research . . . . . . . . . . . . . . . . . . 268

Rigor in Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 270

Variables from the Program Effect Theory . . . . . . 271

Outcome and Impact Dependent Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271

Causal Factors as Independent Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273

Antecedent, Moderating, and Mediating Factors as Variables . . . . . . . . . . 273

Measurement Considerations . . . . . . . . . . . . . . . . . . 275

Units of Observation . . . . . . . . . . . . . . . . . . . . . . 275

Types of Variables (Levels of Measurement) . . . . . . . . . . . . . . . . . . . . . . . 275

Timing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278

Sensitivity of Measures . . . . . . . . . . . . . . . . . . . . 278

Threats to Data Quality . . . . . . . . . . . . . . . . . . . . . . . . 279

Missing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279

Reliability Concerns . . . . . . . . . . . . . . . . . . . . . . . 280

Validity of Measures . . . . . . . . . . . . . . . . . . . . . . 281

Contextual Considerations in Planning the Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281

Evaluation Standards . . . . . . . . . . . . . . . . . . . . . 281

Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282

Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284

Discussion Questions and Activities . . . . . . . . . . . . 284

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285

Chapter 12 Choosing Designs for Effect Evaluations . . . . . . . . . 287

Evaluation Design Caveats . . . . . . . . . . . . . . . . . . . . . 288

Considerations in Choosing a Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289

Using Designs Derived from Multiple Paradigms: An Example . . . . . . . . . . . . . . . . 294

Choosing the Evaluation Design . . . . . . . . . . . . . . . 294

Identifying Design Options . . . . . . . . . . . . . . . 294

Overview of the Decision Tree . . . . . . . . . . . . 295

Designs for Outcome Documentation . . . . 298

Designs for Outcome Assessment: Establishing Association . . . . . . . . . . . . . . . . 301

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

Discussion Questions and Activities . . . . . . . . . . . . 242

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243

Chapter 10 Program Quality and Fidelity: Managerial and Contextual Considerations . . . . . . . . . . . . 245

The Accountability Context . . . . . . . . . . . . . . . . . . . . 246

Program Accountability . . . . . . . . . . . . . . . . . . . 246

Professional Accountability . . . . . . . . . . . . . . . 246

Performance and Quality: Navigating the Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247

Quality Improvement Approaches . . . . . . . . 248

Quality Improvement Tools . . . . . . . . . . . . . . . 248

Relevance to Health Programs . . . . . . . . . . . . 251

Performance Measurement . . . . . . . . . . . . . . . 252

Informatics and Information Technology . . . .253

Creating Change for Quality and Fidelity . . . . . . . 255

Interpreting Implementation Data . . . . . . . . 255

Maintaining Program Process Quality and Fidelity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

Managing Group Processes for Quality and Fidelity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

When and What Not to Change . . . . . . . . . . . 259

Formative Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . 259

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259

Discussion Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 260

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261

SECTION V Outcome and Impact Evaluation of Health Programs 263

Chapter 11 Planning the Intervention Effect Evaluations . . . . . . . . . 265

Developing the Evaluation Questions . . . . . . . . . . 266

Characteristics of the Right Question . . . . . 267

Outcome Documentation, Outcome Assessment, and Outcome Evaluation . . . 268

viii Contents

Issues with Quantifying Change from the Program . . . . . . . . . . . . . . . . . . . . . . 339

Relationship of Change to Intervention Effort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342

Clinical and Statistical Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343

Across Levels of Analysis . . . . . . . . . . . . . . . . . . . . . . . 343

Statistical Answers to the Questions . . . . . . . . . . . 345

Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346

Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348

Association . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349

Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350

Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353

Four Fallacies of Interpretation . . . . . . . . . . . . 353

Ecological Fallacy . . . . . . . . . . . . . . . . . . . . . . . . . 354

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356

Discussion Questions and Activities . . . . . . . . . . . . 356

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357

Chapter 15 Qualitative Methods for Planning and Evaluation . . . . . . . . . . . . . . . 359

Qualitative Methods Throughout the Planning and Evaluation Cycle . . . . . . . . . . . . . . 359

Qualitative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 360

Individual In-Depth Interview . . . . . . . . . . . . 361

Written Open-Ended Questions . . . . . . . . . . . 362

Focus Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363

Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364

Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . 364

Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365

Innovative Methods . . . . . . . . . . . . . . . . . . . . . . 366

Scientific Rigor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368

Sampling for Qualitative Methods . . . . . . . . . . . . . 369

Analysis of Qualitative Data . . . . . . . . . . . . . . . . . . . . 372

Overview of Analytic Process . . . . . . . . . . . . . 372

Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374

Issues to Consider . . . . . . . . . . . . . . . . . . . . . . . . 374

Presentation of Findings . . . . . . . . . . . . . . . . . . . . . . . 375

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376

Designs for Outcome Evaluation: Establishing Causation . . . . . . . . . . . . . . . . . 307

Practical Issues with Experimental Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307

Designs and Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . 309

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311

Discussion Questions . . . . . . . . . . . . . . . . . . . . . . . . . . 312

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312

Chapter 13 Sampling Designs and Data Sources for Effect Evaluations . . . . . . . . . 315

Sampling Realities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315

Sample Construction . . . . . . . . . . . . . . . . . . . . . . . . . . 317

Hard-to-Reach Populations . . . . . . . . . . . . . . . 318

Sample Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318

Calculating Response Rates . . . . . . . . . . . . . . . 319

Sampling for Effect Evaluations . . . . . . . . . . . . . . . . 322

Sampling for Outcome Assessment . . . . . . . 322

Sampling for Outcome Evaluation . . . . . . . . 324

Data Collection Methods . . . . . . . . . . . . . . . . . . . . . . 324

Surveys and Questionnaires . . . . . . . . . . . . . . 325

Secondary Data . . . . . . . . . . . . . . . . . . . . . . . . . . 328

Big Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329

Physical Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330

Discussion Questions and Activities . . . . . . . . . . . . 330

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331

Chapter 14 Quantitative Data Analysis and Interpretation . . . . . . . . . . . . 335

Data Entry and Management . . . . . . . . . . . . . . . . . . 335

Outliers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337

Linked Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337

Sample Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338

Thinking About Change . . . . . . . . . . . . . . . . . . . . . . . 339

Change as a Difference Score . . . . . . . . . . . . . 339

ix Contents

Reporting Responsibly . . . . . . . . . . . . . . . . . . . . . . . . . 392

Report Writing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392

Making Recommendations . . . . . . . . . . . . . . . 394

Misuse of Evaluations . . . . . . . . . . . . . . . . . . . . . 397

Responsible Contracts . . . . . . . . . . . . . . . . . . . . . . . . . 398

Organization–Evaluator Relationship . . . . . . 398

Health Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399

Responsible for Evaluation Quality . . . . . . . . . . . . . 400

Responsible for Dissemination . . . . . . . . . . . . . . . . . 401

Responsible for Current Practice . . . . . . . . . . . . . . . 402

Across the Pyramid . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404

Discussion Questions and Activities . . . . . . . . . . . . 405

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .409

Discussion Questions and Activities . . . . . . . . . . . . 377

Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377

SECTION VI Additional Considerations for Evaluators 381

Chapter 16 Program Evaluators’ Responsibilities . . . . . . . . . . . 383

Ethical Responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . 383

Ethics and Planning . . . . . . . . . . . . . . . . . . . . . . . 383

Institutional Review Board Approval and Informed Consent . . . . . . . . . . . . . . . . . 385

Ethics and Evaluation . . . . . . . . . . . . . . . . . . . . . 387

HIPAA and Evaluations . . . . . . . . . . . . . . . . . . . . 388

Responsible Spin of Data and Information . . . . . 389

Persuasion and Information . . . . . . . . . . . . . . . 389

Information and Sensemaking . . . . . . . . . . . . 391

xi

© Lynne Nicholson/Shutterstock

List of Figures Figure 1-1 The Planning and Evaluation Cycle

Figure 1-2 The Public Health Pyramid

Figure 1-3 The Pyramid as an Ecological Model

Figure 2-1 Effects of Diversity Throughout the Planning and Evaluation Cycle Stage in the Planning and Evaluation Cycle

Figure 3-1 Connections Among Program, Agency, and Community

Figure 3-2 Venn Diagram of Community- Based, Community-Focused, and Community-Driven

Figure 3-3 The Planning and Evaluation Cycle

Figure 4-1 Generic Model of a Theory of Causes

Figure 4-2 Diagram of Theory of Causes/ Determinants of Receiving Immunizations, as Contributing to Adult Immunization Rates, Using the Layetteville Example

Figure 4-3 Diagram of Theory of Causes/ Determinants for Deaths from Gunshot Wounds, as Contributing to Adolescent Death Rates, Using the Layetteville Example

Figure 4-4 Diagram of Theory of Causes/ Determinants for Neural Tube Defects, as Contributing to Rates of Congenital Anomalies, Using the Bowe County Example

Figure 4-5 Theory of Causes/Determinants with Elements of the BPRS Score: Size, Seriousness, and Interventions

Figure 5-1 Model of Program Theory

Figure 5-2 The Effect Theory Showing the Causal Theory Using Community Diagnosis Elements

Figure 5-3 Effect Theory Example: Effect Theory for Reducing the Rate of Congenital Anomalies

Figure 5-4 Two Roots of Program Failure

Figure 6-1 Using Elements of Program Theory as the Basis for Writing Program Objectives

Figure 6-2 Diagram Showing Relationship of Effect Theory Elements to Process and Outcome Objectives

Figure 6-3 Calculations of Options 1 Through 4 Using a Spreadsheet

Figure 6-4 Calculations of Options 5 Through 8 Using a Spreadsheet

Figure 6-5 Calculations of Options 9 and 10 Using a Spreadsheet

Figure 7-1 Amount of Effort Across the Life of a Health Program

Figure 7-2 Diagram of the Process Theory Elements Showing the Components of the Organizational Plan and Services Utilization Plan

Figure 7-3 Process Theory for Neural Tube Defects and Congenital Anomalies Health Problem

Figure 7-4 Effect and Process Theory for Neural Tube Defect Prevention Program

Figure 8-1 Relevance of Process Theory to Economic Evaluations

Figure 8-2 Information System Processes Throughout the Program Planning Cycle

Figure 9-1 Elements of the Process Theory Included in a Process Evaluation

Figure 9-2 Roots of Program Failure

xii List of Figures

Figure 12-2 Decision Tree for Choosing an Evaluation Design, Based on the Design’s Typical Use

Figure 12-3 Three Sources of Program Failure

Figure 13-1 Probability and Nonprobability Samples and Their Usage

Figure 14-1 Contributing Factors to the Total Amount of Change

Figure 14-2 Summary of the Three Decisions for Choosing an Analytic Approach

Figure 14-3 Five Ways That the Rate of Change Can Be Altered

Figure 16-1 Making Recommendations Related to the Organizational and Services Utilization Plans

Figure 16-2 Making Recommendations Related to the Program Theory

Figure 16-3 The Planning and Evaluation Cycle with Potential Points for Recommendations

Figure 9-3 Examples of Organizational Plan Inputs and Outputs That Can Be Measured

Figure 9-4 Examples of Services Utilization Inputs and Outputs That Can Be Measured

Figure 10-1 List of Quality Improvement Tools with Graphic Examples

Figure 11-1 Planning and Evaluation Cycle, with Effect Evaluation Highlights

Figure 11-2 Diagram of Net Effects to Which Measures Need to Be Sensitive

Figure 11-3 Using the Effect Theory to Identify Effect Evaluation Variables

Figure 11-4 Effect Theory of Reducing Congenital Anomalies Showing Variables

Figure 12-1 Relationship Between the Ability to Show Causality and the Costs and Complexity of the Design

xiii

© Lynne Nicholson/Shutterstock

Table 4-3 Global Leading Causes of Disability- Adjusted Life-Years (DALYs) and Years of Life Lost (YLL)

Table 4-4 Numerators and Denominators for Selected Epidemiological Rates Commonly Used in Community Health Assessments

Table 4-5 Existing Factors, Moderating Factors, Key Causal Factors, Mediating Factors, and Health Outcome and Impact for Five Health Problems in Layetteville and Bowe County

Table 4-6 Relationship of Problem Definition to Program Design and Evaluation

Table 4-7 Criteria for Rating Problems According to the BPRS

Table 4-8 Program Prioritization Based on the Importance and Changeability of the Health Problem

Table 4-9 Examples of Sources of Data for Prioritizing Health Problems at Each Level of the Public Health Pyramid

Table 4-10 Examples of Required Existing, Causal, and Moderating Factors Across the Pyramid

Table 5-1 Examples of Interventions by Type and Level of the Public Health Pyramid

Table 5-2 Comparison of Effect Theory, Espoused Theory, and Theory-in-Use

Table 5-3 Examples of Types of Theories Relevant to Developing Theory of Causative/Determinant Factors or Theory of Intervention Mechanisms by Four Health Domains

List of Tables Table 1-1 Comparison of Outcome-Focused,

Utilization-Focused, and Participatory Focused Evaluations

Table 1-2 Evaluation Standards Established by the Joint Commission on Standards for Educational Evaluation

Table 1-3 Fuzzy Aspects Throughout the Planning and Evaluation Cycle

Table 1-4 A Summary of the Healthy People 2020 Priority Areas

Table 2-1 Examples of Cultural Tailoring Throughout the Program Planning and Evaluation Cycle

Table 2-2 Indicators Used to Measure Race in Different Surveys

Table 2-3 Professional Diversity Among Health Professions

Table 2-4 Cultural Continuum with Examples of the Distinguishing Features of Each Stage

Table 3-1 Three Elements of Community, with Their Characteristics

Table 3-2 Summary of the Six Approaches to Planning, with Public Health Examples

Table 3-3 Comparison of Models Developed for Public Health Planning

Table 3-4 A Comparison of the Five Perspectives on Community Health and Needs Assessment

Table 4-1 Haddon’s Typology for Analyzing an Event, Modified for Use in Developing Health Promotion and Prevention Programs

Table 4-2 Quality-of-Life Acronyms and Definitions

xiv List of Tables

Table 5-4 Examples of Types of Theories Relevant to Developing the Organizational Plan and Services Utilization Plan Components of the Process Theory

Table 6-1 Aspects of Process Objectives as Related to Components of the Process Theory, Showing the TAAPS Elements

Table 6-2 Domains of Individual or Family Health Outcomes with Examples of Corresponding Indicators and Standardized Measures

Table 6-3 Bowe County Health Problems with Indicators, Health Outcomes, and Health Goals

Table 6-4 Effect Objectives Related to the Theory of Causal/Determinant Factors, Theory of the Intervention Mechanisms, and Theory of Outcome to Impact, Using Congenital Anomalies as an Example, Showing the TREW Elements

Table 6-5 Effect Objectives Related to the Theory of Causal/Determinant Factors, Theory of the Intervention Mechanisms, and Theory of Outcome to Impact, Using Adolescent Pregnancy as an Example, Showing the TREW Elements

Table 6-6 Matrix of Decision Options Based on Current Indicator Value, Population Trend of the Health Indicator, and Value of Long-Term Objective or Standard

Table 6-7 Framework for Target Setting: Interaction of Data Source Availability and Consistency of Information

Table 6-8 Summary of When to Use Each Option

Table 6-9 Range of Target Values Derived from Options 1 Through 10, Based on the Data from Figures 6-3 Through 6-5

Table 7-1 List of Health Professionals with a Summary of Typical Legal and Regulatory Considerations

Table 7-2 Relationship of Test Sensitivity and Specificity to Overinclusion and Underinclusion

Table 7-3 Examples of Partial- and Full-Coverage Programs by Level of the Public Health Pyramid

Table 7-4 Template for Tracking Services Utilization Outputs Using Example Interventions and Hypothetical Activities

Table 7-5 Hypothetical Logic Model of a Program for Reducing Congenital Anomalies

Table 7-6 Generic Elements of a Business Plan, with Their Purpose and Corresponding Element of the Process Theory and Logic Model

Table 8-1 Formulas Applied for Options A and B

Table 9-1 Methods of Collecting Process Evaluation Data

Table 9-2 Example of Measures of Inputs and Outputs of the Organizational Plan

Table 9-3 Examples of Measures of Inputs and Outputs of the Services Utilization Plan

Table 9-4 Matrix of Undercoverage, Ideal Coverage, and Overcoverage

Table 9-5 Examples of Process Evaluation Measures Across the Public Health Pyramid

Table 10-1 Types of Program Accountability, with Definitions and Examples of Process Evaluation Indicators

Table 10-2 Comparison of Improvement Methodologies and Program Process Evaluation

Table 10-3 Definitions of Terms Used in Performance Measurement

Table 10-4 Partial List of Existing Performance Measurement Systems Used by Healthcare Organizations, with Their Websites

Table 11-1 Three Levels of Intervention Effect Evaluations

Table 11-2 Differences Between Evaluation and Research

xvList of Tables

Table 11-3 Advantages and Disadvantages of Using Each Type of Variable

Table 11-4 Examples of Nominal, Ordinal, and Continuous Variables for Different Health Domains

Table 11-5 Example Time Line Showing the Sequence of Intervention and Evaluation Activities

Table 11-6 Summary of Evaluation Elements

Table 12-1 Contribution of Various Disciplines to Health Program Evaluation

Table 12-2 Summary of Main Designs and Their Use for Individual or Population-Level Evaluations

Table 12-3 Approaches to Minimizing Each of the Three Types of Program Failure

Table 13-1 Probability and Nonprobability Samples and Their Usage

Table 13-2 Comparison of Main Types of Samples with Regard to Implementation Ease, Degree of Representativeness, and Complexity of Sampling Frame

Table 13-3 Example of Data Sources for Each Health and Well-Being Domain

Table 13-4 Interaction of Response Bias and Variable Error

Table 14-1 Calculation of Effectiveness and Adequacy Indices: An Example

Table 14-2 Intervention Efficiency as a Relation of Effect Size and Causal Size

Table 14-3 Factors That Affect the Choice of a Statistical Test: Questions to Be Answered

Table 14-4 Analysis Procedures by Level of Intervention and Level of Analysis

Table 14-5 Commonly Used Parametric and Nonparametric Statistical Tests for Comparison, Association, and Prediction

Table 14-6 Main Types of Comparison Analyses Used by Level of Analysis and Assuming That the Variables Are at the Same Level of Measurement

Table 14-7 Main Types of Association Analyses Used by Level of Analysis, Assuming That Variables Are the Same Level of Measurement

Table 14-8 Example of Statistical Tests for Strength of Association by Level of Measurement, Using Layetteville Adolescent Antiviolence Program

Table 14-9 Examples of Statistical Tests by Evaluation Design and Level of Measurement, with Examples of Variables

Table 14-10 Main Types of Prediction Analyses Used by Level of Analysis, Assuming That Variables Are at the Same Level of Measurement

Table 15-1 Comparison of Qualitative Perspectives with Regard to the Basic Question Addressed and the Relevance to Health Program Planning and Evaluation

Table 15-2 Comparison of Major Qualitative Perspectives with Regard to the Method Used

Table 15-3 Summary of Key Benefits and Challenges to Using Qualitative Methods in Planning and Evaluation

Table 15-4 Sampling Considerations for Each of the Qualitative Methods Discussed

Table 15-5 Summary of Types of Sampling Strategies Used with Qualitative Designs

Table 15-6 Example of Interview Text with Final Coding

Table 15-7 Suggested Qualitative Methods by Pyramid Level and Planning Cycle

Table 16-1 Ethical Frameworks and Principles for Planning Health Programs

Table 16-2 Comparison of Types of IRB Reviews

Table 16-3 Eight Elements of Informed Consent, as Required in 45 CFR 46

Table 16-4 Effect of Rigor and Importance of Claims on Decision Making

Table 16-5 List of Ways to Make Graphs More Interpretable

Table 16-6 Examples of Dissemination Modes, Audiences, and Purposes

xvii

© Lynne Nicholson/Shutterstock

Exhibit 8-3 Break-Even Table Shows Number of Paying Students Needed to Break Even

Exhibit 8-4 Example of a Budget Showing Year-to-Date Variance

Exhibit 8-5 Types of Cost Analyses

Exhibit 9-1 Formulas for Measures of Coverage

Exhibit 9-2 Example of Narrative Background about Coverage and Dosage Measures

Exhibit 9-3 Examples of Coverage Measures Using an Excel Spreadsheet

Exhibit 9-4 Examples of Calculating Dosage for the Congenital Anomalies Prevention Program Using Excel

List of Exhibits Exhibit 2-1 Checklist to Facilitate Development

of Cultural and Linguistic Competence Within Healthcare Organizations

Exhibit 2-2 Checklist to Facilitate Cultural Competence in Community Engagement

Exhibit 7-1 Example of an Abbreviated Time Line for a Short-Term Health Program

Exhibit 7-2 Chapter Text Paragraph Rewritten at an Eighth-Grade Reading Level

Exhibit 8-1 Example of a Scenario Needing a Break-Even Analysis

Exhibit 8-2 Example of a Budget Used for a Break-Even Analysis for Bright Light on an Excel Spreadsheet

xix

© Lynne Nicholson/Shutterstock

the text are relevant to health administrators, medical social workers, nurses, nutritionists, pharmacists, public health professionals, physical and occupational therapists, and physicians.

This textbook grew from teaching experi- ences with both nurses and public health students and their need for direct application of the pro- gram planning and evaluation course content to their work and to their clients and communities. Today programs need to be provided through community-based healthcare settings to address broad public health issues and expand the individ- ual to population focus. The distinction between individual patient health and population health is a prerequisite for the thinking and planning—in terms of aggregates and full populations—by students from clinical backgrounds.

In most graduate health professions programs, students take a research methods course and a statistics course. Therefore, this evaluation text avoids duplicating content related to research methods and statistics while addressing and extending that content into health program de- velopment, implementation, and evaluation. In addition, because total quality management and related methodologies are widely used in healthcare organizations, areas of overlap between quality improvement methodologies and traditional program evaluation approaches are discussed. This includes ways that quality improvement methodologies complement program evaluations. Sometimes evaluations are appropriate; sometimes they are not. Enthusiasm for providing health programs and performing evaluation is tempered with thoughtful notes of caution in the hope that students will avoid potentially serious and costly program and evaluation mistakes.

Preface to the Fourth Edition The fourth edition of Health Program Planning and Evaluation has stayed true to the purpose and intent of the previous editions. This advanced- level text is written to address the needs of professionals from diverse health disciplines who find themselves responsible for developing, implementing, or evaluating health programs. The aim of the text is to assist health profes- sionals to become not only competent health program planners and evaluators but also savvy consumers of evaluation reports and prudent users of evaluation consultants. To that end, the text includes a variety of practical tools and concepts necessary to develop and evaluate health programs, presenting them in language understandable to both the practicing and novice health program planner and evaluator.

Health programs are conceptualized as encompassing a broad range of programmatic interventions that span the social-ecological range, from individual-level to population-level programs. Examples of programs cited through- out the text are specific yet broadly related to improving health and reflect the breadth of public health programs. The examples have been updated once again to reflect current best practices. Maintaining a public health focus provides an opportunity to demonstrate how health programs can target different levels of a population, different determinants of a health problem, and different strategies and interven- tions to address a health problem. In addition, examples of health programs and references are selected to pique the interests of the diverse students and practicing professionals who con- stitute multidisciplinary program teams. Thus, the content and examples presented throughout

xx Preface to the Fourth Edition

Articulating each of the component elements of the program theory sharpens the student’s awareness of what must be addressed to create an effective health program. One element of the program theory is the effect theory, which focuses on how the intervention results in the program effects. The effect theory had its genesis in the concepts of action and intervention hypotheses described by Rossi and Freeman; those concepts were dropped from later editions of their text. We believe these authors were onto something with their effort to elucidate the various path- ways leading from a problem to an effect of the program. Rossi and colleagues’ ideas have been updated with the language of moderating and mediating factors and an emphasis on the intervention mechanisms.

Throughout the current edition of this textbook, emphasis is given to the effect theory portion of the program theory. The effect theory describes relationships among health antecedents, causes of health problems, program interventions, and health effects. The hypotheses that comprise the effect theory need to be understood and ex- plicated to plan a successful health program and to evaluate the “right” elements of the program. The usefulness of the effect theory throughout the planning and evaluation cycle is highlighted throughout this text; for example, the model is used as a means of linking program theory to evaluation designs and data collection. The model becomes an educational tool by serving as an example of how the program theory is manifested throughout the stages of planning and evaluation, and by reinforcing the value of carefully articulating the causes of health problems and consequences of programmatic interventions. Students and novice program planners may have an intuitive sense of the connection between their actions and outcomes, but they may not know how to articulate those connections in ways that program stakeholders can readily grasp. The effect theory and the process theory—the other main element of the program theory—provide a basis from which to identify and describe these connections.

▸ Unique Features The Fourth Edition has retained the three unique features that distinguish this text from other program planning and evaluation textbooks: use of the public health pyramid, consistent use of a model of the program theory throughout the text, and role modeling of evidence-based practice.

The public health pyramid explains how health programs can be developed for individu- als, aggregates, populations, and service delivery systems. Use of the pyramid is also intended as a practical application of the social-ecological per- spective that acknowledges a multilevel approach to addressing health problems. The public health pyramid contains four levels: direct services to individuals; enabling services to aggregates; services provided to entire populations; and, at the base, infrastructure. In this textbook, the pyramid is used as an organizing structure to summarize the content of each chapter in the “Across the Pyramid” sections. In these sections, specific attention is paid to how key concepts in a given chapter might vary across each pyramid level. Summarizing the chapter content in this manner reinforces the perspective that enhancing health and well-being requires integrated efforts across the levels of the public health pyramid. Health program development and evaluation is relevant for programs intended for individuals, aggregates, populations, and service delivery systems, and this fact reinforces the need to tailor program plans and evaluation designs to the level at which the program is conceptualized. Using the pyramid also helps health professionals begin to value their own and others’ contributions within and across the levels and to transcend disciplinary boundaries.

The second unique feature of this text is that one conceptual model of program planning and evaluation is used throughout the text: the program theory. The program theory is like a curricular strand, connecting content across the chapters, and activities throughout the planning and evaluation cycle. The program theory, as a conceptual model, is composed of elements.

xxiPreface to the Fourth Edition

The third unique feature of this text is the intentional role modeling of evidence-based practice. Use of published, empirical evidence as the basis for practice—whether clinical practice or program planning practice—is the professional standard. Each chapter of this book contains substantive examples drawn from the published scientific health and health-related literature. Relying on the literature for examples of programs, evaluations, and issues is consistent with the espoused preference of using scientific evidence as the basis for making programmatic decisions. Each chapter offers multiple examples from the health sciences literature that substan- tiate the information presented in the chapter.

▸ Organization of the Book

The book is organized into six sections, each covering a major phase in the planning and eval- uation cycle. Chapter 1 introduces the fictitious city of Layetteville and the equally fictitious Bowe County. In subsequent chapters, chapter content is applied to the health problems of Layetteville and Bowe County so that students can learn how to use the material on an ongoing basis. In several chapters, the case study is used in the “Discussion Questions and Activities” section to provide students with an opportunity to practice applying the chapter content. In recognition of the availability of parts of the text in digital format, each use of the Layetteville case stands on its own in reference to the chapter’s content.

Section I explores the context in which health programs and evaluations occur. Chap- ter 1 begins with an overview of definitions of health, followed by a historical context. The public health pyramid is introduced and pre- sented as an ecological framework for thinking of health programs. An overview of community is provided and discussed as both the target and the context of health programs. The role of community members in health programs and

evaluations is introduced, and emphasis is given to community as a context and to strategies for community participation throughout the program development and evaluation process. Chapter 2 focuses on the role of diversity in the planning and evaluation cycle and its effects on the delivery and evaluation of health programs. Although a discussion of diversity-related issues could have been added to each chapter, the sensitive nature of this topic and its importance in ensuring a successful health program warranted it being covered early in the text and as a separate chapter. Cultural competence is discussed, particularly with regard to the organization providing the health program and with regard to the program staff members.

Section II contains two chapters that focus on the task of defining the health problem. Chapter 3 covers planning perspectives and the history of health program planning. Effective health program developers understand that approaches to planning are based on assump- tions. These assumptions are exemplified in six perspectives that provide points of reference for understanding diverse preferences for prioritizing health needs and expenditures and therefore for tailoring planning actions to fit the situation best. Chapter 3 also reviews perspectives on conducting a community needs assessment as foundational to decision making about the future health program. Essential steps involved in conducting a community health and needs assessment are outlined as well.

Chapter 4 expands on key elements of a community needs assessment, beginning with a review of the data collection methods appro- priate for a community needs assessment. This discussion is followed by a brief overview of key epidemiological statistics. Using those statistics and the data, the reader is guided through the process of developing a causal statement of the health problem. This causal statement, which includes the notion of moderating and mediating factors in the pathway from causes to problem, serves as the basis for the effect theory of the program. Once the causal statement has been

xxii Preface to the Fourth Edition

developed, prioritization of the problem is needed; four systems for prioritizing in a rational manner are reviewed in Chapter 4.

Following prioritization comes planning, beginning with the decision of how to address the health problem. In many ways, the two chapters in Section III form the heart of planning a successful health program. Unfortunately, students generally undervalue the importance of theory for selecting an effective intervention and of establishing target values for objectives. Chapter 5 explains what theory is and how it provides a cornerstone for programs and evaluations. More important, the concept of intervention is discussed in detail, with attention given to characteristics that make an intervention ideal, including attention to intervention dosage. Program theory is introduced in Chapter 5 as the basis for organizing ideas related to the selection and delivery of the interventions in conjunction. The effect theory element of the program theory is introduced and the components of the effect theory are explained. Because the effect theory is so central to having an effective program interven- tion and the subsequent program evaluation, it is discussed in conjunction with several examples from the Layetteville and Bowe County case.

Chapter 6 goes into detail about developing goals and objectives for the program, with particular attention devoted to articulating the interven- tions provided by the program. A step-by-step procedure is presented for deriving numerical targets for the objectives from existing data, which makes the numerical targets more defendable and programmatically realistic. We focus on distinguishing between process objectives and outcome objectives through the introduction of two mnemonics: TAAPS (Time frame, Amount of what Activities done by which Participants/ program Staff ) and TREW (Timeframe, what portion of Recipients experience what Extent of Which type of change).

Section IV deals with the task of imple- menting a health program. Chapter 7 provides an in-depth review of key elements that consti- tute the process theory element of the program theory—specifically, the organizational plan and services utilization plan. The distinction between

inputs and outputs of the process theory is high- lighted through examples and a comprehensive review of possible inputs and outputs. Budgeting for program operations is covered in this chapter as well. Chapter 8 is devoted entirely to fiscal data systems, including key aspects of budgeting, and informatics. Chapter 9 details how to evaluate the outputs of the organizational plan and the services utilization plan. The practical application of mea- sures of coverage is described, along with the need to connect the results of the process evaluation to programmatic changes. Program management for assuring a high-quality program that delivers the planned intervention is the focus of Chapter 10.

Section V contains chapters that are specific to conducting the effect evaluations. These chap- ters present both basic and advanced research methods from the perspective of a program effect evaluation. Here, students’ prior knowledge about research methods and statistics is brought together in the context of health program and services evaluation. Chapter 11 highlights the importance of refining the evaluation question and provides information on how to clarify the question with stakeholders. Earlier discussions about program theory are brought to bear on the development of the evaluation question. Key issues, such as data integrity and survey construction, are addressed with regard to the practicality of program evaluation. Chapter 12 takes a fresh approach to evaluation design by organizing the traditional experimental and quasi-experimental designs and epidemiological designs into three levels of program evaluation design based on the design complexity and purpose of the evaluation. The discussion of sampling in Chapter 13 retains the emphasis on practicality for program evalua- tion rather than taking a pure research approach. However, sample size and power are discussed because these factors have profound relevance to program evaluation. Chapter 14 reviews sta- tistical analysis of data, with special attention to variables from the effect theory and their level of measurement. The data analysis is linked to interpretation, and students are warned about potential flaws in how numbers are understood. Chapter 15 provides a review of qualitative designs

xxiiiPreface to the Fourth Edition

and methods, especially their use in health pro- gram development and evaluation.

The final section, Section VI, includes just one chapter. Chapter 16 discusses the use of evaluation results when making decisions about existing and future health programs. Practical and conceptual issues related to the ethics issues that program evaluators face are addressed. This chapter also reviews ways to assess the quality of evaluations and the professional responsibilities of evaluators.

Each chapter in the book concludes with a “Discussion Questions and Activities” section. The questions posed are intended to be provoc- ative and to generate critical thinking. At the graduate level, students need to be encouraged to engage in independent thinking and to foster their ability to provide rationales for decisions. The discussion questions are developed from this point of view. In the “Internet Resources” section, links are provided to websites that support the content of the chapter. These websites have been carefully chosen as stable and reliable sources.

▸ Additions to and Revisions in the Fourth Edition

The fourth edition of Health Program Planning and Evaluation represents continuous improve- ment, with corrections and updated references. Classical references and references that remain state of the art have been retained.

The Fourth Edition has retained the original intent—namely, to provide students with the ability to describe a working theory of how the intervention acts upon the causes of the health problem and leads to the desired health results. Some content has been condensed in order to allow enough room to describe current evaluation approaches adequately for both new and experi- enced practitioners. For instance, Chapter 1 now includes participatory evaluations in addition to outcome- and utilization-focused evaluations. In addition to disciplines traditionally recognized

in western medical care, Chapter 2 now includes acupuncture and massage therapy as examples of health professional diversity. Discussion of the nuances of cultural competency has been refined, in light of the continuing importance and challenges of this area. Community strengths have been given more attention in Chapter 3 in recognition of the powerful potential of shifting from a “deficit-based” to an “asset-based” perspec- tive on health planning. Chapter 4 now devotes greater attention to the health evaluation poten- tial of data from social media such as Facebook and Twitter, as well as geospatial data, including attendant concerns about privacy, and also notes implications of the increasingly prevalent public rankings of community health status. Examples of infrastructure-level interventions within the public health pyramid have been added in Chapter 5. Discussion of financial modeling options in Chapter 8 now includes simulation modeling, an exciting if also resource-intensive option to conducting real-world experiments, which are, of course, inevitably expensive themselves. Chapters 9 and 15 include emerging data collection techniques such as participant self-reports, video, photos, and audio recordings that may make public health evaluation more inclusive of the people such interventions seek to serve. Chap- ter 13 includes updates on surveying, reflecting the decreased numbers of people with land-line phones, long a mainstay of health evaluations. Options for online surveying have been updated in Chapter 14; given the rapid evolution of big data such as those available from social media, billing, and medical records, discussion of this topic has been updated in Chapter 13 as well. Finally, Chapter 16 now includes bioethics— the application of ethical and philosophical principles to medical decision making—as an increasingly salient component of responsible health evaluation.

In sum, we have worked hard to sustain this book’s conceptual and empirical rigor and currency in the Fourth Edition while maintaining accessibility for a range of health evaluators. Above all, we hope this book is useful to our readers’ vitally important efforts to improve health.

xxv

© Lynne Nicholson/Shutterstock

Acknowledgments We are indebted to the many people who supported and aided us in preparing this fourth edition of Health Program Planning and Evaluation: A Practical, Systematic Approach for Community Health. We remain grateful to the numerous students over the years who asked questions that revealed the typical sticking points in their acquiring and understanding of the concepts and content, as well as where new explanations were needed. Through their eyes we have learned there is no one way to explain a complex notion or process. Their interest and enthusiasm for planning and evaluating health programs was a great motivator for writing this book.

Several additional colleagues helped fine-tune this text. We are especially indebted to Arden

Handler at the School of Public Health, University of Illinois at Chicago, for taking time to contribute to this textbook. Her devotion to quality and clarity has added much to the richness of otherwise dry material. We remain deeply indebted to Deborah Rosenberg, also at the School of Public Health University of Illinois at Chicago, for sharing her innovative and quintessentially useful work on developing targets for program objectives. Special thanks as well to Joseph Chen, at the University of Texas School of Public Health, for his many contributions to updating the literature cited across many chapters and for his contribution on big data. Last, but not least, thanks to Mike Brown, publisher at Jones & Bartlett Learning, for his encouragement and patience over the years.

xxvii

© Lynne Nicholson/Shutterstock

DHHS U.S. Department of Health and Human Services

DSM-5 Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition

EBM Evidence-based medicine

EBP Evidence-based practice

EHR Electronic health record

EMR Electronic medical record

FTE Full-time equivalent

GAO U.S. Government Accountability Office

GNP Gross Product

GPRA Government Performance and Results Act

HEDIS Healthcare Effectiveness Data and Information Set

HIPAA Health Insurance Portability and Accountability Act

HIT Health information technology

HMOs Health maintenance organizations

HRQOL Health-related quality of life

HRSA Health Resources and Services Administration (part of DHHS)

i-APP Innovation–Adolescent Preventing Pregnancy (Program)

ICC Intraclass correlation

IRB Institutional review board

JCAHO Joint Commission on the Accreditation of Healthcare Organizations

MAPP Mobilizing for Action through Planning and Partnership

MBO Management by objectives

List of Acronyms ABCD Asset-based community

development

ACA Affordable Care Act

AEA American Evaluation Association

AHRQ Agency for Healthcare Research and Quality

ANOVA Analysis of variance

APHA American Public Health Association

BPRS Basic priority rating system

BRFSS Behavioral Risk Factor Surveillance System

BSC Balanced Score Card

CAHPS Consumer Assessment of Health Plans

CARF Commission on Accreditation of Rehabilitation Facilities

CAST-5 Capacity Assessment of Title-V

CBA Cost–benefit analysis

CBPR Community-based participatory research

CDC Centers for Disease Control and Prevention

CEA Cost-effectiveness analysis

CER Cost-effectiveness ratio

CFIR Consolidated Framework for Implementation Research

CFR Code of Federal Regulations

CHIP Community Health Improvement Process

CI Confidence interval

CPT Current Procedural Terminology

CQI Continuous quality improvement

CUA Cost–utility analysis

DALY Disability-adjusted life-year

xxviii List of Acronyms

PSA Public service announcement

QALY Quality-adjusted life-year

RAR Rapid assessment and response

RARE Rapid assessment and response and evaluation

RE-AIM Reach, Effectiveness, Adoption, Implementation, and Maintenance model

RR Relative risk

SAMHSA Substance Abuse and Mental Health Services Administration

SCHIP State Child Health Insurance Program

SES Socioeconomic status

SMART Specific, measurable, achievable, realistic, and time (objective)

TAAPS Time frame, Amount of what Activities done by which Participants/program Staff

TQM Total quality management

TREW Time frame, what portion of Recipients experience what Extent of Which type of change

UOS Units of service

WHO World Health Organization

WIC Special Supplemental Nutrition Program for Women, Infants, and Children

YHL Years of healthy life

YLL Years of life lost

YPLL Years of potential life lost

MCHB Maternal and Child Health Bureau (part of HRSA)

NACCHO National Association of City and County Health Officers

NAMI National Alliance on Mental Illness

NCHS National Center for Health Statistics

NCQA National Commission on Quality Assurance

NFPS National Family Planning Survey

NHANES National Health and Nutrition Examination Survey

NHIS National Health Interview Survey

NIH National Institutes of Health

NPHPS National Public Health Performance Standards

OHRP Office for Human Research Protections

OMB Office of Management and Budgeting

OR Odds ratio

PACE-EH Protocol for Assessing Excellence in Environmental Health

PAHO Pan American Health Organization

PDCA Plan-Do-Check-Act

PEARL Property, economic, acceptability, resource, legality system

PERT Program Evaluation and Review Technique

PPIP Putting Prevention into Prevention

PRECEDE Predisposing, Reinforcing, and Enabling Factors in Community Education Development and Evaluation (model)

SECTION I

The Context of Health Program Development

© Lynne Nicholson/Shutterstock

3

© Lynne Nicholson/Shutterstock

Context of Health Program Development and Evaluation

Health is not a state of being that can easily be achieved through isolated, uninformed, individualistic actions. Health of individ- uals, of families, and of populations is a state in which physical, mental, and social well-being are integrated to enable optimal functioning. From this perspective, achieving and maintaining health across a life span is a complex, complicated, intri- cate affair. For some, health is present irrespective of any special efforts or intention. For most of us, health requires, at a minimum, some level of attention and specific information. It is through health programs that attention is given focus and information is provided or made available, but that does not guarantee that the attention and information are translated into actions or behaviors needed to achieve health. Thus, those providing health programs, however large or small, need to understand both the processes whereby those in need of attention and health information can receive what is needed, and also the processes by which to learn from the experience of providing the health program.

The processes and effects of health pro- gram planning and evaluation are the subjects of this text. The discussion begins here with a brief overview of the historical context. This background sets the stage for appreciating the considerable number of publications on the topic of health program planning and evaluation, and for acknowledging the professionalization of evaluators. The use of the term processes to describe the actions involved in health program planning and evaluation is intended to denote action, cycles, and open-endedness. This chapter introduces the planning and evaluation cycle, and the interactions and iterative nature of this cycle are stressed throughout the text. Because health is an individual, aggregate, and population phenomenon, health programs need to be conceptualized across those levels. The public health pyramid, introduced in this chapter, is used throughout the text as a tool for conceptualizing and actualizing health programs for individuals, aggregates, and populations.

CHAPTER 1

4 Chapter 1 Context of Health Program Development and Evaluation

▸ History and Context An appropriate starting point for this text is reflecting on and understanding what “health” is, along with having a basic appreciation for the genesis of the fields of health program planning and evaluation. A foundation in these elements is key to becoming an evaluation professional.

Concept of Health To begin the health program planning and evaluation cycle requires first reflecting on the meaning of health. Both explicit and implicit meanings of health can dramatically influence what is considered the health problem and the subsequent direction of a program. The most widely accepted definition of health is that put forth by the World Health Organization (WHO), which for the first time defined health as more than the absence of illness and as the presence of well-being (WHO, 1947).

Since the publication of the WHO defini- tion, health has come to be viewed across the health professions as a holistic concept that encompasses the presence of physical, mental, developmental, social, and financial capabil- ities, assets, and balance. This idea does not preclude each health profession from having a particular aspect of health to which it primarily contributes. For example, a dentist contributes primarily to a patient’s oral health, knowing that the state of the patient’s teeth and gums has a direct relationship to his or her physical and social health. Thus the dentist might say that the health problem is caries. The term health problem is used, rather than illness, diagnosis, or pathology, in keeping with the holistic view that there can be problems, deficits, and pathologies in one component of health while the other components remain “healthy.” Using the term health problem also makes it easier to think about and plan health programs for aggregates of individuals. A community, a family, and a school can each have a health problem that is the focus of a health program intervention. The extent to which the health program planners have

a shared definition of health and have defined the scope of that definition influences the nature of the health program.

Health is a matter of concern for more than just health professionals. For many Americans, the concept of health is perceived as a right, along with civil rights and liberties. The right to health is often translated by the public and politicians into the perceived right to have or to access health care. This political aspect of health is the genesis of health policy at the local, federal, and international levels. The extent to which the political nature of health underlies the health problem of concern being programmatically addressed also influences the final nature of the health program.

Health Programs, Projects, and Services What distinguishes a program from a project or from a service can be difficult to explain, given the fluidity of language and terms. The term program is fairly generic but generally connotes a structured effort to provide a specific set of services or interventions. In contrast, a project often refers to a time-limited or experimental effort to provide a specific set of services or interventions through an organizational struc- ture. In the abstract, a service can be difficult to define but generally includes interaction between provider and client, an intangibility aspect to what is provided, and a nonpermanence or transitory nature to what is provided. Using this definition of service, it is easy to see that what is provided in a health program qualifies as a service, although it may not be a health service.

A health program is a totality of an organized structure designed for the provision of a fairly discrete health-focused intervention, where that intervention is designed for a specific target audience. By comparison, health services are the organizational structures through which providers interact with clients or patients to meet the needs or address the health problems of the clients or patients. Health programs, particularly

History and Context 5

in public health, tend to provide educational services, have a prevention focus, and deliver services that are aggregate or population-focused. In contrast, health services exist exclusively as direct services. Recognizing the distinction between health programs and health services is important for understanding the corresponding unique planning and evaluation needs of each.

History of Health Program Planning and Evaluation The history of planning health programs has a different lineage than that of program evaluation. Only relatively recently, in historical terms, have these lineages begun to overlap, with resulting synergies. Planning for health programs has the older history, if public health is consid- ered. Rosen (1993) argued that public health planning began approximately 4,000 years ago with planned cities in the Indus Valley that had covered sewers. Particularly since the Industrial Revolution, planning for the health of populations has progressed, and it is now considered a key characteristic of the discipline of public health.