Program evaluation alternative approaches and practical guidelines 4th edition

Program Evaluation

Alternative Approaches and Practical Guidelines

FOURTH EDITION

Jody L. Fitzpatrick University of Colorado Denver

James R. Sanders Western Michigan University

Blaine R. Worthen Utah State University

Boston Columbus Indianapolis New York San Francisco Upper Saddle River Amsterdam Cape Town Dubai London Madrid Milan Munich Paris Montreal Toronto

Delhi Mexico City São Paulo Sydney Hong Kong Seoul Singapore Taipei Tokyo

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1

-2 69

-5 69

06 -6

Vice President and Editor in Chief: Jeffery W. Johnston Senior Acquisitions Editor: Meredith D. Fossel Editorial Assistant: Nancy Holstein Vice President, Director of Marketing: Margaret Waples Senior Marketing Manager: Christopher D. Barry Senior Managing Editor: Pamela D. Bennett Senior Project Manager: Linda Hillis Bayma Senior Operations Supervisor: Matthew Ottenweller Senior Art Director: Diane Lorenzo Cover Designer: Jeff Vanik Cover Image: istock Full-Service Project Management: Ashley Schneider, S4Carlisle Publishing Services Composition: S4Carlisle Publishing Services Printer/Binder: Courier/Westford Cover Printer: Lehigh-Phoenix Color/Hagerstown Text Font: Meridien

Credits and acknowledgments borrowed from other sources and reproduced, with permission, in this textbook appear on appropriate page within text.

Every effort has been made to provide accurate and current Internet information in this book. However, the Internet and information posted on it are constantly changing, so it is inevitable that some of the Internet addresses listed in this textbook will change.

Copyright © 2011, 2004, 1997 Pearson Education, Inc., Upper Saddle River, New Jersey 07458. All rights reserved. Manufactured in the United States of America. This publication is protected by Copyright, and permission should be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. To obtain permission(s) to use material from this work, please submit a written request to Pearson Education, Inc., Permissions Department, 501 Boylston Street, Suite 900, Boston, MA 02116, fax: (617) 671-2290, email: permissionsus@pearson.com.

Library of Congress Cataloging-in-Publication Data

Fitzpatrick, Jody L. Program evaluation: alternative approaches and practical guidelines / Jody L. Fitzpatrick, James R.

Sanders, Blaine R. Worthen. p. cm.

ISBN 978-0-205-57935-8 1. Educational evaluation—United States. 2. Evaluation research (Social action programs)—

United States. 3. Evaluation—Study and teaching—United States. I. Sanders, James R. II. Worthen, Blaine R. III. Worthen, Blaine R. Program evaluation. IV. Title.

LB2822.75.W67 2011 379.1’54—dc22

2010025390 10 9 8 7 6 5 4 3 2

ISBN 10: 0-205-57935-3 ISBN 13: 978-0-205-57935-8

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1-269-56906-6

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1

-2 69

-5 69

06 -6

Everyone evaluates. As we discussed in Chapter 1, we all form opinions or make judgments about the quality of things we encounter. Such evaluations include everything from the meal we just finished eating or the movie or con- cert we saw last week to more serious endeavors—the program to help students at risk of dropping out at our high school or the parent contact program for par- ents new to our school. Our focus here is not on our individual judgments of something, but on evaluations that are more formal, structured, and public. We connect these personal evaluations with the more formal ones here, though, be- cause the earliest evaluation approaches were concerned, almost exclusively, with judging the quality of something. Those judgments were often derived by

First Approaches: Expertise and Consumer-Oriented Approaches

Orienting Questions

1. What are the arguments for and against using professional judgment as the means for evaluating programs?

2. What are the different types of expertise-oriented approaches? How are they alike and how do they differ?

3. Why is accreditation of institutions of higher education controversial today? How do these controversies reflect the controversies that frequently arise in many evaluations?

4. How is the consumer-oriented evaluation approach like the expertise-oriented approach? How is it different?

5. How do these approaches influence the practice of evaluation today?

126

5

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1-269-56906-6

Chapter 5 • First Approaches: Expertise and Consumer-Oriented Approaches 127

a group of individuals coming together to consider their criteria and the program or product to be judged.

The first modern-day approaches to evaluation were expertise-oriented and consumer-oriented evaluations. These approaches continue to be used today, though not so widely in the professional evaluation field. However, they have influenced the ways we think of evaluation and its purposes and methods. We will review each briefly, with a focus on the most widely used current method— accreditation—to illustrate the key principles of these approaches and how they affected, and continue to affect, evaluation practices.

The Expertise-Oriented Approach

The expertise-oriented approach to evaluation is probably the oldest type of formal, public evaluation and, as its name implies, it relies primarily on professional expertise to judge the quality of an institution, program, product, or activity. For example, the merits of a leadership training program for school principals could be assessed by experts from various fields including leadership, educational adminis- tration, and training who would observe the program in action, examine its mate- rials and underlying theory, perhaps interview some trainers and participants, or, in other ways, glean sufficient information to render a considered judgment about its value.

In another case, the quality of a hospital could be assessed by looking at its spe- cial programs, its operating facilities, its emergency room operations, its in-patient operations, its pharmacy, and so on, by experts in medicine, health services, and hospital administration. They could examine facilities and equipment/supplies of the hospital, its operational procedures on paper and in action, data on the fre- quency and outcomes of different procedures, the qualifications of its personnel, patient records, and other aspects of the hospital to determine whether it is meeting appropriate professional standards.

Although professional judgments are involved to some degree in all evalua- tion approaches, this one is decidedly different from others because of its direct, open reliance on professional expertise as the primary evaluation strategy. Such expertise may be provided by an evaluator or by subject-matter experts, depend- ing on who might offer most in the substance or procedures being evaluated. Usually one person will not own all of the requisite knowledge needed to adequately evaluate the program, institution, or agency. A team of experts who complement each other are much more likely to produce a sound evaluation.

Several specific evaluation processes are variants of this approach, including doctoral examinations administered by a committee, proposal review panels, site visits and conclusions drawn by professional accreditation associations, reviews of institutions or individuals by state licensing agencies, reviews of staff performance for decisions concerning promotion or tenure, peer reviews of articles submitted to professional journals, site visits of educational programs conducted at the behest of the program’s sponsor, reviews and recommendations by prestigious blue-ribbon

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1

-2 69

-5 69

06 -6

128 Part II • Alternative Approaches to Program Evaluation

TABLE 5.1 Some Features of Four Types of Expertise-Oriented Evaluation Approaches

Type of Expertise-Oriented Evaluation Approach

Existing Structure

Published Standards

Specified Schedule

Opinions of Multiple Experts

Status Affected by Results

Formal review system Yes Yes Yes Yes Usually

Informal review system Yes Rarely Sometimes Yes Usually

Ad hoc panel review No No No Yes Sometimes

Ad hoc individual review No No No No Sometimes

panels, and even the critique offered by the ubiquitous expert who serves in a watchdog role.

To impose some order on the variety of expertise-oriented evaluation activ- ities, we have organized and will discuss these manifestations in four categories: (1) formal professional review systems, (2) informal professional review systems, (3) ad hoc panel reviews, and (4) ad hoc individual reviews. Differences in these categories are shown in Table 5.1, along the following dimensions:

1. Is there an existing structure for conducting the review? 2. Are published or explicit standards used as part of the review? 3. Are reviews scheduled at specified intervals? 4. Does the review include opinions of multiple experts? 5. Do results of the review have an impact on the status of whatever is being

evaluated?

Developers of the Expertise-Oriented Evaluation Approach and Their Contributions

It is difficult to pinpoint the origins of this approach, since it has been with us for a very long time. It was formally used in education in the 1800s, when schools be- gan to standardize college entrance requirements. Informally, it has been in use since the first time an individual to whom expertise was publicly accorded ren- dered a judgment about the quality of some endeavor—and history is mute on when that occurred. Several movements and individuals have given impetus to the various types of expertise-oriented evaluations.

Elliot Eisner, an early evaluator discussed later in this chapter, stressed the role of connoisseurship and criticism in evaluation, roles that required exper- tise in the subject matter to be evaluated. James Madison and Alexander Hamilton took on the role of “expert evaluators” in discussing and elaborating on the meaning and merits of the newly proposed Constitution in The Federalist Papers. (They were experts because they were both present and active at the Constitutional Convention that drafted the document. As such, they were also

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1-269-56906-6

Chapter 5 • First Approaches: Expertise and Consumer-Oriented Approaches 129

internal evaluators!) Their writings were influential at the time and are still used by jurists in the U.S. courts to interpret the meanings of the Constitution, illustrating the important actions that can come from reasoned judgments by experts about a product. Accreditation of institutions of higher education is the primary present-day application of expertise-oriented evaluations. The New England Association of Schools and Colleges, which granted the first accredita- tion and continues accreditations for colleges and universities in New England today, began in 1885 when a group of headmasters of preparatory secondary schools began meeting with presidents of colleges in New England to discuss what graduates should know to be prepared for college. Thus, more than 100 years ago, school and college leaders were talking about ways to align their curricula!

Formal Professional Review Systems: Accreditation

Historical Foundations. To many, the most familiar formal professional review system is that of accreditation, the process whereby an organization grants approval of institutions such as schools, universities, and hospitals. Beginning in the late 1800s, regional accreditation agencies in the United States gradually supplanted the borrowed western European system of school inspections. These agencies became a potent force in accrediting institutions of higher education during the 1930s. Education was not alone in institutionalizing accreditation processes to determine and regulate the quality of its institutions. Parallel efforts were under way in other professions, including medicine and law, as concern over quality led to wide-scale acceptance of professionals judging the efforts of those educating fellow professionals. Perhaps the most memorable example is Flexner’s (1910) examination of medical schools in the United States and Canada in the early 1900s, which led to the closing of numerous schools he cited as inferior. As Floden (1983) has noted, Flexner’s study was not accreditation in the strict sense, because medical schools did not participate voluntarily, but it certainly qualified as accreditation in the broader sense: a classic example of pri- vate judgment evaluating educational institutions.

Flexner’s approach differed from most contemporary accreditation efforts in two other significant ways. First, Flexner was not a member of the profession whose efforts he presumed to judge. An educator with no pretense of medical ex- pertise, Flexner nonetheless ventured to judge the quality of medical training in two nations. He argued that common sense was perhaps the most relevant form of expertise:

Time and time again it has been shown that an unfettered lay mind, is . . . best suited to undertake a general survey. . . . The expert has his place, to be sure; but if I were asked to suggest the most promising way to study legal education, I should seek a layman, not a professor of law; or for the sound way to investigate teacher training, the last person I should think of employing would be a professor of education. (Flexner, 1960, p. 71)

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1

-2 69

-5 69

06 -6

130 Part II • Alternative Approaches to Program Evaluation

It should be noted that Flexner’s point was only partially supported by his own study. Although he was a layman in terms of medicine, he was an educator, and his judgments were directed at medical education rather than the practice of medicine, so even here appropriate expertise seemed to be applied.

Second, Flexner made no attempt to claim empirical support for the criteria or process he employed, because he insisted that the standards he used were the “obvious” indicators of school quality and needed no such support. His methods of collecting information and reaching judgments were simple and straightforward: “A stroll through the laboratories disclosed the presence or absence of apparatus, museum specimens, library, and students; and a whiff told the inside story regarding the manner in which anatomy was cultivated” (p. 79).

Third, Flexner dispensed with the professional niceties and courteous criti- cisms that often occur in even the negative findings of today’s accreditation processes. Excerpts of his report of one school included scathing indictments such as this: “Its so-called equipment is dirty and disorderly beyond description. Its outfit in anatomy consists of a small box of bones and the dried-up, filthy frag- ments of a single cadaver. A cold and rusty incubator, a single microscope, . . . and no access to the County Hospital. The school is a disgrace to the state whose laws permit its existence” (Flexner, 1910, p. 190).

Although an excellent example of expertise-oriented evaluation (if expertise as an educator, not a physician, is the touchstone), Flexner’s approach is much like that of contemporary evaluators who see judgment as the sine qua non of evalu- ation and who see many of the criteria as obvious extensions of logic and common sense (e.g., Scriven, 1973).

Accreditation in Higher Education Today. Accreditation in the United States and in many other countries today meets our criteria for an expertise-oriented, formal review system. The systems make use of an existing structure (generally an inde- pendent regional or national accreditation organization in the United States or governmental agencies in other countries), standards published by the organiza- tion responsible for accreditation, a specified schedule (for example, reviews of institutions every 2, 5, or 10 years), and opinions of multiple experts, and the status of the institution, department, college, or school is affected by the results. Accreditation is an excellent example of expertise-oriented evaluation because it uses people with expertise in the subject matter of the program or institution to form a judgment regarding the quality of the entity to be evaluated. The accredi- tation of an institution or program provides consumers and other stakeholders with some indication of the quality of the institution, as judged by experts in the field, and may facilitate summative decisions. For example, many students use an institution’s or program’s accreditation status to aid their decisions about whether to apply to or attend an institution or program. Further, the feedback the accred- itation process provides to the institution can be used for program and institutional improvement and decision making. Thus, the accreditation process serves a formative purpose as well.

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1-269-56906-6

Chapter 5 • First Approaches: Expertise and Consumer-Oriented Approaches 131

Accreditation in the United States is most common for institutions of higher education.1 We will spend a little time describing this process because it has recently become quite political and controversial, and even for those readers not involved in accreditation, the arguments illustrate the types of political issues and choices that often arise in any evaluation. These include disagreements over the purpose of the evaluation (formative or summative); the neutrality and inde- pendence of the experts or evaluators; the criteria to be used to judge the product and, thus, the data to be collected or reviewed; and the transparency of the process (what should be available to the public or other stakeholders outside the organi- zation). These controversies have emerged as the U.S. Department of Education, which has a stake in accreditation through provision of student loans to accred- ited institutions, has begun to take issue with the accreditation practices of the independent regional accrediting bodies that have traditionally reviewed colleges and universities for accreditation.

As noted earlier, in many countries, including Germany, the Netherlands, India, and the countries of the United Kingdom, institutions of higher education are required by law to be accredited. Government agencies, generally through a ministry or department of education, conduct the accreditation process. In some countries, such as Canada, there is no accreditation process for higher education, partly because most institutions of higher education are run by the provincial gov- ernments and that governance is considered to provide sufficient oversight. In the United States, accreditation evolved in a way that very much mirrors U.S. citizens’ distrust of government. With a desire to minimize government’s role nonprofit or voluntary associations carry out the accreditation tasks often fulfilled by government agencies in other countries.

As noted earlier, the New England Association of Schools and Colleges was the first accreditation organization in the United States. Originally established as a mechanism for dialogue between administrators of secondary schools and leaders of colleges in the region in 1885, it eventually evolved into the accrediting association for colleges and institutions in the region (Brittingham, 2009). Other regional associations followed, with each taking responsibility for accrediting institutions of higher education in their region. Today, there are six regional accrediting organiza- tions in the United States, each pursuing similar activities within their region.2 These

1Secondary institutions and school districts are occasionally accredited as well. Some states, for example, are moving to review school districts for accreditation and associations such as AdvancED have been formed out of the North Central and Southern accrediting associations for higher education to focus on accrediting K–12 schools. Further, many private schools are accredited. Our focus is on accreditation in higher education because it has been established for the longest period and its traditions, therefore, illus- trate much about expertise-oriented evaluation and its controversies. 2The major regional accrediting associations in the United States are the Middle States Association of Colleges and Schools, the New England Association of Schools and Colleges, the North Central Association of Colleges and Schools, the Northwest Association of Accredited Schools, the Southern Association of Colleges and Schools, and the Western Association of Schools and Colleges. Although other accredit- ing organizations exist (for example, for religious institutions), these regional accrediting associations are considered the primary accrediting bodies in the United States.

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1

-2 69

-5 69

06 -6

132 Part II • Alternative Approaches to Program Evaluation

associations focus primarily on accrediting institutions of higher education, though often they are also involved in accrediting K–12 schools. Finally, there are many ac- crediting associations that review programs in particular disciplines rather than en- tire institutions. For example, the American Bar Association accredits law schools, the Association of American Medical Colleges accredits medical schools, and the National Council for Accreditation of Teacher Education (NCATE) accredits teacher education programs, with the Teacher Education Accreditation Council (TEAC) emerging as a recent competitor to NCATE.

Accreditation of institutions of higher education by the six regional associa- tions has followed a similar plan and approach, the mission-based approach, since the 1950s. With the mission-based approach, accreditors focus on the extent to which the institution is pursuing and achieving its stated mission. Although each association also has standards for higher education that it uses in the evaluation, the mission-based approach reflects the philosophy of the associations in its eval- uations. Barbara Brittingham describes the mission-based approach and the accreditation process in the United States as “unusually focused on the future” to help the institution improve (2009, p. 18).

The Process of Accreditation. In the first stage of accreditation, the institution prepares a self-study report describing its mission and its progress toward that mis- sion, as well as how the institution meets the standards of the accrediting body. The second major stage is the core of the expertise-oriented approach: a team of peers, faculty, and administrators from other institutions in the region receives the report and conducts a site visit during which they interview faculty, administra- tors, staff, and students; review institutional records on admissions, course curric- ula, student satisfaction and outcomes; observe facilities and classrooms, and so forth. Based on their review of the report and their experiences during the site visit, the team, usually three or four experts, writes a report expressing their views regarding the institution, their recommendations concerning its accreditation sta- tus, and their suggestions for improvement. The site visit report is then reviewed by a standing commission at the accrediting association, which may amend the conclusions. The commission then presents the final conclusions to the institution.

The process is expertise-oriented in several ways: (a) the association has exper- tise concerning standards for higher education, the state and status of other institu- tions, and the practice of accreditation and review; (b) the faculty and administrators who form the site team have expertise in participating in the governance of their own universities and others where they have been employed and receive some training from the association to serve as site reviewers. Therefore, the expertise of the site visit team and the association allows those involved to make use of the standards of the association, their review of the report, and their site visit to form a final judgment of the quality of the institution. This process is a common one followed not only by the regional accrediting organizations but also by the organizations that accredit programs in individual disciplines in higher education and by organizations that accredit other educational institutions, including school districts, private schools, charter schools, secondary schools, vocational schools, and religious schools.

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1-269-56906-6

Chapter 5 • First Approaches: Expertise and Consumer-Oriented Approaches 133

Accreditation Controversies: Accreditation Politicized. So what can be contro- versial here? As one author defending the system notes, “Who better, one might ask, to evaluate the quality of a college or university than those who work in the field?” (O’Brien, 2009, p. 2). O’Brien argues that the evaluation and the relation- ship between the accrediting organizations and the institution should not be adversarial, noting, “The evaluators are not inspectors coming in with their white gloves” (O’Brien, 2009, p. 2). But the history of the controversy traces back to the GI Bill passed by Congress after World War II to provide financial assistance to returning soldiers to attend colleges and universities. The government wanted to ensure that the financial assistance went for worthwhile post secondary educa- tional activities, but did not want to get directly into the business of examining col- leges and universities for quality. So, it decided to rely on the independent regional accrediting associations, which were already reviewing colleges and universities, to determine the institutions students could receive financial aid to attend. Today, with increasing costs of higher education and more and more students attending colleges and universities, U.S. loans to students are big business. The government continues to rely on regional accrediting associations to identify the institutions of higher education that are eligible for aid, but has an increasing stake in the qual- ity of those processes given the large amounts of money distributed in student loans and other forms of aid. In addition, the institutions themselves have a large stake in the process, because many students would not attend an institution that is not accredited, for quality and financial aid reasons.

Through the Higher Education Act, originally passed in 1965, the U.S. govern- ment influences higher education in many areas, from student loans to access. In recent years, many in the U.S. Department of Education have become concerned that accreditations are not sufficiently rigorous in weeding out schools that are perform- ing poorly. Even proponents of the system note that current regional accreditation in the United States carries a “light touch” compared with government evaluations of higher education conducted in other countries (Brittingham, 2009, p. 18).

In 2005, the U.S. Department of Education appointed the Commission on the Future of Higher Education to study four issues critical to higher education, one of which was accountability. In “The Need for Accreditation Reform,” a paper prepared for that report, Robert Dickeson called the current U.S. system of ac- creditation, “a crazy-quilt of activities, processes, and structures that is frag- mented, arcane, more historical than logical, and has outlived its usefulness. More important, it is not meeting the expectations required for the future” (2006, p. 1). He concluded that “any serious analysis of accreditation as it is currently practiced results in the unmistakable conclusion that institutional purposes, rather than public purposes, predominate” (Dickeson, 2006, p. 3). He recommended that Con- gress create a National Accreditation Foundation to accredit institutions of higher education. The final report of the Commission, called the Spellings Commission for then Secretary of Education Margaret Spellings, was quite critical of current accreditation processes (U.S. Department of Education, 2006, http://www2.ed .gov/about/bdscomm/list/hiedfuture/reports/final-report.pdf). The report inspired much controversy and discussion in the higher education community, with

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1

-2 69

-5 69

06 -6

134 Part II • Alternative Approaches to Program Evaluation

organizations such as Phi Beta Kappa and the Association of American Colleges and Universities issuing statements both of support and concern regarding the re- port. The final 2008 amendment of the Higher Education Act ultimately chose to ignore some of these recommendations, but the concerns raised by the Commis- sion will continue (O’Brien, 2009) and, for our purposes, reflect some of the political concerns raised about evaluation today and, in particular, about expertise-oriented evaluation.

The regional accrediting associations see their purpose in evaluating institutions of higher education as primarily formative, helping these institutions improve. They see these goals as the best way to serve institutions, their students, and the public. By helping colleges and universities to improve and better achieve their stated mission, the accrediting associations believe they are helping students to receive a better edu- cation. In contrast, the U.S. Department of Education’s emphasis is summative. It is concerned with maintaining the U.S. position in higher education in the world and in providing educated and skilled graduates for the economy of the twenty-first cen- tury. The Department and other critics see the purpose of accreditation as providing parents, students, and other consumers with information to help them decide which institutions they should attend and where they should spend their tuition dollars. In other words, accreditation should help these consumers make summative decisions about which institutions to choose. Further, accreditation should help make sum- mative decisions about which institutions should continue. One critic notes that in the 60 years since the GI Bill was passed, “a mere handful of schools have been shut down and those largely for financial reasons . . . Meanwhile, on the accreditors’ watch, the quality of higher education is slipping” (Neal, 2008, p. 26). So, the accrediting associations have developed a process that is most useful for formative evaluation when critics see the primary purpose as summative.

Increasing Emphasis on Outcomes. Another area of disagreement concerns the factors that should be considered in accreditation. Today, the emphasis in educa- tion, and in much of evaluation around the world, is on outcomes and impacts. (See Chapter 2.) The Spellings Commission report notes the following:

Too many decisions about higher education—from those made by policymakers to those made by students and families—rely heavily on reputation and rankings derived to a large extent from inputs such as financial resources rather than out- comes. Better data about real performance and lifelong learning ability is absolutely essential if we are to meet national needs and improve institutional performance. (U.S. Department of Education, 2006, p. 14)

Just as K–12 education has moved to measuring student learning by focusing al- most entirely on the extent to which state standards are achieved, the Spellings Commission would like evaluations of institutions of higher education to rely much more heavily on measures of student outcomes.3 Although regional accrediting

3One difference between standards for K–12 education and those for higher education is that the standards for higher education would be national ones, not developed at the state level as K-12 standards are.

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1-269-56906-6

Chapter 5 • First Approaches: Expertise and Consumer-Oriented Approaches 135

associations have begun to require institutions to provide measures of student out- comes and, for accreditations of professional programs, evidence concerning pas- sage of licensing exams or job placements, the regional accreditation process also emphasizes the importance of input and process variables. Input variables include factors such as the quality of faculty, library holdings, IT capacity, classroom space and facilities, student admissions processes and decisions, and other elements that create the academic environment of the institution. Process variables articulated in standards, reviewed in self-reports, and examined by site visit teams include curricula, course requirements, and teaching quality; assistance to students through tutoring, advising, and other mechanisms; faculty-student interactions; intern- ships; and other elements of the learning process. Regional accrediting associations also consider multiple outcomes, including graduation and drop-out rates, time to graduation, knowledge and skills of graduates, and job placements. Accrediting associations argue that they must examine the entire process of higher education to make a valid judgment of the quality of the institution and to provide advice for improvement. Examining only student outcomes does not give the experts in the accreditation process sufficient information to make useful recommendations for how to change the institution, and its inputs and processes, to achieve better outcomes (Murray, 2009).

Neutrality, Transparency, and Purpose in Accreditation. Other criticisms of the current approach concern reviewers’ neutrality or objectivity and the trans- parency of the process. Evaluations are expected to be based on independent judg- ments. Such independence is intended to lead to more objective, and hence more valid, judgments of quality. Generally speaking, expertise-oriented evaluators should not be closely affiliated with the institution or product they are judging. For example, we are suspicious of an expert’s endorsement of a product when we know the expert has a financial relationship with the product’s manufacturer. Consider, for example, current discussions of the objectivity of medical research on the effectiveness of a drug when the research is funded by the pharmaceutical company that developed the drug. But accreditation processes make use of peer reviewers who are faculty and administrators from higher education institutions in the region. Accrediting organizations argue that these experts are in the best po- sition to make the judgments and provide the advice institutions need, because they know what can be accomplished in the environment of such an institution—and how to accomplish it. They have worked in it themselves. Critics, however, are concerned that the closeness of the experts to those being judged and possible competition between institutions or departments present serious conflicts of in- terest that can lead to biased judgments. Judgments as blunt as Flexner’s evalua- tions of medical schools would not see the light of day, at least in written reports.

Concerns over objectivity are heightened by the lack of transparency in the process. The U.S. Department of Education would like data and reports to be far more open, meaning that they would be available to parents, students, and the public and would contain content that is readily understood by nonexperts. For example, the Spellings Commission advocated tables presenting data on the

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1

-2 69

-5 69

06 -6

136 Part II • Alternative Approaches to Program Evaluation

knowledge and skills of graduates and other outcome measures for various colleges and universities. These tables would be available for the public to use in judging the quality of institutions, and for other colleges to use as benchmarks (U.S. Department of Education, 2006). Accreditors rely on the thick descriptions contained in self-study reports and the accreditation report. Defenders of the cur- rent system agree that the system relies heavily on confidentiality but argue that this confidentiality is one of the reasons for its success. Because of it, “institutions can be candid in their self-studies, and teams can be honest in their assessments” (O’Brien, 2009, p. 2). If reports were made public, those writing the self-report would be reluctant to discuss real problems, and accreditation teams would edit their wording for public consumption. Neither would facilitate learning about problems and making recommendations for change.

Thus, accreditation is changing and is controversial. Like many evaluations in recent years, the accreditation of colleges and universities in the United States has moved to an increasing use of mixed methods and a greater focus on outcomes. Controversies concern the purpose of these expertise-oriented evaluations, the stakeholders they serve, the measures that should take priority, the neutrality and objectivity of the judgments of quality, the transparency of the process, and the availability of results to different stakeholders. Regional accrediting associations, which for many years had no competition, are being seriously challenged, not only by the federal government, but also by popular ratings of colleges and univer- sities such as those published by U.S. News and World Report. As a result, accrediting associations are adapting and changing, but, with all their problems, they still remain a useful example of a formal review system using the expertise-oriented evaluation approach.

Other Formal Review Systems. There are numerous examples of other formal re- view systems, particularly in education. For many years, the National Council for Accreditation of Teacher Education (NCATE) has been the primary body to accredit teacher education programs. In 2000, this organization began focusing more on outcomes of such programs by examining knowledge and skills of grad- uates of the program, scores on licensure tests, and evidence that graduates are able to transfer their knowledge and skills to the classroom. The Teacher Educa- tion Accreditation Council (TEAC) has emerged as a competitor to NCATE, but with a similar focus on outcomes (Gitomar, 2007; Murray, 2009).

Some states are beginning to develop systems to review and accredit school districts within their state. For example, the Colorado Department of Education began accrediting districts in 1999 and revised the procedures substantially in 2008. The focus is very much on student outcomes and growth, but includes standards concerning “safe and civil learning environments,” and budget and financial management. Reviewers conclude the process by assigning a district a rating at one of six different levels, from accreditation with distinction to probation and nonac- creditation. Like other formal review systems, the Colorado accreditation process for school districts includes published standards, specified schedules for review (annual for districts with lower ratings, 2 to 3 years for districts at higher levels of

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1-269-56906-6

Chapter 5 • First Approaches: Expertise and Consumer-Oriented Approaches 137

accreditation), site visits by a team of external experts, and the districts’ status being affected by the results (http://www.cde.state.co.us/index_accredit.htm).

Informal Review Systems

Many professional review systems have a structure and a set of procedural guide- lines, and use multiple reviewers. Yet some lack the published standards or speci- fied review schedule of a formal review system.

A graduate student’s supervisory committee for dissertations, theses, or capstone projects is typically composed of experts in the student’s chosen field and is an example of an informal system within expertise-oriented evaluation. Struc- tures within the university, and/or faculty policies, exist for regulating such professional reviews of competence, but the committee members typically deter- mine the standards for judging each student’s performance. Fitzpatrick and Miller- Stevens (2009) have described the development and use of a rubric to assess students’ performance on capstone projects to complete a master’s program in pub- lic administration. But, typically, such criteria do not exist. Instead, the multiple experts on the committee make judgments of the student’s performance, often without discussing their criteria explicitly. And, of course, the status of students is affected by the results.

The systems established for peer reviews of manuscripts submitted to pro- fessional periodicals might also be considered examples of informal review systems, though journals’ procedures vary. Many journals do use multiple reviewers chosen for their expertise in the content of the manuscript. Unlike site visit teams for accreditation or members of a dissertation committee, reviewers do not behave as a team, discussing their reviews and attempting to reach consensus. Instead, a structure exists in the form of an editor or associate editor who selects reviewers, provides a timeframe for their reviews, and makes a final judgment about the manuscript based on the individual reviewers’ comments. However, the schedule, like that for a graduate student’s defense of a dissertation or thesis, is based on the receipt of manuscripts, although reviewers are given a specified time period in which to conduct the review. Many journals, but not all, provide reviewers with some general standards. Of course, the status of the manuscript—whether it is published, revised, or rejected—is affected by the review process.

Ad Hoc Panel Reviews

Unlike the ongoing formal and informal review systems discussed previously, many professional reviews by expert panels occur only at irregular intervals when circumstances demand. Generally, these reviews are related to no institutionalized structure for evaluation and use no predetermined standards. Such professional reviews are usually one-shot evaluations prompted by a particular, time-bound need for evaluative information. Of course, a particular agency may, over time, commission many ad hoc panel reviews to perform similar functions without their collectively being viewed as an institutionalized review system.

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1

-2 69

-5 69

06 -6

138 Part II • Alternative Approaches to Program Evaluation

Panels to Develop Standards. Common examples of ad hoc review panels include panels organized in each state in the United States to develop or revise educational standards for a state or school district, funding agencies to judge pro- posals and make recommendations for funding, and blue-ribbon panels appointed to address particular issues. These ad hoc panel reviews have no rou- tine schedule, but are organized by an agency or organization to receive input from experts on a particular issue. Thus, each of the 50 states has established standards that reflect that state’s expectations regarding what students will know in different subjects at different grades.4 There is considerable variation across the states in their standards, but the standards for each state were originally devel- oped by a panel of experts. These experts typically consist of teachers, educa- tional administrators, policymakers, and experts in the content area. The composition of the committee is intended to include experts with knowledge of the subject matter for which standards are being set and knowledge of the target population. Some sophisticated methods have been developed for the related task of expert committees identifying the cut scores, or scores that divide various test takers into groups based on their performance (Kane, 1995). (See Girard & Impara [2005] for a case study of the cut setting process by an expert panel in a public school district.)

Funding Agency Review Panels. In the United States, most federal government agencies make use of funding panels—panels of experts in the research area to be funded—to read proposals, discuss them, and make recommendations. Generally, the funding agency has developed criteria for the reviewers and, often, members of the team meet in Washington, DC, or other locations to discuss their reactions and attempt to reach some consensus. But the standards for funding vary from discipline to discipline and with the particular funding emphasis. Nevertheless, in the model of expertise-oriented evaluation, experts are coming together to make a judgment about something. Some funding organizations compose committees whose members have different areas of expertise. Thus, committees to review proposals in education can consist of a mix of educational administrators or poli- cymakers, teachers, and researchers. Likewise, committees that review proposals for community development or action can include research experts in the field as well as community members serving as experts on the particular community and its needs.

Blue-Ribbon Panels. Blue-ribbon panels are typically appointed by a high-level government official and are intended to provide advice, not on funding, but on how government should address a particular issue. The Commission on the Future of Higher Education, which was discussed earlier in this chapter, was appointed by the U.S. Department of Education in 2005, at a time when the

4These actions are somewhat in response to the federal legislation commonly known as No Child Left Behind, but many states had developed standards prior to the legislation.

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1-269-56906-6

Chapter 5 • First Approaches: Expertise and Consumer-Oriented Approaches 139

government was concerned with the long-term status of higher education in the United States and needed input from experts in the area. Members of such panels are appointed because of their experience and expertise in the field being studied. They typically are charged with reviewing a particular situation, documenting their observations, and making recommendations for action. Given the visibility of such panels, the acknowledged expertise of panel members is important if the panel’s findings are to be considered credible. At the local level, where ad hoc review panels are frequently used as an evaluative strategy for many endeavors ranging from economic development and environmental policies to school governance, expertise of panel members is no less an issue, even though the re- viewers may be of local or regional repute rather than national renown. Although recommendations of ad hoc panels of experts may have major impact, they might also be ignored, since there is often no formalized body charged with following up on their advice.

Ad Hoc Individual Reviews

Another form of expertise-oriented evaluation is the individual, professional re- view of any entity by any individual selected for his or her expertise to judge the value of the entity and, in some cases, to make recommendations for change or im- provement. Employment of a consultant to perform an individual review of some educational, social, or commercial program or activity is commonplace in many organizations.

Educational Connoisseurship and Criticism

In the previous section, we discussed applications of the expertise-oriented approach in which the experts are not necessarily evaluators. They are experts in something else—the content they are judging. Further, these applications are examples of the expertise-oriented approach, but they were formed and exist independent of the professional evaluation community. In other words, we can study these processes as examples of expertise-oriented evaluation approaches, but those in the evalu- ation community are generally not involved in establishing these activities or in conducting them, as is the case with the other approaches we will discuss. As noted, we have begun our discussion of approaches by focusing on the oldest eval- uation approach, one used for centuries before formal program evaluation emerged, to make judgments about important issues.

But, the expertise-oriented approach has also been part of the discussion of evaluation theories. In the early days of evaluation, Elliot Eisner was a key figure in discussing what evaluation should be, and his writings provide the theoretical foundation for the expertise-oriented approach and connect it to the evaluation literature (Eisner, 1976, 1985, 1991a, 1991b, 2004). Alkin and Christie (2004), in their evaluation tree depicting the origins and theories of evaluation, place Eisner, along with Michael Scriven, at the base of the valuing branch because their em- phasis was on the valuing role of evaluation—determining the value, the merit or

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1

-2 69

-5 69

06 -6

140 Part II • Alternative Approaches to Program Evaluation

worth, of the thing being evaluated. Eisner drew from the arts to describe his ap- proach to evaluation. His perspective was a useful counterpoint to the emphasis in the 1970s on social science methods and program objectives. We will briefly discuss his concepts of connoisseurship and criticism, the fundamentals of his eval- uation approach. These concepts fall within the expertise-oriented approach, because they require expertise in identifying and judging critical components or elements of the thing being evaluated.

The roles of the theater critic, art critic, and literary critic are well known and, in the eyes of many, useful roles. Critics are not without their faults. We may disagree with their views, but their reviews are good examples of direct and efficient application of expertise to that which is judged. Their criticism prompts us to think about the object being evaluated in different ways, even if we continue to disagree with their judgment. That is one goal of a written review or criticism: To prompt us to think about elements of the object that we, as nonexperts, might not have considered. Eisner (1991a) proposes that experts, like critics of the arts, bring their expertise to bear in evaluating the quality of programs in their areas of proficiency. Eisner does not propose a scientific paradigm but rather an artistic one, which he sees as an important qualitative, humanistic, nonscientific supplement to more traditional inquiry methods. He argues that we need to see the thing being evalu- ated from multiple perspectives and that the emphasis on quantitative, reduction- ist methods fails to convey many important qualities of the whole. He notes that numbers play a role in educational evaluation, his area of interest, but also limit what we see:

[W]e should be recognizing the constraints and affordances of any form of rep- resentation we elect to use. Just as a way of seeing is also a way of not seeing, a way of describing is also a way of not describing. The tools we employ for notic- ing have an enormous impact on what it is that we become aware of. If we want a replete, fulsome, generous, complex picture of a classroom, a teacher, or a student, we need approaches to that perception of such phenomena and, in addition, a form of presentation that will make those features vivid. (Eisner, 2004, p. 200)

The key elements of Eisner’s approach are connoisseurship and criticism (Eisner, 1975, 1991b). Connoisseurship is the art of appreciation—not necessar- ily a liking or preference for that which is observed, but rather an ability to notice, “to recognize differences that are subtle but significant in a particular qualitative display” (Eisner, 2004, p. 200). The connoisseur has developed knowledge of the important qualities of the object and the ability to observe and notice them well and to study the relationships among them. The connoisseur, in Eisner’s view, is aware of the complexities that exist in observing something in real-world settings and possesses refined perceptual capabilities that make the appreciation of such complexity possible. The connoisseur’s perceptual acuity results largely from a knowledge of what to look for (advance organizers or crit- ical guideposts) gained through extensive previous experience, education, and reflection on that experience.

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1-269-56906-6

Chapter 5 • First Approaches: Expertise and Consumer-Oriented Approaches 141

The analogy of wine tasting is used by Eisner (1975) to show how one must have many experiences to be able to distinguish what is significant about a wine, using a set of techniques to discern qualities such as body, color, bite, bouquet, flavor, and aftertaste, to judge its overall quality. The connoisseur’s refined palate and gustatory memory of other wines tasted is what enables him or her to distinguish subtle qualities lost on an ordinary drinker of wine and to render judgments rather than mere preferences. Connoisseurs exist in all realms of life, not solely the gusta- tory or artistic. Eisner describes a good coach as a connoisseur of the game who, when watching others at the sport, can recognize subtleties that those with less experience would miss: “We see it displayed in blazing glory in watching a first-rate basketball coach analyze the strengths of the opponents, their weaknesses, as well as the strengths and weaknesses of the team that he or she is coaching” (2004, p. 198).

Connoisseurship does not, however, require a public description or judgment of that which is perceived. The public description is the second part of the Eisner approach. “Criticism,” Eisner states, “is the art of disclosing the qualities of events or objects that connoisseurship perceives” (1979a, p. 197), as when the wine con- noisseur either returns the wine or leans back with satisfaction to declare it of acceptable, or better, quality. Or, more akin to public evaluation, criticism is when the wine critic writes a review of the wine. Evaluators are cast as critics whose con- noisseurship enables them to give a public rendering of the quality and significance of that which is evaluated. Criticism is not a negative appraisal but rather an educational process intended to enable individuals to recognize qualities and char- acteristics that might otherwise have been unnoticed and unappreciated. Criticism, to be complete, requires description, interpretation, and evaluation of that which is observed. “Critics are people who talk in special ways about what they encounter. In educational settings, criticism is the public side of connoisseurship” (Eisner, 1975, p. 13). Program evaluation, then, becomes program criticism. The evaluator is the instrument, and the data collecting, analyzing, and judging are largely hidden within the evaluator’s mind, analogous to the evaluative processes of art criticism or wine tasting. As a consequence, the expertise—training, experience, and credentials—of the evaluator is crucial, because the validity of the evaluation de- pends on the evaluator’s perception. Yet different judgments from different critics are tolerable, and even desirable, since the purpose of criticism is to expand per- ceptions, not to consolidate all judgments into a single definitive statement.

Eisner’s educational criticism focuses on four dimensions that should be portrayed in a criticism: description, development of themes, interpretation, and evaluation. The focus is on expert, and sometimes, detailed description of the factors that are important in judging the quality of the product or program. Obviously, the approach would not be the most direct for clearly establishing cause-and-effect relationships, but it can be useful in helping us to understand the nature of the in- tervention and the manner in which it leads to different outcomes. As Eisner recently stated, “Educational connoisseurship and educational criticism represent an effort to employ what the arts and humanities as partners with the social sciences have to of- fer in advancing our understanding of the process and effect of education. In an age of high-stakes testing, it is a perspective we badly need” (Eisner, 2004, p. 202).

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1

-2 69

-5 69

06 -6

142 Part II • Alternative Approaches to Program Evaluation

Influences of the Expertise-Oriented Approach: Uses, Strengths, and Limitations

Expertise-oriented approaches, generally referred to by other names, are used extensively in the United States and other countries today. Accreditation efforts are changing and expanding. Governments continue to appoint expert commissions to study issues and make recommendations. Often, such commissions help to protect government leaders from the ire of citizens when government needs to address a controversial issue. For example, closing military bases in the United States has been a controversial issue, in spite of the fact that too many bases exist. Congress and the president have resorted to appointing commissions of experts to provide “objective, non-partisan, and independent reviews” of recommendations for major base closures (http:www.brac.gov, homepage). The process has been used five times since the first commission was appointed in 1988, most recently in 2005. Like many blue-ribbon panels, the commissions have included experts in a variety of areas related to the issue. The commissions conduct site visits, seek input from the public and other experts, review information, and make recommendations to the President. The recommendations take effect unless Congress rejects the proposal within 45 days. These commissions have been able to take important actions to improve the efficiency and effectiveness of the placement of military bases.

Collectively, expertise-oriented approaches to evaluation have emphasized the central role of expert judgment, experience, and human wisdom in the evaluative process and have focused attention on such important issues as whose standards (and what degree of transparency) should be used in rendering judgments about programs. Conversely, critics of this approach suggest that it may permit evaluators to make judgments that reflect little more than personal biases. Others have noted that the presumed expertise of the experts is a potential weakness. Those using or contracting for expertise-oriented evaluations should consider carefully the various areas of expertise required for their team of expert judges. Too often the team contains only content experts, people who know various elements of the subject matter to be judged, but may lack experts in the evaluation process itself. The artic- ulation of standards, whether by the contracting organization or by the team of experts, is also important to clarify the criteria and methods used to make the judg- ments requested. Of course, as Elliot Eisner would argue, experts should look beyond the standards and use their connoisseurship to describe, interpret, and judge the dimensions they know to be important to the quality of the product. But, articulated standards help to introduce some consistency across experts and to facil- itate useful discussions among the experts when disagreements do occur.

Eisner’s writings influenced evaluators to think more about the nature of evaluation judgments and the role that experience and connoisseurship can play in helping them to notice important elements of the program or product to be evaluated. However, Eisner did not remain active in the evaluation field, and the approach was used infrequently, generally by his immediate students. Still, we continue to study his writings because of the influences he has had on evaluation practice today. Donmoyer (2005) notes that Eisner’s contributions prompted

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1-269-56906-6

Chapter 5 • First Approaches: Expertise and Consumer-Oriented Approaches 143

evaluators to consider different approaches to evaluation and the implications of each. Eisner also provided an important rationale for qualitative methods at a time when quantitative methods dominated the field. His work was useful in prompt- ing us to consider what we notice in an object. Connoisseurs know the important elements of a particular thing and learn how to form educated opinions about those elements. The connoisseurship-criticism approach also has its critics. Following Eisner’s initial proposals, House (1980) issued strong reservations, cautioning that the analogy of art criticism is not applicable to at least one aspect of evaluation:

It is not unusual for an art critic to advance controversial views—the reader can choose to ignore them. In fact, the reader can choose to read only critics with whom he agrees. A public evaluation of a program cannot be so easily dismissed, however. Some justification—whether of the critic, the critic’s principles, or the criticism—is necessary. The demands for fairness and justice are more rigorous in the evaluation of public programs. (p. 237)

However, more recently, Stake and Schwandt emphasize the importance to evaluation not only of measuring quality but also of conveying quality as it is experienced. Reminiscent of Eisner’s recognition of connoisseurship, they observe that “we do not have good enough standards for recognizing an evaluator’s prac- tical knowledge that arises from a combination of observational skill, breadth of view, and control of bias” (2006, p. 409). They conclude that “as with connoisseurs and the best blue ribbon panels, some of the best examples of synthesizing values across diverse criteria are those that rely on the personal, practical judgment of fair and informed individuals” (2006, p. 409).

The Consumer-Oriented Evaluation Approach

Like the expertise-oriented approach, consumer-oriented evaluation has existed in the practice of individuals making decisions about what to purchase, or trade, for centuries. The approaches are similar in other ways: Their primary purpose is to judge the quality of something, to establish the value, the merit or worth, of a product, program, or policy. Although all evaluations are concerned with deter- mining merit or worth, valuing is the key component of these two approaches.5

Their principal audience is the public. Unlike approaches that will be discussed in other chapters in this section, evaluations relying on these approaches often do not have another audience—a foundation, manager, policymaker, or citizens’ group—who has hired the evaluator to provide them with useful information to

5Other evaluation approaches focus on various types of use, such as stakeholder involvement or organi- zational change, and methodology, such as establishing causality or providing thick descriptions as the central component. These evaluations, too, may ultimately make a judgment of merit or worth, but that judgment, the valuing of the program or product, is not so central to the evaluation approach as it is in expertise-oriented or consumer-oriented evaluation. (See Alkin [2004], Shadish et al. [1991].)

Program Evaluation: Alternative Approaches and Practical Guidelines, Fourth Edition, by Jody L. Fitzpatrick, James R. Sanders, and Blaine R. Worthen. Published by Pearson. Copyright © 2011 by Pearson Education, Inc.

IS B

N 1

-2 69

-5 69

06 -6

make a decision or judgment. Instead, the audience for consumer-oriented and expertise-oriented approaches is a broader one—the purchasing or interested public—and is not directly known to the evaluator. Therefore, the evaluator is the major, often the only, decision maker in the study because he or she does not have other important, direct audiences to serve. But the consumer-oriented approach and the expertise-oriented approach differ dramatically in their methodologies, with the latter relying on the judgments of experts and the arts as a model. On the other hand, consumer-oriented evaluation relies on more transparent and quan- titative methods, with the judgment typically being made by an evaluator, a person with expertise in judging things, but not with the particular content expertise of expertise-oriented or connoisseur evaluations.

Popular examples of consumer-oriented evaluations that the reader will know include Consumer Reports and the U.S. News and World Report ratings of colleges and universities, but examples exist around the world. Which? is a magazine and web site in the United Kingdom that serves a mission similar to that of the Consumers’ Union, the sponsor of Consumer Reports and its web site, in the United States. Both organizations act as consumer advocates and test products to provide information to consumers on the effectiveness of various products.

The Developer of the Consumer-Oriented Evaluation Approach

Consumer-oriented evaluations first became important in educational evalua- tions in the mid to late 1960s as new educational products flooded the market with the influx of funds from the federal government for product development. Michael Scriven is the evaluator best known for prompting professional evalua- tors to think more carefully about consumer-oriented or product evaluations (1974b, 1991c). Scriven, of course, is known for many things in evaluation, and consumer-oriented or product-oriented evaluations represent only one of his contributions. His most important contributions include making evaluators aware of the meaning and importance of valuing in evaluation (Shadish et al. 1991; Alkin, 2004). He often uses examples of product evaluation in his writing to illustrate the nature of valuing and the process of deriving a value in evalua- tion. For many years, he considered Consumer Reports to be “an almost flawless paradigm” in product evaluation. However, he has expressed disappointment with their reluctance to discuss and improve their methodology and has recog- nized PC Magazine and Software Digest as developing more methodologically sound procedures (Scriven, 1991a, p. 281).

Scriven’s approach to determining the value of a product, however, is quite different from Eisner’s connoisseur approach. In fact, Scriven’s critical view of Eisner’s approach illustrates his own priorities. He states that evaluations using the connoisseurship model “may generate a valuable perspective, but it abandons much of the requirement of validity. In particular it is vulnerable to the fallacy of irrele- vant expertise, because connoisseurs are at best a bad guide to merit for the novice— and are also affected by the swing of fashion’s pendulum” (Scriven, 1991a, p. 92).