Humanist 777 bt bold condensed b

Discussion Post - Ethical Dilemma

In this discussion, you will debate an ethical dilemma around test development while relating it to validity and reliability concepts.

Consider the following scenario: A private school utilizes a test with well-established predictive validity for admissions. However, members of a particular group tend to score low on this test. In your initial post, answer the following question:

Is it ethical for the school to continue to use the test? Why or why not?

Do Adjusted Subscores Lack Validity? Don’t Blame the Messenger

Sandip Sinharay1, Shelby J. Haberman1, and Howard Wainer2

Abstract

There are several techniques that increase the precision of subscores by borrowing information from other parts of the test. These techniques have been criticized on validity grounds in several of the recent publications. In this note, the authors ques- tion the argument used in these publications and suggest both inherent limits to the validity argument and empirical issues worth examining.

Keywords

subscores, validity, augmented subscore

Introduction: Subscores and Adjusted Subscores

There are several techniques that increase the precision of subscores by borrowing

information from other parts of the test. These techniques have been criticized on val-

idity grounds in several recent publications such as Skorupski and Carvajal (2010) and

Stone, Ye, Zhu, and Lane (2010). In this note, we question the argument used in these

publications and suggest both inherent limits to the validity argument and empirical

issues worth examining. We begin with an introduction to the techniques that borrow

information from other parts of the test as part of the subscore computation process

and then evaluate the validity arguments advanced recently concerning these

techniques.

Interest in subscores in educational testing reflects their potential remedial and

instructional benefit. According to the National Research Council report ‘‘Knowing

1Educational Testing Service, Princeton, NJ, USA 2National Board of Medical Examiners, Philadelphia, PA, USA

Corresponding Author:

Sandip Sinharay, Educational Testing Service, 12T Rosedale Road, Princeton, NJ 08541, USA

Email: ssinharay@ets.org

Educational and Psychological Measurement

71(5) 789–797 ª The Author(s) 2011

Reprints and permission: sagepub.com/journalsPermissions.nav

DOI: 10.1177/0013164410391782 http://epm.sagepub.com

http://crossmark.crossref.org/dialog/?doi=10.1177%2F0013164410391782&domain=pdf&date_stamp=2011-03-22
What Students Know’’ (2001), the target of assessment is to provide particular infor-

mation about an examinee’s knowledge, skill, and abilities. Subscores have the poten-

tial to provide such information; however, they are too often not reliable enough for

their intended purposes. Several researchers have suggested methods that increase the

precision of subscores by borrowing information from the other related scores or sub-

scores. For example,

• Wainer, Sheehan, and Wang (2000) and Wainer, Vevea, et al. (2001) suggest the augmented subscore that is a function of an examinee’s score on the sub-

scale of interest and that examinee’s score on the remaining subscales.

• Yen (1987) suggested the objective performance index (OPI) that is a weighted average of the observed subscore and an estimate of the observed

subscore obtained using a unidimensional item response theory (IRT) model

for the entire test.

• Haberman (2008a) suggested a weighted average of a subscore and the total score. Sinharay (2010) found that this weighted average is typically very sim-

ilar to the augmented subscore (Wainer et al., 2000).

• Several researchers (de la Torre & Patz, 2005; Haberman & Sinharay, 2010; Luecht, 2003; Yao & Boughton, 2007) suggested using estimated abilities or

their transformations obtained from a multivariate IRT (MIRT) model as sub-

scores. For background on MIRT models, see, for example, Reckase (1997).

The scores obtained from the above-mentioned approaches will be referred to as

‘‘adjusted subscores.’’1 Researchers have found that adjusted subscores are more reli-

able, often substantially so, than the subscores themselves (Dwyer, Boughton, Yao,

Steffen, & Lewis, 2006; Sinharay, 2010; Skorupski & Carvajal, 2010; Stone, Ye,

Zhu, & Lane, 2010).

Recent Criticisms of Adjusted Subscores

The validity of adjusted subscores has been questioned recently. Skorupski and

Carvajal (2010) studied four subscores from a large statewide test and found that

the corresponding OPIs and the augmented subscores (Wainer et al., 2000) were

highly correlated among themselves. The correlations between augmented subscores

were 0.97 or greater and those between the OPIs were all 1.00. Skorupski and Carvajal

(2010) commented that this phenomenon of high correlations among the adjusted sub-

scores (which means that the rank orderings for the four adjusted subscores are very

similar) leads to potential loss of meaning of the subscores and ‘‘reduces, if not elim-

inates, the utility of the subscores for the diagnostic purposes for which they are

intended. This begs the question: Are the augmented subscores providing more useful

information than the raw ones?’’ (p. 372). They went on to comment that ‘‘although

augmentation dramatically improves the reliability of subscores, it may in fact nega-

tively affect the validity of score interpretations’’ (p. 372). In the abstract of their arti-

cle, they commented that the near-perfect correlations among the adjusted subscores

790 Educational and Psychological Measurement 71(5)

‘‘called into question the validity of the resultant subscores, and therefore the useful-

ness of the subscore augmentation process.’’

Stone et al. (2010) studied the four subscores for the spring 2006 assessment of the

Delaware Student Testing Program 8th grade mathematics assessment. They found

the augmented subscores, the OPIs, and the MIRT-based subscores to be highly cor-

related among themselves and commented that ‘‘it may be that adjusted subscale

scores represent the measurement of a construct that is different from the construct

being measured by the unadjusted subscale scores’’ (p. 80). They commented that bor-

rowing information from other subscales causes a ‘‘potential threat to validity’’ of the

adjusted subscores (p. 80).

It seems that Skorupski and Carvajal (2010) and Stone et al. (2010) have criticized

the use of adjusted subscores in general (rather than criticizing their use with their data

sets), and their criticisms might make some practitioners wonder whether it makes

sense to use adjusted subscores at all.

Should One Report Diagnostic Scores for the Tests Considered in Skorupski and Carvajal (2010) and Stone et al. (2010)?

Let us look closely at the tests considered by Skorupski and Carvajal (2010) and Stone

et al. (2010) and ask the question, ‘‘Should one report subscores, or, more generally,

any kind of diagnostic scores for these tests?’’

According to Standard 5.12 of the Standards for Educational and Psychological

Testing (American Educational Research Association, American Psychological Asso-

ciation, & National Council for Measurement in Education,1999), scores should not

be reported for individuals unless the validity, comparability, and reliability of such

scores have been established. This standard applies to subscores as well as to the over-

all or total score. Furthermore, Standard 1.12 of the Standards for Educational and

Psychological Testing (1999) demands that, if a test provides more than one score,

then the distinctiveness of the separate scores should be demonstrated.

Haberman (2008a) suggested an approach to determine if subscores and augmented

subscores have added value over the total score. This approach has been applied in

Lyren (2009); Puhan, Sinharay, Haberman, and Larkin (2010); and Sinharay

(2010). In this approach, a subscore has added value if it is reliable and is distinct

from the other subscores.

Sinharay (2010) applied the approach of Haberman to the data set considered in

Stone et al. (2010) and concluded that none of the original subscores were of added

value and that none of the weighted averages (or augmented subscores) were of added

value. In addition, Stone et al. reported an exploratory factor analysis that suggested

the presence of only one factor in the data set and found the disattenuated correlations

between the subscores to be between 0.96 and 1.03.

Sinharay et al. 791

The disattenuated correlations between the subscores of Skorupski and Carvajal

(2010) were between 0.89 and 0.96, with an average of 0.94. None of the subscores,

weighted averages, and augmented subscores had added value for this data set either.2

These results are enough to conclude that subscores or, more generally, any kind of

diagnostic scores (including adjusted subscores) for the tests considered in Skorupski

and Carvajal (2010) and Stone et al. (2010) will not satisfy professional quality stand-

ards (especially the above-mentioned Standard 1.12 on distinctiveness). Hence, it is

true that the adjusted subscores for these tests lack validity (because of the fact that

Haberman, 2008b, showed that the validity of subscores is limited when the subscores

are either not reliable or are highly correlated with total scores).

However, no reasonable person should blame the adjusted subscores for not being

valid for the tests considered in Skorupski and Carvajal (2010) and Stone et al. (2010).

If the bathroom scale tells us that we need to lose weight, it would be unfair to blame

the scale. The tests considered in Skorupski and Carvajal and Stone et al. were unidi-

mensional and were incapable of producing diagnostic scores of any kind. So it is no

wonder that the adjusted subscores computed from these data are not valid. However,

responsibility for the lack of validity lies not with the adjusted subscores but rather

with the tests and those who try to report any diagnostic subscores from the tests in

the first place. The adjusted subscores are just the messengers of the bad news that

the data are not appropriate for diagnostic score reporting.

A General Defense of Adjusted Subscores

The examples of Skorupski and Carvajal (2010) and Stone et al. (2010) do not repre-

sent a complete picture of the empirical situation, as is evident from a recent review of

subscores for operational tests (Sinharay, 2010). For example, consider the Swedish

Scholastic Assessment Test considered in Lyren (2009), which included subscores

and adjusted subscores that had added value. For this test, the correlation between

the augmented subscores ranged between 0.58 and 0.94, with an average of 0.79.3

These correlations are much higher than the correlations between the unadjusted sub-

scores that ranged between 0.42 and 0.67, with an average of 0.55. However, the cor-

relation between the augmented subscores are much lower than those in Skorupski and

Carvajal (2010) and Stone et al. (2010) and demonstrate that the correlations between

augmented subscores are not always extremely high.

When several subscores of an assessment are adjusted by use of the total score (or

other parts of the test), the adjusted subscores share a common component, the total

score (or score on the other parts), so that the adjusted subscores will always be

more highly correlated than are the original observed subscores.

In general, increased correlations among adjusted subscores do not threaten valid-

ity. If the correlations are very high, then the adjusted subscores are essentially just

versions of the total score, and the test is not able to produce useful diagnostic scores.

If the correlations are not very high, measurement error has been reduced with the

computation of the adjusted subscore (because the variance of adjusted subscore is

less than that of the subscore and hence the reliability is higher). If the measurement

792 Educational and Psychological Measurement 71(5)

error is sufficiently reduced, then the correlation with external criterion scores is likely

to increase rather than decrease when adjusted rather than observed subscores are

employed, although empirical study is needed to verify this observation with real

data (Haberman, 2008b).

To examine the validity issue in a simple setting, it is helpful to consider parallel

forms. The subscore on a parallel form is a basic validity criterion for the correspond-

ing observed and adjusted subscores on the original form. For example, let us consider

the test TC2 considered in Table 1 of Sinharay (2010). The test, which measured

achievement in a discipline, had 200 multiple choice items and three subscores,

each having 66 or 67 items. We split the test into two tests, say Test A and Test B,

of length 100 items each. Tests A and B were made roughly parallel in difficulty

and content. We then computed the subscores and augmented subscores for Tests A

and B. All three of the augmented subscores have added value for both Tests A and

B according to the criteria of Haberman (2008a). Table 1 shows some correlations.

The table shows that any subscore on Test B (or A) has a higher correlation with

the corresponding augmented subscore on parallel Test A (or B) than with the corre-

sponding subscore on parallel Test A (or B).4 For example, the correlation between

subscore 1 on Test B and augmented subscore 1 on Test A is 0.88, which is larger

than 0.85, the correlation between subscore 1 on Test B and subscore 1 on Test A.

Figure 1, which is like Figure 4 of Skorupski and Carvajal (2010), shows the subscore

profiles (top panel) and the profiles of augmented subscores (bottom panel) of five

randomly chosen examinees. Although the three observed subscores for each exam-

inee vary more than the three augmented subscore for the examinee, the profiles of

augmented subscores are not all parallel, unlike in Figure 4 of Skorupski and Carvajal.

Some of the profiles of augmented subscores even intersect with each other.

Thus, the two facts—(a) the adjusted subscores (augmented subscores in this case)

estimate the subscores on a parallel form better and (b) the profiles of the adjusted sub-

scores are not all parallel—show that adjusted subscores did not ‘‘lose their meaning’’

or ‘‘have their utility reduced or eliminated’’ (as commented in Skorupski & Carvajal,

2010, p. 372) and did not represent a construct different from that measured by the

subscores (as mentioned in Stone et al., 2010).

Therefore, for a test that was designed to report diagnostic scores (e.g., the Swedish

Scholastic Assessment Test or the test TC2 considered above), it is straightforward to

gather evidence that supports the proposed interpretation of the adjusted subscores and

Table 1. Correlations Among Subscores and Augmented Subscores

Subscore

Correlation Between a Subscore on Test A

and the Corresponding Subscore on Test B

Correlation Between an Augmented Subscore on Test

A and the Corresponding Subscore on Test B

Correlation Between a Subscore on Test A and the Corresponding Augmented

Subscore on Test B

1 0.85 0.88 0.87 2 0.79 0.84 0.82 3 0.83 0.85 0.84

Sinharay et al. 793

it will not be difficult to stand up to any criticism of the adjusted subscores as long as

the accumulated evidence is evaluated in an evenhanded way (Kane, 2006, mentioned

the need to stand up to criticism in establishing validity).

To make the validity claim foolproof, it is important also to collect empirical evi-

dence concerning validity of subscores and adjusted subscores. Haberman (2008b)

suggested some theoretical results on the validity of subscores, but those results do

not obviate the need for data on validity. Although modern concepts of validity of tests

consider many aspects of test content, intended use, and consequences of use (Kane,

2006; Messick, 1989), a mature testing program requires empirical evidence that

a reported test score is adequately related to appropriate criterion scores.

If the adjusted subscores have lower correlations with appropriate criterion varia-

bles than the total scores or the original subscores, then there is justification to criticize

them for lack of validity. However, until that can be demonstrated, we think that it is

premature to criticize their validity based on any current findings. It does not seem that

any of the validity standards of the Standards for Educational and Psychological Test-

ing (American Educational Research Association, American Psychological

Figure 1. Subscore profiles of five randomly chosen examinees

794 Educational and Psychological Measurement 71(5)

Association, & National Council for Measurement in Education, 1999) have been vio-

lated by the use of the adjusted subscores.

We agree with Stone et al. (2010) that score users may not like or understand the

dependence of, say, a reading subscore on a speaking subscore. However, many score

users do not understand measurement concepts such as Cronbach’s alpha or equiper-

centile equating and that has not deterred the testing companies from reporting reli-

ability values or equated scores. In addition, we believe that it would not be

difficult to make an argument that, for example, a common language skill is required

to answer both reading and listening items, which will justify the adjustment of listen-

ing subscores using reading subscores in addition to the listening subscores. Think of

a test for which (a) the subscores are reliable and distinct, (b) the adjusted subscores

have higher reliability than the subscores, (c) there is a strong evidence of criterion

validity of adjusted subscores, and (d) the subscales are somewhat connected concep-

tually (e.g., language skills such as reading and listening). Here, it makes sense to

adjust subscores. In our opinion, for such a test, it is possible for the testing company

to make a claim about the validity of the adjusted subscores that is strong enough to

overcome the above-mentioned potential problem of explanation of adjusted sub-

scores to users and to convince the users that the adjusted subscores are reliable, valid,

and useful.

Authors’ Note

Any opinions expressed in this article are those of the authors and are not necessarily those of

Educational Testing Service or National Board of Medical Examiners.

Acknowledgements

The authors are grateful to Dan Eignor, Wendy Yen, Gautam Puhan, and George Mercoulides

for their helpful comments and to William Skorupski for generously sharing with us some sum-

mary of a data set.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interests with respect to the authorship and/or

publication of this article.

Funding

The research of the first two authors was funded by Educational Testing Service (ETS), which is

the company that these two authors work for.

Notes

1. A better name would have been ‘‘augmented subscores,’’ but that corresponds to the scores

described in Wainer et al. (2000).

Sinharay et al. 795

2. We thank William Skorupski for generously sharing with us some summary of their data that

allowed us to perform these computations.

3. The correlations among OPIs or MIRT-based subscores would be of similar magnitudes.

4. The same result would have been obtained for OPIs and MIRT-based subscores.

References

American Educational Research Association, American Psychological Association, & National

Council on Measurement in Education. (1999). Standards for educational and psychological

testing. Washington, DC: American Educational Research Association.

de la Torre, J., & Patz, R. J. (2005). Making the most of what we have: A practical application of

multidimensional IRT in test scoring. Journal of Educational and Behavioral Statistics, 30,

295-311.

Dwyer, A., Boughton, K. A., Yao, L., Steffen, M., & Lewis, D. (2006). A comparison of sub-

scale score augmentation methods using empirical data. Paper presented at the annual meet-

ing of the National Council on Measurement in Education, San Francisco, CA.

Haberman, S. J. (2008a). When can subscores have value? Journal of Educational and Behav-

ioral Statistics, 33, 204-229.

Haberman, S. J. (2008b). Subscores and validity (ETS Research Report No. RR-08-64). Prince-

ton, NJ: Educational Testing Services.

Haberman, S. J., & Sinharay, S. (2010). Reporting of subscores using multidimensional item

response theory. Psychometrika, 75, 209-227.

Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed.,

pp. 18-64). Westport, CT: Praeger.

Luecht, R. M. (2003, April). Applications of multidimensional diagnostic scoring for certifica-

tion and licensure tests. Paper presented at the meeting of the National Council on Measure-

ment in Education, Chicago, IL.

Lyren, P. (2009). Reporting subscores from college admission tests. Practical Assessment,

Research, and Evaluation, 14, 1-10.

Messick, S. (1989). Validity. In R. L. Linn (Ed.) Educational measurement (3rd ed., pp. 13-

103). Washington, DC: National Council on Measurement in Education and American

Council on Education.

National Research Council. (2001). Knowing what students know: The science and design of

educational assessment. Washington, DC: National Academies Press.

Puhan, G., Sinharay, S., Haberman, S. J., & Larkin, K. (2010). Comparison of subscores based

on classical test theory. Applied Measurement in Education, 23, 1-20.

Reckase, M. D. (1997). The past and future of multidimensional item response theory. Applied

Psychological Measurement, 21, 25-36.

Sinharay, S. (2010). How often do subscores have added value? Results from operational and

simulated data. Journal of Educational Measurement, 47, 150-174.

Skorupski, W. P., & Carvajal, J. (2010). A comparison of approaches for improving the reliabil-

ity of objective level scores. Educational and Psychological Measurement, 70, 357-375.

Stone, C. A., Ye, F., Zhu, X., & Lane, S. (2010). Providing subscale scores for diagnostic infor-

mation: A case study when the test is essentially unidimensional. Applied Measurement in

Education, 23, 63-86.

Wainer, H., Sheehan, K., & Wang, X. (2000). Some paths toward making praxis scores more

useful. Journal of Educational Measurement, 37, 113-140.

796 Educational and Psychological Measurement 71(5)

Wainer, H., Vevea, J. L., Camacho, F., Reeve, B. B., Rosa, K., Nelson, L., Swygert, K. A., . . .

Thissen, D. (2001). Augmented scores—‘‘Borrowing strength’’ to compute scores based on

small numbers of items. In D. Thissen & H. Wainer (Eds.), Test scoring (pp. 343-387). Mah-

wah, NJ: Lawrence Erlbaum.

Yao, L., & Boughton, K. A. (2007). A multidimensional item response modeling approach for

improving subscale proficiency estimation and classification. Applied Psychological Mea-

surement, 31, 83-105.

Yen, W. M. (1987, June). A Bayesian/IRT index of objective performance. Paper presented at

the annual meeting of the Psychometric Society, Montreal, Quebec, Canada.

Sinharay et al. 797

<< /ASCII85EncodePages false /AllowTransparency false /AutoPositionEPSFiles true /AutoRotatePages /None /Binding /Left /CalGrayProfile (Dot Gain 20%) /CalRGBProfile (sRGB IEC61966-2.1) /CalCMYKProfile (U.S. Web Coated \050SWOP\051 v2) /sRGBProfile (sRGB IEC61966-2.1) /CannotEmbedFontPolicy /Error /CompatibilityLevel 1.3 /CompressObjects /Off /CompressPages true /ConvertImagesToIndexed true /PassThroughJPEGImages true /CreateJobTicket false /DefaultRenderingIntent /Default /DetectBlends true /DetectCurves 0.1000 /ColorConversionStrategy /LeaveColorUnchanged /DoThumbnails false /EmbedAllFonts true /EmbedOpenType false /ParseICCProfilesInComments true /EmbedJobOptions true /DSCReportingLevel 0 /EmitDSCWarnings false /EndPage -1 /ImageMemory 1048576 /LockDistillerParams false /MaxSubsetPct 100 /Optimize false /OPM 1 /ParseDSCComments true /ParseDSCCommentsForDocInfo true /PreserveCopyPage true /PreserveDICMYKValues true /PreserveEPSInfo true /PreserveFlatness true /PreserveHalftoneInfo false /PreserveOPIComments false /PreserveOverprintSettings true /StartPage 1 /SubsetFonts true /TransferFunctionInfo /Apply /UCRandBGInfo /Remove /UsePrologue false /ColorSettingsFile () /AlwaysEmbed [ true /ACaslon-Bold /ACaslon-BoldItalic /ACaslon-Italic /ACaslon-Ornaments /ACaslon-Regular /ACaslon-Semibold /ACaslon-SemiboldItalic /AdobeCorpID-Acrobat /AdobeCorpID-Adobe /AdobeCorpID-Bullet /AdobeCorpID-MinionBd /AdobeCorpID-MinionBdIt /AdobeCorpID-MinionRg /AdobeCorpID-MinionRgIt /AdobeCorpID-MinionSb /AdobeCorpID-MinionSbIt /AdobeCorpID-MyriadBd /AdobeCorpID-MyriadBdIt /AdobeCorpID-MyriadBdScn /AdobeCorpID-MyriadBdScnIt /AdobeCorpID-MyriadBl /AdobeCorpID-MyriadBlIt /AdobeCorpID-MyriadLt /AdobeCorpID-MyriadLtIt /AdobeCorpID-MyriadPkg /AdobeCorpID-MyriadRg /AdobeCorpID-MyriadRgIt /AdobeCorpID-MyriadRgScn /AdobeCorpID-MyriadRgScnIt /AdobeCorpID-MyriadSb /AdobeCorpID-MyriadSbIt /AdobeCorpID-MyriadSbScn /AdobeCorpID-MyriadSbScnIt /AdobeCorpID-PScript /AGaramond-BoldScaps /AGaramond-Italic /AGaramond-Regular /AGaramond-RomanScaps /AGaramond-Semibold /AGaramond-SemiboldItalic /AGar-Special /AkzidenzGroteskBE-Bold /AkzidenzGroteskBE-BoldEx /AkzidenzGroteskBE-BoldExIt /AkzidenzGroteskBE-BoldIt /AkzidenzGroteskBE-Ex /AkzidenzGroteskBE-It /AkzidenzGroteskBE-Light /AkzidenzGroteskBE-LightEx /AkzidenzGroteskBE-LightOsF /AkzidenzGroteskBE-Md /AkzidenzGroteskBE-MdEx /AkzidenzGroteskBE-MdIt /AkzidenzGroteskBE-Regular /AkzidenzGroteskBE-Super /AlbertusMT /AlbertusMT-Italic /AlbertusMT-Light /Aldine401BT-BoldA /Aldine401BT-BoldItalicA /Aldine401BT-ItalicA /Aldine401BT-RomanA /Aldine401BTSPL-RomanA /Aldine721BT-Bold /Aldine721BT-BoldItalic /Aldine721BT-Italic /Aldine721BT-Light /Aldine721BT-LightItalic /Aldine721BT-Roman /Aldus-Italic /Aldus-ItalicOsF /Aldus-Roman /Aldus-RomanSC /AlternateGothicNo2BT-Regular /AmazoneBT-Regular /AmericanTypewriter-Bold /AmericanTypewriter-BoldA /AmericanTypewriter-BoldCond /AmericanTypewriter-BoldCondA /AmericanTypewriter-Cond /AmericanTypewriter-CondA /AmericanTypewriter-Light /AmericanTypewriter-LightA /AmericanTypewriter-LightCond /AmericanTypewriter-LightCondA /AmericanTypewriter-Medium /AmericanTypewriter-MediumA /Anna /AntiqueOlive-Bold /AntiqueOlive-Compact /AntiqueOlive-Italic /AntiqueOlive-Roman /Arcadia /Arcadia-A /Arkona-Medium /Arkona-Regular /ArrusBT-Black /ArrusBT-BlackItalic /ArrusBT-Bold /ArrusBT-BoldItalic /ArrusBT-Italic /ArrusBT-Roman /AssemblyLightSSK /AuroraBT-BoldCondensed /AuroraBT-RomanCondensed /AuroraOpti-Condensed /AvantGarde-Book /AvantGarde-BookOblique /AvantGarde-Demi /AvantGarde-DemiOblique /Avenir-Black /Avenir-BlackOblique /Avenir-Book /Avenir-BookOblique /Avenir-Heavy /Avenir-HeavyOblique /Avenir-Light /Avenir-LightOblique /Avenir-Medium /Avenir-MediumOblique /Avenir-Oblique /Avenir-Roman /BaileySansITC-Bold /BaileySansITC-BoldItalic /BaileySansITC-Book /BaileySansITC-BookItalic /BakerSignetBT-Roman /BaskervilleBE-Italic /BaskervilleBE-Medium /BaskervilleBE-MediumItalic /BaskervilleBE-Regular /Baskerville-Bold /BaskervilleBook-Italic /BaskervilleBook-MedItalic /BaskervilleBook-Medium /BaskervilleBook-Regular /BaskervilleBT-Bold /BaskervilleBT-BoldItalic /BaskervilleBT-Italic /BaskervilleBT-Roman /BaskervilleMT /BaskervilleMT-Bold /BaskervilleMT-BoldItalic /BaskervilleMT-Italic /BaskervilleMT-SemiBold /BaskervilleMT-SemiBoldItalic /BaskervilleNo2BT-Bold /BaskervilleNo2BT-BoldItalic /BaskervilleNo2BT-Italic /BaskervilleNo2BT-Roman /Baskerville-Normal-Italic /BauerBodoni-Black /BauerBodoni-BlackCond /BauerBodoni-BlackItalic /BauerBodoni-Bold /BauerBodoni-BoldCond /BauerBodoni-BoldItalic /BauerBodoni-BoldItalicOsF /BauerBodoni-BoldOsF /BauerBodoni-Italic /BauerBodoni-ItalicOsF /BauerBodoni-Roman /BauerBodoni-RomanSC /Bauhaus-Bold /Bauhaus-Demi /Bauhaus-Heavy /BauhausITCbyBT-Bold /BauhausITCbyBT-Heavy /BauhausITCbyBT-Light /BauhausITCbyBT-Medium /Bauhaus-Light /Bauhaus-Medium /BellCentennial-Address /BellGothic-Black /BellGothic-Bold /Bell-GothicBoldItalicBT /BellGothicBT-Bold /BellGothicBT-Roman /BellGothic-Light /Bembo /Bembo-Bold /Bembo-BoldExpert /Bembo-BoldItalic /Bembo-BoldItalicExpert /Bembo-Expert /Bembo-ExtraBoldItalic /Bembo-Italic /Bembo-ItalicExpert /Bembo-Semibold /Bembo-SemiboldItalic /Benguiat-Bold /Benguiat-BoldItalic /Benguiat-Book /Benguiat-BookItalic /BenguiatGothicITCbyBT-Bold /BenguiatGothicITCbyBT-BoldItal /BenguiatGothicITCbyBT-Book /BenguiatGothicITCbyBT-BookItal /BenguiatITCbyBT-Bold /BenguiatITCbyBT-BoldItalic /BenguiatITCbyBT-Book /BenguiatITCbyBT-BookItalic /Benguiat-Medium /Benguiat-MediumItalic /Berkeley-Black /Berkeley-BlackItalic /Berkeley-Bold /Berkeley-BoldItalic /Berkeley-Book /Berkeley-BookItalic /Berkeley-Italic /Berkeley-Medium /Berling-Bold /Berling-BoldItalic /Berling-Italic /Berling-Roman /BernhardBoldCondensedBT-Regular /BernhardFashionBT-Regular /BernhardModernBT-Bold /BernhardModernBT-BoldItalic /BernhardModernBT-Italic /BernhardModernBT-Roman /BernhardTangoBT-Regular /BlockBE-Condensed /BlockBE-ExtraCn /BlockBE-ExtraCnIt /BlockBE-Heavy /BlockBE-Italic /BlockBE-Regular /Bodoni /Bodoni-Bold /Bodoni-BoldItalic /Bodoni-Italic /Bodoni-Poster /Bodoni-PosterCompressed /Bookman-Demi /Bookman-DemiItalic /Bookman-Light /Bookman-LightItalic /Boton-Italic /Boton-Medium /Boton-MediumItalic /Boton-Regular /Boulevard /BremenBT-Black /BremenBT-Bold /BroadwayBT-Regular /CaflischScript-Bold /CaflischScript-Regular /Caliban /CarminaBT-Bold /CarminaBT-BoldItalic /CarminaBT-Light /CarminaBT-LightItalic /CarminaBT-Medium /CarminaBT-MediumItalic /Carta /Caslon224ITCbyBT-Bold /Caslon224ITCbyBT-BoldItalic /Caslon224ITCbyBT-Book /Caslon224ITCbyBT-BookItalic /Caslon540BT-Italic /Caslon540BT-Roman /CaslonBT-Bold /CaslonBT-BoldItalic /CaslonOpenFace /CaslonTwoTwentyFour-Black /CaslonTwoTwentyFour-BlackIt /CaslonTwoTwentyFour-Bold /CaslonTwoTwentyFour-BoldIt /CaslonTwoTwentyFour-Book /CaslonTwoTwentyFour-BookIt /CaslonTwoTwentyFour-Medium /CaslonTwoTwentyFour-MediumIt /CastleT-Bold /CastleT-Book /Caxton-Bold /Caxton-BoldItalic /Caxton-Book /Caxton-BookItalic /CaxtonBT-Bold /CaxtonBT-BoldItalic /CaxtonBT-Book /CaxtonBT-BookItalic /Caxton-Light /Caxton-LightItalic /CelestiaAntiqua-Ornaments /Centennial-BlackItalicOsF /Centennial-BlackOsF /Centennial-BoldItalicOsF /Centennial-BoldOsF /Centennial-ItalicOsF /Centennial-LightItalicOsF /Centennial-LightSC /Centennial-RomanSC /Century-Bold /Century-BoldItalic /Century-Book /Century-BookItalic /CenturyExpandedBT-Bold /CenturyExpandedBT-BoldItalic /CenturyExpandedBT-Italic /CenturyExpandedBT-Roman /Century-HandtooledBold /Century-HandtooledBoldItalic /Century-Light /Century-LightItalic /CenturyOldStyle-Bold /CenturyOldStyle-Italic /CenturyOldStyle-Regular /CenturySchoolbookBT-Bold /CenturySchoolbookBT-BoldCond /CenturySchoolbookBT-BoldItalic /CenturySchoolbookBT-Italic /CenturySchoolbookBT-Roman /Century-Ultra /Century-UltraItalic /CharterBT-Black /CharterBT-BlackItalic /CharterBT-Bold /CharterBT-BoldItalic /CharterBT-Italic /CharterBT-Roman /CheltenhamBT-Bold /CheltenhamBT-BoldCondItalic /CheltenhamBT-BoldExtraCondensed /CheltenhamBT-BoldHeadline /CheltenhamBT-BoldItalic /CheltenhamBT-BoldItalicHeadline /CheltenhamBT-Italic /CheltenhamBT-Roman /Cheltenham-HandtooledBdIt /Cheltenham-HandtooledBold /CheltenhamITCbyBT-Bold /CheltenhamITCbyBT-BoldItalic /CheltenhamITCbyBT-Book /CheltenhamITCbyBT-BookItalic /Christiana-Bold /Christiana-BoldItalic /Christiana-Italic /Christiana-Medium /Christiana-MediumItalic /Christiana-Regular /Christiana-RegularExpert /Christiana-RegularSC /Clarendon /Clarendon-Bold /Clarendon-Light /ClassicalGaramondBT-Bold /ClassicalGaramondBT-BoldItalic /ClassicalGaramondBT-Italic /ClassicalGaramondBT-Roman /CMR10 /CMR8 /CMSY10 /CMSY8 /CMTI10 /CommonBullets /ConduitITC-Bold /ConduitITC-BoldItalic /ConduitITC-Light /ConduitITC-LightItalic /ConduitITC-Medium /ConduitITC-MediumItalic /CooperBlack /CooperBlack-Italic /CooperBT-Bold /CooperBT-BoldItalic /CooperBT-Light /CooperBT-LightItalic /CopperplateGothicBT-Bold /CopperplateGothicBT-BoldCond /CopperplateGothicBT-Heavy /CopperplateGothicBT-Roman /CopperplateGothicBT-RomanCond /Copperplate-ThirtyThreeBC /Copperplate-ThirtyTwoBC /Coronet-Regular /Courier /Courier-Bold /Courier-BoldOblique /Courier-Oblique /Critter /CS-Special-font /DellaRobbiaBT-Bold /DellaRobbiaBT-Roman /Della-RobbiaItalicBT /Della-RobbiaSCaps /Del-NormalSmallCaps /Delphin-IA /Delphin-IIA /Delta-Bold /Delta-BoldItalic /Delta-Book /Delta-BookItalic /Delta-Light /Delta-LightItalic /Delta-Medium /Delta-MediumItalic /Delta-Outline /DextorD /DextorOutD /DidotLH-OrnamentsOne /DidotLH-OrnamentsTwo /DINEngschrift /DINEngschrift-Alternate /DINMittelschrift /DINMittelschrift-Alternate /DINNeuzeitGrotesk-BoldCond /DINNeuzeitGrotesk-Light /Dom-CasItalic /DomCasual /DomCasual-Bold /Dom-CasualBT /Ehrhard-Italic /Ehrhard-Regular /EhrhardSemi-Italic /EhrhardtMT /EhrhardtMT-Italic /EhrhardtMT-SemiBold /EhrhardtMT-SemiBoldItalic /EhrharSemi /ELANGO-IB-A03 /ELANGO-IB-A75 /ELANGO-IB-A99 /ElectraLH-Bold /ElectraLH-BoldCursive /ElectraLH-Cursive /ElectraLH-Regular /ElGreco /EnglischeSchT-Bold /EnglischeSchT-Regu /ErasContour /ErasITCbyBT-Bold /ErasITCbyBT-Book /ErasITCbyBT-Demi /ErasITCbyBT-Light /ErasITCbyBT-Medium /ErasITCbyBT-Ultra /Euclid /Euclid-Bold /Euclid-BoldItalic /EuclidExtra /EuclidExtra-Bold /EuclidFraktur /EuclidFraktur-Bold /Euclid-Italic /EuclidMathOne /EuclidMathOne-Bold /EuclidMathTwo /EuclidMathTwo-Bold /EuclidSymbol /EuclidSymbol-Bold /EuclidSymbol-BoldItalic /EuclidSymbol-Italic /EUEX10 /EUFB10 /EUFB5 /EUFB7 /EUFM10 /EUFM5 /EUFM7 /EURB10 /EURB5 /EURB7 /EURM10 /EURM5 /EURM7 /EuropeanPi-Four /EuropeanPi-One /EuropeanPi-Three /EuropeanPi-Two /EuroSans-Bold /EuroSans-BoldItalic /EuroSans-Italic /EuroSans-Regular /EuroSerif-Bold /EuroSerif-BoldItalic /EuroSerif-Italic /EuroSerif-Regular /Eurostile /Eurostile-Bold /Eurostile-BoldCondensed /Eurostile-BoldExtendedTwo /Eurostile-BoldOblique /Eurostile-Condensed /Eurostile-Demi /Eurostile-DemiOblique /Eurostile-ExtendedTwo /EurostileLTStd-Demi /EurostileLTStd-DemiOblique /Eurostile-Oblique /EUSB10 /EUSB5 /EUSB7 /EUSM10 /EUSM5 /EUSM7 /ExPonto-Regular /FairfieldLH-Bold /FairfieldLH-BoldItalic /FairfieldLH-BoldSC /FairfieldLH-CaptionBold /FairfieldLH-CaptionHeavy /FairfieldLH-CaptionLight /FairfieldLH-CaptionMedium /FairfieldLH-Heavy /FairfieldLH-HeavyItalic /FairfieldLH-HeavySC /FairfieldLH-Light /FairfieldLH-LightItalic /FairfieldLH-LightSC /FairfieldLH-Medium /FairfieldLH-MediumItalic /FairfieldLH-MediumSC /FairfieldLH-SwBoldItalicOsF /FairfieldLH-SwHeavyItalicOsF /FairfieldLH-SwLightItalicOsF /FairfieldLH-SwMediumItalicOsF /Fences /Fenice-Bold /Fenice-BoldOblique /FeniceITCbyBT-Bold /FeniceITCbyBT-BoldItalic /FeniceITCbyBT-Regular /FeniceITCbyBT-RegularItalic /Fenice-Light /Fenice-LightOblique /Fenice-Regular /Fenice-RegularOblique /Fenice-Ultra /Fenice-UltraOblique /FlashD-Ligh /Flood /Folio-Bold /Folio-BoldCondensed /Folio-ExtraBold /Folio-Light /Folio-Medium /FontanaNDAaOsF /FontanaNDAaOsF-Italic /FontanaNDCcOsF-Semibold /FontanaNDCcOsF-SemiboldIta /FontanaNDEeOsF /FontanaNDEeOsF-Bold /FontanaNDEeOsF-BoldItalic /FontanaNDEeOsF-Light /FontanaNDEeOsF-Semibold /FormalScript421BT-Regular /Formata-Bold /Formata-MediumCondensed /ForteMT /FournierMT-Ornaments /FrakturBT-Regular /FrankfurterHigD /FranklinGothic-Book /FranklinGothic-BookItal /FranklinGothic-BookOblique /FranklinGothic-Condensed /FranklinGothic-Demi /FranklinGothic-DemiItal /FranklinGothic-DemiOblique /FranklinGothic-Heavy /FranklinGothic-HeavyItal /FranklinGothic-HeavyOblique /FranklinGothicITCbyBT-BookItal /FranklinGothicITCbyBT-Demi /FranklinGothicITCbyBT-DemiItal /FranklinGothicITCbyBT-Heavy /FranklinGothicITCbyBT-HeavyItal /FranklinGothic-Medium /FranklinGothic-MediumItal /FranklinGothic-Roman /Freeform721BT-Bold /Freeform721BT-BoldItalic /Freeform721BT-Italic /Freeform721BT-Roman /FreestyleScrD /FreestyleScript /Freestylescript /FrizQuadrataITCbyBT-Bold /FrizQuadrataITCbyBT-Roman /Frutiger-Black /Frutiger-BlackCn /Frutiger-BlackItalic /Frutiger-Bold /Frutiger-BoldCn /Frutiger-BoldItalic /Frutiger-Cn /Frutiger-ExtraBlackCn /Frutiger-Italic /Frutiger-Light /Frutiger-LightCn /Frutiger-LightItalic /Frutiger-Roman /Frutiger-UltraBlack /Futura /FuturaBlackBT-Regular /Futura-Bold /Futura-BoldOblique /Futura-Book /Futura-BookOblique /FuturaBT-Bold /FuturaBT-BoldCondensed /FuturaBT-BoldCondensedItalic /FuturaBT-BoldItalic /FuturaBT-Book /FuturaBT-BookItalic /FuturaBT-ExtraBlack /FuturaBT-ExtraBlackCondensed /FuturaBT-ExtraBlackCondItalic /FuturaBT-ExtraBlackItalic /FuturaBT-Heavy /FuturaBT-HeavyItalic /FuturaBT-Light /FuturaBT-LightCondensed /FuturaBT-LightItalic /FuturaBT-Medium /FuturaBT-MediumCondensed /FuturaBT-MediumItalic /Futura-CondensedLight /Futura-CondensedLightOblique /Futura-ExtraBold /Futura-ExtraBoldOblique /Futura-Heavy /Futura-HeavyOblique /Futura-Light /Futura-LightOblique /Futura-Oblique /Futura-Thin /Galliard-Black /Galliard-BlackItalic /Galliard-Bold /Galliard-BoldItalic /Galliard-Italic /GalliardITCbyBT-Bold /GalliardITCbyBT-BoldItalic /GalliardITCbyBT-Italic /GalliardITCbyBT-Roman /Galliard-Roman /Galliard-Ultra /Galliard-UltraItalic /Garamond-Antiqua /GaramondBE-Bold /GaramondBE-BoldExpert /GaramondBE-BoldOsF /GaramondBE-CnExpert /GaramondBE-Condensed /GaramondBE-CondensedSC /GaramondBE-Italic /GaramondBE-ItalicExpert /GaramondBE-ItalicOsF /GaramondBE-Medium /GaramondBE-MediumCn /GaramondBE-MediumCnExpert /GaramondBE-MediumCnOsF /GaramondBE-MediumExpert /GaramondBE-MediumItalic /GaramondBE-MediumItalicExpert /GaramondBE-MediumItalicOsF /GaramondBE-MediumSC /GaramondBE-Regular /GaramondBE-RegularExpert /GaramondBE-RegularSC /GaramondBE-SwashItalic /Garamond-Bold /Garamond-BoldCondensed /Garamond-BoldCondensedItalic /Garamond-BoldItalic /Garamond-Book /Garamond-BookCondensed /Garamond-BookCondensedItalic /Garamond-BookItalic /Garamond-Halbfett /Garamond-HandtooledBold /Garamond-HandtooledBoldItalic /GaramondITCbyBT-Bold /GaramondITCbyBT-BoldCondensed /GaramondITCbyBT-BoldCondItalic /GaramondITCbyBT-BoldItalic /GaramondITCbyBT-BoldNarrow /GaramondITCbyBT-BoldNarrowItal /GaramondITCbyBT-Book /GaramondITCbyBT-BookCondensed /GaramondITCbyBT-BookCondItalic /GaramondITCbyBT-BookItalic /GaramondITCbyBT-BookNarrow /GaramondITCbyBT-BookNarrowItal /GaramondITCbyBT-Light /GaramondITCbyBT-LightCondensed /GaramondITCbyBT-LightCondItalic /GaramondITCbyBT-LightItalic /GaramondITCbyBT-LightNarrow /GaramondITCbyBT-LightNarrowItal /GaramondITCbyBT-Ultra /GaramondITCbyBT-UltraCondensed /GaramondITCbyBT-UltraCondItalic /GaramondITCbyBT-UltraItalic /Garamond-Kursiv /Garamond-KursivHalbfett /Garamond-Light /Garamond-LightCondensed /Garamond-LightCondensedItalic /Garamond-LightItalic /GaramondNo4CyrTCY-Ligh /GaramondNo4CyrTCY-LighItal /GaramondThree /GaramondThree-Bold /GaramondThree-BoldItalic /GaramondThree-BoldItalicOsF /GaramondThree-BoldSC /GaramondThree-Italic /GaramondThree-ItalicOsF /GaramondThree-SC /GaramondThreeSMSIISpl-Italic /GaramondThreeSMSitalicSpl-Italic