Strengths and weaknesses of rating scales psychology

EVALUATION OF TEST MATERIALS AND PROCEDURES

Running Head: EVALUATION OF TEST MATERIALS AND PROCEDURES 1.

6

EVALUATION OF TEST MATERIALS AND PROCEDURES

Test Items and Format

The Autism Spectrum Rating Scale is a norm-referenced test, meaning that the ASRS looked at what behaviors are most commonly presented in ASD and in an individuals age range. The scale allows for rapid assessment, a family member, caregiver, teacher, or any adult figure that has known the child for at least 4 weeks can administer the test. Administering the test can take place in a home, school, parks, or on vacation. If the individual requires the ASRS is available in English or Spanish, allowing for administration in either format. An online format or a paper copy is available that is available for administration and scoring. Also, a software system is available in regards to scoring. This assessment is intended for individuals who have an Autism Spectrum Disorder and are between the ages of 2-18 years of age. There are two different forms that can be utilized according to age and attention span. Children are dived into two groups, group one is designed for children between 2-5 years of age and group two contains children between 6-18 years of age. To make a test measure fair one looks at the content found within the assessment and the types of formatting options available. Myers and McReynolds (2014) conducted a research study regarding behavior-rating scales to see how effective it is in identifying children with autism. Why, one may ask is it important to identify children at a young age with ASD? To provide accurate and early intervention services to individuals with an autism spectrum diagnosis, as early intervention has been proven to be most effective. Autism Spectrum Disorder is difficult to diagnose, because ASD is a spanning a broad range of severity across multiple ages and developmental level. To evaluate the effectiveness of a broadband behavior rating scale within rural United States. Participants were diagnosed with a developmental disorder and presented behavior issues. Out of the 156 participants 59 males and 11 females with ASD were present. Psychological evaluations were conducted for children between birth and up to 8 years of age for 30 months. The parents of the children between two years of age and 71 months were also asked to participate. 169 agreed and one parent chose not to participate. The results of the research showed that the most effective scales for diagnosing ASD are: withdrawn/ depressed, social problems, thought problems, aggressive behavior, pervasive developmental problems, anxiety, functional communication, and hyperactivity. If one is to compare the results of this research study one can see how the ASRS addresses each one of the characteristics Myers, and McReynolds (2014) found to be most effective in addressing ASD and behavior issues (Myers, Gross, and McReynolds, 2014). The content of the ASRS is broken down into 13 categories. Skills tested are separated into the following subsets: social/communication, unusual behaviors, peer socialization, social/ emotional reciprocity, stereotypy, adult socialization, behavioral rigidity, atypical language, sensory sensitivity, attention/ self-regulation, and short form score (Goldstein and Naglieri 2010a). The questions the ASRS address are based on parent and teacher ratings of 2560 children from the United States. The short forms are compromised of 15 questions that have been shown through research to be the most effective when measuring behavior. 71 questions are included in the long form of the assessment (Jones, 2013). When either a short or long form is used a parent or caregiver questionnaire is also filled out (2010b). Parental or caregiver questionaries’ are in the form of a 5-point Likert response scale (2009a). The score of the assessment is then based on the DSM V criteria. The format of the ASRS can be given in English or in Spanish. Also, there is an ASRS non-verbal assessment that can be utilized. Using the ASRS scoring guide can provide three different reports. The first report it can provide is the interpretive report, which is a detailed result from the administration of the test. A comparative report is the second report the software is able to produce, which involves a multi-rater perspective from two or more assessments of the ASRS. Lastly the software has the capability to provide progress-monitoring reports, which provide an overview of change over time by using at most four assessments of the ASRS. When an individual is evaluating the positives of the ASRS, the format of the test is simple, logical, easy to administer, and great at identifying ASD/ interventions/ as well as providing ongoing evaluations. The publishing website overs the option to purchase manuals, scoring guides, as well as take classes on scoring methods. The test presents a high rate of inter-rater reliability, class B instruments, a high rate of validity is seen from the scientific community, as well as positive feedback from parents and teachers. The negatives regarding the test are that a parent or a caregiver cannot complete the scoring method. A class level of a C is required in regards to scoring. Level C consists of a master degree in a health-related field or a bachelor’s degree in occupational therapy (2009b).

When one is summarizing the quality and appropriateness of the test items the ASRS, provides a comprehensive evaluation for individuals with ASD. The quality found with the ASRS is highly recommended and the statistical data prove that the assessment is reliable, valid, and accurate. The format is broad and covers a wide array of typical behavior problems associated with ASD. The directions when utilizing this assessment are simple and easy to follow. A manual is available to purchase to assist in administration. Answer sheets and score reports can all be received using an online format.

Fair and Appropriate Materials

To minimize offensive content or language with the ASRS, a study that asked teachers and parents to identify appropriate subscales. Also, the content was analyzed by the test developers to ensure that the assessment met the standards of the DSM-V (Goldstein and Naglieri 2010a). The DSM-V uses person-first language to reduce negative aspects in language. Recently, the ASRS was tested in China in regards to norms, the study conducted a pilot study before the implementation of the actual study to reduce offensive language and address content fairness (Zhou, Zhang, Zou, Luo, Xia, Wu, Wang, 2017).

In regards to appropriate modifications the ASRS is accessibility (Cohen, and Swerdlik, 2018). Positive aspects of test materials can be seen in the formatting that serves to support individuals that speak English or Spanish. Having a test that can only be utilized in two different languages is beneficial because it allows for a variety of individuals to utilize this assessment. The ASRS also identifies individuals with mild, moderate, and severe learning disabilities (2010b). Individuals that speak infrequently or not at all can also utilize the Autism Spectrum Rating Scale (Goldstein and Naglieri 2010c). Allowing for a prorating method to be utilized when working with an individual with no verbal or limited verbal function. Reliability and validity of the prorated ASRS the values are similar and in some cases higher. A negative aspect that was noted was the limited information for individuals with physical disabilities. Simek, and Wahlberg, (2011) also evaluated the validity of the Autism Spectrum Rating Scale. The article wanted to ensure that the ASRS has fairness in regards to criterion-related and construct validity. In regards to criterion validity, children with ASD were compared to children throughout the United States and children with other clinical diagnoses. Lastly, this article addresses construct validity, which shows that the ASRS is broken up into parts. In the 2-5 year ASRS a two-part solution was found to be most suitable. The first category addresses socialization and communication. The second category is related to stereotypical behaviors, sensory stimulation, and rigidity. When administering the ASRS to children between 6-18 year olds a third category is included which is the category of self-regulation. The research shows that the ASRS is effective and efficient to use with this population.

AERA (2014) mentions that in order to ensure fairness that standard 4.8 is followed. Standard 4.8 mentions that a test review process must include empirical analyses and that expert’s judge and review scoring methods. The ASRS, uses the direct observation, Likert scales, Cronbach’s Alpha, and T- scores are utilized to perform empirical analyses. The stability of the T-score was also evaluated and showed that the scores obtained in time one and time two had a standard deviation of one 90% of the time Standard 4.10 in the AERA (2014) also talks about the psychometric properties of items within a test and utilizing them to ensure fairness by documenting them.

Technology

In regards to technology, the Autism Spectrum Rating Scale (ASRS) is innovative and effective. The ASRS uses many components that require the use of technology. Within the ASRS one findings Cronbach's Alpha and T-Scores, which provide statically evaluations. Technology allowed test developers to standardize the instruments found within the ASRS. The test developers were able to locate pertinent information such as norms associated with Autism Spectrum Disorder, it provided an understanding of atypical behaviors present in individuals with ASD, how to validate the parent rating scales, and through others research it provided test developers information as to which subsets needed to be included in the formal assessment (Goldstein and Naglieri 2010a).

To ensure fairness the usage of technology has been beneficial. The ASRS, allows users to access the scoring guide using an online software system. This not only helps make the format easier for individual users but also allows for quick and easy scoring methods to be performed. Knowing that the ASRS, can be scored quickly and effectively is beneficially for users who need to gather accurate results. The scoring guide provides users with fairness. Also, technology allows for appropriateness, as mentioned above technology has allowed for norm-based ratings to be established. Technology also provided individuals who administer the ASRS a computer to easily facilitate the assessment at a variety of location such as the beach, school, home, etc.. This is how technology has benefited and continues to benefit the ASRS (2010b).

Synthesis of Findings

Throughout this paper, I have identified major strengths and weaknesses in regards to the test items and materials. First, the ASRS has a language barrier. The test contents are only available in an English and Spanish version. This limited the influence this test can have on individuals because the format of the test cannot be given to individuals who do not speak English or Spanish. However, the fact that the ASRS accommodates individuals with non-verbal or limited verbal social skills is a major strength. Researching the validity of behavior rating scales ensured me that the ASRS has taken appropriate measures in regards to fairness. Another strength noted was the to score the ASRS a master degree or a bachelors degree in occupational therapy is needed; this provides accurate and reliable scores. In regards to scoring, the fact that individuals have the option to choose from three different result formats: progress-monitoring reports, interpretive report, and comparative reports is a strength. A weakness that was noted was that often time’s individuals with ASD have other co-morbidities and I was unable to locate any research studies in regards to individuals with ASD and cerebral palsy or ADHD. The test is unable to speak to the fairness when conducting the ASRS with an individual with multiple co-morbidities. Overall, the ASRS has excellent reliability, validity, and accuracy (Goldstein and Naglieri 2010b).

Conclusion and Recommendations

Throughout this paper I have discussed specific components of the Autism Spectrum Rating Scale, specifically addressing the test items, appropriateness, and materials. Overall, my evaluation of the ASRS in regards to the materials, test items, and appropriateness are great. Through conducting my only research I know that the ASRS is effective at identifying diagnostic criteria, interventions, and ongoing monitoring of individuals with Autism Spectrum Disorder.

The three recommendations that I believe the ASRS should implement would be to fist require test administers to undergo certain training in regards to facilitating the assessment. According, to AERA (2014) standard 6.1 mentions that administrators should follow specific guidelines in regards to administering the assessment. I feel as if there is not a well-defined guideline to follow for parents and caregivers that administer the test. Secondly, AERA (2014) according to standard 6.4 talks about the test environment and how it needs to be controlled to a certain degree. The computer format of the ASRS allows the testing environment to take place anywhere, however, the test needs to have specific guidelines for those that administer the test can follow. Lastly, it is my recommendation to make the test format available in more than English and Spanish, having limited availability in language means low generalizability.