Http garote bdmonkeys net bsri html

Eye Ubiquitous/photolibrary

chapter 4

Survey Designs— Predicting Behavior

Chapter Contents

• Introduction to Survey Research • Questionnaire Design • Sampling from the Population • Analyzing Survey Data

new66480_04_c04_p133-172.indd 133 10/31/11 10:38 AM

CHAPTER 4Introduction

In a highly influential book published in the 1960s, the sociologist Erving Goffman (1963) defined stigma as an unusual characteristic that triggers a negative evaluation. In his words, “the stigmatized person is one who is reduced in our minds from a whole and usual person to a tainted, discounted one” (1963, p. 3). People’s beliefs about stigmatized characteristics exist largely in the eye of the beholder but have substantial influence on social interactions with the stigmatized (see Snyder, Tanke, & Berscheid, 1977). A large research tradition in psychology has been devoted to understanding both the origins of stigma and the consequences of being stigmatized. According to Goffman and others, the characteristics associated with the greatest degree of stigma have three features in common, all of which apply to the label of ADHD: they are highly visible, they are perceived as controllable, and they are misunderstood by the public.

Recently, researchers have taken considerable interest in people’s attitudes toward mem- bers of the gay and lesbian community. Although these attitudes have become more pos- itive over time, this group still encounters harassment and other forms of discrimination on a regular basis (see National Gay Task Force, 1984). One of the top recognized experts on this subject is Gregory Herek, professor of psychology at the University of Califor- nia at Davis (http://psychology.ucdavis.edu/herek/). In a 1988 article, Herek conducted a survey of heterosexuals’ attitudes toward both lesbians and gay men, with the goal of understanding the predictors of negative attitudes. Herek approached this research question by constructing a scale to measure attitudes toward these groups. In three stud- ies, participants were asked to complete this attitude measure, along with other existing scales assessing attitudes about gender roles, religion, and traditional ideologies.

Herek’s (1988) research revealed that, as hypothesized, heterosexual males tended to hold more negative attitudes about gay men and lesbians than heterosexual females. However, the same psychological mechanisms seemed to explain the prejudice in both genders. That is, negative attitudes were associated with increased religiosity, more traditional beliefs about family and gender, and fewer experiences actually interacting with gay men and lesbians. These associations meant that Herek could predict people’s attitudes toward gay men and lesbians based on knowing their views about family, gen- der, and religion, as well as their past interactions with the stigmatized group. Herek’s primary contribution to the literature in this paper was the insight that reducing stigma toward gay men and lesbians “may require confronting deeply held, socially reinforced values” (1988, p. 473). And this insight was only possible because people were asked to report these values directly.

TX_

TX

new66480_04_c04_p133-172.indd 134 10/31/11 10:38 AM

http://psychology.ucdavis.edu/herek/
CHAPTER 4Introduction

Research: Making an Impact Kinsey Reports

Alfred Kinsey’s research on human sexuality is an example of social research that changed the way soci- ety thought about the complex issue of sex, affecting how people viewed their own behavior in light of social norms. Kinsey’s research, particularly two books on male and female sexuality known together at the Kinsey Reports, illuminated the discrepancies between the assumptions made by a “moral public” and the actual behavior of individuals. His shift in the approach to studying sex—applying sci- entific methods and reasoning rather than basing conclusions on medical speculation and dogmatic opinions—changed the nature of sex research and the general public’s view of sex for decades to come.

Kinsey’s major contribution was in challenging the prevailing assumptions about sexual activity in the United States and obtaining descriptive data from both men and women that described their own sexual practices (Bullough, 1998). By collecting actual data instead of relying on speculation, Kinsey made the study of sexuality more scientifically based. The results of his surveys revealed a variety of sexual behaviors that shocked many members of society and redefined the sexual morality of modern America.

Until Kinsey’s research, the general Victorian viewpoint was that women should not show any inter- est in sex and should submit to their husband without any sign of pleasure (Davis, 1929). Kinsey’s data challenged society’s assumption that women were asexual. His studies revealed that 25% of the women studied had experienced an orgasm by the age of 15 and more than half by the age of 20 (Kinsey, 1953). Eventually, these results were bundled into the various elements that fueled the women’s movement of the 1960s and encouraged further examination of female sexuality (Bullough, 1998).

Kinsey’s data also contributed to the budding gay and lesbian liberation movement. Until the Kinsey Reports, studies of human sexuality were based on the assumption that homosexuals were mentally ill (Bullough, 1998). When Kinsey’s data revealed that many males and females practiced homosexu- ality to some degree, he suggested that sexuality was more of a continuum than a series of catego- ries one fit into. In addition, the Kinsey Reports revealed that the number of extramarital relation- ships people were having was higher than most expected. Forty percent of married American males reported having an extramarital relationship (Kinsey, 1953).

These ideas, though controversial, led society to take a realistic look at the actual sexual practices of its members. The topic of sexuality became less dogmatic as society became more open about sexual activities and preferences.

Kinsey’s data not only encouraged social change but also revolutionized the way in which scientists study sexuality. By examining data and studying sex from an unbiased standpoint, Kinsey successfully made the study of human sexuality into a science. His research not only changed our way of studying sexual behavior but also allowed society to become less restrictive in their expectations of “normal” sexual behavior.

In this chapter, we continue our journey along the continuum of control, moving on to survey research, in which the primary goal is either describing or predicting atti- tudes and behavior. For our purposes, survey research refers to any method that relies on people’s reports of their own attitudes, feelings, and behaviors. So, for example, in Herek’s (1988) study, the participants reported their attitudes toward lesbians and gay

new66480_04_c04_p133-172.indd 135 10/31/11 10:38 AM

CHAPTER 4Section 4.1 Introduction to Survey Research

men, rather than these attitudes being somehow directly observed by the researchers. Compared to the descriptive designs we discussed in Chap- ter 3, survey designs tend to have more control over both data collection and question content. Thus, survey research falls somewhere between purely descriptive research (Chapter 3) and the explanatory research involved in experimental designs (Chapter 5). This chapter provides an overview of survey research from conceptualiza- tion through analysis. We will cover the types of research questions that are best suited to survey research and provide an overview of the deci- sions to consider in designing and conducting a survey study. We will then cover the process of data collection, with a focus on selecting the

people who will complete your survey. Finally, we will cover the three most common approaches for analyzing survey data, bringing us back full circle to addressing our research questions.

4.1 Introduction to Survey Research

Whether you are aware of it or not, you have been encountering survey research for most of your life. Every time your telephone rings during dinnertime, and the person on the other end of the line insists on knowing your household income and favorite brand of laundry detergent, he or she is helping to conduct survey research. When news programs try to predict the winner of an election two weeks early, these reports are based on survey research of eligible voters. In both cases, the researcher is trying to make predictions about the products people buy or the candidates they will elect based on people’s reports of their own attitudes, feelings, and behaviors.

Surveys can be used in a variety of contexts and are most appropriate for questions that involve people describing their attitudes, their behaviors, or a combination of the two. For example, if you wanted to examine the predictors of attitudes toward the death pen- alty, you could ask people their opinions on this topic and also ask them about their polit- ical party affiliation. Based on these responses, you could test whether political affiliation predicted attitudes toward the death penalty. Or, imagine you wanted to know whether students who spent more time studying were more likely to do well on their exams. This question could be answered using a survey that asked students about their study habits and then tracked their exam grades. We will return to this example near the end of the chapter, as we discuss the process of analyzing survey data to test our hypotheses about predictions.

The common thread behind these two examples is that they require people to report either their thoughts (e.g., opinions about the death penalty) or their behaviors (e.g., the hours they spend studying). Thus, in deciding whether a survey is the best fit for your research

Stockphoto/thinkstock

Surveys are used to describe or predict attitudes and behavior.

new66480_04_c04_p133-172.indd 136 10/31/11 10:38 AM

CHAPTER 4Section 4.1 Introduction to Survey Research

question, the key is to consider whether people will be both able and willing to report these things accurately. We will expand on both of these issues in the next section.

Distinguishing Features of Surveys

Survey research designs have three distinguishing features that set them apart from other designs. First, all survey research relies on either written or verbal self-reports of peo- ple’s attitudes, feelings, and behaviors. This means that researchers will ask participants a series of questions and record their responses. This approach has several advantages, including being relatively straightforward and allowing access to psychological processes (e.g., “Why do you support candidate X?”). However, researchers are also cautious in their interpretation of self-report data because participants’ responses can reflect a combination of their true attitude and their concern over how this attitude will be perceived. Scientists refer to this as social desirability, which means that people may be reluctant to report unpopular attitudes. So, if you were to ask people their attitudes about different racial groups, their answers might reflect both their true attitude and their desire not to appear racist. We return to the issue of social desirability and discuss some tricks for designing questions that can help to sidestep these concerns and capture respondents’ true attitudes.

The second distinguishing feature of survey research is that it has the ability to access internal states that cannot be measured through direct observation. In our discussion of observational designs in Chapter 3, we learned that one of the limitations of these designs was a lack of insight into why people do what they do. Survey research is able to address this limitation directly: By asking people what they think, how they feel, and why they behave in certain ways, researchers come closer to capturing the underlying psychologi- cal processes. However, people’s reports of their internal states should be taken with a grain of salt, for two reasons. First, as mentioned, these reports may be biased by social desirability concerns, particularly when unpopular attitudes are involved. Second, there is a large literature in social psychology suggesting that people may not be very accu- rate at understanding the true reasons for their behavior. In a highly cited review paper, psychologists Richard Nisbett and Tim Wilson (1977) argued that we make poor guesses about why we do things, and those guesses are based more on our assumptions than on any real introspection. Thus, survey questions can provide access to internal states, but these should always be interpreted with caution.

Third, on a more practical note, survey research allows us to collect large amounts of data with relatively little effort and few resources. Many of the descriptive designs we discussed in Chapter 3 require observing one person at a time, and the same will hold true when we get to experimental designs in Chapter 5. Survey research designs stand out as the most efficient (or, at least, the most potentially efficient) because surveys can be distributed to large groups of people simultaneously. However, their actual efficiency depends on the decisions made during the design process. In reality, efficiency is often in a delicate balance with the accuracy and completeness of the data.

Broadly speaking, survey research can be conducted using either verbal or written self- reports (or a combination of the two). Before we dive into the details of writing and for- matting a survey, it is important to understand the pros and cons of administering your survey as an interview (i.e., a verbal survey) or a questionnaire (i.e., a written survey).

new66480_04_c04_p133-172.indd 137 10/31/11 10:38 AM

CHAPTER 4Section 4.1 Introduction to Survey Research

Interviews

An interview involves a verbal question-and- answer exchange between the researcher and the participant. This verbal exchange can take place either face-to-face or over the phone. So, our tele- marketer example from above represents an inter- view because the questions are asked verbally via phone. Likewise, if you are approached in a shopping mall and asked to answer questions about your favorite products, you’re experiencing a survey in interview form because the questions are administered verbally. And, if you have ever taken part in a focus group, in which a group of people gives their reactions to a new product, the researchers are essentially conducting an inter- view with the group.

Interview Schedules Regardless of how the interview is administered, the interviewer (i.e., the researcher) has a predetermined plan, or script, for how the interview should go. This plan, or script, for the progress of the interview is known as an interview schedule. When conducting an inter- view—including those telemarketing calls—the researcher/interviewer has a detailed plan for the order of questions to be asked, along with follow-up questions depending on the participant’s responses.

Broadly speaking, there are two types of interview schedules. A linear (also called “struc- tured”) schedule will ask the same questions in the same order for all participants. In contrast, a branching schedule unfolds more like a flowchart, with the next question dependent on participants’ answers. A branching schedule is typically used in cases with follow-up questions that only make sense for some of the participants. For example, you might first ask people whether they have children; if they answer “yes,” you could then follow-up by asking how many children.

One danger in using a branching schedule is that it is based partly on your assumptions about the relationships between variables. Granted, it is fairly uncontroversial to only ask people with children to indicate how many children they have. But imagine the follow- ing scenario. Let’s say you first ask participants for their household income, and then ask about their political donations:

• “How much money do you make? $18,000? OK, how likely are you to donate to the Democratic Party?

• ”How much money do you make? $250,000? OK, how likely are you to donate money to the Republican Party?”

The assumption that is implicit in the way that these questions branch is that wealthier people are more likely to be Republicans and less wealthy people to be Democrats. This might be supported by the data, or it might not. But by planning the follow-up questions in

Jochen Sands/Thinstock

Interviews are conducted verbally.

new66480_04_c04_p133-172.indd 138 10/31/11 10:38 AM

CHAPTER 4Section 4.1 Introduction to Survey Research

this way, you are unable to capture cases that do not fit your stereotypes (i.e., the wealthy Democrats and the poor Republicans). The lesson here is to be careful about letting your biases shape the data collection process.

Advantages and Disadvantages of Interviews Interviews have a number of advantages over written surveys. For one, people are often more motivated to talk than they are to write. Many years ago, in my days as an under- graduate research assistant, I was dispatched to a local shopping mall to interview people about their experiences in romantic relationships. I had no trouble at all recruiting partici- pants, many of whom would go on and on (and on, and on) about recent relationships— one woman confided to me that she had just left an abusive spouse earlier in the week! For better or for worse, these experiences would have been more difficult to capture in writ- ing. Related to this, people’s oral responses are typically richer and more detailed than their written responses. Think of the difference between asking someone to “describe their views on gun control” versus “indicate on a scale of 1 to 7 the degree to which you support gun control.” The former is more likely to capture the richness and subtlety involved in people’s attitudes about guns. On a practical note, using an interview format also allows you to ensure that respondents understand the questions. If written questionnaire items are poorly worded, people are forced to guess at your meaning, and these guesses intro- duce a big source of error variance. But if an interview question is poorly asked, people find it much easier to ask the interviewer to clarify. Finally, using an interview format allows you to reach a broader cross-section of people and to include those who are unable to read and write—or, perhaps, unable to read and write the language of your survey.

Interviews also have two clear disadvantages compared to written surveys. First, inter- views are more costly in terms of both time and money. It certainly used more of my time to go to a shopping mall than it would have taken to mail out packets of surveys (but no more money—these research assistant gigs tend to be unpaid!). Second, the interview format allows many opportunities to glean personal bias from the interview. These biases are unlikely to be deliberate, but participants can often pick up on body language and subtle facial expressions when the interviewer disagrees with their answers. These cues may lead them to shape their responses in order to make the interviewer happier. The best way to understand both the pros and cons of interviewing is that both are a con- sequence of personal interaction. The interaction between interviewer and interviewee allows for richer responses but also the potential for these responses to be biased. As a researcher, you have to weigh these pros and cons and decide which method is the best fit for your survey. In the next section, we turn our attention to the process of administering surveys in writing.

Questionnaires

A questionnaire is a survey that involves a written question-and-answer exchange between the researcher and the participant. The exchange is a bit different from what we saw with interview formats—in this case, the questions are designed ahead of time, then given to participants, who write their responses and return the questionnaire to the researcher. In the next section, we will discuss details for designing these questions. Before we get there, let’s take a quick look at the process of administering written surveys.

new66480_04_c04_p133-172.indd 139 10/31/11 10:38 AM

CHAPTER 4Section 4.1 Introduction to Survey Research

Distribution Methods Questionnaires can be distributed in three primary ways, each with its own pattern of advantages and disadvantages:

Distributing by mail: Until recently, one common way to distribute surveys was to send paper copies through the mail to a group of participants (see the section on “Sampling” for more discussion on how this group is selected). Mailing surveys is relatively cheap, and relatively easy to do, but is unfortunately one of the worst methods when it comes to response rates. People tend to ignore questionnaires that they receive in the mail, dismissing them as one more piece of junk. There are a few tricks available to researchers to increase response rates, including providing incentives, making the survey interesting, and making it as easy as possible to return the results (e.g., with a postage-paid envelope). However, even using all of these tricks, researchers consider themselves extremely lucky to get a 30% response rate from a mail survey. That is, if you mail 1,000 surveys, you will be doing well to receive 300 back. Because of this low return on investment, researchers have begun using other methods for their written surveys.

Distributing in person: Another option is to distribute a written survey in person, simply handing out copies and asking participants to fill them out on the spot. This method is certainly more time consuming; a researcher has to be stationed for long periods of time in order to collect data. In addition, people are less likely to answer the questions honestly because the presence of a researcher makes them worry about social desirability. Last, the sample for this method is limited to people who are in the physical area at the time that questionnaires are being handed out. As we will discuss later, this might lead to problems in the composition of the sample. On the plus side, however, this method tends to result in higher compliance rates because it is harder to say no to someone face-to-face than it is to ignore a piece of mail.

Distributing online: Most recently, a number of researchers have begun to distribute their questionnaires over the Internet. In this case, the questionnaire is designed and posted on a web page, and then participants are directed to this web page to complete the questionnaire. The advantages of online distribution are clear—this method is easiest for both researchers and participants and may give people a greater sense of anonymity. Because this method is relatively new, the jury is still out on whether online distribution results in biased samples or biased responses. For readers interested in more information on designing and conducting Internet research, Sam Gosling and John Johnson’s (2010) recent book provides an excellent resource. In addition, several groups of psychological researchers have been attempting to understand the psychology of Internet users (you can read about recent studies on this website: http://www.spring.org.uk/2010/10/internet-psychology.php).

Jupiterimahes/thinkstock

Questionnaires can be distributed in three primary ways: by mail, in person, or online.

new66480_04_c04_p133-172.indd 140 10/31/11 10:38 AM

http://www.spring.org.uk/2010/10/internet-psychology.php
CHAPTER 4Section 4.2 Questionnaire Design

Advantages and Disadvantages of Questionnaires Just as with interview methods, written questionnaires have their own set of advan- tages and disadvantages. Written surveys allow researchers to collect large amounts of data with little cost or effort, and they can offer a greater degree of anonymity than interviews. Anonymity can be a particular advantage in dealing with sensitive or poten- tially embarrassing topics. That is, people may be more willing to answer a question- naire about their alcohol use or their sexual history than they would be to discuss these things face-to-face with an interviewer. On the downside, written surveys miss out on the advantages of interviews because there is no one available to clarify confusing ques- tions. Fortunately, there is one relatively easy way to minimize this problem—write questions that are as clear as possible. In the next section, we turn our attention to the process of questionnaire design.

4.2 Questionnaire Design

One of the most important elements in conducting survey research is in deciding how to construct and assemble the questionnaire items. In some cases, you will be able to use questionnaires that other researchers have developed in order to answer your research questions. For example, many psychology researchers use standard scales that measure self-esteem, prejudice, depression, or stress levels. The advantage of

these ready-made measures is that other people have already gone to the trouble of making sure they are valid and reliable. So, if you are interested in the relationship between stress and depres- sion, you could distribute the Perceived Stress Scale (Cohen et al., 1983) and the Beck Depression Inventory (Beck et al., 1996) to a group of partici- pants and move on to the fun part of data analyses.

However, in many cases, there is no perfect mea- sure for your research question—either because no one has studied the topic before or because the current measures are all flawed in some way. When this happens, you will need to go through the process of designing your own questions. In this section, we discuss strategies for writing questions and choosing the most appropriate response format.

Five Rules for Better Questions

Each of the rules listed below is designed to make your questions as clear and easy to understand as possible in order to minimize the potential for error variance. We discuss each one and illustrate them with contrasting pairs of items, consisting of “bad” items that do not follow the rule, and then “better” items that do.

iStockphoto/thinkstock

Using simple language is one way to craft an effective questionnaire.

new66480_04_c04_p133-172.indd 141 10/31/11 10:38 AM

CHAPTER 4Section 4.2 Questionnaire Design

1. Use simple language. One of the simplest and most important rules to keep in mind is that people have to be able to understand your questions. This means you should avoid jargon and specialized language whenever possible.

BAD: “Have you ever had an STD?” BETTER: “Have you ever had a sexually transmitted disease?” BAD: “What is your opinion of the S-CHIP program?” BETTER: “What is your opinion of the State Children’s Health Insurance Program?”

It is also a good idea to simplify the language as much as possible, so that people spend time answering the question rather than trying to decode your meaning. For example, words like assist and consider can be replaced with words like help and think. This may seem odd—or perhaps even condescending to your partici- pants—but it is always better to err on the side of simplicity. Remember, if people are forced to guess at the meaning of your questions, these guesses add error vari- ance to their answers.

2. Be precise. Another way to ensure that people understand the question is to be as precise as possible in your wording. Questions that are ambiguous in their wording will introduce an extra source of error variance into your data because people may interpret these questions in varying ways.

BAD: “What kind of drugs do you take?” (Legal drugs? Illegal drugs? Now? In college?) BETTER: “What kind of prescription drugs are you currently taking?” BAD: “Do you like sports?” (Playing? Watching? Which sports??) BETTER: “How much do you like watching basketball on television?”

3. Use neutral language. It is important that your questions be designed to measure your participants’ attitudes, feelings, or behaviors rather than to manipulate these things. That is, you should avoid leading questions, which are written in such a way that they suggest an answer.

BAD: “Do you beat your children?” (Clearly, beating isn’t good; who would say yes?) BETTER: “Is it acceptable to use physical forms of discipline?” BAD: “Do you agree that the president is an idiot?” (Hmmm. . . I wonder what the researcher thinks. . . ) BETTER: “How would you rate the president’s job performance?”

This guideline can also be used to sidestep social desirability concerns. If you suspect that people may be reluctant to report holding an attitude, such as using corporal punishment with their children, it helps to phrase the question in a non- threatening way—”using physical forms of discipline” versus “beating your chil- dren.” Many current measures of prejudice adopt this technique. For example, McConahay’s “modern racism” scale contains items such as “Discrimination