Testing Tomorrow's Surveys Today

Computers may be the future of survey data collection, but first U-M researchers need to develop the tools that will make respondents feel at ease with the machines.

By Diane Swanbrow
ISR Sampler Magazine

In a brand-new facility in the remodeled Perry School on Packard Street, the Institute for Social Research is experimenting with the future of survey research.

They call her Victoria, and she's not like any survey interviewer you've ever seen.

Victoria is an avatar, a talking, expressive animated agent on a computer screen who can administer a survey. With support from the National Science Foundation, ISR researchers are analyzing how Victoria influences respondents in self-administered computerized interviews.

"We're doing a series of lab experiments to explore which features of these agents improve respondent performance and satisfaction, and which hurt," says ISR researcher Fred Conrad, principal investigator of the project.

As face-to-face and telephone interviews conducted by real people become more expensive and more difficult to complete, survey research is increasingly moving toward self-administered questionnaires, using the web and automated telephone systems. These kinds of methods are not only more efficient, they're also more likely to elicit truthful responses to sensitive questions. But many details are unresolved, including how natural a computerized voice needs to sound and how animated an avatar's facial expression needs to be, in order to encourage responses.

Instead of building a series of costly animated avatars with varying features and capacities, Conrad and colleagues at the U-M, the New School for Social Research, and the University of Memphis are using what they call a Wizard-of-Oz technique. They are simulating animated avatars, like Victoria, using videos of real people. The human interviewers actually interact with test respondents, but respondents only see an animated version of the interviewer.

"Because respondents believe they are interacting with actual animated agents and not human interviewers, this research will help us understand the way new data collection methods blur the traditional distinction between self- and interviewer-administered surveys," says Conrad. More generally, the research should help in designing user interfaces that promote high-quality data. And that may not always mean using animated agents, such as when questions are sensitive.

Another cutting-edge project being conducted at the new facility is the use of eye-tracking to study survey response processes.

Roger Tourangeau and colleagues at the University of Maryland/University of Michigan/Westat Joint Program in Survey Methodology are investigating visual context effects - how images affect survey responses in web surveys – and other issues concerning the visual character of web surveys.

In a recent experiment with 117 respondents recruited through ads and flyers, Tourangeau and colleagues, including Fred Conrad and Mick Couper at ISR, employed TOBII, an unobtrusive eye-tracking device from Sweden that uses near-infrared beams and video to capture participant eye movements without the need for cumbersome lenses or helmets. In one study, they investigated the issue of "banner blindness," the assumption that visual images are not as influential and interesting when they appear in a website survey header as when they are located in the question area.

TOBII found that subjects were indeed less likely to look at pictures in a website header: 81% of the subjects looked at the pictures in the question area versus 64% who looked at the pictures in the header. Subjects also spent more time looking at pictures in the question area than in the header. Tourangeau and colleagues found that the content of the picture mattered, too. Subjects looked at photos of a happy woman more often than they looked at photos of a sad one, and for longer periods of time. But oddly enough, those who looked at the happy woman said they were less happy themselves than those who looked at the sad woman - a contrast effect, according to Tourangeau.

The new instrument development lab includes an observation room with one-way mirrors that permits viewing the computer testing stations and the conference room where focus groups and one-on-one interviews are held to make sure respondents understand survey questions the way investigators intend.

Since the new ISR Perry Building also houses classrooms for the Program in Survey Methodology, the ISR Summer Institute, and the ICPSR Summer Program in quantitative Methods, the lab is expected to advance ISR's educational mission as well as enhance its research on survey methods.

"The new facilities offer a great mix of services that knit together survey methods research, education, and survey operations," says Patty Maher, associate director of the operations unit.