Other Questions That Arise in Discussions with Potential Sponsors
At UMTRI, we pay special attention to the selection of test subjects. Standard practice is to test both young (30 and under) and mature drivers (65 and above) with an equal number of men and women in each age group. Increasingly, the older population is being partitioned into two age groups, 65-75, and over 75. In some cases, a middle aged group has also been examined. Drivers are recruited either from newspaper ads or from previous subject lists. Participants come from Ann Arbor, Ypsilanti, Saline, and other surrounding communities, which provides a mix of subjects from various educational, occupational, and socioeconomic groups. We usually do not sample based on income or other variables. Such factors are often irrelevant to driver performance. They result in samples that are overcontrolled and cannot be obtained at a reasonable cost.
Even though Ann Arbor is a college town, it is extremely rare that only students participate in an experiment. Where special populations are sampled (e.g., truck drivers, garage hoist operators, etc.), UMTRI has been successful in recruiting desired participants. However, for reasons of cost and scheduling uncertainty, we generally do not recruit samples of owners of a particular product.
The number of people required for an experiment depends on the issues to be explored. Preferably, 100 respondents would be required in order to obtain a desired percentage statistic. However, for performance studies (e.g., the time to read a display) fewer people (typically 20-30) are needed with numerous responses obtained from each participant. Typically over 1-2 hours are needed to get better performance estimates (response times, error rates, glance durations, etc.). For example, the time it takes to read an in-vehicle display is primarily affected by driver age and physical characteristics such as visual acuity. Human performance is largely unaffected by the personal characteristics that influence preference for vehicles and vehicle features (education, income, etc.). Many engineers unfamiliar with human factors research think that every experiment should require hundreds of drivers. However, from a scientific perspective, this is not necessary to obtain meaningful results.
For virtually every project, UMTRI produces a detailed, scholarly technical report describing the problem, how the project was conducted, the results including statistical analyses, and conclusions or recommendations. The number of pages in a final report varies typically from 50-300 pages. Our reports must meet the standard criteria of being "publicly replicable". That is, given the final report, a reasonably knowledgeable person should be able to duplicate the results within the limits of statistical error (and reach the same conclusions).
In addition, all reports include an Americanized version of a Japanese style A3 summary. An A3 summary is a highly graphical, faxable representation, usually on two 8-1/2 x 11 pages, summarizing the project issues, methods, and results with minimal text to facilitate translation. A3 summaries are particularly useful for managers and others not directly involved in a project who need a quick overview.
To assure that reports are of high quality, every UMTRI technical report undergoes a rigorous internal review before being released to sponsors. All draft reports must first be reviewed by the Human Factors Division Head, the UMTRI Library, and either the Director or Assistant Director. It usually takes about a month for reports to be reviewed and revised.
About a month before a final report is sent to a sponsor, a (nearly final) draft copy is sent to the sponsor for review. Sponsors may suggest changes for the final report and identify points that may have been overlooked. The number of final copies provided to the sponsor is usually specified in the contract, with 10 copies being typical, though the number has varied from 3 to 100. Sponsors are free to produce additional copies for themselves or others.
After a 6-month waiting period, released reports are sent to the National Technical Information Service for cataloging and may be disseminated to others. (With special dispensation from the Vice President for Research, that waiting period can be negotiated upward.) If the report contains patentable information, public release of that portion can be delayed for a reasonable period to allow for patent filing. Over the last 15 years, there has been only 1 Human Factors Division project where anything patentable was considered. The project director may introduce additional delays (for example, a year until a related product is announced). UMTRI is careful about not releasing proprietary information and includes in reports only details necessary for the complete description of human performance studies. Thus, when information becomes publicly available this represents a compromise of the University's need to publicly disseminate research results and the need of industrial sponsors to maintain a competitive advantage.
In keeping with the University's policy on the general dissemination of research findings (a common policy of all research universities), the results of most studies are presented at conferences (such as the Society of Automotive Engineers and the Human Factors and Ergonomics Society) and published in journals (such as Human Factors, Ergonomics, and the International Journal of Vehicle Design). Normal lags between the completion of research contracts and publications in scholarly journals is 1-2 years. Sponsors may be asked to support travel to such meetings and the production of journal articles, and their support is acknowledged. Publications which have received sponsor support are submitted to them for comment.
Our experience has been that publications invariably are of great benefit to the sponsor. They provide a concise description of the results and serve to further disseminate the results within the sponsor's organization. Further, standard publication lags mean the results are not published until well after the associated product has been marketed.
In addition to reports, papers, and journal articles, UMTRI has also produced videotapes of its research. Some sponsors have found videotapes to be more informative reviews of test protocols than written reports. The Human Factors Division has modest facilities for producing 1/2-inch VHS tape. If a sponsor wants a broadcast-quality videotape, one can be produced by the Michigan Engineering Television Network. Using the industry rule of thumb of $2000/minute, a videotape of a single project produced typically costs $10,000 to $20,000.
The most effective means of keeping sponsors informed of progress are regular phone contacts and electronic mail. Written progress reports are produced if requested by the sponsor. Sponsors should realize that producing progress reports eats into project funds and time. When pilot testing is in progress, sponsors are urged to visit UMTRI to review the protocol.
Has the sponsor given us materials (instrument panels, headlamps, etc.) to keep us current of their interests independent of specific research projects? When potential sponsors take the initiative, it indicates they are very interested in a long-term collaborative relationship.
Is the work basic research or are the results generalizable? While we do all types of experiments, the University does not encourage routine product tests that do little to advance science.
Is funding budgeted for the project? Only when funding is set aside is it reasonably certain a sponsor will pursue a project. If funding is not budgeted, the next available opportunity for new funding is the next fiscal year, on average 6 months away.
How much money is involved? There is a certain amount of work that must be done to administer a project regardless of its size. Small projects may be less cost-effective in that a great deal of time is spent figuring out what to do and little time doing it.
Does the sponsor agree with the University policy on information dissemination? Proprietary research is not encouraged at the University of Michigan. To verify there are grounds for an agreement, early on the potential sponsor may be asked for a letter accepting the University's policies on information dissemination and other matters.
Does the sponsor want a report on the research with conclusions and recommendations? We rarely just collect data (serve as "subject runners"). Carrying out the analysis and producing reports with conclusions is part of doing quality scientific research. Further, usually people who collected the data are in the best position to interpret it. We have no problems including disclaimers concerning the conclusions for sponsors.
Does the sponsor's liaison have Human Factors research experience? Experienced liaisons understand the difficulties in conducting such work, where problems can occur, and the benefits of basis scientific research.