MJCSL Special Issue (2000)

There has been a flood of service-learning research in the 1990s and most studies have focused on the impact of service-learning programs on students. Unfortunately, this research is weakest in both concept and methodology precisely in the areas where we need the most guidance if we are to design powerful academic programs. We know that service has something important to contribute to personal and social development; we have less reason to be confident that uniting it with academic work improves learning. Intellectual outcomes — knowledge, cognitive development, problem-solving skills, and transfer of learning — are at the heart of the school and college mission and yet we know relatively little about how they are affected by service-learning. So while we will touch on the many student outcomes documented in the literature, the primary focus of our recommendations for future research will address gaps in our understanding of the academic learning goals of service-learning and the instructional processes needed to achieve these goals.

What We Know

Just in the past ten years there have been several national studies that address the impact of service-learning on students (Astin & Sax, 1998; Eyler & Giles, 1999; Gray, Ondaatje, Geschwind, Fricker, Goldman, Kaganoff, Robyn, Sundt, Vogelsang, & Klein, 1999; Melchior, 1997) and dozens of smaller scale studies (Alt & Medrich, 1994; Andersen, 1999; Eyler, Giles & Gray, 1999). Most of these have explored the impact of service-learning on such personal qualities as efficacy, interpersonal skills, reduced stereotyping, and on social responsibility or sense of commitment to future service. This body of research consistently shows a small but positive effect of service-learning on these outcomes. Studies which have examined the impact of quality differences in service-learning have found that programs with more opportunity for reflection, substantive links between coursework and service, and ethnic and cultural diversity have a stronger impact (Eyler & Giles, 1999; Gray et al, 1999; Mabry, 1998). While we lack longitudinal studies that show a link between academic service-learning and later civic involvement, there is evidence in the youth development literature (Youniss, McLellan & Yates, 1997), as well as the work of Astin, Sax and Avalos (1999) in higher education, that volunteer service leads to subsequent community involvement. The mediating factor appears to be the development of civic identity, i.e. the personal efficacy and social responsibility that are the outcomes of both community service and service-learning. The effect of service-learning on cognitive outcomes has been less well studied and relatively little attention has been given to defining learning outcomes that would be expected to be enhanced by service participation. Most of the reports of learning are based on student self reports or faculty testimony (Eyler, Giles & Gray, 1999). Where attempts have been made to use grades as measures of learning, the evidence is mixed (Berson & Younkin, 1996; Markus, Howard & King, 1993; Miller, 1994; Kendrick, 1996). In some cases where positive results are reported the method of calculating grades has differed for treatment and comparison students so that results can be attributed to different standards or ‘extra credit’ for service rather than increased learning (Balazadeh, 1996; Sugar & Livosky, 1988). In other studies, it is hard to disentangle the effects of selectivity on reports of better grades; do better students choose to become engaged in service or does service or service-learning lead to better grades (Astin & Sax, 1998; Melchior, 1997)? While few studies have attempted to articulate learning outcomes that might be particularly affected by the integration of service and learning, a handful of small studies have pointed the way and have used measures that allow students to demonstrate learning. They have provided evidence that service-learning has an impact on complexity of problem analysis, identification of locus of problem or solution, use of information to support arguments, creation of practical strategies for community action, cognitive moral development, and critical thinking (Batchelder & Root, 1994; Boss, 1996; Eyler & Giles, 1999; Eyler & Halteman, 1981). While these studies are encouraging, their small scale, lack of finely differentiated treatment conditions, and lack of replication limits their usefulness. We also have limited evidence to guide selection of service-learning strategies to enhance learning. Most studies of student outcomes have simply used ‘service-learning’ as the predictor variable and ‘service- learning’ covers dramatically different experiences ranging from a single visit to a soup kitchen, to a few hours a week of ‘extra credit’ for tutoring, to an intensive semester focused on working with a community partner on a social problem. It is hard to argue that experiences so varied should be linked to improved learning, except insofar as doing worthwhile service may lead to increased interest or engagement in study. And because outcomes have not been well specified, studies often link service experiences with outcomes that bear no logical relationship to them. Stacking soup cans in a food pantry is not likely to have an impact on public speaking; tutoring kids in math may have no impact on a multiple choice test in sociology; clearing a trash-filled city block is obviously not connected to improved critical thinking capacity. A few studies have explored the impact of program characteristics (Eyler & Giles, 1997). For example, twenty years ago Conrad and Hedin (1980) found that opportunity for frequent reflection was the best predictor of outcomes. Subsequent research has examined the impact of program characteristics such as amount of reflection, placement quality, application, feedback, ethnic and cultural diversity, and program duration (Eyler, Giles & Gray, 1999), but these are fairly gross measures of pedagogy. There have been no systematic attempts to test alternative, theoretically-anchored models of instruction, reflection or project planning. We know reflection is a good thing — but we don’t know how to structure reflection and integrate it with service to maximize learning — or what that learning might look like.

top

What We Need to Know

The national survey studies and single program efforts of the past decade have provided a map of service- learning and its impact on students, but it is akin to mapping terrain with a 30,000 foot fly over. We don’t have the detailed information that will help design programs that enhance cognitive outcomes. An important task for the next wave of research is to provide this information by designing research programs that clearly articulate intellectual outcomes of service-learning and provide information on the best way to structure the instructional process to attain those goals. Important way to move the field forward is to engage scholars in related fields in the process of designing and implementing the next wave of research.

Defining and Measuring the Cognitive Outcomes of Service-Learning

We are preparing students for an increasingly turbulent society, which requires adaptability, sophisticated knowledge, lifelong learning and problem solving skills (Davis & Meyer, 1998; Vaill, 1997). The traditional intellectual tasks of the classroom offer a poor match with the kinds of learning that students will need to practice in the workplace and the community (Resnick, 1987). But these descriptions of needed learning characteristics are remarkably similar to the qualities that have long been identified with service-learning (Astin, 1995; Barber, 1994; Ehrlich, 1997). While there are pedagogies for helping students amass factual knowledge that are substantially less demanding on the teacher than the rigors of service-learning, we need to explore outcomes which service-learning is uniquely fitted to promote. Service-learning, which at its best allows students to confront issues and problems in complex natural contexts, appears to be ideally suited to help students develop a deeper understanding of subject matter, a practical knowledge of how community decision-making processes work, and strategies for transferring knowledge and problem solving skills to new situations. There is also reason to believe that the ability to recognize ill-structured problems (Voss & Post, 1988) embedded in messy social contexts, to sort out conflicting information and views, and to resolve an issue while understanding that such resolutions are inherently tentative all rest on attaining advanced levels of cognitive development (Lynch, 1996). The ability to function in the face of uncertainty requires critical thinking abilities more advanced than those typically attained by American college students (King, 1992), and there is reason tothink that the challenges and support provided in some service-learning programs may facilitate this development (Eyler & Giles, 1999; Kitchener, Lynch, Fischer & Wood, 1997). A primary task for service-learning research then is to refine our definition of appropriate intellectual outcomes and to design measurements that are convincing. Our traditional reliance on self-report of learning, application and critical thinking is inadequate to the task. Self-report of learning is not only a weak measure of the complex cognitive outcomes we expect from service-learning, it also confuses satisfaction with learning. For example, Barron and her colleagues (1998) discuss a community project where elementary students visited local streams and monitored water quality as part of a unit on water pollution; students and faculty were enthusiastic aout the value of the project. Yet when students were tested on their knowledge, they demonstrated little understanding and had a number of misconceptions about water quality, including the belief that natural streams should test free of microorganisms as is true with swimming pools. It isn’t enough to enjoy a community project, and we don’t know that students have learned from service because they tell us that they have learned. Other items and scales suitable for use on surveys also fail to measure complex understanding, problem solving skills and cognitive development that may be the primary intellectual contribution of well designed service-learning. Insofar as traditional grades measure ‘replicative learning’ or memory of content, they are also weak measures of the outcomes we expect to see enhanced by service-learning. The challenge for educators is to help students develop the ability to apply what they have learned in new settings — and to problem solve. Tests of memory or simple application when a problem is identified for the student do not test these abilities. Problems in society are not identified and labeled; they don’t come with a syllabus and questions at the end of the chapter. Much of transfer of learning involves being able to recognize the need to systematically explore the new problem, i.e. preparing for transfer is “preparation for future learning” (Bransford & Schwartz, in press). In order to test the effectiveness of a particular service-learning program we might try to discover if students are able to generate a sensible strategy for tackling a new problem. We might examine whether they are able to monitor their own problem solving progress and are aware of the need to seek out further expertise. These are outcomes rarely tested in conventional classroom tests, but which are closely matched to the benefits expected from community-based learning projects. What is needed are measures that allow students to show us, rather then tell us, that they have attained greater understanding, ability to apply their knowledge, problem solving skills and cognitive development. There may be extant measures for testing some expected outcomes; others might be created specifically for a course or program, especially when our concern is with deeper understanding of particular subject matter or learning transfer. Creative measurement of students’ deep understanding and ability to transfer knowledge is something that cognitive scientists have been struggling with for some time; building on their work in project and problem-based learning will advance our ability to measure the learning outcomes of service-learning.

Testing Alternative Strategies for Structuring Reflection and Service Experiences

Once we have clearly identified and developed methods to measure the most important academic outcomes, the next step will be to identify the specific organizational and instructional strategies that increase the power of service-learning to add to these outcomes. In recent years, learning scientists — particularly those interested in situated cognition and project-based learning — have experimented with a variety of methods for helping students learn from experience. Their attention has focused on such elements of instructional design as student goal setting, scaffolding that supports student and teacher learning, frequent opportunities for formative self-assessment, and social organization that promotes participation and results in a sense of agency (Barron et al., 1998). One of the particular strengths of service-learning is that it produces what Dewey (1938) called an ‘educative experience’ i.e. it engages students in worthwhile activity which stimulates intellectual curiosity; thus it is likely to create social arrangements which lead to motivation and a sense of agency. But little attention has been paid in service-learning research to testing effective ways to help students set goals, to provide appropriate cues and information for problem solving or to facilitate the development of self monitoring skills. As David Moore (1999) notes in a recent analysis of student experiences during school to work programs, the learning activities and reflection that practitioners assume take place actually are often not occurring for students. There is a belief in the power of real world experience, but little guidance on how to increase the ability of students to learn from experience and integrate it with other sources of information. While both service-learning practitioners and cognitive scientists understand the importance of learning in complex contexts, cognitive scientists have tended to emphasize control of the many elements of the instructional situation and have often used simulated or highly structured experiences, whereas service-learning practitioners stress working in the inherently messier real world. Some cognitive scientists are venturing outside simulated experience to community-based projects so as to try to identify the factors that make such projects effective. For example, in a study exploring the learning from building and launching model rockets in a science class, it became clear that students who worked to meet needs of a client for information about alternative rocket designs learned more from their experience than students who simply enjoyed watching things explode into the air (Bransford & Schwartz, in press). The goal setting process used in that study, i.e. meeting the needs of a client, is consistent with one of the basic tenets of service- learning practice — reciprocity — a true partnership between students and community members (Sigmon, 1979). This is just one example of something that would be useful to test; exploring the impact of goal setting with community partners on the learning outcomes of service-learning projects might well yield some concrete guidance for practitioners. Cognitive scientists have experimented with a variety of ways to provide scaffolding or support for student problem solving and for providing them with experiences that prepare them to learn from projects like those undertaken in many service-learning classes. (Barron et al., 1998; Schwartz & Bransford, 1998). Simulations, exposure to contrasting case studies, and sequencing of increasingly complex projects are all instructional techniques that may enhance the ability of students to observe more clearly in a complex community setting and to perform at a higher level. Service-learning practitioners need to know how to design instruction to prepare students to get the most and give the most in their service projects. The ability to monitor one’s own learning and realize when it is necessary to seek further information is a vital element in the ability to understand complex subject matter and to transfer learning to new settings (Barron et al., 1998). It is also characteristic of higher levels of critical thinking ability (Lynch, 1996). Self-monitoring is part of the reflection that has long been regarded as central to service-learning, but we know relatively little about how to structure reflection for maximum effect. Learning scientists and service-learning professionals could profit from working together to design and test techniques for enhancing reflection.

top

What Do We Need to Do This Research?

A number of groups have sponsored efforts to identify ‘next steps’ in service-learning practice and research and it has been the topic of countless symposia at professional meetings. These efforts have tended to bog down, covering the same ground and surfacing perennial conflicts over process and ideology. We need to create a process to engage the expertise of people who have developed methodologies for studying field-based instruction and to bring them together with experienced service-learning researchers in order to design and implement a coordinated research program that will produce practical results for the field. Such a project would involve three major phases: the design of a research program, its implementation, and the dissemination of the results in ways that facilitate use.

An Interdisciplinary Task Force to Design Service-Learning Research

A task force that teams experts on learning, problem solving and transfer of learning with service-learning practitioners is an important first step in developing a concrete research program focused on the learning in service-learning. A small group — perhaps 8-10 service-learning researchers/practitioners and half a dozen learning scientists — would meet several times to identify appropriate learning outcomes of service-learning and to identify pedagogical approaches to curriculum design and reflection that would be tested across schools and colleges. This group would design a series of coordinated sequential experiments. Early efforts might be oriented towards specifying and measuring the unique outcomes that service-learning would be expected to produce. Later studies would systematically examine alternative ways of designing service-learning curricula and managing reflection. Participants would test these elements of effective design in a variety of settings, e.g., high school as well as college classes, but the studies would be designed to be comparable and help us accumulate knowledge about good practice.

Implementation of Research Program

Once the task force has crafted the research design, participants would implement it at their local institutions. The research program would be coordinated across sites, with centrally-provided resources, such as managing the scoring of complex assessment tools and the data analysis. Additional support would be needed at each research site to manage the application of the treatment pedagogy to the specific service- learning program, the implementation of the experiments and the data collection. Data from early waves of the research would help shape the later studies; each site would participate in several studies as treatments and measures were refined.

Dissemination of Research-Based Instructional Guidelines

The research process should yield approaches to measurement that would help practitioners build convincing assessment into their courses. In addition, the testing of alternative methods of program organization, instruction and reflection should also provide a basis for specific suggestions for design and implementation of service-learning courses. Practitioner materials would be created based on the research results and disseminated through professional conferences and through Learn and Serve’s National Service-Learning Clearinghouse.

Key Questions

Over the past decade we have accumulated a lot of evidence about the impact of service-learning on college students, but this research has relied on surveys and other simple measures which do not capture the most important intellectual outcomes of this experience. We know that service-learning has a small but consistent impact on attitudes and perceptions of self, but we have less evidence for its impact on learning and cognitive development and no evidence of its effect on lifelong learning and problem solving in the community. We also have few studies about different approaches to service-learning that would provide guidance to instructors on how best to optimize the impact of their service-learning courses on students. This suggests a number of conceptual and empirical questions for future research. First we need to have a clear idea of the intellectual outcomes that one might expect from well designed service-learning classes. Some questions, rooted in experiential learning theory, need more definitive empirical answers. These include:

• Can service-learning produce greater interest and engagement in subject matter?
• Can service-learning contribute to a deeper understanding of subject matter?
• Can service-learning promote cognitive development of post formal reasoning capacity?
• Can service-learning lead to increased ability to solve new problems?
• Can service-learning contribute to the development of practical political and community action skills and understanding?
• Will problem solving, community action and learning skills as well as knowledge acquired through service-learning be used by students throughout their lives in the community?

Second, we need to design convincing measures of the outcomes on which we choose to focus. Most of the intellectual outcomes we might expect from service-learning are what Winter, McClelland and Stewart (1981) would call “processes for operating on and using information rather than mere knowledge of facts and information” (p. 24). They noted that in assessing the benefits of a liberal education that “a concern for quick and easy measurement often has usurped a concern for the meaningful content of what is measured” (p. 22). Certainly the tendency in the service-learning research literature to assess critical thinking with a Likert response item or learning with self-report deserves this same criticism. Attention needs to be given to convincing measures of important outcomes. We need to ask question such as:

• How can we design measures of understanding and problem solving that allow students to demonstrate their competence rather than simply testify to it?
• How can we embed authentic assessment measures into the service-learning experience itself?
• What measures are appropriate for assessing long term community use for problem solving, community action and learning skills as well as knowledge acquired through service-learning?
• What tools used by cognitive scientists, developmental psychologists and others can be adapted for use in measuring important intellectual outcomes of service-learning?

Third, and perhaps most importantly for practitioners, we need to explore how particular instructional designs and techniques might enhance the intellectual power of service-learning. While current research hints at the importance of reflection, little attention has been given to providing empirical evidence of particular approaches to reflection. We need answers to questions such as:
• What kinds of preparatory activities increase the learning impact of service-learning?
• What kinds of scaffolding or support do students need to integrate their community experience and the subject matter objectives of the course?
• What techniques for reflection enhance various outcomes and when and under what circumstances are they employed to greatest effect?
• How do we increase students’ engagement in personal reflection and self-monitoring of their learning?
• How can projects be designed to increase students’ sense of engagement with community partners?
• How can community projects be shaped to enhance understanding of academic subject matter?
• What strategies for service-learning increase the likelihood of long term community use of problem solving, community action and learning skills?

top

Conclusion

Questions like these need to be answered to enable instructors to design experiences that will help students attain the cognitive objectives of service-learning courses. In order to create a theory-based and empirically-tested body of knowledge about design and implementation of effective service-learning, teams of researchers and practitioners need to conduct experimental studies that allow them to isolate particular instructional techniques and test their effectiveness. This body of knowledge can be created by combining the expertise of learning scientists already exploring problem-based learning with service-learning scholars. The past decade has established a firm empirical base; we know that service-learning has a small but consistent impact on a number of important outcomes for students. Now we need to push ahead to empirically answer questions about improving the academic effectiveness of service-learning.

References

Astin, A.W., Sax, L., & Avalos, J. (In press). Long term effects of volunteerism during the undergraduate years. Review of Higher Education.

Astin, A.W. & Sax, L. (1998). How undergraduates are affected by service participation. Journal of College Student Development, 39 (3), 251-263.

Barber, B. (1990). Service, citizenship and democracy: Civic duty as an entailment of civil right. In W. Evers (Ed.), National Service Pro and Con (pp. 27-43). Stanford: Hoover Institution Press.

Barron, B.J.S., Schwartz, D. L., Vye, N. J., Moore, A., Petrosino, A., Zech, L., Bransford, J.D. & the cognition and technology group at Vanderbilt. (1998). Doing with understanding: Lessons from research on problem and project-based learning. The Journal of the Learning Sciences, 7 (3&4), 271-311.

Batchelder, T. H. & Root, S. (1994). Effects of an undergraduate program to integrate academic learning and service: Cognitive, prosocial cognitive and identity outcomes, Journal of Adolescence, 17, 341-356.

Boss, J. A. (1994). The effect of community service work on the moral development of college ethics students. Journal of Moral Education, 23 (2), 183-198.

Boyer, E. L. (1994, March 9). Creating the new American college. The Chronicle of Higher Education, A48.

Bransford, J. D. & Schwartz, D.L. (in press) Rethinking transfer: A simple proposal with multiple implications. Review of Research in Education.

Conrad, D. & Hedin, D. (1980). Executive summary of the final report of the experiential education evaluation project. Center for Youth Development and Research, University of Minnesota.

Davis, S. & Meyer, C. (1998). BLUR: The speed of change in the connected economy. Masschusetts: Addison Wesley.

Dewey, J. (1938). Experience and education. New York: Collier Books. Ehrlich, T. (1997). Civic learning: Democracy and education revisited. Educational Record, Summer/Fall, 57-65.

Eyler, J., & Giles, D. E., Jr. (1997). The importance of program quality in service-learning. In A. Waterman (Ed.), Service-learning: Applications from the research (pp. 57-76). Hilldale, N.J: Lawrence Erlbaum Associates.

Eyler, J. & Giles, D. E. Jr. (1999). Where’s the learning in service-learning. San Francisco: Jossey-Bass.

Eyler, J., Giles, D. E. Jr. & Gray, C. (1999). At a glance: Summary and annotated bibliography of recent service-learning research in higher education. Minneapolis: Learn & Serve America National Service-Learning Clearinghouse.

Eyler, J. & Halteman, B. (1981). The impact of a legislative internship on students’ political skill and sophistication. Teaching Political Science, 9, 27-34. Gray, M. J., Ondaatje, E. H., Geschwind, S., Fricker, R., Goldman, C., Kaganoff, T., Robyn, A., Sundt, M., Vogelsang, L., & Klein, S. (1999). Combining service and learning in higher education. Santa Monica, CA: Rand.

Greene, D. & Diehm, G. (1995). Educational and service outcomes of a service integration effort. Michigan Journal of Community Service Learning, 2, 54-62. Kendrick, J.R. (1996). Outcomes of service-learning in an introduction to sociology course. Michigan Journal of Community Service Learning, 2, 72-81. King, P. (1992). How do we know? Why do we believe? Liberal Education, 78 (1), 2-9.

Kitchener, K., Lynch, C., Fischer, K. R & Wood, P. (1993). The developmental range of reflective judgment: The effect of contextual support and practice on developmental stages. Developmental Psychology, 29 (5), 893- 906.

Lynch, C.L. (1996). Facilitating and assessing unstructured problem solving. Journal of College Reading and Learning, 27, 16-27.

Mabry, J.B. (1998). Pedagogical variations in service-learning and student outcomes: How time, contact and reflection matter. Michigan Journal of Community Service Learning, 5, 32-47.

Markus, G.B., Howard, J. & King, D. (1993). Integrating community service and classroom instruction enhances learning: Results from an experiment. Educational Evaluation and Policy Analysis, 15(4), 410-419.

Miller, J.(1994). Linking traditional and service-learning courses: Outcome evaluation using two pedagogically distinct models. Michigan Journal of Community Service Learning, 1, 29-36.

Moore, D. T. (1999). Behind the wizard’s curtain: A challenge to the true believer. N.S.E.E. Quarterly, 25 (1), 1, 23-27.

Myers-Lipton, S.J. (1996). Effect of a comprehensive service- learning program on college students’ level of modern racism. Michigan Journal of Community Service Learning, 3, 44-54.

Resnick, L. (1987). The 1987 presidential address: Learning in school and out. Educational Researcher, 16(9), 13-20. Schwartz, D.L. & Bransford, J.D. (1998) A time for telling. Cognition and Instruction, 16 (4), 475-522.

Sigmon, R. (1979). Service-learning: Three principles. Synergist, 8, 9-11.

Vaill, P. (1997). Learning as a way of being: Strategies for surviving in a world of permanent whitewater. San Francisco: Jossey-Bass.

Voss, J. & Post, T. (1988). On the solution of ill structured problems. In M.Chi & R. Glaser (Eds.), The nature of expertise. Hilldale, N.J.: Lawrence Erlbaum Associates.

Winter, D.G., McClelland, D.C. & Stewart, A.J. (1981). A new case for the liberal arts. San Francisco: Jossey-Bass.

Youniss, J, McLellan, J. A. & Yates, M. (1997). What we know about engendering civic identity. American Behavioral Scientist, 40 (5), 620-631.

Author

JANET S. EYLER is Associate Professor of the Practice of Education, Peabody College of Vanderbilt University. With Dwight E. Giles, Jr. she co-directed a national FIPSE-funded service-learning research project “Comparing Models of Service-Learning” and a Corporation for National Service research project on learning outcomes for college students. Where’s the Learning in Service-Learning, published by Jossey-Bass in 1999, is drawn from these studies. The publication of A Practitioner’s Guide to Reflection in Service Learning: Student Voices and Reflection (1996), with Giles and Angela Schmiede, resulted from the Corporation grant. She has served as a consultant in program evaluation for businesses and universities as well as a consultant in service-learning practice and evaluation for colleges and universities, foundations, and professional societies. At present, she is a member of both the Campus Compact National Research Advisory Board and the Research Advisory Board of the Learning In Deed initiative sponsored by the Kellogg Foundation.

top