Most of the research on case-based science teaching at the university level involves comparative studies, which contrast student perceptions of case-based science teaching with more traditional teaching. Sometimes this type of research is being done to convince colleagues that case-based science teaching is equally effective if not better than traditional teaching. Such research has also been “reactive,” where proponents of case-based instruction gather evidence to support their position and counter criticism from opponents (Williams 1992), rather than attempting to understand the dynamics and processes of case-based teaching. And, of course, all instructors who are trying innovative teaching methods are interested themselves in knowing if the time and energy they are investing in the new method is worth the effort.

But whatever the motivation for the research, there are many ways to go wrong. For example, sometimes instructors teaching only one class will conduct a pretest and a posttest without collecting any process data. In this situation we cannot tell without additional qualitative data on processes whether the gain is due to the case, practice with the test, or other uncontrolled factors (Campbell and Stanley 1966; Pressley, Graham, and Harris Forthcoming).

It is natural for faculty to select paradigms with which they have familiarity, and in the sciences, measurement is a careful, precise aspect of the research method. In science we have had a bias toward counting, as expressed in this quote from Sir William Thompson, Lord Kelvin (1824–1907):

When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge of it is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced it to the stage of science.

However, measurement is not as precise in educational research as in the sciences, and a strict focus on quantitative measurement may limit methodology as well as research questions.

In the social sciences and education, such quantitative measures are sometimes difficult to generate; in any case, a statement about the nature and estimated magnitude of error must be made in order to signal the level of certainty with which conclusions have been drawn. Perhaps most importantly, the reasoning about evidence should identify, consider, and incorporate, when appropriate, the alternative, competing explanations or rival “answers” to the research question. (Shavelson and Towne 2002, p. 68).

Williams (1992) highlighted problems teachers face in assessing student learning when using case-based and problem-based approaches. First, even though the students are learning via the case-based method, they are often being assessed using standardized objective tests, predominately multiple-choice assessments. Second, classes with a large number of students make it difficult to use written tests of performance skills throughout the semester. However, because it is important to investigate whether students are able to abstract the general principles from the study of cases so that they are able to apply those principles and transfer them to other situations, we need to develop better and efficient methods for assessment.

A factor that influences the design of classroom research is a bias towards comparison conditions, or what some researchers refer to as “horse race” studies—that is, research examining which approach “wins” in terms of student perceptions or achievement. The “winner” in these studies varies, depending on the kind of knowledge measured. For example, comparing two classes of students, one that experienced case-based teaching and one that experienced traditional lectures, might show no differences on a standardized test of critical thinking or the standard multiple-choice exam. How are we to interpret this? Without pretest information, it is impossible to judge gains between the two groups. Moreover, if the assessment task is not aligned with the goals of case-study teaching, the results will not be valid.

In the following three examples, we illustrate some problems with comparative studies that are not well controlled for instructor bias, not well planned to measure particular outcomes (comparing apples to oranges), and have such weak interventions they yield data impossible to interpret.

Examples of how not to do a study

Scenario I
One kind of problematic study involves instructor bias that influences students’ responses to survey items, leading to self-report bias. A typical example would be an instructor who uses case-based instruction for the first time and is very enthusiastic and decides to survey students to see if their perceptions of the value of case-based teaching match his own. The instructor quickly constructs a brief questionnaire, often with leading questions and/or questions that call for judgments based on experiences students may not have gained. After the class is concluded, this instructor might ask students whether the discussions he led in case-study teaching required them to think critically. If he was explicit about this being one of his course goals, and had developed good rapport with the class, students may agree with this perception, just because they want to tell him what they know he wants to hear, regardless of their awareness of any growth in critical thinking on their part. This can be problematic, especially if students have “operationalized” critical thinking totally differently than the instructor.

Another difficulty with surveys is given in the next example in which the instructor might ask students whether they liked his case-based instruction more than they would have liked traditional lecture. This is an impossible question for students to answer accurately, since they have not experienced a traditional lecture with that particular class. A variation on this same theme is asking students to compare a previous biology course to this one; this kind of leading question is not likely to provide meaningful information, due to the inherent time bias and memory problems associated with prior schooling. Moreover, students generally respond positively to this kind of question because they do not really know how they might have liked traditional lecture in a particular class and may want to please the instructor, given the instructor’s obvious bias toward cases.

Scenario II
In this second example, the instructor compares grades from the case-study section that she taught with grades from all sections of introductory biology. She hopes to show that the case-study section had fewer Ds, Fs and withdrawal grades than the sections taught with more traditional lectures, and to argue that the case-study section provided evidence that students who do not traditionally do well in introductory science courses learned more with case-study teaching. Now, suppose the case-study section does meet her expectations. Let’s presume that the instructor did not use tests; rather, her students completed open-ended assignments she scored with rubrics. Clearly there are several problems here: Different kinds of tests (open-ended essays or projects or papers) may produce different grades than multiple-choice tests. Furthermore, different instructors have different levels of enthusiasm, knowledge and clarity, as well as different levels of skill in facilitating student ideas. Given these kinds of extraneous variables, it is difficult to know precisely what grade comparisons among sections of a particular course really mean if that is the only kind of data collected.

Scenario III
This type of problematic study uses an intervention that is both too weak and too brief to make a difference. For example, an instructor who is new to the case method decides to try one case discussion in one section of a course and to use a lab that he has used for years in another section of the same course. To see whether cases improve students’ critical thinking, he administers the multiple-choice exam he has always used that primarily tests for factual scientific knowledge of terms at the end of the unit.

This design is troublesome for four reasons: First, a brief exposure to a case is unlikely to have a strong effect on student thinking or knowledge. Second, the instructor is comparing a pedagogical method in which he is inexperienced (cases) with a method (labs) he is very experienced in using, so his skill may be a confounding factor. Third, the instructor is using the same assessment without considering whether that test measures the kind of critical thinking or conceptual understanding he assumes cases are developing. Finally, if the instructor had consulted the literature, he would have found a meta-analysis showing that the type of exam given to students generally determines gain-score effects in their critical thinking (Dochy et al. 2003).

Comparative studies of whether case treatment is better than control treatment (typically large lecture) may not add to the knowledge base on case-study teaching unless we also examine in what contexts, with what populations, and under what conditions these interactive methods affect learning and/or motivation. Furthermore, to demonstrate whether significant differences are practical, meaningful differences, researchers also need to supply effect size information to augment statistical tests. Effect sizes provide a perspective on the importance of a treatment effect, independent of sample size.

What do we know in spite of the problems?

Because they have been easy to count, we have considerable information about faculty and student perceptions of the value of case-based teaching, student attendance in class, and perceptions of changes in student learning and motivation. In a recent national survey we found faculty believed that students’ critical thinking increased and their understanding deepened when learning via case-based instruction (Yadav et al. 2006). Faculty reported students in the classes using case studies demonstrated stronger critical-thinking skills (89.1%), were able to make connections across multiple content areas (82.6%), and developed a deeper understanding of concepts (90.1%). Most of the faculty perceived when they used case-based teaching that students were better able to view an issue from multiple perspectives (91.3%), and were more engaged in the class when using cases (93.8%).

Research by Hoag and colleagues (2005) has shown that students tend to attend class more on days when cases, rather than lectures, are used. However, on days in which students discussed cases, Hoag gave points for turning in responses to the cases, so this extraneous variable (points) is a confounding factor in interpreting these results. Furthermore, students believe that content is easier to remember and apply and they enjoy it more when using case studies (Hoag et al. 1999). While perceptual outcomes based on self-report data have been relatively easy to measure, a greater challenge has been to study what students actually learn using the case-study teaching approach (Lundeberg 1999).

Performance in learning (other than overall grades) is not often assessed in case-based teaching because it requires careful construction of appropriate measurement tasks to assess student understanding and academic performance. Do students learn more from a case study approach to teaching science as compared to a more traditional approach using primarily lectures? The answer depends on how one measures learning. According to a recent meta-analysis examining 43 research studies on problem-based learning (one variation of case-based teaching used primarily in medical schools), there are significant differences on skills gained (that is, clinical application of knowledge and higher-order thinking). All of the skill differences favored problem-based learning. However, there were no significant differences in knowledge on standardized measures of accumulated knowledge gained in medical school (Dochy et al. 2003). Given that the case-study approach focuses on depth of understanding rather than breadth, this result is not surprising. When the assessment task was open-ended (e.g., recall, short answer, simulation, oral, essay) and/or when the assessment task measured critical thinking rather than basic knowledge, students in the PBL group showed higher gains in performance as compared to the traditional group. Essentially, the higher the level of knowledge and thinking required on the assessment task, the more likely that case-based teaching will produce greater gains in student understanding.

To measure critical thinking requires careful formulation of questions, and to construct such questions may require preliminary research. For example, a group of physicians and faculty teaching physiology (Michael et al. 2002) collected open-ended written explanations from students to discern reasons for some prevalent misconceptions in physiology. From this qualitative research, they constructed a two-tier test that assessed both misconceptions as well as reasons for these misconceptions. In physics, Mazur (1998) advocated assessing students’ understanding of basic physics concepts using what he called paired problem testing. In the paired problem testing, students are assessed using a traditional problem and a qualitative conceptual problem. Mazur found that students would solve the traditional problems easily by plugging the numbers into right equation. However, they did not fare well when solving the conceptual problems, indicating that students had little understanding of what the equations used to solve traditional problems really meant. We have also found in our research on teaching with multimedia cases in biology that the kind of assessment task matters greatly. Tests that require students to engage in higher-order thinking, such as interpreting data from simulations and cases, generally produce higher performance among students than do traditional multiple-choice course exams (Bergland et al. Forthcoming).

Creating assessments such as the ones described above provides a way to assess whether students form conceptual understanding of important, complex principles. Einstein reminded us, “Not all that counts can be counted; not all that is counted, counts.” Rather than relying on tests that can be easily scored, or simply calculating differences in grades among sections, researchers may need to begin with more open-ended assessments to understand students’ reasoning and their application of knowledge.

Future questions worth investigating using classroom research

With cases being used more and more for science instruction, it is important to address what kind of research questions we might ask and what gaps in the literature need to be filled. There has been very little empirical research investigating how students learn from case-based teaching. How do students represent knowledge in a particular discipline? Previous research has indicated that problem solvers represent problems by category and these categories in turn have an influence on problem solving. Experts and novices differ in their categorization of problems, the process of inquiry, and problem-solving techniques (Chi, Feltovich, and Glaser 1981). Researchers studying case-based instruction might explore the influences of group tasks, social processes, and the social influence on group effectiveness when using cases. If we want to understand case-based instruction, we need to know more about who learns what from cases and why. Examples of fruitful questions to pursue include the following:

What misconceptions do students bring to instruction, and how are they affected by different case-based approaches designed to dispel them? When in the instructional process are case-based approaches most promising? More specifically, how much background knowledge is necessary for case-based learning to be effective?

Are some case-based approaches particularly appropriate for online or computer-based settings versus discussions led by the instructor? Are these approaches more appropriate in some disciplines and contexts than in others? Under what conditions do video, computer simulations, and other emerging representational tools enhance or interfere with the case-based learning experience?

Future research in case-based instruction might also promote a deeper understanding of case-teaching approaches that make science accessible to all students, promote the learning of disciplinary concepts and habits of mind, and portray science in relevant, interdisciplinary contexts (Lundeberg and Moch 1995; NRC 1996; Handelsman et al. 2004). In all such studies, important questions concern how case-based methods facilitate understanding and engagement for different student populations (Lesh and Lovitts 2000).

Faculty members use cases in a variety of ways, so it is important to understand the kind of cases used, how a case is presented, and how student learning is assessed. For example, we are investigating how learning might be affected by the use of video cases about HIV/AIDS compared to text-based cases (Dirkin et al. 2005). We, the authors, argue that it is important for researchers to describe the context of case study teaching, including the kind of cases used, in order to understand the issues and processes surrounding the use of case-based instruction.

Mary A. Lundeberg (mlunde@msu.edu) is a professor in the Department of Teacher Education and Aman Yadav is a doctoral candidate in the Learning, Technology, and Culture program in the Department of Teacher Education at Michigan State University in East Lansing, Michigan.

References

Bergland, M., M.A. Lundeberg, K. Klyczek, D. Hoffman, J. Emmons, C. Martin, K. Marsh, J. Sweet, J. Werner, and M. Jarvis-Uetz. Forthcoming. Exploring biotechnology using case-based multimedia. American Biology Teacher.
Campbell, D.T., and J.C. Stanley. 1966. Experimental and quasi-experimental designs for research. Chicago: Rand McNally.
Chi, M., P. Feltovich, and R. Glaser. 1981. Categorization and representation of physics problems by experts and novices. Cognitive Science 5 (2): 121–52.
Dirkin, K., A. Yadav, M. Phillips, M. Koehler, K. Hilden, and M. Lundeberg. 2005. Is the movie better than the book? Differences in engagement and delayed recall of video and text cases in science. In Proceedings of world conference on educational multimedia, hypermedia and telecommunications 2005, 4196–201. Norfolk, VA: AACE.
Dochy, F., M. Segers, P. Van den Bossche, and D. Gijbels. 2003. Effects of problem-based learning: A meta-analysis. Learning and Instruction 13 (5): 533–68.
Handelsman, J., D. Ebert-May, R. Beichner, P. Bruns, A. Chang, R. DeHaan, J. Gentile, S. Lauffer, J. Stewart, S.M. Tilghman, and W.B. Wood. 2004. Scientific teaching. Science 304 (5670): 521–22.
Hoag, K.A., J.K. Lillie, and R. Hoppe. 2005. Piloting case-based instruction in a didactic clinical immunology course. Clinical Laboratory Science 18 (4): 213–20.
Lesh, R., and B. Lovitts. 2000. Research agendas: Identifying priority problems and developing useful theoretical perspectives. In Handbook of research design in mathematics and science education. eds. A. Kelly and R. Lesh, 45–71. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Lundeberg, M.A. 1999. Discovering teaching and learning through cases. In Who learns what from cases and how: The research base for teaching and learning with cases. eds. M.A. Lundeberg, B.B. Levin, and H. Harrington, 3–24. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Lundeberg, M.A., B.B. Levin, and H.L. Harrington. 1999. Where do we go from here? Reflections on methodologies and future directions. In Who learns what from cases and how: The research base for teaching and learning with cases. eds. M.A. Lundeberg, B.B. Levin, and H. Harrington, 231–40. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Lundeberg, M.A., and S. Moch. 1995. The influence of social interaction on cognition: Connected learning in science. Journal of Higher Education 66 (3): 310–335.
Mazur, E. 1998. Moving the mountain: Impediments to change. In Proceedings of the third annual NISE forum. ed. S.B. Millar, 91–93. Washington, DC: NISE.
Michael, J.A., M.P. Wenderoth, H.I. Modell, W. Cliff, B. Horwitz, P. McHale, D. Richardson, D. Silverthorn, S. Williams, and S. Whitescarver. 2002. Undergraduates’ understanding of cardiovascular phenomena. Advances in Physiological Education 26: 72–84.
National Research Council (NRC). 1996. National science education standards. Washington, DC: National Academy Press.
Pressley, M., S. Graham. and K. Harris. Forthcoming. The state of educational intervention research as viewed through the lens of literacy intervention. British Journal of Educational Psychology.
Shavelson, R.J., L. Towne, and the Committee on Scientific Principles for Education Research, eds. 2002. Scientific research in education. Washington, DC: National Academy Press.
Shulman, L.S. 1992. Towards a pedagogy of cases. In Case methods in teacher education, ed. J.H.Shulman, 1–33. New York: Teachers College Press.
Williams, S.M. 1992. Putting case-based instruction into context: Examples from legal and medical education. The Journal of Learning Sciences 2 (4): 367–427.
Yadav, A., M.A. Lundeberg, K. Dirkin, N. Schiller, and C.F. Herreid. 2006. National survey of faculty perceptions of case-based instruction in science. Paper presented at the annual meeting of American Educational Research Association, San Francisco, CA.

Acknowledgment

This material is based upon work supported by the NSF under Grant No. 0341279. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.