Skip to main content
 

Evaluating Information: The Impact of Major, Class Standing, and Experience With Primary Literature

Journal of College Science Teaching—January/February 2020 (Volume 49, Issue 3)

By Shawn Stover and Michelle Mabry


Social media platforms and other internet sites may propagate inaccurate or misleading information. This misinformation encourages speculation, rumors, and mistrust in established science. In the current study, we attempted to determine whether biology and environmental science students at a small college are able to transfer their critical thinking skills to the assessment of online information. Students were presented with examples of internet-based information sources and asked to rate their reliability. Regardless of major, class standing, or experience with primary literature, students accurately rated the internet sources. Students also provided written rationales for their ratings. Although freshmen and sophomores had unsophisticated rationales, juniors and seniors were likely to base their rationales on the presence or absence of evidence, peer review, and publication in a scientific journal. Juniors or seniors with some experience in the analysis of primary literature were most likely to provide sophisticated, evidence-based rationales for their ratings.

 

In a systematic review, Moorhead et al. (2013) identified several benefits of social media for health-related communication, including increased accessibility of health information, the potential for enhanced emotional support, and the possibility of improved public health surveillance. Furthermore, social media can be used effectively to engage the public and communicate health-related messages. During the 2009 H1N1 pandemic, for example, the Centers for Disease Control (CDC) used Facebook to educate the public about the disease and the importance of vaccination (Kass-Hout & Alhinnawi, 2013).

Unfortunately, social media platforms and other internet sites can propagate inaccurate or misleading information (Moorhead et al., 2013; Rutsaert et al., 2013). In some cases, facts may be combined with half-truths to generate “informational blends” (Bessi et al., 2015; Rojecki & Meraz, 2014). Consequently, the unregulated environment of the internet encourages speculation, rumors, and mistrust in the scientific/medical establishment (Del Vicario et al., 2016; Sunstein & Vermeule, 2009). Recently, in the midst of a particularly bad flu season, an article with the title “CDC Doctor: ‘Disastrous’ Flu Shot Is Causing Deadly Flu Outbreak” was published online. The article quoted an anonymous doctor at the CDC who claimed that the common denominator for people dying of the flu was the fact that they all had received flu shots. The entire story was a fabrication, yet it went viral on the internet (Keslar, 2018). Likewise, stories promoting “alternative” cancer treatments reach millions of readers online, even though actual laboratory research suggests that the use of alternative treatments instead of conventional care could double the risk of death from cancer (Johnson, Park, Gross, & Yu, 2018).

False information is easily disseminated online. On Twitter, the top 1% of false stories can reach as many as 100,000 people, but accurate stories rarely reach 1,000. Although true stories may inspire joy or sadness, fake stories tend to inspire fear or disgust. Those stories are much more likely to be retweeted (Vosoughi, Roy, & Aral, 2018). Once the false information is out there, it can be difficult to eradicate. Online campaigns meant to debunk erroneous ideas have actually reinforced the misconceptions through a phenomenon known as the “backfire effect.” Criticism can increase the familiarity of myths, overwhelm believers with too much information, or threaten the overarching worldview held by those advocating the myths (Bessi et al., 2015; Cook & Lewandowsky, 2011). Internet users tend to be embedded in “homogeneous clusters.” Their sense of identity may be greatly influenced by politics or religion, and these group allegiances can promote the formation of narratives that are self-confirming (Del Vicario et al., 2016; Stover, 2014).

Between January 2015 and June 2016, a group of researchers at Stanford University administered surveys to middle school, high school, and college students across 12 states to assess the ability to evaluate information on the internet. For example, college students at six universities were asked to: (a) decide if certain websites could be trusted, (b) verify online claims about controversial topics, (c) identify strengths and weaknesses of online videos, and (d) indicate whether specific tweets were reliable sources of information. Results of the surveys indicated that students, in general, are unable to critically evaluate internet content. Although students may be “fluent” in social media, they cannot effectively judge the information they find online. Many students made broad statements about the dangers of social media, without actually investigating the online content (Wineburg McGrew, Breakstone, & Ortega, 2016).

Davis and Elkins College (D&E) is a small, private liberal arts college that emphasizes small class sizes and strong faculty–student interactions. In the Department of Biology and Environmental Science, we introduce our students to the critical analysis of primary scientific literature as early as the sophomore year, and we continue to stress the importance of evidence-based analysis and data interpretation throughout the curriculum. In the current study, we attempt to determine whether our students are able to transfer their critical thinking skills to the assessment of online information.

Methods

In accord with official guidelines regarding research and educational practices involving human participants, this study was approved by the D&E Institutional Review Board. A total of 134 students, from six different biology/environmental science courses at D&E, were surveyed between the spring of 2017 and the spring of 2018. Each student signed a consent form and provided information regarding their major, class standing (freshman, sophomore, etc.), and the specific biology/environmental science courses they had completed. They were then presented with five paper-based examples of internet information sources. Some appeared exactly as they did online. In others, the names of real people and products were changed.

The first item was a Facebook post with a link to a Skeptical Inquirer article (Novella, 2014) regarding health risks associated with genetically modified organisms (GMOs). The article cited several studies that demonstrated the safety of GMOs. Results of the studies were summarized in the survey. The second item was the following Twitter post: “It should be pretty obvious that the idea of global warming is nonsense.” The post included a picture of icicles forming on a gutter above a window (Figure 1). Item number three was the abstract of a review article published in Proceedings of the National Academy of Science, USA (Wilson & Nowak, 2014). The article presented several lines of evidence indicating that natural selection drives the evolution of ant life cycles. The fourth item was a mock web page for an online vendor of dietary supplements. The site highlighted a product called “Snake Berry Tablets.” According to the site, the product is used to boost metabolism and prevent excess fat deposition. No information regarding scientific research, clinical trials, or FDA approval was provided. The final item was a press release from the American Physiological Society describing a study published in the American Journal of Physiology—Cell Physiology regarding the fitness and muscle characteristics of aging athletes (Power et al., 2016).

FIGURE 1
Image accompanying Twitter post in survey item 2.

Image accompanying Twitter post in survey item 2.

After each item, there was the statement: “The information in this (information source) is reliable.” Based on what was presented in the example, students decided how much they agreed with the statement by selecting a value from 1 (strongly disagree) to 5 (strongly agree) on a five-level, Likert-type scale. Students then briefly explained why they agreed or disagreed in a sentence or two. Likert responses were analyzed according to major, class standing, and experience with primary scientific literature, and median values were calculated. Using a rubric (Figure 2), students’ “Why?” responses were categorized by the two authors as Beginning, Emerging, or Mastery and were assigned 1, 2, or 3 points, respectively, to generate composite scores unrelated to the Likert responses. If a student were assigned a 3 (Mastery level) for each of the five survey items, that student would receive a composite score of 15, the highest possible score. Conversely, if a student were assigned a 1 (Beginning level) for each of the five items, that student would receive a score of 5, the lowest possible score. In an effort to ensure interrater reliability, Cohen’s kappa coefficient (κ) was calculated for the categorical evaluation of student rationales. Both authors categorized rationales, and κ > 0.70 was considered reliable. Composite rubric scores were analyzed according to major, class standing, and experience with primary scientific literature, and mean values were calculated. Composite scores were subjected to a multiple comparison analysis of variance (ANOVA). Fisher’s least significant difference test was employed to compare specific groups within the ANOVA. An alpha level of P < .05 was regarded as statistically significant. Composite score data are expressed as mean ± standard deviation.

FIGURE 2
Rubric used to analyze “Why?” responses.

Rubric used to analyze “Why?” responses.

Results

Median Likert-type scores indicated that students, regardless of major, class standing, or experience with primary literature, ranked the survey items similarly in terms of reliability (Figures 35).

FIGURE 3
Median Likert-type scores by major (biology, environmental science, exercise science, nursing, and other).

Median Likert-type scores by major (biology, environmental science, exercise science, nursing, and other).

FIGURE 4
Median Likert-type scores by class standing.

Median Likert-type scores by class standing.

FIGURE 5
Median Likert-type scores by experience with primary scientific literature.

Median Likert-type scores by experience with primary scientific literature.

The calculated value for κ was 0.74, indicating good interrater reliability. Mean rubric composite scores demonstrated no significant differences between majors (Figure 6). However, upperclassmen (juniors and seniors) had significantly higher composite scores than underclassmen (freshmen and sophomores; Figure 7). Furthermore, students who had some experience analyzing primary scientific literature had significantly higher composite scores than students with no experience (Figure 8).

FIGURE 6
Mean rubric composite scores by major (biology, environmental science, exercise science, nursing, and other). The maximum score is 15.

Mean rubric composite scores by major (biology, environmental science, exercise science, nursing, and other). The maximum score is 15.

FIGURE 7
Mean rubric composite scores by class standing. Values with different superscripts are significantly different (<i>P</i> < .05). The maximum score is 15.

Mean rubric composite scores by class standing. Values with different superscripts are significantly different (P < .05). The maximum score is 15.

FIGURE 8
Mean rubric composite scores by experience with primary scientific literature. Values with different superscripts are significantly different (<i>P</i> < .05). The maximum score is 15.

Mean rubric composite scores by experience with primary scientific literature. Values with different superscripts are significantly different (P < .05). The maximum score is 15.

Discussion

Although the Stanford study described students’ ability to critically evaluate internet information as “poor,” our results were more encouraging. Regardless of major, class standing, or experience with primary literature, students accurately evaluated internet sources in terms of reliability. All groups of students correctly deemed the second (Twitter post) and fourth (vender website) items on the survey as least reliable, the third (abstract of review article) and fifth (press release describing results of research study) items as most reliable, and the first (Facebook post linking to scientific magazine) item somewhere in between (Figures 35). The first item was designed to be the most challenging in terms of assessing reliability. On one hand, it is a Facebook post, which is definitely not a primary source of scientific information. On the other hand, the post did link to a secondary source, which cited multiple primary sources of information.

Underclassmen had unsophisticated rationales for their ratings, as indicated by rubric composite scores (Figure 7). Many based their ratings on mistrust of social media and other internet sites, as well as trust of well-known scientists and universities. Quite a few underclassmen commented on the prestige of Harvard University (the institution from item 3) as a rationale for high reliability. Upperclassmen, however, were much more likely to base their rationales on the presence or absence of evidence, peer review, and publication in a scientific journal.

As expected, experience analyzing primary scientific literature effectively prepares students in the evaluation of scientific information (Figure 8). All the students with experience in primary literature were junior or senior biology/environmental science majors. To fulfill departmental requirements, biology and environmental science majors at D&E must complete Current Topics, a one-credit course usually taken in the junior year that involves the analysis and discussion of current research articles. Students carefully analyze recent journal articles, which are given to them one week prior to an in-class discussion. Students and a faculty facilitator then discuss the article and compare notes. The goal is for students to become comfortable with graphical representations of data and the interpretation of experimental results. In addition to Current Topics, students are expected to analyze primary scientific literature in five of the six 300-level biology courses that are offered on a regular basis for juniors and seniors. By the end of the senior year, we expect our majors to be able to interpret scientific data, judge the merits of research according to evidence presented, and recognize the differences between actual science, pseudoscience, and anecdote. Results of the current study indicate that most are able to transfer their critical thinking skills to the assessment of online information.

The incorporation of primary literature analysis into the undergraduate curriculum appears to facilitate critical examination of science-related media by biology and environmental science students. Results of our study also indicate that students, regardless of major, tend to think more critically following the sophomore year, after 2 years of exposure to college-level science and general education courses. In addition, recent research has suggested that explicit teaching of the nature of science can have a positive effect on students’ learning, as well as their ability to transfer knowledge to socioscientific issues (Khishfe, 2014). For example, a better grasp of the nature of science might help students comprehend and appreciate scientific arguments associated with topics like anthropogenic global warming and the safety of GMOs. Finally, an understanding of common reasoning fallacies, including confirmation bias, might make it easier for students to understand the public’s denial of scientific evidence supporting concepts like biological evolution and childhood vaccinations (Stover, 2016). If faculty can promote a good working knowledge of the nature of science, a familiarity with common reasoning mistakes, and an ability to analyze and interpret primary scientific literature, our students should be able to accurately evaluate scientific issues presented by print, broadcast, and online media by the time they graduate.

Shawn Stover (stovers@dewv.edu) is a professor of biology and Michelle Mabry is a professor of biology, both in the Department of Biology and Environmental Science at Davis and Elkins College in Elkins, West Virginia. 

References

Bessi A., Coletto M., Davidescu G. A., Scala A., Caldarelli G., & Quattrociocchi W. (2015). Science vs conspiracy: Collective narratives in the age of misinformation. PLoS ONE, 10(2), e0118093.

Cook J., & Lewandowsky S. (2011). The debunking handbook. St. Lucia, Australia: University of Queensland. Retrieved from

Del Vicario M., Bessi A., Zollo F., Petroni F., Scala A., Calarelli G.,…Quattrociocchi W. (2016). The spreading of misinformation online. Proceedings of the National Academy of Sciences, USA, 113, 554–559.

Johnson S. B., Park H. S., Gross C. P., & Yu J. B. (2018). Use of alternative medicine for cancer and its impact on survival. Journal of the National Cancer Institute, 110, 121–124.

Kass-Hout T. A., & Alhinnawi H. (2013). Social media in public health. British Medical Bulletin, 108, 5–24.

Keslar L. (2018, June 18). Going viral: The rise of fake medical news. Proto. Retrieved from

Khishfe R. (2014). Explicit nature of science and argumentation instruction in the context of socioscientific issues: An effect on student learning and transfer. International Journal of Science Education 36, 974–1016.

Moorhead S. A., Hazlett D. E., Harrison L., Carroll J. K., Irwin A., & Hoving C. (2013). A new dimension of health care: Systematic review of the uses, benefits, and limitations of social media for health communication. Journal of Medical Internet Research, 15(4), e85.

Novella S. (2014). No health risks from GMOs. Skeptical Inquirer, 38(4), 19–21.

Power G. A., Minozzo F. C., Spendiff S., Filion M. E., Konokhova Y., Purves-Smith M. F.,…Rassier D. E. (2016). Reduction in single muscle fiber rate of force development with aging is not attenuated in world class older masters athletes. American Journal of Physiology—Cell Physiology, 310, C318–C327.

Rojecki A., & Meraz S. (2014). Rumors and factitious informational blends: The role of the web in speculative politics. New Media and Society 18, 25–43.

Rutsaert P., Regan A., Pieniak Z., McConnon A., Moss A., Wall P., & Verbeke W. (2013). The use of social media in food risk and benefit communication. Trends in Food Science and Technology 30, 84–91.

Stover S. K. (2014). Changing long held beliefs. Skeptic, 19(3), 54–56.

Stover S. K. (2016). Two wrongs make a right: Using pseudoscience and reasoning fallacies to complement primary literature. Journal of College Science Teaching 45(3), 23–27.

Sunstein C., & Vermeule A. (2009). Conspiracy theories: Causes and cures. Journal of Political Philosophy 17, 202–227.

Vosoughi S., Roy D., & Aral S. (2018). The spread of true and false news online. Science 359(6380), 1146–1151.

Wilson E. O., & Nowak M. A. (2014). Natural selection drives the evolution of ant life cycles. Proceedings of the National Academy of Science 111(35), 12585–12590.

Wineburg S., McGrew S., Breakstone J., & Ortega T. (2016). Evaluating information: The cornerstone of civic online reasoning. Stanford Digital Repository. Retrieved from

Postsecondary

Asset 2