Skip to main content

Evaluating Information: The Impact of Major, Class Standing, and Experience With Primary Literature

Journal of College Science Teaching—January/February 2020 (Volume 49, Issue 3)

By Shawn Stover and Michelle Mabry

Social media platforms and other internet sites may propagate inaccurate or misleading information. This misinformation encourages speculation, rumors, and mistrust in established science. In the current study, we attempted to determine whether biology and environmental science students at a small college are able to transfer their critical thinking skills to the assessment of online information. Students were presented with examples of internet-based information sources and asked to rate their reliability. Regardless of major, class standing, or experience with primary literature, students accurately rated the internet sources. Students also provided written rationales for their ratings. Although freshmen and sophomores had unsophisticated rationales, juniors and seniors were likely to base their rationales on the presence or absence of evidence, peer review, and publication in a scientific journal. Juniors or seniors with some experience in the analysis of primary literature were most likely to provide sophisticated, evidence-based rationales for their ratings.


In a systematic review, Moorhead et al. (2013) identified several benefits of social media for health-related communication, including increased accessibility of health information, the potential for enhanced emotional support, and the possibility of improved public health surveillance. Furthermore, social media can be used effectively to engage the public and communicate health-related messages. During the 2009 H1N1 pandemic, for example, the Centers for Disease Control (CDC) used Facebook to educate the public about the disease and the importance of vaccination (Kass-Hout & Alhinnawi, 2013).

Unfortunately, social media platforms and other internet sites can propagate inaccurate or misleading information (Moorhead et al., 2013; Rutsaert et al., 2013). In some cases, facts may be combined with half-truths to generate “informational blends” (Bessi et al., 2015; Rojecki & Meraz, 2014). Consequently, the unregulated environment of the internet encourages speculation, rumors, and mistrust in the scientific/medical establishment (Del Vicario et al., 2016; Sunstein & Vermeule, 2009). Recently, in the midst of a particularly bad flu season, an article with the title “CDC Doctor: ‘Disastrous’ Flu Shot Is Causing Deadly Flu Outbreak” was published online. The article quoted an anonymous doctor at the CDC who claimed that the common denominator for people dying of the flu was the fact that they all had received flu shots. The entire story was a fabrication, yet it went viral on the internet (Keslar, 2018). Likewise, stories promoting “alternative” cancer treatments reach millions of readers online, even though actual laboratory research suggests that the use of alternative treatments instead of conventional care could double the risk of death from cancer (Johnson, Park, Gross, & Yu, 2018).

False information is easily disseminated online. On Twitter, the top 1% of false stories can reach as many as 100,000 people, but accurate stories rarely reach 1,000. Although true stories may inspire joy or sadness, fake stories tend to inspire fear or disgust. Those stories are much more likely to be retweeted (Vosoughi, Roy, & Aral, 2018). Once the false information is out there, it can be difficult to eradicate. Online campaigns meant to debunk erroneous ideas have actually reinforced the misconceptions through a phenomenon known as the “backfire effect.” Criticism can increase the familiarity of myths, overwhelm believers with too much information, or threaten the overarching worldview held by those advocating the myths (Bessi et al., 2015; Cook & Lewandowsky, 2011). Internet users tend to be embedded in “homogeneous clusters.” Their sense of identity may be greatly influenced by politics or religion, and these group allegiances can promote the formation of narratives that are self-confirming (Del Vicario et al., 2016; Stover, 2014).

Between January 2015 and June 2016, a group of researchers at Stanford University administered surveys to middle school, high school, and college students across 12 states to assess the ability to evaluate information on the internet. For example, college students at six universities were asked to: (a) decide if certain websites could be trusted, (b) verify online claims about controversial topics, (c) identify strengths and weaknesses of online videos, and (d) indicate whether specific tweets were reliable sources of information. Results of the surveys indicated that students, in general, are unable to critically evaluate internet content. Although students may be “fluent” in social media, they cannot effectively judge the information they find online. Many students made broad statements about the dangers of social media, without actually investigating the online content (Wineburg McGrew, Breakstone, & Ortega, 2016).

Davis and Elkins College (D&E) is a small, private liberal arts college that emphasizes small class sizes and strong faculty–student interactions. In the Department of Biology and Environmental Science, we introduce our students to the critical analysis of primary scientific literature as early as the sophomore year, and we continue to stress the importance of evidence-based analysis and data interpretation throughout the curriculum. In the current study, we attempt to determine whether our students are able to transfer their critical thinking skills to the assessment of online information.


In accord with official guidelines regarding research and educational practices involving human participants, this study was approved by the D&E Institutional Review Board. A total of 134 students, from six different biology/environmental science courses at D&E, were surveyed between the spring of 2017 and the spring of 2018. Each student signed a consent form and provided information regarding their major, class standing (freshman, sophomore, etc.), and the specific biology/environmental science courses they had completed. They were then presented with five paper-based examples of internet information sources. Some appeared exactly as they did online. In others, the names of real people and products were changed.

The first item was a Facebook post with a link to a Skeptical Inquirer article (Novella, 2014) regarding health risks associated with genetically modified organisms (GMOs). The article cited several studies that demonstrated the safety of GMOs. Results of the studies were summarized in the survey. The second item was the following Twitter post: “It should be pretty obvious that the idea of global warming is nonsense.” The post included a picture of icicles forming on a gutter above a window (Figure 1). Item number three was the abstract of a review article published in Proceedings of the National Academy of Science, USA (Wilson & Nowak, 2014). The article presented several lines of evidence indicating that natural selection drives the evolution of ant life cycles. The fourth item was a mock web page for an online vendor of dietary supplements. The site highlighted a product called “Snake Berry Tablets.” According to the site, the product is used to boost metabolism and prevent excess fat deposition. No information regarding scientific research, clinical trials, or FDA approval was provided. The final item was a press release from the American Physiological Society describing a study published in the American Journal of Physiology—Cell Physiology regarding the fitness and muscle characteristics of aging athletes (Power et al., 2016).

Median Likert-type scores indicated that students, regardless of major, class standing, or experience with primary literature, ranked the survey items similarly in terms of reliability (Figures 35).

Median Likert-type scores by major (biology, environmental science, exercise science, nursing, and other).

Median Likert-type scores by major (biology, environmental science, exercise science, nursing, and other).

Median Likert-type scores by class standing.

Median Likert-type scores by class standing.

Median Likert-type scores by experience with primary scientific literature.

Median Likert-type scores by experience with primary scientific literature.

The calculated value for κ was 0.74, indicating good interrater reliability. Mean rubric composite scores demonstrated no significant differences between majors (Figure 6). However, upperclassmen (juniors and seniors) had significantly higher composite scores than underclassmen (freshmen and sophomores; Figure 7). Furthermore, students who had some experience analyzing primary scientific literature had significantly higher composite scores than students with no experience (Figure 8).

Mean rubric composite scores by major (biology, environmental science, exercise science, nursing, and other). The maximum score is 15.

Mean rubric composite scores by major (biology, environmental science, exercise science, nursing, and other). The maximum score is 15.

Shawn Stover ( is a professor of biology and Michelle Mabry is a professor of biology, both in the Department of Biology and Environmental Science at Davis and Elkins College in Elkins, West Virginia. 


Bessi A., Coletto M., Davidescu G. A., Scala A., Caldarelli G., & Quattrociocchi W. (2015). Science vs conspiracy: Collective narratives in the age of misinformation. PLoS ONE, 10(2), e0118093.

Cook J., & Lewandowsky S. (2011). The debunking handbook. St. Lucia, Australia: University of Queensland. Retrieved from

Khishfe R. (2014). Explicit nature of science and argumentation instruction in the context of socioscientific issues: An effect on student learning and transfer. International Journal of Science Education 36, 974–1016.

Moorhead S. A., Hazlett D. E., Harrison L., Carroll J. K., Irwin A., & Hoving C. (2013). A new dimension of health care: Systematic review of the uses, benefits, and limitations of social media for health communication. Journal of Medical Internet Research, 15(4), e85.

Novella S. (2014). No health risks from GMOs. Skeptical Inquirer, 38(4), 19–21.

Power G. A., Minozzo F. C., Spendiff S., Filion M. E., Konokhova Y., Purves-Smith M. F.,…Rassier D. E. (2016). Reduction in single muscle fiber rate of force development with aging is not attenuated in world class older masters athletes. American Journal of Physiology—Cell Physiology, 310, C318–C327.

Rojecki A., & Meraz S. (2014). Rumors and factitious informational blends: The role of the web in speculative politics. New Media and Society 18, 25–43.

Stover S. K. (2014). Changing long held beliefs. Skeptic, 19(3), 54–56.

Stover S. K. (2016). Two wrongs make a right: Using pseudoscience and reasoning fallacies to complement primary literature. Journal of College Science Teaching 45(3), 23–27.

Sunstein C., & Vermeule A. (2009). Conspiracy theories: Causes and cures. Journal of Political Philosophy 17, 202–227.

Vosoughi S., Roy D., & Aral S. (2018). The spread of true and false news online. Science 359(6380), 1146–1151.

Wilson E. O., & Nowak M. A. (2014). Natural selection drives the evolution of ant life cycles. Proceedings of the National Academy of Science 111(35), 12585–12590.

Wineburg S., McGrew S., Breakstone J., & Ortega T. (2016). Evaluating information: The cornerstone of civic online reasoning. Stanford Digital Repository. Retrieved from


Asset 2