Education researchers usually publish about “what works.” However, there is another side to education that is more difficult to understand. Students have numerous causes, reasons, and excuses for withdrawing from school or dropping a class, yet there are a certain number of students who do the assigned classwork and are still unsuccessful.

This study evaluates students from seven institutions as part of a continuing 4-year study. Specifically, the focus is on unsuccessful Hispanic students who attended Hispanic-Serving (HSI) or emerging Hispanic-Serving (eHSI) Institutions. By “unsuccessful,” we mean that the students did not succeed at a level necessary to advance to the next general chemistry course. Results provide support for identifying students during the first week of class who may possibly complete first-semester general chemistry (Chem I) with a grade of D or F. Two diagnostics were selected to assess students’ automaticity skills (what they can do without the use of a calculating device) in two areas: basic arithmetic and quantitative reasoning (QR). By identifying these students early in the semester, there is time to help the students succeed before it is too late.

The purpose of this study is to explore the personal characteristics of Hispanic students who consistently attended class and did required assignments but finished the course with a grade of D or F (n = 69 or the 13% of the population who were unsuccessful completers). Even though this is a small sample of the total population, further study provides insights into important mental-math skills that may have gone dormant due to lack of use from as early as seventh grade.

Hispanic students compose the largest percentage of ethnic minorities in most of the study team’s universities (Texas Higher Education Coordinating Board, 2019). Hispanic students who were omitted from this study include those who were successful (the majority of the classes evaluated), post-baccalaureates, those who did not sign the Institutional Review Board (IRB) release form, those who failed to be present at one or both diagnostic test days, and those who failed to complete the demographic survey about their ethnicity and gender. This brings the percentage of omitted Hispanic students at emerging HSIs (eHSIs) to 63% of the overall Hispanic participants and the overall percentage at HSIs to 73%. Our focus is on the 37% and 27%, respectively, or the unsuccessful completers.

Purpose of study

There is only a small number of published research articles that address students’ abilities in lower-level chemistry courses and how they may relate to students’ Hispanic ethnicity (Báez-Galib et al., 2005; Fernandez & Middlecamp, 1999; Garcia et al., 1993; Mason & Mittag, 2001). There is ongoing interest in retaining female students and students from racial and ethnic minority backgrounds in STEM fields, so more studies are needed regarding these populations (Zeldin et al., 2008). This study aims to recognize why some Hispanic students, whether they attend an HSI or eHSI, are unsuccessful even when they completed the Chem I course. To be classified as an HSI, at least 25% of the undergraduate population of a university must be students from Hispanic backgrounds. Four universities that participated in this investigation are considered to be eHSIs (those with a minimum of 16% Hispanic enrollment), with the other three institutions recognized as HSIs (Table 1). One of the HSIs has an enrollment of more than 34,000 students, and the other two HSIs are schools with smaller enrollments (around 5,000) that are considered medium-sized schools (Tai et al., 2005). Schools with a population of fewer than 5,000 undergraduates are considered small. Given the wide variety of participating institutions, this research population provides a representative view of a large, majority-minority state, lending credibility to this study.

One reason given by the state commissioner of education for the significant decline in SAT scores over the past few decades (1987–2016), to a 30-year low of 944/1600 (Petros et al., 2017), is that many more students are being encouraged to take the SAT who have previously never considered going to college; as this “college-going culture” increases, the number of minority students is having an overall impact on the state’s performance, as minorities typically score lower on the SAT than their nonminority contemporaries (Stutz, 2012). There is little evidence to suggest that remediation has an impact or adequately prepares students to complete a degree (Cohen & Kelly, 2020; Habley et al., 2012).

Even though there are no reported differences between White non-Hispanic and Hispanic students being engaged and persistent until Year 2, historically underserved students in postsecondary institutions benefit the most in terms of grades and persistence (Kuh et al., 2007) when targeted programs are implemented. Tai et al. (2005) reported data gathered from 12 institutions, with 1,500 students included in the data. Findings included that gender did not make a difference, nor did race, except for Latino students, in which case the last high school mathematics course completed and parental education were prime factors among the variables assessed that did affect outcomes. Academic discipline (i.e., doing homework, attending class, class participation, taking responsibility for course performance, and working with others) and commitment to college (i.e., college is important and completion is essential) are the strongest predictors of retention outcomes, even when compared to students’ ability levels when they enter college (Habley et al., 2012).

Methods

The universities chosen to participate in this study are part of a larger group of institutions that form the Networking for Science Advancement (NSA) collaborative, whose team is composed of chemical education researchers who have teaching experience ranging from 4 to 45 years. Data in this study are from the NSA team’s institutions. In the current study, Chem I students completed two calculator-free assessments used as diagnostics to determine their automaticity ability: the Math-Up Skills Test (MUST; 20 items in 15 minutes) and quantitative reasoning (QR; 20 items in 30 minutes). (The MUST assessment is available in Williamson et al., 2020; the questions selected for the QR quiz were obtained from the NSF DUE 1140562 project maintained by Eric Gaze.) Demographic indicators collected included gender, ethnicity, precollege academic preparation (i.e., participation in one or more high school chemistry courses), mathematics course(s) concurrently enrolled or degree requirements completed, parents’ education, and where the students lived during high school based on the first two digits of their reported ZIP code. More than 94% of the unsuccessful students in this IRB-approved study (n = 554) who agreed to participate attended public schools prior to admittance into one of the seven universities. Having experienced an isomorphic high school curriculum, as mandated by the state’s education agency, the students are mostly assumed to have completed equivalent secondary school experiences when enrolled in similar classes. As seen in Figure 1, the two diagnostic assessments used in this study give a strong indication that the unsuccessful students entered university life with similar MUST and QR abilities regardless of whether they attend an HSI or eHSI. Note the similar up and down flow of the lines on these diagrams. Different students at different universities struggle with the same problems on the MUST and the QR quizzes, indicating that students bring similar understandings and misconceptions reflective of their precollege backgrounds to Chem I. (The last section of Figure 1b, Images, appears to be students’ most difficult section and is being investigated further.)

IRB guidelines prohibit instructors from requiring that students complete the diagnostic quizzes, so some instructors elect to have an activity grade stated on their syllabi, which means students are encouraged to complete a particular number of the total activities available but may decline to participate if they choose. At the end of the semester, final course grades were collected for students from each class as an output measure of course completion. Students in this study have at least one thing in common: They failed to qualify for continuance from Chem I to Chem II because they had not met the required expectation of an average of 69.5% or better.

Research question

Existing knowledge provides the framework for new learning and is a known predictor of learning success in most disciplines (Shell et al., 2010). In prior publications, the authors of this study have reported that the automaticity skills students bring to Chem I and II may be more important than was once thought (Albaladejo et al., 2018; Mamiya et al., 2022; Powell et al., 2020; Williamson et al., 2020). Broad demographic categories (e.g., personal characteristics and environmental factors) were analyzed. The primary contributors to attrition are students’ personal characteristics, whereas institutional factors (i.e., environmental) make only a minimal contribution to attrition and retention (Habley et al., 2012; Wilkerson, 2008).

Based on prior research published by the NSA team (Williamson et al., 2020), MUST scores appear predictive of Chem I success about 80% of the time and thus will be presented as supportive of students’ prior mathematics knowledge when accompanied by the mathematics courses in which students are currently enrolled. Prior knowledge in chemistry is identified as the last course completed in high school (regular, pre-Advanced Placement [AP], or AP/International Baccalaureate [IB]/dual enrollment). Parental education background determines which students are considered first-generation (i.e., no parent has completed a degree). Students choose which degree in a STEM discipline to pursue, known as their entry college, but the choice of major is limited by which degrees are available. Chem I courses in this study are for science majors, yet about one quarter of these students do not consider themselves STEM majors.

The following research question was addressed: How do automaticity abilities—as assessed by the MUST and QR diagnostic instruments—of unsuccessful (grade of D/F) Hispanic students who attend eHSIs (n = 39) or HSIs (n = 30) affect students’ lack of course success when specific personal characteristics are taken into consideration?

Instruments

In the current study, we are interested in the influence of automaticity in two areas: basic arithmetic (as assessed by the MUST) and QR, which has a strong quantitative literacy (QL) component. The MUST is a 15-minute, hand-graded assessment that includes 20 open-ended questions. Its practical use for Chem I has been well established (Petros et al., 2017; Albaladejo et al., 2018; Williamson et al., 2020; Mamiya et al., 2022), and it shows that students’ automaticity skills when they enter the course are a valid predictor of how successful they will be in Chem I, a gateway course for pursuing several specific STEM degrees. The QR instrument requires 30 minutes and uses a multiple-choice format with three distinct parts: arithmetic problems, typical algebra-based word problems, and problems focused on the use of images (e.g., diagrams, graphs, and charts). There are two versions of each quiz; no statistically significant difference (p < 0.05) between versions was found on either assessment. Table 2 presents the diagnostic scores for all students (n = 692) by institution type and self-reported ethnicity (Hispanic or non-Hispanic).

The initial version of the automaticity quiz that became the MUST was published by Hartman and Nelson (2016). The open-ended MUST has a good internal consistency with a Cronbach’s alpha and KR-20 of 0.856 (n = 1,073; Williamson et al., 2020). In the present study, the KR-20 was calculated for MUST (n = 554), r_kr20 = 0.870 (good internal consistency, very high reliability). Selection of the QR questions was based on a project that contributes to the National Numeracy Network database (Gaze & Misener, 2012). For the QR quiz, the r_kr20 = 0.730 (acceptable internal consistency, high reliability) The Cohen’s d for the MUST is 1.40 (large), and the Cohen’s d for the QR is 0.784 (medium). The QR scores had varied impact, from very large for Hispanic and non-Hispanic students who attended HSIs to a medium effect for Hispanic students attending eHSIs. Overall, the Cohen’s d results support that the QR has a large effect on Hispanic students’ averages (1.03) and a medium effect for non-Hispanic students (r = 0.690). Hispanic students’ MUST and QR scores showed a strong correlation (r = 0.665); the assessments did not include repeated questions.

Research setting

The fall-semester Chem I courses are considered to be “on sequence.” The instructors involved in this study conducted their classes as they would under nonexperimental conditions. Some professors teach large-group lectures (n > 200) and some teach small groups (n < 35), some give online homework and others do not, and some only lecture while others conduct a more active classroom. No restrictions were placed on how the classes were taught, making this study more realistic than some controlled studies.

Findings

In Table 2, note that both the MUST and QR quizzes are scored out of a maximum of 20 points and the scores from each diagnostic aligned from low to high. Non-Hispanic students tend to enter Chem I with higher MUST and QR scores, indicating greater automaticity skills.

The studied population (Table 3) consists of only 69 Hispanic students because they are the only “unsuccessful” students who completed the course. In this study, about half of the first-generation students at eHSIs are female; at the HSIs, a much larger percentage of unsuccessful female students were considered first generation (i.e., 52.4% at eHSIs vs. 92.9% at HSIs). At the HSIs, many more female students than male students were unsuccessful, and all but one of the first-generation students were female. At the eHSIs, the distributions were more evenly split between genders. In Table 3, a high percentage of first-generation female students at HSIs (13/14; approximately 93%) were not successful in Chem I.

Table 4 displays no statistically significant difference between the groups’ averages on items evaluated. However, at eHSIs, students showed more prior knowledge as measured by the MUST and QR, but their overall course average was slightly lower than the HSI students’ average. For comparison purposes, Table 2 reports the overall MUST and QR scores for the combined group of successful and unsuccessful Hispanic students as 6.8 and 11.7, respectively. As shown in Table 4, MUST scores of 5.3 and QR scores of 11.2 for unsuccessful Hispanic students provide evidence that these unsuccessful students on average entered with lower automaticity knowledge (MUST and QR scores) than their classmates who were successful, supporting that both of these instruments can be used to identify at-risk students. For comparison, successful Hispanic students (n = 399) entered the course with MUST average of 5.9 (SD = 4.9) and a QR average of 11.6 (SD = 3.5) and finished the class with a course average of 84.1 (SD = 8.8). Successful Hispanic students’ scores on diagnostic tests were significantly lower (p < 0.05) than their non-Hispanic classmates’ scores (MUST = 8.3 [SD = 5.0] and QR = 13.2 [SD = 3.4]).

Personal characteristics

Gender

Male Hispanic students entered Chem I with statistically higher MUST and QR scores than female Hispanic students but finished the course with a lower class average, though it was not significantly different (Table 5). The greatest effect of prior knowledge on both the MUST and QR appears to come from male Hispanic students enrolled at eHSIs, and yet this group, which would be predicted to finish with the highest course average, finished instead with the lowest. The highest course average was earned by male Hispanic students enrolled at HSIs. At eHSIs, male Hispanic students’ diagnostic test scores are significantly higher than those of their female counterparts, but the female Hispanic students completed the class with a higher overall average.

Prior knowledge (chemistry and mathematics)

Prior chemistry courses from students’ high school experiences were evaluated. The two most populated groups are students who took either regular chemistry (n = 31) or pre-AP chemistry (n = 31). No statistical difference was found between any of the class averages or diagnostic quiz scores. Two students reported not having had any prior course experience in chemistry, and both were from a state where 3 years of science are required for college-bound students, so it is possible that these students took physics instead of chemistry to satisfy the mandate. Four students (5.8%) reported having had AP or IB chemistry courses, yet their course averages were lower than the two students who had not taken chemistry in high school.

At most universities, completion of college algebra, chosen as the cutoff mark for this study, is considered a necessary prerequisite for entering Chem I. Students who responded that they were not enrolled in a mathematics course or were in elementary statistics or college algebra were placed into one group (n = 36), and those who were ready to enroll in precalculus or higher (n = 33) were placed into the comparison group. Table 6 shows that the MUST and QR scores and the class averages align, with the lower scores going to the students who have not completed college algebra and the higher scores earned by those who are ready for precalculus or a higher level of mathematics. The only statistical difference (p < 0.05) seen was between the MUST means identified. On average, students enrolled in precalculus or higher outperformed the less-prepared group, with the greater contribution coming from students at eHSIs, based on their MUST scores. What is interesting is that 52% of these students (n = 36) had enrolled in Chem I without having met necessary prerequisites. Poor advising may also be a reason behind poor performance. Sometimes students have technically met the prerequisite for enrollment, but they may have taken the class many months (or even years) prior to enrollment, and proven knowledge decay over time was possibly not taken into consideration.

Students’ entry college

Sixteen students enrolled in the Chem I classes had no intent to complete a STEM degree, as indicated by the college within the university in which they enrolled. It is possible that these students are premedical and are majoring in a subject such as psychology or another social science. STEM majors presented with higher MUST and QR scores but finished with lower averages, without statistical difference. There was no statistically significant difference in mean scores for students when they entered Chem I. The subgroup that entered the class with higher MUST and QR scores (STEM majors at eHSIs) finished with the lowest class average of 57%. These students’ QR score compares well to the average Hispanic student (Table 2), who had an average score of 11.7, but the MUST score mean was lower than that of the average successful student.

Parents’ education

Students were asked to respond to the questions “Have any of your parents/guardians obtained a 4-year degree?” and “Have any of your grandparents obtained a 4-year degree?” Both of these questions were asked because we observed that some students lived with their grandparents. If the answer to the question about their parents’ education was “no” but the response to the question about the grandparents’ education was “yes,” students were not counted as first-generation students. To be considered part of the first-generation category, the answers to both questions had to be “no.” (However, none of the unsuccessful students in this study reported that their grandparents held a degree when their parents did not.) There was almost an even split between students with no one in their family holding a 4-year degree and those who had at least one parent with a degree. The first-generation students came to class less prepared than their classmates who had higher MUST and QR scores (without statistical difference), but it is important to remember that all of the students in this study were not successful. The final class average of the first-generation students was slightly higher than those not considered to be first-generation, even though the average MUST and QR scores were lower than those for the groups whose parents held a 4-year degree. In all cases, the students at the eHSIs entered with higher MUST and QR scores, but the stronger class average (62%) was earned by students who were not first-generation students and attended HSIs. The MUST and QR scores were in perfect alignment: First-generation students who attended HSIs entered with the lowest MUST and QR scores.

Significance of findings

The research question concerned the automaticity abilities of unsuccessful Hispanic students who attended HSIs (n = 30) and those who were enrolled at eHSIs (n = 39) as assessed by the MUST and QR diagnostics. Of the factors investigated (gender, prior knowledge in chemistry and mathematics, intended major, and first-generation status), there were 10 groups (ignoring single-member groups), which were split into groups of students who attended either an HSI or an eHSI. Six groups were the same with regard to which diagnostic was higher. Of the remaining four groups, two had higher MUST scores and two had higher QR scores. The groups that did not match included female Hispanic students (those at eHSIs had higher MUST scores but those at HSIs had higher QR scores); students who had taken regular high school chemistry (students at eHSIs had higher QR scores and those at HSIs had higher MUST scores); students who co-enrolled in college algebra or lower (eHSI students had higher QR scores but HSI students had higher MUST scores); and students who were not pursuing a STEM major (eHSI students had higher MUST scores and those at HSIs had higher QR scores). Our investigation of the remaining factors found that unsuccessful Hispanic students at eHSIs entered Chem I with higher MUST and QR scores than those enrolled at HSIs. Overall, unsuccessful Hispanic students at HSIs completed the course with a higher average (61%) than their counterparts at eHSIs (58%), though the difference was not statistically significant. With regard to the personal demographics addressed, we found that students at eHSIs entered Chem I with better arithmetic automaticity than QR ability (greater literacy emphasis) about 70% of the time. The characteristics of the weakest unsuccessful student are as follows: male, completed pre-AP high school chemistry, yet to complete college algebra, pursuing a STEM major, and has at least one parent who holds a degree. A simple correction that can be made is for advisors to prevent students from enrolling in Chem I if they have not completed college algebra and to not accept an equivalent course.

Discussion: Successful high-impact practices (HIPs)

The NSA partner identified as the comparison-control class teaches at one of the largest Tier One (Carnegie Classification) research universities in the United States. This eHSI hosts a special program that offers accepted students with high financial need and below-average SAT scores (this university typically admits students with a score of at least 1200) to participate in the Texas Interdisciplinary Plan (TIP; Tough, 2014). Entrance into this program is based on 14 variables, such as unmet financial need (family income less than $28,000), SAT score, class rank, and parents’ background (first-generation status). This program offers smaller class sizes (approximately 150 students) for general chemistry sections (the norm for this university is more than 400 students per class) and an additional 1 to 2 hours of structured time for extra help (i.e., these students meet for 4 to 5 hours per week instead of the 3 hours per week that are more common at other institutions; for instance, if a chemistry course offers a recitation hour, the total time per week might be 4 hours per week). This program also provides students with advisors and community-building exercises, along with a $5,000 scholarship per year, paid in monthly installments.

Of those attending the comparison-control university (Table 7), about 55% are of Hispanic ethnicity and 49% reported having first-generation status, with 57.1% of the first-generation students being Hispanic. These percentages are higher than those for the other eHSIs and are in line with the Hispanic population at the HSIs. These indicators prompted the selection of this class from an eHSI for our comparison section because it, like the other HSI partners, has a large Hispanic enrollment. As shown in Table 7, there were only two students (both female) out of 128 who did not succeed; both entered with MUST and QR scores one standard deviation below the respective means.

The intervention implemented by the instructors allows them to help develop students’ automaticity skills by requiring that some activities and exams must be completed without a calculator. The “carrot” to get them to agree to this intervention is that the MCAT is a calculator-free assessment and a majority of these students seek a career in the health professions (Powell & Mason, 2021). It is also evident that students might benefit from more QR preparation because more of our daily lives make use of quantitative reasoning, including the grocery store, the gas station, banking, and sports news, to name a few.

Conclusion

Completion rate is a main measure of student success in college (Lipka, 2019). In the context of this study, it is also important to remember that education is valued by many Mexican American families, which is known to have an effect on achievement (Duran & Weffer, 1992). In this study, we evaluated the success of a sizable number of students from seven universities in a large majority-minority state, with a large percentage of the population being of Hispanic descent. Demographics aside, the primary point is that with brief and straightforward assessments administered at the beginning of the semester, instructors can identify students who may benefit from targeted additional help, as was made available at the comparison-control university. The MUST has a greater effect size than the QR assessment, indicating that basic arithmetic skills, when committed to memory, may be one of the most important skills all students bring with them to Chem I, a gateway class to STEM majors. The data collected for this study contribute to the knowledge that at-risk students can be identified early in the semester, giving educators a pathway to improve students’ engagement and set them up for future success.

Acknowledgments

The first author wishes to acknowledges the National Science Foundation under Grant No. 1725674 for supporting this work. Any opinions, findings, and conclusions or recommendations expressed in this article are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Adrian Villalta-Cerdas is an assistant professor in the Department of Chemistry at Sam Houston State University in Huntsville, Texas. Anton Dubrovskiy is an associate professor, co-director of the UHCLTeach program, and program chair of chemistry in the Department of Physical and Applied Sciences at University of Houston-Clear Lake in Houston, Texas. Deborah Rush Walker is an associate professor of instruction in the Department of Chemistry at The University of Texas at Austin in Austin, Texas. Blain Mamiya is a lecturer in the Department of Chemistry & Biochemistry at Texas State University in San Marcos, Texas. G. Robert Shelton is an assistant professor of chemistry in the Department of Science and Mathematics at Texas A&M University in San Antonio, Texas. Cynthia B. Powell is professor of chemistry in the Department of Chemistry and Biochemistry at Abilene Christian University in Abilene, Texas. Susan Broadway is an inventory supervisor, Rebecca Weber is a senior lecturer and undergraduate advisor, and Diana Mason (diana.mason@unt.edu) is professor emeritus of chemistry education, all in the Department of Chemistry at the University of North Texas in Denton, Texas.