The K–16 academic journey to a career in science, technology, engineering, and mathematics (STEM) is often described as having serious barriers, such as uninspired teaching, an unwelcome environment, and poor math and science preparation that lead to the under-representation of ethnic minorities and women in these fields. At the same time, national and global demand for workers in STEM fields is growing rapidly. STEM attitudes and career interests typically decline in the middle school years due to the reduction in emphasis on mastery, engagement, and school culture, making this a critical time for STEM interventions (National Academies of Sciences, Engineering, and Medicine 2016; Vedder-Weiss and Fortus 2011).

In this article we describe the Saturday Seminars for Urban Scholars Program (USP), which is an out-of-school program for inner city youth in Albany, New York. The program aims to increase the number of African Americans, Latinx individuals, and women in STEM fields by supporting the development of positive attitudes toward STEM and careers in these areas. We accomplish this by providing:

1. long-term, high-quality STEM experiences;
2. supportive STEM role models as teachers and mentors;
3. a positive program climate that is appropriately challenging and supportive;
4. STEM content taught through multiplayer gaming environments; and
5. free access to the program and free transportation to attend.

USP provides a complex environment to examine important questions about what works for successful out-of-school STEM programs. For example, the program allows elementary and middle school students (grades 5–8) to work with experienced high school mentors who previously participated in the program for multiple years. In addition, college mentors from a variety of academic majors serve as assistants, course developers, and teachers. Although the majority of experiences are STEM-based, we include some experiences in arts and humanities. This helps draw a diversity of middle school participants, including those who have an interest in the program but also have some trepidation about the STEM focus.

Since 2012 the program has collected data on participants’ general attitudes about school and specific attitudes about learning STEM topics and pursuing STEM careers. Program attendance, science course-taking in high school, and school achievement (including standardized test scores before, during, and after leaving the program) provide a longitudinal perspective on program effects.

This article focuses on the program as it pertains to elementary and middle school students. We describe features of the program that align with best practices in STEM education. We then present a high-level overview of findings with a focus on evidence of the relationship between participation in the program and STEM attitudes, academic performance on state exams, and science course-taking in high school.

Program overview

Program participants

USP operates in conjunction with Albany City School District. Since 2012, USP has focused on STEM content and activities for underserved, under-represented upper-elementary and middle school students from this district. Participating students are bused to the Siena College campus 14 Saturdays during the academic year for up to 70 contact hours per year. At the program, students participate in brief morning and afternoon meetings as a group, take morning and afternoon classes (approximately one and a half hours each), and eat lunch in the college cafeteria.

The school district partner is classified as a high-needs and low resource capacity district, with 58% of K–12 students considered economically disadvantaged. Students in the district have low graduation rates (65%) compared to the rest of the state (80%). Figure 1 shows the ethnic diversity of the school district and the program (NYS Education Department 2018). USP has similar percentages of white and Latinx students, more African-American students, and fewer Asian students than the district. More females (58%) than males (42%) participate in USP. Participating students live in similar neighborhoods as nonparticipants in the school district, and these areas tend to have above average levels of poverty^[1]. Seventy-five percent of USP students and 73% of district students live in high-poverty neighborhoods, respectively. Most students (93%) begin participating as fifth or sixth graders. Students in fifth and sixth grades comprise 41% of participants, and seventh and eighth graders comprise 59%. Most students (92%) participate for two or more years, with two years of participation amounting to 28 Saturdays, or approximately 140 contact hours.

Participation with the district

USP is funded by both the school district and Siena College. The district provides 75% of the support, with the college contributing the rest. The program is free of charge to participants, including transportation, instruction, snacks, and lunch. District teachers, science coordinators, and parents are encouraged to visit and attend the program with their students, providing an opportunity for these decision-makers to understand the nature of the program and observe student engagement.

To recruit students, we work with the science curriculum coordinator and fifth- and sixth-grade teachers at the three elementary schools we serve in the district (Note: Any student attending any of the elementary and middle schools in the district is eligible to participate and students from other schools learn about the program through word-of-mouth). A few weeks before the start of the program, we go to each of the fifth- and sixth-grade classrooms to show a PowerPoint presentation along with videos of students in USP classes from the previous year. Classroom teachers assist in recruitment by distributing applications to all fifth- and sixth-grade students, encouraging them to attend, reminding them to return applications, and returning completed applications to us. All students are encouraged to sign up regardless of academic performance, behavior, or special education classification. The program is able to serve approximately 60 students each year. Individualized Education Plans (IEPs) or 504 plans govern the support provided for students with special needs. Parents who request special accommodations are encouraged to seek support through the district’s existing pathways. Students with IEP or 504 plans who attend the program without support have been successful in USP. We believe this is due to the high ratio of college mentors to participants. Students are encouraged through e-mail and phone contact with parents and teachers to return to the program during their middle school years.

We provided the participating school district with a list of participants, as well as their attendance and course-taking in USP. The district provided aggregated data for participants’ standardized English language arts (ELA) and math assessments (during their elementary and middle school years), and Regents exams (during their high school years), as well as graduation rates for participants and nonparticipants. Following Family Educational Rights and Privacy Act guidelines, all school district data used in this report were provided in aggregate form. Analyses focused on group comparisons between program participants and their nonparticipating peers.

Staffing: Supportive STEM role models as teachers and mentors

Siena College faculty, STEM professionals from local businesses, and STEM teachers from local schools develop the majority of the curricula for the program. To recruit these individuals, we take several approaches. We reach out to area teachers who have participated in USP as mentors. We attend local STEM events and gaming conferences to identify individuals who would be a good fit for our program. We network with STEM professionals who have an interest in outreach, including parents of students in our STEM camp program. In addition to creating opportunities for middle school students to engage in STEM-based projects, faculty provide leadership development opportunities for undergraduate STEM mentors and high school mentors. College STEM mentors who participate for several years have the opportunity to develop and teach lessons of their own. Additional college mentors also attend the program to complete fieldwork hours from their preservice teacher education classes.

In-person, full-day training prior to the program introduces mentors to the Saturday schedule, as well as the program climate and expectations. Expectations include, for example, attending regularly, assisting in the classes, chaperoning on buses, and behaving as a role model. In addition, mentors are provided with technology training for coding, gaming, and hands-on technology classes. Training continues throughout the program through e-mail and in-person meetings with the class teachers, as needed.

Key program elements

STEM experiences that are high quality and evidence-based

The National Research Council (2015) recommendations are a guide to creating high-quality STEM experiences. According to its report, successful programs provide opportunities for engagement socially, emotionally, and intellectually by responding to students’ experiences and interests while taking cultural practices into consideration. Additionally, literature on effective programs for inner-city youth has guided efforts to shape the program climate. For example, Heath and McLaughlin (1993) studied effective youth organizations and concluded that inner-city youth “resist programs that try to control their behavior, that label them as deviant or deficient, that ignore their culture and context, that offer them little in the way of personal accomplishment and skill, or that hold them to low levels of expectation and accomplishment” (p. 222). Yet, such youth are apt to take advantage of programs that provide opportunities for them to shape their self-expectations, explore their identity, and provide links to education and work opportunities. We use strategies from sources similar to the ones mentioned here to create high-quality, evidence-based experiences.

Serious gaming is a foundation for curriculum design

A frontier in educational technology that is emerging in public consciousness, called “serious gaming,” challenges teachers to connect their instruction to the everyday games students use in their lives. Zyda (2005) provided a definition: “Serious game: a mental contest, played with a computer in accordance with specific rules, that uses entertainment to further government or corporate training, education, health, public policy, and strategic communication objectives” (p. 25). Serious games include educational tools as well as those that were developed primarily as entertainment, but have been adapted or redesigned for education purposes. Educational versions of popular games exist, such as Minecraft: Education Edition, KerbalEdu, and CivilizationEDU. There are also online educational systems such as TeacherGaming’s subscription service, which features over 35 classroom games, lesson plans, and smart learning analytics that allow teachers to track progress and performance. These and other resources are used in USP to connect academic content with gaming contexts that are familiar to students.

A serious gaming approach is used in USP to connect academic content to students’ lives so that activities show how learning about STEM can fit into and affirm their developing sense of identity through use of multiplayer games (e.g., Minecraft). Serious gaming can be an effective instructional method because it provides an authentic experience that supports diverse 21st-century skills, such as problem solving, critical thinking, and innovation.

Gee (2005) refers to such experiences as creating an affinity space, or an in-person, blended online space where people gather to pursue shared interests in a common endeavor. Affinity spaces provide opportunities for people to “relate to each other primarily in terms of common interests, endeavors, goals or practice, not in terms of race, gender, age, disability or social class” (Gee 2005, p. 225). In our Minecraft physics worlds, students and mentors participate in a common endeavor to learn physics concepts through their engagement in the challenges presented in Minecraft. They participate at any level, ranging from novice to expert, with respect to their skills in Minecraft as well as their physics knowledge.

One example is a Minecraft world that uses quantum mechanics blocks^[1]. Students and mentors work together using their newly acquired understanding of quantum mechanics principles (observational dependence, superposition, and entanglement) to complete different challenges that require understanding STEM content to advance through the gaming experience. One student described this class as his favorite. He said it was the hardest class he had taken and he had to really understand the quantum mechanics concepts while working with other students to be successful.

Material that is enriching and challenging

The program emphasizes challenge and enrichment—as opposed to remediation—to guide instructional decisions. Evidence does not support an assumption that under-represented youth lack capacity or intellectual ability. Instead, measures of their ability and performance suggest they underperform. Years of research on gifted education practices have demonstrated that all students can benefit from a challenging, enriched curriculum (Renzulli 1999).

USP creates a challenging environment in several ways. Rather than providing easy tasks to ensure success, we structure experiences in which mistakes are normalized as part of a scientific process. Success requires persistence, problem solving, and frustration tolerance. This method of teaching is valued by students in the program. One student described how he liked that “It was very interactive and you didn’t really do it for us, you let us learn for ourselves and that really helped with the process.” Because students are allowed to struggle and take on challenges, we think they learn to value and understand that the often arduous and time-consuming process of solving problems is a necessary dimension of reaching workable solutions.

A returning participant explained that she “learned that not every problem has to be solved immediately and that taking a break and looking back on it and getting a fresher perspective is always great.” She added, “Teamwork is all around very helpful, but sometimes you have to work individually to work out kinks in your own work.” 

Students have choices

Experience has led us to provide new programming options for each of our three sessions throughout each academic year. Participants choose two STEM courses (morning and afternoon) from a list of eight to 10, which they explore for four to six weeks. During the academic year, participants can take up to six different classes. Offering a variety of courses allows students to learn new topics, work with different students and mentors, and direct their own STEM learning. Long-attending USP students report that they continue in the program because there are always new classes to take.

Courses evolve for variety and novelty

Classes are continually added and past topics are reintroduced. Favorite classes are offered frequently for new students and extended to allow returning students to continue to learn about the topic. Participants are referred to as “Urban Scholars” and projects are given creative titles that emphasize academic content in nontraditional ways—for example, “Landscape Painting With Gouache” rather than “Art,” and “Projectiles and Motion in Minecraft.”

For example, in a class called “Particle Physics in Minecraft,” middle school students are introduced to quarks and leptons, the building blocks of everything in the universe. The class contains four lessons, each lasting about one and a half hours. In the first lesson, students engage in a simulation about the scale of the universe, watch a video about particle physics, and complete a hands-on activity with puzzle pieces to understand how quarks and leptons go together. In the next two lessons, students enter a Minecraft world where participants combine quarks in sets of three to create the building blocks of atoms, and they construct specific atoms with the right number of protons and neutrons. On the last day, participants learn about fission and fusion and create a particle accelerator in the Minecraft world.

A class called FLOSS Desktops for Kids, an incubator project of the Open Source Initiative, adds to the program’s hands-on options. Students are given recycled desktop computers that they disassemble and reassemble three times to become familiar and comfortable with computer hardware. As part of the curriculum, college and high school mentors work alongside students to ensure they learn about safe handling of computer equipment. For example, students learn to always unplug the computer before opening it, and they learn to never open the power supply as the capacitors inside may store significant amounts of energy. Students install an open-source operating system and software. They learn to download and use software that aligns with their interests (see Figure 2). At the end of the class, students take the computers home “for keeps.”

^[1] Quantum blocks were added to Minecraft through a mod that was collaboratively developed by CalTech’s Institute for Quantum Information and Matter, TeacherGaming, Google , E-Line Media, and Daniel Ratcliffe.

Positive, supportive STEM program climate

USP purposefully creates a positive space where students are recognized for their talents and interests. Problems in the education of inner-city youth can arise from issues of representation. The meaning ascribed to students and their behavior (how they are defined, described, and characterized in an education setting) affects how educators think about them and their needs, and shapes the activities that are developed for them. Views of students shape both education climate and the sense of belonging students develop.

Training of project leaders and college and high school mentors is designed to emphasize a positive climate—one that is supportive of students’ needs and respectful of their interests. Mentors learn to create caring relationships by sharing their stories and listening to those of the middle school participants. Stories are a powerful gateway to understanding the lives of others and to building influential relationships (McColgan, Colesante, and Clark 2019). This is particularly important for inner-city students who come from low-performing schools with systemic barriers, such as racism, low expectations, and teacher negativity, that dissuade students from advanced courses in STEM topics.

Group meetings occur at the start and end of each Saturday. The meetings focus on program themes such as collaboration, self-regulation, STEM literacy, responsibility, and respect. Shout-outs are used to compliment students or mentors for solving difficult problems, assisting others, or skillfully handling frustrations. Project leaders and college mentors model giving praise that is informative, not merely supportive. For example, when a student gives a shout-out that is a vague comment about how another student helped them, they are asked to elaborate on what the other person did that was helpful. When someone says they overcame an obstacle, they are asked to describe the obstacle and what another student did to help them overcome it.

In 2006 students in the program developed a mission statement that continues to be used in the program. It is recited each Saturday morning by a participant in the program. The mission statement is:

We choose to achieve our highest potential in order to bring out the best in ourselves and others.

We see challenges as opportunities, and we work creatively and respectfully to meet them.

We are responsible for our words, our actions, and our success.

We are confident! We are motivated!

We are on a quest to be our best!

We are on a quest to be our best!

The mission is used to provide a guide for expectations throughout the day. It is also used to direct attention to the common values and optimistic perspective of the program. At the conclusion of the day, shout-outs are focused specifically on elements of the mission.

The climate in individual classes is one of collaboration, similar to a STEM graduate school lab where people at different points in their education and career work together on a common theme or project. Project leaders deliver the lessons, college and high school STEM mentors provide support, and preservice education mentors work alongside students. Collectively, new content is explored, learned, and applied through hands-on activities and the use of the serious gaming approach to teaching previously described. A guided collaborative inquiry approach characterizes many of the learning experiences, in which students exercise choice, work in small teams (two or three people), and design methods to understand a concept or phenomenon. 

Outcomes

Data collection and primary questions

USP is evaluated annually for reporting to the school district. In the spring of each year, students provide feedback on the climate of the program, their attitudes toward STEM, and their interest in STEM careers. Recently, we began examining the impact of the program on NYS Assessments in ELA and Mathematics. Proficiency on these assessments was studied during participants’ middle school years, while they are participating in the program. Because former students have progressed through high school, we also examine standardized NYS Regents exams, choice of electives, and graduation rates.

Two key questions guide the evaluation of the program:

1. How do program participants compare to nonparticipants in the same school district?
2. How does attendance in the program relate to participants’ attitudes and interest in STEM?

How do USP students compare to non-USP students in standardized assessments?

Prior to the program

We compared participants with nonparticipants in the same district using a variety of indicators to determine the differences before participants joined the program. The indicators suggest that participants are similar to nonparticipants in disciplinary referrals, prior achievement, growth scores in math, and grades in school as judged by their teachers. For example, there was no difference in Northwest Evaluation Association scores before student participation and after one year of participation (NWEA 2005). The participants in the program did not have higher course grades than their peers. Violent and Disruptive Incident Reporting scores were analyzed, and there were no differences for participants in the program and nonparticipants from the district.

During the program

Middle school standardized ELA and math assessments

In New York State (NYS), students take standardized tests in ELA and mathematics at the end of each year from third to eighth grade. There are four proficiency levels to report student performance relative to NYS standards. Levels 1 and 2 indicate that a student is not proficient, whereas levels 3 and 4 indicate that a student is proficient. Performance is reported as a percentage of students passing, which is a sum of levels 3 and 4 (more information about the NYS Education Department testing program can be found online).

A comparison of ELA and math proficiency of participants and nonparticipants in the program is shown in Figure 3. Differences comparing participants and nonparticipants were significant for both tests (p < 0.0001).

STEM attitudes and career interest survey

We examined program attendance as it relates to measures of students’ interest in STEM and obtaining a career in STEM fields. Students in USP take the STEM Semantics survey (Tyler-Wood, Knezek, and Christensen 2010). For each STEM topic (science, technology, engineering, and mathematics) and for obtaining a career in a STEM field, students choose answers along a scale from 1 to 7 for the same semantic differentials (e.g., Fascinating–Mundane, Appealing–Unappealing, Boring–Interesting). The five scales have adequate indices of internal consistency reliability, with a Cronbach Alpha ranging from 0.84 for the technology scale to 0.89 for the science scale.

Each scale was correlated to the number of sessions a student attended in all years of the program (2011–2019). Number of sessions attended was found to be positively correlated to attitudes for science (r = 0.14, p = 0.011), engineering (r = 0.11, p = 0.040), and interest in a STEM career (r = 0.14, p = 0.010). The more sessions a student attended, the more positive were their attitudes about science, engineering, and a career in STEM. Figure 4 presents the means for these outcomes as a function of attendance. Attitudes toward mathematics and technology were nonsignificant (p > 0.05).

After leaving the program

High school Regents Exams

To receive a diploma, NYS requires students to take specific courses and achieve passing performance on standardized assessments called Regents Exams. Regents Exams are administered in all core subjects during the high school years. Unlike the middle school achievement tests, which report levels of performance relative to state standards, Regents Exams are scaled for comparability and reported for a passing score (a score of 65 or higher) and passing with mastery (a score of 85 or higher). To receive a Regents diploma with advanced designation, students must take additional courses and additional state Regents Exams. A Regents diploma with advanced designation is an indicator that the student is planning to pursue a college degree. More information about NYS high school graduation requirements can be found online.

The topics of the Regents courses include living environment (biology), Earth science, chemistry, physics, algebra, geometry, algebra 2, global history, U.S. history, and ELA. The passing rates on the state exams for USP participants and nonparticipants for test years from 2013 to 2017 are shown in Figure 5. Participants in the program passed at a higher rate than nonparticipants (p < 0.0001 ). Participants were also more likely to take advanced STEM electives that are not required for graduation, including honors and Advanced Placement courses (McColgan, Colesante, and Robin 2018).

Graduating with advanced designation

In many low-income and urban districts, including the one described in this article, graduation rates are low. Graduation rates for students in USP are higher than the average for students in the district. A higher percentage of USP participants (43%) graduate with advanced designation than nonparticipants (13%), an indicator of college readiness. Because students begin the program in fifth or sixth grade, and our analysis begins in the 2010–11 academic year, only a few cohorts have moved through high school to graduation. Still, participants have higher graduation rates (93%) than nonparticipants (62%) for the two years with data available (2016 and 2017). This is a preliminary indicator that participation in USP is positively related to graduation rates and college readiness.

Conclusions and future directions

Because USP does not align its curriculum with the school district or provide guidance past middle school, we were surprised to find academic differences, such as higher standardized ELA and math scores, higher rates of science course-taking in high school, and higher high school graduation rates between participants and non-participants. USP provides experiences that go beyond the standard curriculum and do not replicate the school’s instructional process or remediate students’ skills. It is primarily an enrichment program designed to enhance school experiences through a challenging and supportive atmosphere for learning.

Still, more research is needed on this program and other out-of-school time programs similar to it, using multiple measures to assess impact. Our next step involves further examination of participant and nonparticipant comparisons. We are also planning subgroup analyses using different measures of program intensity (measures of attendance and program involvement) to examine conditional program effects. For example, we will be examining whether there are differential impacts for students with varying levels of perceived family support, general and science-specific self-efficacy, and attitudes toward school. We plan to examine perceived levels of challenge and support in the program as moderators of program outcomes. We expect these analyses will lead to stronger evidence of causal impacts and areas where impacts are not seen.

Michele W. McColgan (mmccolgan@siena.edu) is director at the Siena Center for Informal Teaching and Learning in Science and the Arts and associate professor in the Department of Physics and Astronomy, both at Siena College in Loudonville, New York. Robert J. Colesante (rcolesante@siena.edu) is an associate at the Siena Center for Informal Teaching and Learning in Science and the Arts and professor in the Department of Education, both at Siena College in Loudonville, New York. Albert G. Andrade (andrade@cbaalbany.org) is an associate at the Siena Center for Informal Teaching and Learning in Science and the Arts at Siena College in Loudonville, New York, and a middle school teacher at Christian Brothers Academy in Albany, New York. Kenneth Robin (kenrobin23@gmail.com) is a senior research scientist at the Center for Human Services Research at the State University of New York at Albany in Albany, New York.