Abstract
To address the lack of a classroom observation protocol aligned with integrated STEM, the author team developed one to measure the degree of integrated STEM instruction implemented in K-12 science and engineering classrooms. This study demonstrates how our instrument can be used to uncover the dimensions of integrated STEM instruction practiced in K-12 classrooms and to determine which protocol items are associated with each of these dimensions. This article reports on the results of a principal component analysis (PCA) using 2030 K-12 classroom observation videos. PCA revealed two core dimensions of integrated STEM education. Real-world problem-solving includes 21st century skills and STEM practices necessary for developing solutions to real-world problems. Nature of Integrated STEM includes items that promote integration between the real-world context, students’ personal experiences, STEM careers, and STEM content. The authors’ analysis also suggests the possibility of an additional dimension of integrated STEM involving technology practices in STEM.
1 Introduction
Integrated STEM (science, technology, engineering, and mathematics) education is a global phenomenon with countries around the world working to engage students in interdisciplinary approaches to science learning (e.g., Australian Curriculum, Assessment, and Reporting Authority, 2016; Bascope et al., 2020; European Commission, 2015; Li, Yao, et al., 2020; National Research Council [NRC], 2012). Advocates of K-12 STEM education argue that teaching approaches which integrate disciplinary STEM content can greatly improve student learning (Jong et al., 2020). Integrated STEM instruction can also better prepare students to address 21st century problems, such as climate change, health, and the environment, which are inherently interdisciplinary in nature (e.g., Moore et al., 2020; National Academy of Engineering [NAE] and NRC, 2014). The inclusion of engineering in K-12 science education standards (e.g., NRC, 2012) further demonstrates the need for an integrated approach to STEM instruction. However, no single accepted definition of integrated STEM instruction exists, nor do researchers or educational practitioners agree on what integrated STEM looks like in practice (Moore et al., 2020).
This debate about definitions of integrated STEM instruction has hampered the development of protocols to observe and measure integrated STEM classroom practices. Guimarães and da Silva Lima (2021) conducted a systematic review of classroom observation protocols relevant to engineering education in active learning environments. This review uncovered 68 classroom protocols with four primary foci: (a) analysis of the emotional and instructional environment, (b) classroom management, (c) assessment of teaching and learning, and (d) observation of teacher and student behaviors. The protocols were also ranked using five dimensions (data collection, resources, training, robustness, and deployment). This ranking primarily focuses on logistical concerns such as cost and training requirements, whereas for integrated STEM researchers, our interest is alignment with common features of K-12 integrated STEM education. Additionally, existing classroom observation instruments focus solely on individual STEM disciplines (primarily science or mathematics) or good teaching practices while overlooking the nature of STEM integration entirely. For example, the Reformed Teaching Observation Protocol (RTOP; Sawada et al., 2002) and the Classroom Observation Protocol for Undergraduate STEM (COPUS; Smith et al., 2013) were all designed to evaluate the teaching of either science or mathematics content. Few protocols have attended specifically to engineering content (Guimarães & da Silva Lima, 2021); these protocols include the Science and Engineering Classroom Learning Observation Protocol (Dringenberg et al., 2012), the Classroom Observation Protocol for Engineering Design (COPED) (Wheeler et al., 2019), a modified RTOP (Love et al., 2017), and the Engineering Design-based Science Teaching Observation Protocol (EDSTOP) (Capobianco et al., 2018). While they target different grade levels, for example the EDSTOP was designed for use in elementary classrooms and the COPED was designed for use in secondary (grades 7–12) classrooms, these engineering protocols measure the same components of engineering design advocated within K-12 policies (e.g., NGSS Lead States, 2013; NRC, 2012). However, these instruments are ill-suited for use in either elementary or secondary classrooms that feature STEM integration because they either focus solely on the engineering design process or treat science and engineering as separate disciplines within a single protocol. Without a classroom observation protocol aligned with frameworks for integrated STEM, the field cannot move forward and offer useful recommendations to promote integrated STEM in K-12 classrooms.
To address these issues, we developed an observation protocol (Dare et al., 2021) designed to measure the degree of integrated STEM instruction present in K-12 science and engineering classrooms. The purpose of the present study is to show how our instrument can be used to uncover the primary dimensions of integrated STEM instruction practiced in K-12 classrooms and to determine which items are associated with each of these dimensions. We report the results of principal component analysis (PCA) used to extract the principal components or primary dimensions of integrated STEM instruction from scores generated using our instrument to observe K-12 classroom practices.
2 Theoretical Framework
Understanding how best to enact integrated STEM instruction has proven challenging partly because of the debate in the literature about the nature of integrated STEM education (Li, Wang, et al., 2020; Moore et al., 2020). Although there is a lack of consensus in the literature regarding how best to define and implement integrated STEM instruction, it is clear that integrated STEM is a multidimensional construct. The development of our STEM observation protocol was guided by our framework (Roehrig, Dare, Ellis et al., 2021) which includes seven key characteristics of integrated STEM: (a) focus on real-world problems, (b) centrality of engineering, (c) context integration, (d) content integration, (e) STEM practices, (f) 21st century skills, and (g) informing students about STEM careers. In the following section, we first situate our framework by discussing engineering as a discipline within K-12 settings. This is followed by a brief literature review for each characteristic of the framework; a more detailed description can be found in Roehrig, Dare, Ellis et al. (2021).
Given that engineering design is central within our conception of integrated STEM, it is important to first consider the nature of engineering and its representation within K-12 science standards (e.g., NGSS Lead States, 2013; NRC, 2012). Merrill and colleagues (2008) identified constraints, optimization, and predictive analysis as three core engineering concepts that are important to be addressed in high school settings. Constraints are factors such as cost, feasibility, materials, and environmental considerations that need to be considered by students throughout the design process. Optimization has the goal of producing the best design within the stated criteria and constraints. Predictive analysis occurs as students consider possible design solutions in light of scientific and mathematical principles to determine the potential of different designs. Within the Next Generation Science Standards (NGSS; NGSS Lead States, 2013), these engineering concepts are embedded into the scientific and engineering practices, as well as the two disciplinary core ideas specific to engineering: (a) defining and delimiting an engineering problem and (b) optimizing the design solution. Dearing and Daugherty (2004) also identified core engineering concepts for K-12 settings, with topics such as interpersonal-skills, working within constraints, brainstorming, and product design assessment being chief among them. Unlike scientific concepts, engineering concepts are represented as practices; even within the two disciplinary core ideas, engineering is positioned as practices (Cunningham & Carlsen, 2014). As such, engineering represents both a specific form of a real-world problem as a context, as well as specific STEM practices related to engaging in the engineering design process.
2.1 Focus on Real World Problems
Common across definitions of integrated STEM is a focus on real-world problems (Moore, Stohlmann, et al., 2014; Kelley & Knowles, 2016; Kloser et al., 2018). Real-world problems are inherently interdisciplinary in nature, thus they provide a context for learning and applying concepts from multiple disciplines. Additionally, real-world problems are complex in nature, leading to multiple possible solutions that provide a context for learning that promotes creativity and critical thinking.
Framing integrated STEM instruction through real-world problems provides motivation for student learning as many students find it difficult to relate to STEM content through traditional, single-disciplinary approaches (Djonko-Moore et al., 2018; Kelley & Knowles 2016). However, care needs to be taken that these real-world problems are aligned with students’ interests and lived experiences. For example, a focus on societal issues such as health, the environment, and social justice is more relevant and motivational for females and students of color, compared to traditionally male-oriented problems that focus on technical aspects, such as designing cars and rockets (Djonko-Moore et al., 2018; Schellinger et al., 2018).
2.2 Centrality of Engineering
Given the prominence of engineering within STEM policy documents (e.g., NRC, 2012), real-world problems are often portrayed as engineering design challenges within STEM curricula (Stohlmann, et al., 2014; Berland & Steingut, 2016). Developing solutions to an engineering design challenge relies on using and developing understanding of content from multiple disciplines (e.g., Thibaut et al., 2018) and engaging in engineering design practices (Berland & Steingut, 2016; NAE and NRC, 2014). As students iteratively test their designs, they are expected to reflect on how well their design addresses the client’s needs and use their knowledge of STEM content and data from iterative testing to refine their solutions (Siverling et al., 2019). Thus, it is critical that K-12 students have opportunities to fully engage in the iterative engineering design process and engage in at least one cycle of redesign (Wendell et al., 2017).
2.3 Context Integration
Context integration occurs through the explicit connections of STEM concepts and practices to the real-world problems that engage learners in applying and expanding their knowledge of the STEM disciplines (Berland & Steingut, 2016). There needs to be clear alignment between the engineering design challenge or real-world problem. Further, specific content learning objectives need to be included, as without explicit integration between the problem context and content learning goals, students will resort to tinkering (a form of trial and error) (McComas & Burgin, 2020; Roehrig, Dare, Ring-Whalen et al., 2021).
2.4 Content Integration
In addition to context integration, integrated STEM lessons should also explicitly address integration across STEM content areas to help students “build knowledge and skill both within the disciplines and across disciplines” (NAE and NRC, 2014, p. 5). Although teachers may understand the connections across the different disciplines within an integrated STEM lesson, students often struggle to make these connections on their own and rarely spontaneously recognize them without support (Tran & Nathan, 2010). Because of this, it is critical for teachers to make these connections explicit in their instruction (English, 2016; Kelley & Knowles, 2016).
2.5 STEM Practices
Integrated STEM instruction should directly engage students in STEM practices such as scientific and engineering practices (NRC, 2012), evidence-based reasoning (Siverling et al., 2019), and the creation, collection, manipulation, analysis, and visualization of data (Weintrop et al., 2016). Such practices are “a representation of what practitioners do as they engage in their work, and they are a necessary part of what students must do to learn a subject and understand the nature of the field” (Reynante et al., 2020, p. 3). This emphasis on teaching students how to “do STEM” highlights an important pedagogical shift in STEM education away from teaching STEM as the rule-based application of a well-established body of facts and toward a greater appreciation of the complexities that face STEM practitioners in professional settings. It is important that students are given the opportunity to exercise agency when engaging in integrated STEM (Berland & Steingut, 2016; Miller et al., 2018).
2.6 21st Century Skills
Integrated STEM instruction should support the development of 21st century skills (e.g., Moore et al., 2014; Sias et al., 2017). This is essential because the future STEM workforce needs employees to have strong 21st century skills such as critical thinking, communication, collaboration, and creativity (Charyton, 2015). Beyond STEM workforce considerations, 21st century skills are considered critical for any person “to adapt and thrive in an ever-changing world” (Stehle & Peters-Burton, 2019, p. 2). Engaging in developing solutions to real-world problems and engineering design challenges inherently incorporates creativity and critical thinking as there is no single correct solution to these complex problems (Stretch & Roehrig, 2021; Simpson et al., 2018). Iterative testing and learning from failure lead to stronger designs and innovation through the application of creativity and critical thinking (Simpson et al., 2018).
Within K-12 classrooms, students are expected to work collaboratively within small groups to co-construct knowledge of STEM content and design solutions to real-world problems (e.g., Moore et al., 2014). This is especially important given that small group activities account for more than half of instructional time during integrated STEM units (Wieselmann et al., 2020). Within these small groups, students are expected to develop negotiated design solutions that synthesize across differing understandings of the same problem space (Wendell et al., 2017).
2.7 STEM Career Awareness
As previously noted, integrated STEM allows students to engage in authentic STEM practices and 21st century skills, both of which are critical elements in promoting the development of positive identities toward STEM (Kitchen et al., 2018). This is important given the policy focus on STEM workforce readiness as research shows that STEM interest, attitude, and identity, not academic performance in STEM, predict sustained pursuit in the STEM disciplines (Tai et al., 2016). Given the goal of promoting future participation in STEM careers, integrated STEM education should not only engage students in authentic STEM practices, but also expose students to details about STEM careers (Jahn & Myers, 2014; Luo et al., 2021).
3 Methods
3.1 Overview of Instrument Development
In prior research, we applied the theoretical framework described above (Roehrig, Dare, Ellis et al., 2021) to a sample of classroom observations drawn from a repository of over 2,000 videos obtained in a professional development (PD) grant project to design an observation instrument that could be used to measure the degree of integrated STEM instruction occurring in K-12 science and engineering classrooms as described in Dare et al. (2021). The videos used to develop our instrument (and later to generate data for the analysis reported below) were recorded in the classrooms of teachers in grades 3–9 (primarily elementary teachers, elementary science specialists, and middle school science teachers) recruited from five school districts (representing urban, inner-ring suburban, and outer-ring suburban environments in the midwestern United States) to complete a three-week PD workshop designed to promote science learning through engineering design activities and the development of integrated STEM curriculum that centralized engineering as the integrator of STEM content (Moore, Glancy et al., 2014; Moore, Stohlmann, et al., 2014). Each video in this repository represents a single instructional period of roughly 50-minutes in length recorded daily for the entirety of a curriculum unit that ranged between one and several weeks of instruction. Through inductive analysis of a sample of this dataset, we developed a 10-item instrument with four scoring levels (0–3) per item designed to measure the seven characteristics of integrated STEM outlined in our theoretical framework above (see Dare et al., 2021 for details on the instrument development process). Table 1 provides a summary of these 10 items and Figure 1 demonstrates how they align with our seven characteristics of integrated STEM. Figure 1 shows that Item 1 measures the extent to which “context integration” occurs in a given observation; Item 2 measures the extent to which instruction is grounded in “real-world problems;” Items 3, 6, 8, and 9 measure engagement in various STEM practices; Items 4 and 7 measure the development of two important 21st century skills; Item 5 measures “content integration” across STEM disciplines; and Item 10 measures attempts to elevate “STEM career awareness.” It is important to emphasize that each characteristic of integrated STEM is represented by items that can be scored through direct observation of whole class instruction (Dare et al., 2021). A thorough exploration of some characteristics would require additional data, including observation of small group work and collection of student work. For example, when considering 21st century skills, Items 4 and 7 assess the degree to which critical thinking and collaboration are present in a lesson. Other 21st century skills, such as creativity, would require examination of students’ work to understand how creativity was evident in the lesson.
Brief description of STEM observation protocol items
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Brief description of STEM observation protocol items
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Brief description of STEM observation protocol items
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Alignment of observation protocol items and characteristics of integrated STEM
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Alignment of observation protocol items and characteristics of integrated STEM
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Alignment of observation protocol items and characteristics of integrated STEM
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Of note in Figure 1, is the overlap of the characteristic of the centrality of engineering with the characteristics of real-world problems and STEM practices. As described in our theoretical framework, engineering provides context for learning as a specific example of a real-world problem, Item 2, as well as specific content and practices necessary for the development of solutions for an engineering design challenge, the specific engineering practices described in Item 3. Thus, no single item or set of items measure the centrality of engineering because we determined that this was already implicitly captured in Items 2 and 3. Since it would have been redundant and confusing to measure these things twice, the centrality of engineering became an implied rather than directly measured characteristic of our instrument.
Once our instrument was developed, we then assigned a team of seven coders to use our instrument to independently score a new random sample of roughly 100 classroom videos drawn from our repository to evaluate item reliability (see Dare et al., 2021). The seven coders included Gillian H. Roehrig, who has published widely on the topic of K-12 STEM integration, two post-doctoral students and four graduate students. The post-doctoral and graduate students all had taken graduate coursework on integrated STEM and were specifically trained on the use of the STEM-OP by the project leaders. All items achieved an inter-rater reliability above our acceptability threshold of Krippendorff’s
3.2 Principal Component Analysis
PCA is a dimension reduction technique commonly used to simplify and facilitate the understanding of multivariate data analyzed across large numbers of cases. PCA was chosen over alternative analysis techniques, such as exploratory factor analysis (EFA), item-response theory (IRT), or latent class analysis (LCA), because we designed the protocol to measure integrated STEM instruction as a formative rather than reflective concept. We conceptualized the observed individual items of the protocol to be the formative causes of the various aggregated dimensions of integrated STEM instruction that we hoped to uncover in our analysis rather than these reflective dimensions being the latent causes of our observed individual protocol items. In other words, we designed each individual protocol item to measure something unique to integrated STEM instruction with the intention of seeing which items tended to “hang together” within a classroom observation rather than trying to uncover latent dimensions of integrated STEM instruction using multiple measures of the same concept.
The data for the PCA included 2,030 classroom video-observations generated from the previously described PD program. This sample size far exceeds the minimum 10-to-1 case-to-item ratio recommendations of Costello and Osborne (2005). The data set represents a wide range of teachers (106 separate teachers), classroom settings (434 earth science, 597 life science, and 999 physical science classrooms), curriculum units (48 in total), and grades (6 lower elementary, 879 upper elementary, 1071 middle school, and 74 high school observations). Each video represents a single instructional period of roughly 50-minutes in length recorded daily for the entirety of curriculum units ranging between one and several weeks of instruction. This reflects the reality of classroom observations such that, in many cases, they are conducted only on one day of instruction rather than for a full unit of instruction.
To further reinforce scoring reliability among our coders, 200 classroom videos were coded in triads among six coders with each triad assigning two coders to score the same video independently and the third coder serving as a neutral arbiter to facilitate the process of coming to consensus on items whenever the two coders disagreed. We then assigned these same six coders to independently score the remaining classroom videos in our repository to produce the full dataset used in our analysis reported below.
4 Results
This section presents the result of our principal component analysis (PCA) used to identify the primary dimensions of integrated STEM instruction practiced in our observed K-12 classroom data. After reporting the results of our PCA, we then provide an explanation of the underlying dimensions uncovered in our analysis interpreted through the lens of our theoretical framework to contextualize and ground these findings within the pre-existing literature.
4.1 Principal Component Extraction
We first used the default principal component extraction method in the SPSS statistical software package to analyze the correlation matrix of our observed item scores using promax rotation to aid in the interpretation of our component loadings. We chose to rotate our solution using oblique rotation under the assumption that the dimensions uncovered in our analysis were likely to correlate in a classroom setting since each is likely to support the other in effective instructional practice. We chose to analyze the correlation matrix to extract our principal components given that our instrument items were all equivalently scaled by default. Analysis of the eigenvalues using the Kaiser-Guttman rule and the accompanying scree plot show that a maximum of three underlying dimensions of integrated STEM instruction were present in our observed K-12 classrooms (see Table 2 and Figure 2). These three principal components captured a cumulative percentage of the total variance of 60.7%, with the first component responsible for 34.7% of the overall explained variance, the second component responsible for an additional 15.1%, and the third component responsible for an additional 11.0%.
Total variance explained and eigenvalues of extracted principal components
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Total variance explained and eigenvalues of extracted principal components
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Total variance explained and eigenvalues of extracted principal components
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Screen plot of eigenvalues of extracted principal components
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Screen plot of eigenvalues of extracted principal components
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Screen plot of eigenvalues of extracted principal components
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
4.2 Principal Component Loadings and Interpretations
Table 3 reports the rotated loadings of the STEM-OP items on the three principal components extracted in our PCA with loadings less than 0.40 suppressed to facilitate interpretation.
Protocol item loadings on first three extracted principal components analyzed using the correlation matrix and rotated using Promax rotation
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Protocol item loadings on first three extracted principal components analyzed using the correlation matrix and rotated using Promax rotation
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Protocol item loadings on first three extracted principal components analyzed using the correlation matrix and rotated using Promax rotation
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Table 3 shows that Items 3, 4, 6, 7, and 8 load most strongly on the first principal component with no overlapped loading of these items across the other two principal components. Items 1, 2, 5, and 10 load most strongly on the second principal component with Item 10 also loading negatively on the third principal component. Item 9 loads most strongly on this third and final principal component but in an inverse manner to Item 10.
Protocol item loadings on first three extracted principal components analyzed using the correlation matrix and reported without rotation
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Protocol item loadings on first three extracted principal components analyzed using the correlation matrix and reported without rotation
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Protocol item loadings on first three extracted principal components analyzed using the correlation matrix and reported without rotation
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Table 4 shows the component matrix of our PCA or the unrotated solution. For transparency, we report the component matrix to demonstrate the consistency of our item loadings. The emphasis here is on the degree of similarity between our rotated and unrotated solutions to determine the extent to which our results hold without rotating our solution.
The results of the unrotated solution in Table 4 are nearly the same as the rotated solution presented in Table 3 with two minor exceptions. First, Item 2 cross-loads onto Component 1 and Item 5 cross-loads onto Component 3 rather than both items loading solely onto Component 2 as in our rotated solution. Second, Item 10 loads solely onto Component 2 and no longer negatively cross-loads onto Component 3 as in our rotated solution. This means that there are no negative loadings in the unrotated solution to interpret. As such, the interpretation of Component 1 remains essentially the same with the exception of the addition of Item 2, and the interpretation of Component 2 is fundamentally the same as in the rotated solution. The only major difference between the rotated and unrotated solutions is Component 3 which went from being composed of Item 9 and Item 10 inversely loaded to Item 9 and Item 5 loading together positively.
STEM-OP item loadings on first three extracted principal components analyzed using the covariance matrix and reported without rotation
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
STEM-OP item loadings on first three extracted principal components analyzed using the covariance matrix and reported without rotation
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
STEM-OP item loadings on first three extracted principal components analyzed using the covariance matrix and reported without rotation
Citation: Research in Integrated STEM Education 1, 1 (2023) ; 10.1163/27726673-00101004
Table 5 shows the results of using the covariance matrix to extract our principal components rather than the correlation matrix. As with the unrotated solution, we report these results to demonstrate the consistency of our item loadings regardless of the method used to extract our principal components. Although our instrument items were designed to measure STEM instruction using the same 4-point scale and the correlation matrix is preferred when items are equivalently scaled, we report the results of using the covariance matrix to extract our principal components simply to demonstrate the extent to which our findings hold even if our scale equivalency assumption happened to be violated in our data due to issues like skewed item-score distributions. In other words, we see this as the most conservative way to report the possible underlying dimensions of integrated STEM instruction in that it relies on fewer assumptions than the correlation matrix approach. Thus, we believe these findings demonstrate the “bare minimum” dimensions of integrated STEM we can reliably infer from our classroom observation data.
The only fundamental difference between the correlation and covariance extraction approach is the disappearance of Component 3 and the failure of Item 9 to load onto any of the remaining components. Even after relaxing the scale equivalency assumption, Component 1 and Component 2 remain exactly the same as in the unrotated solution using the correlation matrix to extract these components and the only difference between the covariance result and the rotated solution reported above is the cross-loading of Item 2 onto both Components 1 and 2 rather than Component 2 alone.
5 Discussion
The PCA results reported in the previous section demonstrate how our instrument can be used to uncover the primary dimensions of integrated STEM instruction present in observed K-12 classrooms. They also show that two core dimensions emerged from the analysis of our data centered on items that consistently load onto Component 1 and Component 2, with the possibility of a third dimension centered on instructional practices related to Item 9. In this section, we provide an interpretation of these dimensions to ground our findings in the existing literature and theoretical conceptualization of integrated STEM instruction. We first interpret the two core dimensions represented in Components 1 and 2 that we label “Real-World Problem Solving” and “Nature of STEM integration”, respectively, before interpreting the possibility of a third dimension centered on Technology Practices in STEM.
5.1 Real-World Problem Solving
Component 1 represents the first core dimension of integrated STEM instruction that includes Items 3 (developing multiple solutions), 4 (cognitive engagement in STEM), 6 (student agency), 7 (student collaboration), 8 (evidence-based reasoning), and possibly 2 (contextualizing student learning). These items describe behaviors representative of Real-World Problem Solving, such as the application, analysis, and evaluation of STEM concepts (Item 4), the use of evidence-based reasoning (Item 8), collaborative construction of knowledge and design solutions (Item 7), and the development, evaluation, and redesign of multiple solutions (Item 3). In high quality integrated STEM lessons these behaviors require students to exercise agency in their use of STEM practices (Item 6), often with a real-world problem or engineering design challenge being used to contextualize engagement (Item 2) in these STEM practices.
It is important to underscore that this dimension represents the practices that students engage in as they work to develop solutions to a real-world problem and/or engineering design challenge. Thus, the loading of Item 6 (student agency) in this core dimension is theoretically appealing, as it is illustrative of students having agency in determining possible solutions. Engaging students in engineering design is a shift from working on well-defined problems with single, correct solutions to working on open-ended problems with multiple solution pathways (Jonassen et al., 2006), which means that solutions cannot be developed through structured, routine procedures.
The possible loading of Item 2 in the unrotated and covariance matrix solution of this dimension makes sense as a strategy to engage students in problem-solving for real-world purposes that encourages multiple solution pathways due to the complexity and open-ended nature of real-world problems. This requires students to engage in the 21st century skills of collaboration and critical thinking (Items 4 and 7) throughout an integrated STEM lesson and as they engage in STEM practices (Items 6 and 8) to generate possible design solutions and engage in analysis of evidence to iteratively improve design solutions (Item 3 and 8) (Simpson et al., 2018; Stretch & Roehrig, 2021).
Engaging in an integrated STEM lesson contextualized through an engineering design challenge allows students to develop and exercise important skills needed for the 21st century, especially as they relate to STEM. Critical thinking is implicit across items within the dimension of Real-World Problem Solving. The levels of Items 3, 4, 7, and 8 each mirror Bloom’s taxonomy in describing a continuum of cognitive engagement, where at the highest levels, students are expected to apply what they have learned to develop possible design solutions and improve their designs through iterative analysis and evaluation (Sharunova et al., 2020). The loading of Item 3 suggests that engaging students in proposing and iteratively testing solutions to an engineering design challenge provides a unique context for the development of 21st century STEM skills.
5.2 The Nature of STEM Integration
Component 2 represents the second core dimension of integrated STEM instruction present in our classroom observations that includes Items 1 (relating content to students’ lives), 2 (contextualizing student learning), 5 (integrating STEM content), and 10 (STEM career awareness). The concurrent loading of these items suggests that integration takes multiple forms within a STEM lesson.
As discussed in our theoretical framework and development of our instrument, engineering represents a context for student learning and specific content to be learned based on K-12 standards (Ekiz-Kiran & Aydin-Gunbatar, 2021). Context integration (Item 2) represents the expectation that students learn and apply STEM content to address the real-world problem and/or engineering design challenge. As students engage in the engineering design process they learn and apply engineering practices and content in conjunction with content from the other STEM disciplines (Item 5). Given the dual role of engineering as context and content in integrated STEM lessons, it is not surprising that Items 2 and 5 both load onto our second core dimension, the Nature of Integrated STEM. The loading of Items 1 and 10 suggest that in addition to engineering design challenges providing a context for student learning, lessons can also be contextualized using students’ personal experiences and STEM careers.
The loading of Item 1 onto this dimension suggests that students’ personal experiences are attended to by teachers in helping students understand the real-world context. For example, some observed lessons in our data set focused on place-based issues, such as environmental issues in local lakes and rivers and run-off from a school parking lot. Such place-based pedagogies are grounded in the idea that content matters in helping students to interpret and understand real-world problems and can empower students to take action within their locality (Nieto & Bode, 2007).
STEM career awareness (Item 10) also loads into the core dimension of Nature of STEM Integration. STEM careers are a specific connection to the real-world and as such they represent a form of context integration. The presence of connections to STEM careers is aligned with the policy goals for integrated STEM intending to promote interest in STEM careers. For example, in our video repository, teachers integrated information about the kinds of STEM professionals that worked on the type of real-world problem at hand, as well as discussing how specific STEM professionals engaged in addressing real-world problems.
5.3 Technology Practices in STEM
The only item on our instrument that failed to load consistently with any of the other items in a given component was Item 9, which measures the use of technology to facilitate data practices associated with both science and engineering. We included this item on our instrument because data practices are central to knowledge construction and the work of STEM professionals (Duschl et al., 2007) by way of the creation, collection, manipulation, analysis, and visualization of data (Weintrop et al., 2016). The rapid growth of data and the need to effectively manage large data sets further necessitates that students have opportunities to learn how to properly use technology to facilitate these data practices (Ellis et al., 2020). Given the importance of data practices in STEM, it is somewhat surprising that Item 9 failed to load onto the component representing the Real-World Problem Solving dimension of integrated STEM. This may be because technology practices were evidenced in only 403 classroom observations (approximately 20%) in our dataset so the opportunities for this item to correlate with other items were simply lower in our observed data. The low prevalence of Item 9 may also be due to the fact that student engagement in data practices was often limited to the testing and evaluation of possible design solutions (accounting a limited number of lessons within a given unit) and would therefore not be expected to occur in as many lessons within an integrated STEM unit as any other potentially related items. Several of the observed integrated STEM units also asked students to collect data without the assistance of technology (primarily elementary lessons) or required students to qualitatively analyze their design with respect to stated criteria and constraints rather than using data collection and analysis to evaluate design solutions, leading to the overall absence of Item 9 in many classroom observations.
While our PCA results suggest the possibility of a third dimension of integrated STEM centered on data practices, more research is needed to determine the extent to which data practices are likely to correlate with other items on our instrument or are more likely to occur in isolation or possibly even correlate with other aspects of integrated STEM not measured by our instrument. The analysis of more classroom data that feature technology assisted data practices would aid our efforts to better understand the role of technology and data practices and to develop additional items representative of data and technology practices. Although our findings are limited in the sense of having fewer observations of Item 9 than we would have desired, it is still worth noting that data practices were more likely to have occurred in our observed classrooms in the presence of teachers making connections across disciplines (Item 5), in the absence of teachers enhancing student awareness of STEM careers (Item 10), and somewhat haphazardly in regards to the areas the remaining items on our instrument measure.
6 Conclusions
Principal component analysis of data from our instrument reveals two core dimensions of integrated STEM (Real-World Problem Solving and Nature of STEM Integration). Interestingly, Item 2 (contextualizing student learning) serves as a bridge across the two core dimensions, loading into both dimensions. This is notable as the most common feature of integrated STEM within the literature is the use of a real-world problem (e.g., Kelley & Knowles, 2016; Kloser et al., 2018; Moore et al., 2020). Real-world problems represent both a form of integration and the necessary context for application of STEM practices and 21st century skills.
As described above, the dimension of Real-World Problem Solving includes observable student behaviors related to STEM practices and 21st century skills that are consistent with the integrated STEM literature (e.g., Moore, Stohlmann, et al., 2014; Kelly & Knowles, 2016; NAE and NRC, 2014) and necessary for proposing, testing, and refining solutions to real-world problems, which given the current NGSS reforms are most often presented as engineering design challenges.
The dimension representing the Nature of STEM Integration incorporates four different aspects of integration: (a) context integration which connects the target content to the context of the real-world problem or engineering design challenge (Item 2) (e.g., Moore, Stohlmann, et al., 2014; Kelley & Knowles, 2016), (b) content integration which connects across the disciplines (Item 5) (e.g., Moore, Stohlmann, et al., 2014; English, 2016; Kelley & Knowles, 2016), (c) connections to students’ lived experiences (Item 1) (e.g., Djonko-Moore et al., 2018), and (d) connections to STEM careers (Item 10) (Jahn & Myers, 2014; Luo et al., 2021).
7 Limitations
The work described above includes both theoretical and practical limitations. Our findings are strongly conditioned by our theoretical conceptualization of integrated STEM instruction, and we are aware that ours is not the only way to define this concept. We have sought to identify a core consensus in the literature despite a range of understandings of integrated STEM education. We realize that some educators and researchers may disagree with the emphasis placed on engineering and engineering design; however, within the realm of science classrooms within the United States, the NGSS calls for the integration of engineering and engineering practices and as such a focus on engineering design challenges is aligned with these current reform efforts (Moore, Stohlmann, et al., 2014; NAE and NRC, 2014). We encourage others to explore ways in which this instrument could be modified for additional classroom spaces, such as in mathematics or computer science classrooms.
As suggested previously, an important limitation lies in the dataset that we used during the development and subsequent testing processes. These classroom observations are bound by the previously described project that used two specific frameworks of integrated STEM education (Moore, Glancy et al., 2014; Moore, Stohlmann, et al., 2014). Our theoretical framework (Roehrig, Dare, Ellis et al., 2021) draws on research relevant to expectations of K-12 science teachers and addresses standards relevant to all grade levels (K-12). For example, students at all ages are expected to engage in science and engineering practices (NGSS Lead States, 2013; NRC, 2012). However, our dataset was limited primarily to upper elementary and middle school classrooms in the United States, so our findings may not be applicable to lower elementary classrooms, high school classrooms, or international settings. Future research should explore the use of the STEM observation protocol in lower elementary and high school settings. Additionally, exploration of the use of the protocol beyond classroom research purposes would be beneficial. For example, future research could consider the application of the protocol in teacher education settings or professional learning communities within schools to assess its use in guiding teacher learning with respect to integrated STEM.
Acknowledgements
This research was made possible by the National Science Foundation grants 1854801, 1812794, and 1813342. The findings, conclusions, and opinions herein represent the views of the authors and do not necessarily represent the view of personnel affiliated with the National Science Foundation.
References
Australian Curriculum, Assessment, and Reporting Authority (2016). ACARA STEM Connections Project Report. Retrieved from www.australiancurriculum.edu.au/media/3220/stem-connections-report.pdf.
Bascope, M., Reiss, K., Morales, M., Robles, C., Reyes, P., Duque, M. I., & Andrade, J. C. (2020). Latin American STEM policy: A review of recent initiatives on STEM education in four Latin American countries. In Johnson, C. C., Mohr-Schroeder, M. J., Moore, T. J., & English, L. (Eds.), Handbook of research on STEM education (pp. 443–458). Routledge.
Berland, L. K., & Steingut, R. (2016). Explaining variation in student efforts towards using math and science knowledge in engineering contexts. International Journal of Science Education, 38(18), 2742–2761. www.doi.org/10.1080/09500693.2016.1260179.
Capobianco, B. M., DeLisi, J., & Radloff, J. (2018). Characterizing elementary teachers’ enactment of high‐leverage practices through engineering design‐based science instruction. Science Education, 102(2), 342–376.
Charyton, C. (2015). Creative engineering design: The meaning of creativity and innovation in engineering. In C. Charyton (Ed.), Creativity and innovation among science and art: A discussion of the two cultures (pp. 135–152). Springer-Verlag Publishing. www.doi.org/10.1007/978-1-4471-6624-5_7.
Costello, A. B., & Osborne, J. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research, and Evaluation, 10(1), 7.
Cunningham, C. M., & Carlsen, W. S. (2014). Teaching engineering practices. Journal of Science Teacher Education, 25, 197–210.
Dare, E. A., Hiwatig, B., Keratithamkul, K., Ellis, J. A., Roehrig, G. H., Ring-Whalen, E. A., Rouleau, M. D., Faruqi, F., Rice, C., Titu, P., Li, F., Wieselmann, J. R., & Crotty, E. A. (2021, July). Improving integrated STEM education: The design and development of a K-12 STEM observation protocol (STEM-OP) (RTP). In Proceedings of the 2021 ASEE Annual Conference & Exposition. https://peer.asee.org/improving-integrated-stem-education-the-design-and-development-of-a-k-12-stem-observation-protocol-stem-op-rtp.pdf.
Dearing, B. M., & Daugherty, M. K. (2004). Delivering engineering content in technology education. The Technology Teacher, 64(3), 8–11.
Djonko-Moore, C., Leonard, J., Holifield, Q., Bailey, E., & Almughyirah, S. (2018). Using culturally relevant experiential education to enhance urban children’s knowledge and engagement in science. Journal of Experiential Education, 41(2), 137–153.
Dringenberg, E., Wertz, R., Purzer, S. & Strobel, J. (2012). Development of the Science and Engineering Classroom Learning Observation Protocol. In Proceedings of American Society of Engineering Education (ASEE) Annual Conference & Exposition, San Antonio, TX.
Duschl, R. A., Schweingruber, H. A., & Shouse, A. W. (2007). Taking science to school: learning and teaching science in grades K-8. National Academies Press, Washington, DC.
Ekiz-Kiran, B., & Aydin-Gunbatar, S. (2021). Analysis of engineering elements of K-12 science standards in seven countries engaged in stem education reform. Science & Education, 30, 1–34.
Ellis, J., Wieselmann, J., Sivaraj, R., Roehrig, G., Dare, E., & Ring-Whalen, E. (2020). Toward a productive definition of technology in science and STEM education. Contemporary Issues in Technology and Teacher Education, 20(3), Article 1 https://www.citejournal.org/volume-20/issue-3-20/science/toward-a-productive-definition-of-technology-in-science-and-stem-education/.
English, L. D. (2016). STEM education K-12: Perspectives on integration. International Journal of STEM Education, 3(1), 3.
European Commission (2015). Science education for responsible citizenship. Brussels, Belgium: European Union.
Guimarães, L. M., & da Silva Lima, R. (2021). A systematic literature review of classroom observation protocols and their adequacy for engineering education in active learning environments. European Journal of Engineering Education. DOI: 10.1080/03043797.2021.1937946.
Jahn, J. L. S., & Myers, K. K. (2014). Vocational Anticipatory Socialization of Adolescents: Messages, Sources, and Frameworks That Influence Interest in STEM Careers. Journal of Applied Communication Research, 42(1), 85–106.
Jonassen, D. H., Strobel, J., & Lee, C. B. (2006). Everyday problem solving in engineering: Lessons for engineering educators. Journal of Engineering Education, 95(2), 1–14.
Jong, C., Priddie, C., Roberts, T., & Museus, S. D. (2020). Race-related factors in STEM: A review of research on educational experiences and outcomes for racial and ethnic minorities. In C. C. Johnson, M. J. Mohr-Schroeder, T. J. Moore, & L. D. English (Eds). Handbook of research in STEM education (pp. 278–288). Routledge.
Kelley, T. R., & Knowles, J. G. (2016). A conceptual framework for integrated STEM education. International Journal of STEM Education, 3(11), 1–11.
Kitchen, J. A., Sonnert, G., & Sadler, P. M. (2018). The impact of college- and university-run high school summer programs on students’ end of high school STEM career aspirations. Science Education, 102(3), 529–547.
Kloser, M., Wilsey, M., Twohy, K. E., Immonen, A. D., & Navotas, A. C. (2018). “We do STEM”: Unsettled conceptions of STEM education in middle school S.T.E.M. classrooms. School Science and Mathematics, 118(8), 335–347.
Li, J., Yao, J., Luo, T., & So, W. W. M. (2020). STEM Policy in Asia. In Johnson, C. C., Mohr-Schroeder, M. J., Moore, T. J., & English, L. (Eds.), Handbook of research on STEM education (pp. 416–427). Routledge.
Li, Y., Wang, K., & Froyd, J. E. (2020). Research and trends in STEM education: A systematic review of journal publications. International Journal of STEM Education, 7(11), 1–16.
Love, T. S., Wells, J. G., & Parkes, K. A. (2017). Examining the Teaching of Science, and Technology and Engineering Content and Practices: An Instrument Modification Study. Journal of Technology Education, 29(1), 45–66.
Luo, T., So, W. W. M., Wan, Z. H., & Li, W. C. (2021). STEM stereotypes predict students’ STEM career interest via self-efficacy and outcome expectations. International Journal of STEM Education, 8(36). www.doi.org/10.1186/s40594-021-00295-y.
McComas, W. F., & Burgin, S. R. (2020). A Critique of “STEM” Education Revolution- in-the-Making, Passing Fad, or Instructional Imperative? Science & Education, 29, 805–829.
Merrill, C., Custer, R. L., Daugherty, J., Westrick, M., & Zeng, Y. (2008). Delivering Core Engineering Concepts to Secondary Level Students. Journal of Technology Education, 20(1), 48–64.
Miller, E., Manz, E., Russ, R., Stroupe, D., & Berland, L. (2018). Addressing the epistemic elephant in the room: Epistemic agency and the next generation science standards. Journal of Research in Science Teaching, 55(7), 1053–1075.
Moore, T. J., Johnston, A. C., & Glancy, A. W. (2020). STEM integration: A synthesis of conceptual frameworks and definitions. In C. C. Johnson, M. J. Mohr-Schroeder, T. J. Moore, & L. D. English (Eds). Handbook of research in STEM education (pp. 3–16). Routledge.
Moore, T. J., Glancy, A. W., Tank, K. M., Kersten, J. A., & Smith, K. A. (2014). A framework for quality K-12 engineering education: Research and development. Journal of Pre-College Engineering Education Research, 4(1), 1–13.
Moore, T. J., Stohlmann, M. S., Wang, H.-H., Tank, K. M., Glancy, A., & Roehrig, G. H. (2014). Implementation and integration of engineering in K-12 STEM education. In J. Strobel, S. Purzer, & M. Cardella (Eds.), Engineering in precollege settings: Research into practice. Rotterdam, the Netherlands: Sense Publishers.
National Academy of Engineering and National Research Council (2014). STEM integration in K-12 education: Status, prospects, and an agenda for research. Washington: National Academies Press.
National Research Council. (2012). A framework for K-12 science education: Practice, crosscutting concepts, and core ideas. Washington, DC: National Academies Press.
NGSS Lead States. (2013). Next Generation Science Standards: For states, by states. Washington, DC: National Academies Press.
Nieto, S., & Bode, P. (2007). Affirming diversity: The sociopolitical context of multicultural education (5th ed.). Boston: Allyn & Bacon.
Reynante, B. M., Selbach-Allen, M. E., & Pimentel, D. R. (2020). Exploring the promises and perils of integrated STEM, through disciplinary practices and epistemologies. Science & Education, 29, 785–803.
Roehrig, G. H., Dare, E. A., Ellis, J. A., & Ring-Whalen, E. A. (2021). Beyond the Basics: A Detailed Conceptual Framework of Integrated STEM. Disciplinary and Interdisciplinary Science Education Research, 3, 11 https://doi.org/10.1186/s43031-021-00041-y.
Roehrig, G. H., Dare, E. A., Ring-Whalen, E. A., & Wieselmann, J. R. (2021). Understanding Coherence and Integration in Integrated STEM Curriculum. International Journal of STEM Education, 8, (2) https://doi.org/10.1186/s40594-020-00259-8.
Sawada, D., Piburn, M. D., Judson, E., Turley, J., Falconer, K., Benford, R., & Bloom, I. (2002). Measuring reform practices in science and mathematics classrooms: The reformed teaching observation protocol. School Science and Mathematics, 102(6), 245–253.
Schellinger, J., Billington, B., Britsch, B., Santiago, A., Carter, S., & Hughes, R. (2018). Gender equitable teaching and learning in STEM spaces. Paper presented at the National Association for Research in Science Teaching (NARST) Conference, Atlanta, GA.
Sharunova, A., Wang, Y., Kowalski, M., & Qureshi, A. J. (2020). Applying Bloom’s taxonomy in transdisciplinary engineering design education. International Journal of Technology and Design Education. www.doi.org/10.1007/s10798-020-09621-x.
Sias, C. M., Nadelson, L. S., Juth, S. M., & Seifert, A. L. (2017). The best laid plans: Educational innovation in elementary teacher generated integrated STEM lesson plans. The Journal of Educational Research, 110(3), 227–238. www.doi.org/10.1080/00220671.2016.1253539.
Simpson, E., Bradley, D., & O’Keeffe, J. (2018). Failure is an option: an innovative engineering curriculum. International Journal of Building Pathology and Adaptation, 36(3), 268–282.
Siverling, E. A., Suazo-Flores, A., Mathis, C. A. and Moore, T. J. (2019). Students’ use of STEM content in design justifications during engineering design-based STEM integration. School Science and Mathematics, 119, 457–474.
Smith, M. K., Jones, F. H., Gilbert, S. L., & Wieman, C. E. (2013). The Classroom Observation Protocol for Undergraduate STEM (COPUS): a new instrument to characterize university STEM classroom practices. CBE-Life Sciences Education, 12(4), 618–627.
Stehle, S. M., & Peters-Burton, E. E. (2019). Developing student 21st Century skills in selected exemplary inclusive STEM high schools. International Journal of STEM Education, 6 www.doi.org/10.1186/s40594-019-0192-1.
Tai, R. T., Liu, C. Q., Maltese, A. V., & Fan, X. (2006). Planning early for careers in science. Science Education, 312(5777), 1143–1144.
Thibaut, L., Knipprath, H., Dehaene, W., & Depaepe, F. (2018a). How school context and personal factors relate to teachers’ attitudes toward teaching integrated STEM. International Journal of Technology & Design Education, 28(3), 631–651. www.doi.org/10.1007/s10798-017-9416-1.
Tran, N. A., & Nathan, M. J. (2010). Pre-college engineering studies: an investigation of the relationship between pre-college engineering studies and student achievement in science and mathematics. Journal of Engineering Education, 99(2), 143–157.
Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., & Wilensky, U. (2016). Defining computational thinking for mathematics and science classrooms. Journal of Science Education and Technology, 25(1), 127–147.
Wendell, K. B., Wright, C. G., & Paugh, P. (2017). Reflective decision‐making in elementary students’ engineering design. Journal of Engineering Education, 106(3), 356–397.
Wheeler, L. B., Navy, S. L., Maeng, J. L., & Whitworth, B. A. (2019). Development and validation of the classroom observation protocol for engineering design (COPED). Journal of Research in Science Teaching, 56(9), 1285–1305.
Wieselmann, J. R., Dare, E. A., Ring-Whalen, E. A., & Roehrig, G. H. (2020). “I just do what the boys tell me”: Exploring small group student interactions in an integrated STEM unit. Journal of Research in Science Teaching, 57(1), 112–144.