Math in Focus: Impact Study 2020–2021 School Year

This study included two private schools that were implementing Math in Focus^®: Singapore Math^® by Marshall Cavendish^® 2020 during the 2020–2021 School Year. The first site was a K–12 private school located in southwestern NJ serving approximately 350 students. The second site was a PK–12 private school located in Central Florida serving approximately 1,150 students. Given each site had similar characteristics and student body, data was combined for ease of interpretation and analysis.

During the 2020–2021 schools, both schools implemented Math in Focus® 2020 as their core mathematics program at grades K-6. The Central Florida school was in its second year of using the copyright and had been using a version of Math in Focus for over five years. The current school year was the second year of implementation for the NJ site for Math in Focus.

Math in Focus^®: Singapore Math^® by Marshall Cavendish^® is an innovative mathematics program for Grades K–8 + Accelerated that focuses on problem-solving and the use of model drawing to drive the acquisition and application of mathematical skills. A majority of students were taught "in-classrooms" at both sites, with teachers indicating they used using Math in Focus every day for instruction.

Across both sites, there was a total of 303 students. A total of 288 students (95%) in grades K-6 met the inclusion criterion for this analysis. To be Included, students needed to complete both the beginning of the year (BOY) and end of the year (EOY) administrations of HMH Growth Measure and had verified student demographic data.

This sample consisted of almost an equal number of female students (n = 146, 50.7%) and male students (n = 142, 49.43). A majority of students (60.0%) were identified as Caucasian with 14.6% being African American, 10.1% Hispanic, and 4.9% Asian American. None of the students were eligible for free or reduced lunch. Five percent of the sample were identified as English Language learners and 5.9% reported receiving accommodations.

HMH GROWTH MEASURE

HMH Growth Measure is a research-based and adaptive benchmark assessment for Math. It uses assessment performance to connect to HMH's curriculum and instruction resources. HMH Growth Measure is a computer-adaptive test (CAT). The test adjusts item difficulty to students' responses. The assessment ends once the student has completed up to 35 questions, including skips, and the tests take on average 45 minutes.

HMH Growth Measure provides benchmark assessments up to three times a year, and it is recommended that districts use HMH Growth Measure at the beginning of the year (Fall), in the middle of the year (Winter), and at the end of the year (Spring).

The domain performance for classes and students represents the major standards covered by each subject and grade-level test. The domains align tightly to common standard sets like the College and Career Readiness (CCR) Standards. HMH Growth Measure delivers valid and reliable achievement scores, including an HMH Scaled Score and performance categories.

TEACHER SURVEY

Nineteen teachers (90.5%) completed both the baseline survey in the fall of 2020, and the end of year survey in the spring of 2021.

At the beginning of the year, teachers reported using several practices and strategies on a 5-point scale (1 = None to 5 Considerable). These surveys were then categorized as problem-solving practices (8 items) and assessment practices (4 items). Also, teachers reported using student learning activities to teach math skills (5 Items) and their use of math student learning activities to teach math conceptual-understanding (6 Items).

Several scales assessed educational technology use on a 5-point scale (1 = Strongly Disagree to 5 Strongly Agree). These scales included teacher attitudes towards the effectiveness of educational technology (7 items, α = .89), teacher self-efficacy using educational technology (6 Items α = .86), and teacher use of educational technology to teach mathematics (6 Items α = .86).

IMPLEMENTATION AND FIDELITY

Teachers reported use of Math In Focus digital components (α = .58), on a five-point scale (1 Never to 5 Daily) at both the beginning of the year and end of the school year. Further, study teachers logged a bi-weekly implementation report on each lesson completed and the components and resources utilized. The Implementation Teacher Logs allowed the researchers to monitor Math in Focus usage during the study period. These logs were developed as a worksheet to provide teachers with easy "anytime" access in a user-friendly format.

Overall, teachers reported increases in the use of effective instructional practices, as well as implementation of evidence-based learning actives. Overall, teachers reported moderate, to moderate-high usage of these teaching practices and strategies over the course of the school year.

Furthermore, even though teacher ratings at the beginning of the year were quite strong, from the beginning of the year to the end of the year, teachers reported a rise in usage of these practices in the classroom. However, given the limited number of teachers (N = 19) only two changes were significantly different.

Teachers reported a significant increase in the use of mathematical practices focusing on improving student mathematics skills over the course of the year, t(18) = -2.87, p < .01. In addition, teachers reported on the additional use of educational technology to teach mathematics from fall to spring surveys, t(18) = -2.04, p = .05. Two other changes over the course of the school year were marginally significant, including the use of problem solving practices and assessment practices from the fall of 2020 to spring 2021 (see Table 2).

At both sites, instruction occurred in-person for all teachers and grade levels. While a few students remained remote for the beginning of school year, all students returned to in-classroom instruction by the middle of the spring semester.

At the beginning of the year, teachers reported infrequent use of most Math in Focus technology components. The most frequently used components included Teachers Online Resources (85.7%) and the Teacher Guide eBook (66.7%). These trends did not change over the course of the year.

Teacher logs were reviewed and analyzed to determine fidelity of implementation during the regular school year.

Implementation fidelity categories were determined by comparing the number of lessons/modules in which they used each of the components or resources in the Math in Focusprogram out of the documented chapters that were completed across all grade levels over the school year.

If teachers used a component or resource with 70% or more of the total completed chapters, they were classified as "high" implementers, 50%-69% were categorized as "moderate", and less than 50% usage were categorized as “low” implementers.

The Chapter Overview Components & Program Resources (see Table 3) that the majority of teachers used with high fidelity included the Teacher Edition, the Student Edition, and the Chapter Planning Guide.

Based on teacher usage of components and lesson coverage, teachers were also categorized as either low (n = 2, 10.5%), moderate (n= 14, 73.7%), or high (n = 3, 15.8%) implementers of Math in Focus 2020.

Students made significant gains from the beginning of year (M = 55.44, SD = 13.51) to the end of year (M = 68.07, SD = 15.84), t (287) = -17.97, p < .001, d = 1.06.

Figure 1 displays the gains made on HMH Growth Measure Scaled Scores from the beginning of the year to the end of the year for each grade level. Follow-up analyses indicated that there was statistically significant growth (p < .05) at grades K-5. While students at grade 6 did improve, the change did not reach typical levels of statistical significance (t(28) = 1.81, p = .08).

In addition, performance categories were examined to determine If students made progress over the course of the year. Students are categorized as either below-level, on-level, or above-level based on their scaled scores at each adminstration.

As Table 4 displays, there were signficant changes in student performance categories over the course of the year.

One-way chi-square analysis revealed that there was a significantly smaller percentage of students labeled as below-level at the Spring 2021 administration, 𝜒²(1) = 28.13, p < .001. In addition, there was a significant increase in the percentage of students identified as above-level in the spring of 2021, when compared to the fall of the school year, 𝜒²(1) = 54.23, p < .001. There was no change in the percentage of students categorized as on-level by the end of the school year.

More than half of the students participating in the study (n = 149, 51.7%) improved at least one performance category from the beginning of the year to the end of the year. The starkest improvement came from students identified as below-level at the beginning of the year, with 71.3% of these students increasing at least one performance category from the fall to spring test administrations.

This exploratory study of two implementations of Math in Focus^® and Singapore Math^® by Marshall Cavendish^® 2020 found that most teachers implemented the program with moderate to strong fidelity. Teacher usage of the program was associated with the use of evidence-based teaching practices and student learning activities.

Further, the program's usage was associated with statistically significant gains in student test scores overall and across all grade levels included in this analysis. Student performance categories, as measured by HMH Growth Measure, also increased significantly.

This initial research provides encouraging results suggesting that using Math in Focus^® 2020 is associated with improved student math achievement scores and improved teaching outcomes. More rigorous research is needed to determine the context and conditions in which the program's usage leads to greater increases in student achievement.