Increasing the Reach of Promising Dropout Prevention Programs: Examining the Trade-offs Between Scale and Effectiveness
Trial Snapshot
- Phase
- Not Applicable
- Sponsor
- University of Chicago
- Enrollment
- 6,600
- Primary Endpoint
- Difference in math achievement
Study Overview
Brief Summary
The inability to consistently deliver at large scale promising education interventions is an important contributing cause to inequality in the U.S. The research team applies insights from price theory and field-based randomized controlled trials to examine the effect of implementing a promising academic skills development program at large scale before implementing at scale. The project is designed to provide evidence of direct scientific and policy value for attempts to scale up a specific intervention, but also stimulate a much more thorough investigation of social policy scale-up challenges by refining these methods and demonstrating their feasibility and value.
The research team examines the challenge of program scale up for a promising intervention studied in Chicago at medium scale in the past - SAGA tutoring. Past work has demonstrated that SAGA's intensive, individualized, during-the-school-day math tutoring can generate very large gains in academic outcomes in a short period, even among students who are many years behind grade level. This study will explicitly explore the extent to which there is a trade-off between effectiveness and scale for this intervention. By taking advantage of the power of random sampling, this study will also allow for observation of the program's effectiveness as if it were running at three-and-a-half times the proposed scale in a subset of the study population.
Detailed Description
The University of Chicago Education Lab and Crime Lab New York research teams are carrying out a randomized controlled trial during the 2016-17 and 2017-18 academic years to build on previous collaborations with the Chicago Public Schools (CPS), the New York City Department of Education, and SAGA Innovations that have found that SAGA's intensive, individualized, during-the-school-day tutoring can generate very large gains in academic outcomes in a short period of time, even among students who are many years behind grade level. This research suggests the promise of this approach for improving the academic skills and educational attainment of disadvantaged youth, even once they have reached adolescence. However, to truly affect outcomes at the local and national level, SAGA would have to be rolled out on a much greater scale than researchers have been able to study in Chicago. Yet little is known about how to take promising interventions to scale. This study seeks to build the science of scale-up, by examining the extent to which this individualized tutoring program can be implemented at an even greater scale and by explicitly exploring the trade-offs between effectiveness and scale.
The SAGA Innovations program expands on the nationally recognized innovation of high- dosage, in-school-day tutoring developed in Match Education's charter school in Boston. The tutoring program meets as a scheduled course, Math Lab, once a day during the normal school day, and is provided in addition to a student's regular math class. Students work two-on-one (two students with one tutor) with the same full-time, professional tutor for the entirety of the school year. The content of the tutoring sessions is aligned with what students are learning in their regular math courses, but is also targeted to address individual gaps in math knowledge. Also following the original model developed by Match Education, SAGA tutors use frequent internal formative assessments of student progress to individualize instruction.
A previous randomized controlled trial conducted by the University of Chicago research team found that one year of this intervention, delivered in AY2013-14 in the Chicago Public Schools, generated between one and two extra years of academic growth in math, over and above what the normal U.S. high school student learns in one year (Cook et al., 2015; Reardon, 2011). The estimated effects for math achievement are on the order of 0.19 to 0.30 standard deviations, depending on the exact test and norming used. The intervention also improved student grades in math by 0.58 points on a 1-4 grade point scale, compared to a control mean of 1.77. These gains are particularly important because of the growing evidence on the importance of math specifically for short- and medium-term success in school, and for long-term life outcomes such as employment and earnings (Duncan et al., 2007).
This study aims to build upon the investigators' previous evaluations of the program, and will provide insight into the ability of this program to serve youth at a much larger scale. Specifically, this study aims to answer the following research questions:
- What is the effect of implementing an evidence-based individualized tutoring program at larger scale?
- What is the relationship between the effect of the program and the scale at which the program is implemented?
Study Design
- Study Type
- Interventional
- Allocation
- Randomized
- Intervention Model
- Parallel
- Primary Purpose
- Treatment
- Masking
- Single (Outcomes Assessor)
Eligibility Criteria
- Sex
- All
- Accepts Healthy Volunteers
- No
Inclusion Criteria
- •Chicago Public School and New York City Department of Education high schools students attending schools in low-income communities. Schools in the study are chosen in collaboration with the Chicago Public Schools and New York City Department of Education based on criteria such as dropout rate, test scores, scores on academic rating scale, etc.
- •School administrators are enthusiastic about the program and agree to terms and conditions of the experimental design
- •Male and female youth within these schools who are rising 9th and 10th graders in academic year (AY) 2016-17 and 2017-18
- •Applicants who apply to be a tutor for SAGA Innovations
Exclusion Criteria
- •In Chicago (where the randomized controlled trial is being run), youth who have missed >60% of days during AY2015-16 or AY2016-17 (through March), and so would not be expected to show up in school enough during intervention years (AY2016-17 and AY2017-2018) to benefit from school-based programming
- •In Chicago, youth who have failed >75% of classes during AY2015-16 and AY2016-17 (through March)
- •In Chicago, youth who have designations for autism, "educable mentally handicapped," and/or traumatic brain injury
Arms & Interventions
Control group
These youth will receive standard mathematics instruction and support (including possibly other tutoring interventions), but not the daily, intensive, during-the-school-day math tutoring provided by SAGA.
Scale-up SAGA math tutoring
These youth will receive intensive, daily mathematics tutoring, and students will be paired with scale-up tutors. Tutors will be hired using the randomization process, and will be randomly assigned to youth.
Intervention: SAGA Innovations (Other)
Scale-up SAGA math tutoring
These youth will receive intensive, daily mathematics tutoring, and students will be paired with scale-up tutors. Tutors will be hired using the randomization process, and will be randomly assigned to youth.
Intervention: Scale-up tutors (Other)
Standard SAGA math tutoring
These youth will receive intensive, daily mathematics tutoring, and students will be paired with tutors hired via SAGA's standard process, which does not involve randomization. Tutors will be randomly assigned to youth.
Intervention: SAGA Innovations (Other)
Standard SAGA math tutoring
These youth will receive intensive, daily mathematics tutoring, and students will be paired with tutors hired via SAGA's standard process, which does not involve randomization. Tutors will be randomly assigned to youth.
Intervention: Standard Tutors (Other)
Outcomes
Primary Outcomes
Difference in math achievement
Time Frame: 1-year, 2-years, 3-years
Performance on math standardized achievement tests
Secondary Outcomes
- Difference in math course grades(1-year, 2-years)
- Difference in absentee rate(1-year, 2-years, 3-years)
- Difference in index of schooling outcomes(1-year, 2-years, 3-years)
- Difference in student misconduct(1-year, 2-years, 3-years)
- Difference in total courses failed(1-year, 2-years, 3-years)
- Difference in math courses failed(1-year, 2-years, 3-years)
- Difference in non-math courses grades(1-year, 2-years, 3-years)
- Difference in non-math courses failures(1-year, 2-years, 3-years)
- Difference in school persistence(1-year, 2-years, 3-years)
- Difference in violent crime arrests(1-year, 2-years, 3-years)
- Difference in other arrests (property, drug, and other crimes)(1-year, 2-years, 3-years)
- Difference in standardized test score achievement(1-year, 2-years, 3-years)
- Difference in high school graduation rate(2-years, 3-years, 4-years)
- Difference in college enrollment rate(3-years, 4-years, 5-years, 6-years, 7-years, 8-years)