Evidence-Based Interventions in Education (Part II): A Tour of Check & Connect’s What Works Clearinghouse Report

When we refer to Check & Connect as an evidence-based intervention, we mean that it has rigorous scientific research behind it to suggest that it’s an effective student engagement and dropout prevention intervention.

The What Works Clearinghouse report on Check & ConnectMany people learn about Check & Connect and its evidence from the What Works Clearinghouse (WWC), where it’s recognized as the only dropout prevention program to show Positive Effects for Staying in School (2006). Let’s explore the WWC criteria for rating the effectiveness of interventions and their rating of Check & Connect further. We’ll refer directly to the WWC Intervention Report for Check & Connect.

Research 

Screen image of the WWC report on Check & Connect and its research.

Screen image of the WWC report on Check & Connect and its research.

When looking at the research behind an intervention, the WWC looks for studies that meet their standards for rigorous research. The WWC reviewed six studies of Check & Connect:

  • One of them (Sinclair, Christenson, Evelo, & Hurley, 1998) met the WWC’s evidence standards, which means that it was a well-designed, well-implemented randomized control trial (in which participants were randomly assigned to Check & Connect or to a control condition of business as usual).
  • A second study (Sinclair, Christenson, & Thurlow, 2005), also a randomized control trial, met the WWC evidence standards with reservations because slightly more than 30% of the students assigned to participate in the study refused to participate or moved out of the district. The purpose of random assignment is to ensure equivalence between the treatment group (those receiving Check & Connect) and the control group (those not receiving Check & Connect) so that any effects from the treatment can be linked to that treatment and not student characteristics such as socioeconomic status or students’ test scores. If a large number of students from either group leave the study, this may affect the equivalence of the groups, which in turn may lead to a biased estimate of the effect size (that is, the relationship between the intervention and the expected outcomes).
  • The remaining four studies did not meet the WWC’s evidence standards because they were not randomized control trials, did not serve middle or high school students (Check & Connect has also been used with elementary school students), and/or did not examine the outcomes WWC has deemed relevant to its Dropout Prevention topic: Staying in School, Progressing in School, and Completing School (Check & Connect studies have looked at a wide range of outcomes, including engagement, attendance, truancy, grades, behavior, literacy, and perceptions of parental involvement in education).

Effectiveness

Screen image of the WWC report on Check & Connect and its effectiveness.

Screen image of the WWC report on Check & Connect and its effectiveness.

After the WWC identifies studies that meet their standards, they rate the effectiveness of the intervention based on the findings of those studies. The Effectiveness portion of the intervention report is what you want to look at to find out whether an intervention is truly evidence-based. Specifically, when looking at intervention ratings, you want to look for Positive Effects and Potentially Positive Effects.

  • Check & Connect received a Positive Effects rating for Staying in School, meaning that more than one study (Sinclair et al., 1998, 2005) demonstrated that students receiving Check & Connect were significantly less likely to have dropped out of school by the end of 9th grade and by the end of 12th grade than those not receiving Check & Connect.
  • Check & Connect received a Potentially Positive Effects rating for Progressing in School, meaning that one of those two studies (Sinclair et al., 1998) showed that students receiving Check & Connect earned significantly more credits toward graduation than their peers who didn’t receive Check & Connect.

Check & Connect also received a rating of No Discernible Effects for Completing School for the Sinclair et al., 2005 study. This means that no significant difference was found between four-year graduation rates for the Check & Connect students and the control students. However, it’s important to keep in mind that the studies examined by the WWC were conducted with high school students in special education, who have until age 21 to graduate from high school. Although participating in Check & Connect had no discernible effect on the four-year graduation rate, significantly more Check & Connect students than control students graduated within five years of entering high school. Given the poor graduation rate for students in special education (57% graduate with a regular diploma by the time they’re 21; NCES, 2008), it’s very important to look beyond a four-year graduation rate for these students, and it matters when a program can improve their graduation rate.

A couple further considerations about the WWC Intervention Report on Check & Connect and notes on more kinds of evidence on Check & Connect we are currently gathering or plan to gather in the future:

  • Research is typically conducted in a somewhat artificial, controlled way that doesn’t always resemble real-world implementation of a program. In the research studies reviewed by the WWC, Check & Connect was implemented in real schools but with full-time or part-time, well-qualified, dedicated mentors (here, “dedicated” means “hired to be a Check & Connect mentor”; we hope that all Check & Connect mentors are “dedicated” in the sense of being “committed” to their C&C mentor role!) paid using federal grant funds. The evidence reviewed by the WWC thus suggests that Check & Connect is effective in schools when paid, dedicated mentors are utilized. We know that some schools implementing Check & Connect have limited resources and can’t afford to hire dedicated mentors. Is Check & Connect effective with volunteers or with existing school staff who have other responsibilities serving as mentors? We think so, but we don’t have the evidence from randomized control trials to support that assertion yet—we only have evidence from individual schools’ evaluations of such implementations at their sites. We plan to conduct more studies to gather this evidence.
  • Evidence-based interventions typically have evidence behind them for a specific group of students. Most of Check & Connect’s evidence is from students with disabilities. Does Check & Connect work as well for students in general education? We think so, but we won’t know until we have evidence to support this hypothesis. We’re currently gathering this evidence through efficacy trials in San Diego, Chicago, San Jose, and Montréal.

More Evidence

The WWC is just one database that recognizes Check & Connect as an evidence-based intervention. There are many other databases of evidence-based interventions in education, such as the National Dropout Prevention CenterAttendance Works and Social Programs That Work, and each has their own standards for recognizing interventions as evidence-based (or whatever term they use to suggest evidence-based, such as effective, exemplary, research-based, or scientifically-based). See Check & Connect in National Databases (or do a web search using the terms “evidence-based educational interventions”) for more information about Check & Connect and other evidence-based interventions in education.

To learn even more about the evidence behind Check & Connect, check out our Research page, where we’ve summarized our research findings and provided links to our research publications and information on our current research projects.

Also, as we noted in Part I of this series on evidence-based interventions in education, we know that some of the best evidence of effectiveness for an intervention comes from those on the ground implementing it. For example, the Check & Connect sites we’ve profiled in our Community Spotlights have reported evidence such as improved graduation rates, improved persistence in school, improved GPAs, fewer unexcused absences, improved attendance, and improved engagement at their sites. We’d love to hear from you about what evidence you have collected at your site to demonstrate the effectiveness of Check & Connect for your students. Please share in our comment section below!

References

National Center for Education Statistics. (2008). The condition of education, NCES 2008-031. Washington, DC: Author. Retrieved from http://nces.ed.gov/pubs2008/2008031.pdf

Sinclair, M. F., Christenson, S. L., Evelo, D. L., & Hurley, C. M. (1998). Dropout prevention for youth with disabilities: Efficacy of a sustained school engagement procedure. Exceptional Children, 65(1), 7–21.

Sinclair, M. F., Christenson, S. L., & Thurlow, M. L. (2005). Promoting school completion of urban secondary youth with emotional or behavioral disabilities. Exceptional Children, 71(4), 465–482.


About the Authors: This post was co-authored by Angie Pohl, Ph.D., and Chris Opsal, M.A., members of the Check & Connect team at the Institute on Community Integration, University of Minnesota.

© 2013 Regents of the University of Minnesota. All rights reserved.
The University of Minnesota is an equal opportunity educator and employer.

One thought on “Evidence-Based Interventions in Education (Part II): A Tour of Check & Connect’s What Works Clearinghouse Report

  1. Pingback: Evidence-Based Interventions in Education (Part I): An Overview | Attend, Engage, Invest

Comment on this post (comments are moderated)

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s