Part II: How Were Sample Intervention Programs Selected?

The Need for Examples of Effective Interventions

The need for examples of effective interventions that can be used as guides to assist educators, administrators, and policymakers in implementing school-based dropout interventions is clear. While there have been many reports describing promising dropout interventions, only a small portion of these offer conclusions that are based on methodologically sound analysis. It is from this list that examples of interventions were selected for inclusion in this document. It is important to note that this is not a comprehensive list of effective interventions. The interventions described serve as examples that show some evidence of effectiveness and meet criteria (described below) for inclusion in this document. Given the nature of applied research and the number of variables that cannot be controlled, exact replication of any program is difficult. The reader is encouraged to carefully consider the population for whom each intervention was successful and the contextual variables that may have influenced effectiveness. Each of the examples must be considered in relation to the needs, demographics, resources, and other circumstances of local schools or districts. Rather than rate the merits of each intervention, information is intended to facilitate critical review of the intervention programs/strategies by those implementing such programs. References and contact information are provided for those wanting more detailed information.

Search Process & Initial Criteria

The identification process for strategies and programs to include as examples in this NCSET Essential Tool occurred in two stages. First, an initial list of programs to review was generated from a search that relied on information from four sources. Programs were included in the initial pool if they

Focused on dropout prevention;
Appeared to include evidence of effectiveness using a research or evaluation design; and
Results were published (or the study was completed) between 1988 and 2003.

The sources used to conduct the initial search are described below.

Integrative review of prevention and intervention studies addressing dropout described in professional journals (Lehr et al., 2003). Authors of this review conducted computerized searches of an in-house database of more than 600 documents and online databases including Educational Resources Information Center (ERIC), Psychological Abstracts, and Education Abstracts. Search terms linked to dropout and school completion included “dropout prevention” along with the descriptors “school engagement,” “school completion,” “achievement,” and “disabilities.” A list of more than 300 unduplicated citations from 1980 to 2001 was compiled. The list of articles was further refined using the following criteria: (a) was published in a professional journal, (b) focused on a dropout prevention or intervention program, and (c) included impact data on the described program. Studies with an intervention focus and an emphasis on impacting truancy or dropout were included. General summaries of the literature, nonexperimental studies, and policy reports were omitted. Additional articles were collected through a review of references cited in rejected articles and a computer search (using the same key terms) to identify articles published during the coding period. Forty-five studies were included in the final review.
Literature search conducted by What Works Transition Research Synthesis Project. This current project (funded by OSEP) is engaged in a review and synthesis of research conducted in the past 20 years. The project, based at the University of Colorado, works in collaboration with the Evidence for Policy and Practice Information Coordination Centre at the University of London and the Campbell Collaboration at the University of Pennsylvania. What Works plans to produce several research-based syntheses in the context of transition for youth with disabilities including academic outcomes, dropout prevention (“risk and resiliency”), and transition outcomes. After conducting a systematic search of the literature using ERIC, Psych Info, and Medline databases for studies on dropout prevention and students with disabilities, What Works provided a list of 232 references and associated abstracts. A review of the abstracts from this list yielded 10 articles that were selected for further review.
Web-based search for documents on dropout intervention and prevention produced by the Office of Special Education Programs (OSEP), Employment and Training Administration (ETA), and the Institute of Education Sciences (IES, formerly the Office of Educational Research and Improvement, or OERI). Project staff conducted a Web-based search for dropout intervention and prevention programs funded by OSEP, ETA, and IES. The key terms “dropout intervention,” “dropout prevention,” and “dropout program” were employed to search each of these federal government Web sites. This search collected additional relevant documents describing empirical studies of dropout intervention. Titles and/or abstracts of nearly 200 documents were reviewed to determine their relevance for further review. Abstracts referring to intervention programs and data on effectiveness were obtained for further examination.
A search of ERIC documents for reviews of dropout interventions. The authors of this document also conducted a search of ERIC documents for comprehensive reviews of dropout intervention and prevention programs completed between 1988 and 2003. A search was conducted using the key terms “dropout program,” “review,” and “data,” yielding 69 documents. The abstracts of these reviews were carefully considered. Several promising reviews that appeared relevant were obtained. These documents were then examined to determine if they included evidence of effectiveness and, if so, were added to the list of initial references for further consideration.

Raising the Bar

After generating a pool of interventions that met initial selection criteria, additional criteria for inclusion in this document were applied. Use of these additional criteria “raises the bar” by selecting only those interventions clearly supported by empirical evidence (see Table 3 for desirable components). Many publications describe and recommend effective or model programs. However, upon closer examination, many of these interventions were not supported by research or evaluation data and/or did not measure enrollment status. Furthermore, quality research must be conducted in an objective manner, and results should be tested by other researchers.

Policies directed at implementing large scale programming with significant associated costs ought to be based on research that is conceptually and methodologically sound. Sometimes, programs are promoted despite a lack of supporting data. This project’s review of the literature yielded, for example, that a program was mandated despite the fact that data showed an increase in truancy for high school students who were in the program and no evidence of improved graduation rates (http://www.edpriorities.org/Pubs/Opinion/Letters95/Let95_Lernfare.html). It is the responsibility of educators, administrators, and policymakers to require that claims of success be supported by adequate research before adopting those strategies or interventions. We used rigorous criteria to select interventions for inclusion in this document because the implications of the decisions being made to address school completion for students at risk of school failure are important. The stringent criteria excluded some promising (but not proven) programs. The selection process may also have unintentionally excluded some programs that would have met the criteria because the available documents did not include the necessary information.

Final Parameters for Selection

The second stage of the identification process further refined the list of programs to be included as examples of intervention programs. Final criteria for selection are listed below.

Focus on dropout prevention/intervention (manipulation of independent variable).
Measurement of an outcome variable tied to enrollment status (e.g., graduation, current attendance in school, dropout rate).
Evidence of effectiveness using a research design (preferably randomized control or trial—but not exclusively) that yields statistically significant results in favor of the intervention on one or more dependent variables, including enrollment status.
Results published in a professional journal or comprehensive government report (to ensure critical review of the intervention and its accessibility to interested users).

Abstracts: Coding & Definitions

The categories for describing interventions were selected based on a review of coding manuals including the Procedural and Coding Manual for Identification of Evidence-based Interventions (Kratochwill et al., 2002), and guidelines for examining the conceptual and methodological quality of intervention (Crane, 1998). In addition, compilations of other evidence-based programs (e.g., Safe and Sound: An Educational Leader’s Guide to Evidence-Based Social and Emotional Learning Programs, CASEL, 2002) were examined. Coding areas from these sources were considered and used to derive categories suitable for this document. The categories selected and subsequent coding is intended to be descriptive rather than evaluative.

This document includes an abstract of each sample intervention and a chart summarizing information across interventions. The contact person listed for each intervention reviewed the chart and relevant abstract for accuracy. Table 3 lists the information that is included in the abstract and/or the summary chart.

Table 3: Abstracts: Coding & Definitions
*Intervention Program or Strategy.* The title of the program is listed. In cases where the intervention did not have a formal title, the type of program is listed using summary descriptors.
*Background.* History and purpose of the intervention is briefly described.
*Intervention Description.* The intervention is briefly described.
*Outcome Variables.* Outcome variables that were significantly impacted are listed.
Population and Setting. The population intended for the intervention is briefly described with regard to grade level and any targeted subgroups (e.g., students with disabilities). In addition, the setting in which the intervention has been implemented is described (e.g., urban).
*Evidence of Effectiveness.* Research showing evidence of effectiveness is described. The availability of the following information is noted. Research design. Three types of research design are identified. These include: (a) studies incorporating random selection and assignment of participants to two or more groups; (b) studies incorporating random assignment of participants (no random selection) to treatment or control groups; and (c) pre-post studies including measures and comparisons of outcome data before and after an intervention with a single group. Statistical significance. A mathematical determination that indicates the presence of an effect that is unlikely to have resulted from chance alone. When key outcomes are statistically significant, the intervention/program is assumed to have had an effect. Effect size. This indicator of effectiveness measures the amount of impact attributed to the program or intervention, and is not influenced by sample size. Durability of effects. Evidence indicating program effects persisted after the termination of the intervention. Treatment integrity. Information indicating whether the treatment or intervention was implemented as specified. Sample size. The sample size used in the research studies is specified. Use of an external evaluator. The use of an evaluator external to program development and implementation is noted. Multiple sites or studies. Implementation of the program in more than one setting or more than one evaluative study is noted.
*Implementation Considerations.* Practical considerations for implementation are described in terms of support personnel required, training, additional resources, and the estimated duration of the intervention. In addition, information about the availability of a manual or training is provided. When available, information about cost is also included.
*Contact Information.* Names and contact information of individuals associated with the interventions (and in some cases Web addresses) are listed.
*References.* Additional references regarding the interventions are included.