Each year, the Institute of Education Sciences (IES) invites applications for education and special education research projects that address one of five research goals, including Efficacy and Replication (Goal 3) and Effectiveness (Goal 4). We write to request your feedback on whether Goals 3 and 4 as currently configured are meeting the needs of the field, or whether we should consider making changes to incentivize and support more replication and effectiveness studies.
The requirements and recommendations for Goal 3 and Goal 4 projects are described in the Request for Applications for Education Research (84.305A) and Special Education Research (84.324A). Briefly, Goal 3 supports the evaluation of fully-developed interventions to determine whether they produce a beneficial impact on student education outcomes relative to a counterfactual when they are implemented under ideal or routine conditions by the end user in authentic education settings. Goal 4 supports the independent evaluation of fully-developed interventions with prior evidence of efficacy to determine whether they produce a beneficial impact on student education outcomes relative to a counterfactual when they are implemented by the end user under routine conditions in authentic education settings. To date, IES has funded 321 Goal 3 studies and 18 Goal 4 studies.
In October 2016, IES convened a group of experts for a Technical Working Group (TWG) to discuss the broad question of what should come after an efficacy study (a summary of the TWG meeting (PDF) and a blog post summarizing the discussion are available on the IES website). The TWG agreed that more emphasis on replication research is needed, and recognized that replication may take different forms, including re-analysis of original datasets; direct replications (in which the elements and conditions of the original study are repeated as closely as possible); and conceptual replications (in which the parameters of the original study are modified in some way, such as altering an aspect of the intervention to improve outcomes or re-testing an intervention on a different population or in a different context). About half of all Goal 3 and Goal 4 studies funded by IES have been conceptual replications; re-analysis of datasets and direct replications are relatively rare.
The TWG considered various actions IES could take to increase the visibility and support it provides for replication studies. The TWG also discussed whether the requirements for a Goal 4 study – in particular, the emphasis on an independent evaluation of an intervention under routine conditions – are impeding efforts to build more evidence on interventions that have prior evidence of efficacy. Finally, the TWG considered whether more attention is needed to examine causal mechanisms of interventions, variability across studies or study sites, and strategies to produce greater impacts and/or to help end users implement and sustain interventions with fidelity.
As we consider the TWG’s comments and recommendations, we would also like to invite input from the field on the following questions:
- What can IES do to encourage more visible and systematic replication research?
- How can IES revise the current requirements for Goal 4 to encourage more effectiveness studies? Specifically:
- How important is IES’ current requirement for an independent evaluation, that is, an evaluation carried out by individuals who did not and do not participate in the development or distribution of the intervention? Are there other ways that an independent evaluation might be conceptualized?
- Do we need a different definition of routine conditions? Currently, IES uses the term to refer to conditions under which an intervention is implemented that reflect (1) the everyday practice occurring in classrooms, schools, and districts; (2) the heterogeneity of the target population; and (3) typical or standard implementation support.
- Does IES pay sufficient attention to collecting and analyzing data on program implementation under the current research goal structure? If not, how can IES better support this type of research?
- Does IES place enough emphasis on examining causal mechanisms and variation in impacts under the current goal structure? If not, what can IES do to better support work in these areas?
Please send your feedback to [email protected] by Monday, October 2, 2017. We also encourage you to forward this request to anyone interested in providing suggestions. Thank you.
Joan McLaughlin
Commissioner
National Center for Special Education Research
Thomas Brock
Commissioner
National Center for Special Education Research
Delegated the Duties of IES Director