Best available evidence: three complementary approaches.
Article Type:
Report
Subject:
Educational programs (Management)
Decision-making (Educational aspects)
Authors:
Slocum, Timothy A.
Spencer Trina D.
Detrich Ronnie
Pub Date:
05/01/2012
Publication:
Name: Education & Treatment of Children Publisher: West Virginia University Press, University of West Virginia Audience: Professional Format: Magazine/Journal Subject: Education; Family and marriage; Social sciences Copyright: COPYRIGHT 2012 West Virginia University Press, University of West Virginia ISSN: 0748-8491
Issue:
Date: May, 2012 Source Volume: 35 Source Issue: 2
Topic:
Event Code: 200 Management dynamics Computer Subject: Company business management
Geographic:
Geographic Scope: United States Geographic Code: 1USA United States

Accession Number:
292882219
Full Text:
Abstract

The best available evidence is one of the three critical features of evidence-based practice. Best available evidence is often considered to be synonymous with extremely high standards for research methodology. However, this notion may limit the scope and impact of evidence-based practice to those educational decisions on which high quality intervention research is plentiful and definitive leaving educators with little guidance on the majority of the decisions they face. If the mandate to use best available evidence is taken to mean that educators should use the best of the evidence that is available, then evidence-based practice can address virtually all educational decisions. This paper discusses three complementary approaches to identifying the best available evidence and deriving recommendations from research. These include (a) conducting systematic reviews to identify empirically supported treatments, (b) using methods other than systematic reviews to summarize evidence, and (c) considering research on "treatments" that are not multi-component packages. A conscientious combination of these approaches can increase the breadth of research that is relevant to educational decision-making.

Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise.

The basis for selecting treatments has long been a concern in education and other human service disciplines. Frequently this concern is couched in terms of a research to practice gap (Hoagwood, Burns, & Weisz, 2002). Since the enactment of No Child Left Behind (2001) considerable attention has been given to evidence-based practice as a framework for making decisions, including those about the selection of treatments. Evidence-based practice has been defined as the integration of (1) best available evidence (2) professional judgment and (3) client values and context (Sackett, Straus, Richardson, Rosenberg, & Haynes, 2000). Elsewhere in this special issue, Spencer, Detrich and Slocum (2012 [this issue]) provide a comprehensive discussion of evidence-based practice, This paper provides an in-depth examination of the construct best available evidence. There are two critical reasons best available evidence, as an important facet of evidence-based practice, requires further consideration. First, misunderstanding the construct may lead to a narrowing of the concept of evidence-based practice to a list of treatments. Secondly, the construct of evidence has multiple meanings and can vary across perspectives (i.e., researcher, practitioner, client). A more fully elaborated understanding of the term may influence the treatment decisions and practices of evidence-based educators.

Most definitions of evidence-based practice include the term best available evidence, but they rarely go on to define this critical term. There are several important aspects of the concept best available evidence that should be explicitly recognized. First, the available evidence can be best according to its relevance to the immediate practical situation. Other things being equal, we should be guided by the evidence that most closely matches our situation in terms of characteristics of the students (age, ability level, educational placement, etc.), setting (public school, home, classroom, playground, etc.), and implementers (professional roles, level of training, level of motivation to solve problem, etc.), as well as the outcomes to be accomplished (reading comprehension, math reasoning, social behavior, etc.). A second aspect of best available evidence is the strength of the evidence--the amount and methodological rigor of the evidence. Research that strongly demonstrates that an intervention is functionally related a particular outcome is better than evidence in which other factors may have contributed to the outcome. And when there is a great deal of research demonstrating that an intervention has a given effect, this is better evidence than when there is less research showing this effect.

Further, the term best available evidence suggests that the evidence-based practitioner will rely on the strongest and most relevant evidence that is available. For some questions, the practitioner may be able to take advantage of evidence that is very strong and highly relevant; but in other cases the best evidence may be weaker and only indirectly relevant. The reliance on best available evidence is rooted in the assumption that basing decisions on evidence, even if it is imperfect, is more effective than the alternatives. The idea that practitioners should base decision-making on the best available evidence is very different from the idea that they should only attend to evidence when it is of the highest quality, highest relevance, and highly plentiful.

In this paper we argue that when evidence-based practice is restricted by the current list of treatments that are well-supported by research, the scope is limited to the point that it is not relevant to many of the important decisions that educators must make--decisions that could be more effective if they were informed by the best available evidence. We argue that there are several different and mutually complementary approaches to capturing important lessons from the best available evidence, that each has its own strengths and weaknesses, and that the evidence-based practice movement might be well served to recognize and embrace a combination of these approaches. These complementary approaches include (a) conducting systematic reviews to identify empirically supported treatments, (b) using methods other than systematic reviews to summarize evidence, and (c) considering research for different types of "treatments" as evidence.

Empirically Supported Treatment Reviews

As suggested earlier, there are several types of evidence that can inform the decisions of practitioners. A particular type of systematic review of research literature has come to be strongly associated with identifying empirically supported treatments for use in evidence-based practice. We will refer to this type of review as an empirically supported treatment review. It is highly objective and uses well-defined procedures to minimize arbitrary decisions and judgments by reviewers. Each of the critical steps in the review process is guided by specific procedures and, as much as possible, decisions about review procedures and criteria are made before the review process has begun. The empirically supported treatment review is characterized by systematic and replicable procedures for (a) searching for relevant literature, (b) screening literature for relevance to the review, (c) rating methodological quality of each relevant study, (d) rating outcomes of each study, and (e) rating the degree to which use of the intervention (for the specific population, outcome, and context) is supported by the literature. This is the type of review reported by the What Works Clearinghouse (WWC, http://ies.ed.gov/ncee/wwc/) in their intervention reports, posted on the Best Evidence Encyclopedia (n.d.), described in the influential papers by Odum et al. (2005), Gersten et al. (2005), Homer et al. (2005) and published in the special issue of Exceptional Children on empirically supported treatments (they used the term "evidence-based practices" rather than "empirically supported treatments") for reading, math, writing, and behavior (Cook, Tanker-sley, & Landrum, 2006).

The empirically supported treatment review process begins with explicit, systematic, and thorough procedures for identifying literature that may be relevant to the review question. There are two important criteria for this phase of the process; that the methods are explicitly described and that they are likely to locate all relevant literature. This includes thorough searches of computer databases and examinations of the references of these studies for additional relevant references. Often, these procedures also include contacting researchers and developers of interventions in order seek out additional research.

Second, studies located through the search of literature are screened for their relevance to the review question. Screening criteria include the relevance of the treatment, participant population, measures, and setting. For each of these variables, there must be a clear specification of what is considered relevant and what is considered to be outside the scope of the review. For example, if the review is of repeated readings interventions, the reviewer must specify in advance which variations on this intervention are included and which are excluded. They must also specify how they will handle situations in which other features (e.g., a strong motivation system for improvements in reading rate, or simultaneous instruction in decoding) are combined with the intervention. This screening results in a pool of studies that are presumably directly relevant to the question.

Third, studies that meet screening criteria are evaluated for methodological quality -- whether they provide a sound basis for evaluating the effects of the treatment. Ratings of methodological quality begin with the design of the study. Experimental designs (i.e., randomized clinical trials) and quasi-experimental designs are almost always included. Single subject designs and regression discontinuity designs are often, but not always, included. In addition to the basic design of the research, reviewers also consider other factors such as comparability of control groups, attrition, reliability of measures, fidelity with which the treatment was implemented, possible confounding factors, and appropriateness of the analysis of results. Again, the standards for acceptability on each of these factors are explicitly set in advance. Studies that meet these standards are considered to provide valid evidence about the effects of the treatment under review. Some review systems rate the quality of studies on a scale. For example, WWC (2011) categorizes studies as (1) meets evidence standards, (2) meets evidence standards with reservations, and (3) does not meet evidence standards.

Fourth, studies that are found to be relevant and of adequate quality are examined and the outcomes are summarized. Typically, when considering group designs such as randomized clinical trials (RCT), reviewers consider two issues, statistical significance and effect size. Statistical significance addresses the question of whether the results obtained would commonly occur by chance. Effect size addresses the question of whether the effects are large enough to be meaningful and important for educational practice. Gersten et al. (2005) suggested that effect sizes .4 or greater are necessary for effects to be considered educationally or clinically significant. Reviewers rate outcomes on a scale such as positive effect, no clear effect, and negative effect. Reviews of single subject research typically do not determine statistical significance. For years there has been considerable discussion about appropriate statistical methods for analyzing single subject research but no acceptable measure has been developed (Kazdin, 1976; Kratochwill & Brody, 1978; Gentile, Roden, & Klein, 1972). Instead single subject researchers attend to repeated demonstrations of clear changes in behavior (e.g., Kratochwill et al., 2010).

Fifth, the overall strength of the body of evidence related to the intervention is rated on a pre-established scale. Ratings are based on the number of studies of acceptable quality that show positive effects, no clear effects, and negative effects. For example, the WWC (2011) rates an intervention as having positive effects if two studies of acceptable quality (including one of high quality) show statistically significant positive effects and no studies show negative effects that are large in size or are statistically significant. Some reviews also compute an overall effect size by averaging the effect sizes found in all studies. For example, Gersten et al. (2005) suggested that four acceptable-quality or two high-quality studies with a weighted effect size significantly greater than zero should be required to consider a treatment to be empirically supported. Rating schemes for single subject research typically require more studies and do not include statistical significance (see Horner et al., 2005; Kratochwill et al., 2010). As a result of this review process, particular treatments are given a rating indicating how well the reviewed evidence supports them. If many treatments are rated, lists are often created to show the level of support found for each treatment.

As we noted earlier, one general feature of this type of review is that procedures and criteria are defined prior to the beginning of the review process. This often involves two levels of specification. For example, the WWC (2011) has general review procedures that are applied to all topics, populations, and interventions--these are described in their Procedures and Standards Handbook (WWC, 2011). in addition, the details of review procedures and specific standards are established in a protocol for each review topic. WWC reviews of early reading programs are guided by a protocol designed to accommodate the nature of the literature on that topic; a different protocol guides the review of middle school math interventions. Reviews published in journals are generally not as explicit about these two levels of procedural specification but they often have a similar structure. For example, the evidence-based practice reviews published in the special issue of Exceptional Children (Cook et al., 2006) used Gersten et al. (2005) and Homer et al. (2005) as general procedures and made further specification of details of the reviews.

Like any research or review process, empirically supported treatment reviews can be carried out with greater or lesser quality. The fact that a review following the general model of empirically supported treatment review is no indication of its quality or the validity of its conclusions. A growing literature describes and discusses numerous specific methods for conducting empirically supported treatment reviews in education (e.g., Briggs, 2008; Cook, Tankersley, & Landrum, 2009; Gersten et al., 2005; Homer et al., 2005; Lloyd, Pullen, Tankersley, & Lloyd, 2006; Slavin, 2008; WWC, 2008) and a smaller literature explicitly addresses features of high quality reviews (Schlosser, Wendt, & Sigafoos, 2007) and resulting treatment guidelines (Psychological Association, 2002). Each of the five steps described above includes numerous specific decisions required to plan and carry out a review. Any of these decisions could significantly change the outcome of the review.

Several variations on these methods have been reported. For example, the National Autism Center's (2009) National Standards Project (NSP) did not begin their review with specific targeted programs or interventions. Instead, the NSP evaluated the entire body of research on interventions for children with autism. They began with abstracts of over 7000 articles and followed a procedure similar to that described above. The abstracts were screened based on inclusion/exclusion criteria resulting in 775 studies that were reviewed in detail. Only after individual studies were reviewed for relevance, scientific merits, and outcomes, were specific treatments categorized. The categories of treatments were derived by examining the treatments in the acceptable literature, not the other way around. Once treatment categories were established, standards for strength of evidence were applied to these categories. This is just one example of the variations on the empirically supported treatment review. Strengths and Limitations of Empirically Supported Treatment Reviews

Strengths and Limitatins of Empirically Supported Treatment Reviews

Because they identify and review evidence in a thorough and systematic manner with dear procedures for selecting and weighing evidence, empirically supported treatment reviews are the gold standard for identifying treatments that have strong and directly relevant research support. The highly objective quality of empirically supported treatment reviews is an important strength because it makes the review process transparent and replicable, and it minimizes arbitrary or idiosyncratic decisions. It is an important antidote to the potential bias (conscious or unconscious) of reviews in which article selection and criteria for conclusions are personal decisions of the reviewer. In addition, because each step in the review process is clearly laid out, they can be examined to judge whether a particular review has been conducted with high quality. This also allows for ongoing improvement of the methods of review. As reviewers find better ways to conduct each step in the review, these improved procedures must be clearly described; and therefore, they can be used and further refined by future reviewers.

Empirically supported treatment reviews are not a panacea for identifying the best available evidence to guide evidence-based practice. Like any high-quality tool, these reviews are very effective for doing a certain job and less effective for doing other jobs. One of the keys to effective tool use is to select the right tool for the particular task. Therefore, it is useful to distinguish between questions that are well suited to this type of review and those questions for which this type of review is less effective.

Empirically supported treatment reviews are very effective for identifying the amount of high quality and directly relevant research that supports (or opposes) use of a particular type of treatment for a particular group of students to achieve a defined outcome. When numerous studies have been conducted on a particular treatment (for the particular group of students to achieve the defined outcome) empirically supported treatment reviews can provide practitioners with a clear summary of the best available evidence. This style of review is particularly effective for sifting through large literature bases and identifying the treatments that are supported by high quality and directly relevant research. However, there are many educational situations for which this type of review is not as useful--these are situations in which there is little or no high quality research that is directly relevant to the practitioner's specific question. When empirically supported treatment reviews are conducted in these situations, they result in the conclusion that no treatment for the specified problem can be considered to be empirically supported. This is important information; however, it does not directly help the practitioner solve his or her problem. This kind of conclusion from an empirically supported treatment review does not provide any guidance as to what type of treatment is supported by the best of the evidence that does exist. Further, such a conclusion could be taken by practitioners to mean that (a) none of the treatments are effective or (b) there is no useful evidence at all. If evidence-based practice is limited to evidence provided by this type of review, it will offer no guidance to practitioners in the many (perhaps the vast majority) of the important decisions they must make. There are at least three types of common situations in which this type of review provides no guidance. First, there are many specific practical problems that might be addressed by an existing treatment, but none of the treatments have sufficient high-quality and directly relevant evidence to be considered empirically supported. Second, when they are implementing a well-supported intervention, practitioners face myriad detailed questions of how to adapt the intervention for use in their particular situation. They must distinguish between critical features that must be implemented in a specific way in order to produce the effects found in the research and variable features which may be adjusted without threatening the effectiveness of the treatment. Having identified features that can be adjusted to increase effectiveness, practitioners need to have guidance about how those factors can be sensibly modified. Empirically supported treatment reviews do not provide a basis for making these decisions (Cook, Tankersley, & Harjusola-Webb, 2008). Third, practitioners face many challenges that are not addressed by any defined treatment package. In these instances, they must construct interventions in response to the particular problem and circumstances rather than selecting an existing intervention. Empirically supported treatment reviews (as currently conceived) offer little guidance to these practitioners because their locally constructed program has not been subjected to previous research. Educators cannot put their decision-making on hold until programs are developed for all their needs and these programs are empirically evaluated in multiple high-quality studies. A challenge, then, for evidence-based practice is how to bring the best available evidence to bear on these kinds of problems that are not addressed by empirically supported treatment reviews. We suggest that there may be great value in considering additional approaches to reviewing the research literature and supporting practitioners in making important educational decisions based on the best evidence that is available.

Other Methods for Reviewing Evidence

The most important strengths of empirically supported treatment reviews are the sources of their limitations. The objectivity and exclusive focus on research that is both of high quality and directly relevant to the treatment and situation that are the hallmarks of empirically supported treatment reviews mean that this type of review is unable to give any consideration to research that, although not meeting the specifications for strength and relevance, may give important indications about what kinds of treatments are most likely to be effective. Therefore, it would be worthwhile to consider methods for reviewing research that can glean recommendations for practice from the best available evidence when that evidence is not captured by empirically supported treatment reviews.

Narrative reviews, expert panels, and empirically supported practice guide processes are additional means for summarizing evidence and guiding the decisions of practitioners. These sorts of reviews can provide guidance in circumstances when there is no evidence from an empirically supported treatment review and they can elaborate on the recommendations from such reviews. The methods for these reviews are much less structured and clearly defined than those for empirically supported treatment reviews. Because of the looser structure, they allow reviewers to weigh evidence that is indirectly relevant to a particular question generalizing from related areas of research even if those related areas are somewhat distant from the specific population or setting. In addition, these reviews can speak to implementation and broader systems level issues that are not addressed by empirically supported treatment reviews.

Narrative Reviews of Research

The use of narrative reviews to identify the best available evidence on a question of relevance to educational practice is perhaps best exemplified by the National Association of School Psychology (NASP,2008) series Best Practices in School Psychology. In this approach, the editors select authors for chapters, each of which is titled "Best Practices in ..." The authors discuss themes and cite literature that, in their expert opinion, is most relevant and important. There are difficulties with this approach. First, it is usually not clear how the editors select the authors for the chapters. It may well be that the authors are, in fact, experts in the particular area but the selection process is usually not well described. This lack of transparency raises questions about the credibility of the claim that the authors are authoritative and unbiased experts on the topic and the best available evidence supports the practices described in a chapter.

The narrative nature of the review process creates a second set of problems. Again, there is little transparency regarding how the authors select the themes, interventions and particular studies to review. The reader has little assurance that the author has not ignored areas of research that are not consistent with their philosophical orientation or do not confirm their perspective. The lack of transparency and relatively undefined review procedures allow for the possibility of author bias. Each author brings perspectives to the writing task that are influenced by training and experience. It is unreasonable to expect any author to be completely neutral when reviewing and drawing conclusions about a literature base. In the traditional narrative review there are no inherent constraints to mitigate these biases. This leaves readers with "trusting" the authors and the "trust" may be a function of the shared biases of the reader and author.

Best Practice Committees

A second approach to identifying and summarizing evidence in areas where empirically supported treatment reviews do not give useful guidance is to form a committee of experts and stakeholders to establish best practice recommendations. The idea is that by involving multiple experts, concerns about bias and limited knowledge base associated with a single author may be reduced; however, questions about selection of committee members can arise. The assumption is that the members of the committee are experts but as described above the process for identifying experts is often not transparent. Committees are generally assembled with attention to diversity of perspectives. If the group is insufficiently diverse, then the recommendations can be dismissed for failing to have input from key perspectives.

In addition, there is often a lack of transparency about how the practices are actually determined. Since the recommendations reflect the perspectives of a group, the decision-making process becomes very important. The make-up of the group can have significant influence on how decisions are made and what is defined as best practice. If the group is drawn broadly in an attempt to include all perspectives, it may be difficult for the committee to reach consensus about which research base to consider, the process for deciding what is a best practice, and the resulting best practice recommendations. If the committee is drawn narrowly, it may sample a portion of the research base that is consistent with the members' perspective or it may reflect a narrow view of best practices and may not be perceived as credible by the larger community. Further, consumers may assume that the expert panelists' recommendations reflect the research literature when there may have been no formal review of the literature, or the review may be unstructured and subject to the potential errors and biases described above.

Narrative reviews and best practice committee recommendations are certainly limited by issues of transparency and potential bias, but still may serve as useful approaches to identifying and summarizing the best available evidence when EST reviews are insufficient. It is important to recognize that, until recently, these were the most common approaches for making research-based recommendations. Advancements in EBP have increased the awareness of the limitations inherent in these earlier approaches. Yet, there will always be a need for reviews that are able to glean and compile the best available evidence from a complicated and imperfect research base. Consequently, improved alternatives are emerging.

Practice Guides: A New Way to Fill Gaps

Practice guides are a promising alternative to narrative reviews and best practice committees for filling in some of the gaps left by empirically supported treatment reviews. They are a relatively recent development and combine some of the flexibility and latitude for expert judgment of narrative and best practice reviews with some of the rigor of empirically supported treatment reviews. Practice guides originated with the U.S. Institute of Medicine (Field & Lohr, 1992) and are considered to be the gold standard for summarizing the best available evidence in medicine (Eddy & Hasselblad, 1994; Holland, 1995; Schriger, 1995; Woolf, 1995). This approach was used to develop guidelines for educating children with autism (National Research Council, 2001) and is similar to practice guides published by the Institute for Educational Sciences (IES, n.d.; e.g., Shanahan et al., 2010). IES described the goal of their practice reviews bringing "the best available evidence and expertise to bear on the types of systemic challenges that cannot currently be addressed by single interventions or programs" (Shanahan et al., 2010, p. 43). The distinguishing feature of practice guides is that they combine expert judgment with a systematic review of relevant research and report the strength of evidence for each recommendation (based on objective standards).

There are several specific methods for producing practice guides. Often, the process of developing practice guides begins with a systematic review of the literature, just as in empirically supported treatment reviews. The systematic review becomes the basis for identifying best practices. Once the strength of evidence for each intervention is determined, the panel of experts makes recommendations of best practices. Usually, these recommendations extend beyond the identification of the intervention and also include details about implementation and methods for adapting the intervention so that it is a better contextual fit (e.g., Shanahan et al., 2010). The strength of evidence supporting each recommendation is provided--even when the evidence is limited.

IES publishes practice guides that provide a set of recommendations developed by a panel of experts and explicitly linked to specific research evidence. The process of developing these practice guides provides latitude for experts to examine the full range of evidence and formulate recommendations based on patterns of findings and logical connections among bits of evidence. Then, once recommendations are formulated, the strength of evidence supporting each recommendation is explicitly rated and thoroughly discussed. IES (e.g., Shanahan et al., 2010) has defined three levels of evidence that may be associated with recommendations:

Importantly, the very stringent standards of the WWC (2011) are applied to the studies that may justify claims of strong or moderate evidence. As a result, although the process of developing recommendations is very different from the WWC (2011) reviews of specific programs and interventions, the evidence standards share important elements. Practice guides conspicuously display these ratings of the strength of evidence for each recommendation in the text. Extensive discussion and documentation of the rationale for the ratings, including tables listing individual studies and their characteristics, are also provided in an appendix. Draft practice guides are subjected to external peer review. Reviewers are instructed to examine whether the evidence cited in support of the recommendations is up-to-date and whether evidence contradicting the recommendations might have been overlooked. In addition, reviewers evaluate whether the level of evidence proposed for each recommendation is justified (e.g., Shanahan et al., 2010).

Table 1 lists the recommendations and levels of evidence from 2 of the 14 practice guides that had been published by July 2011. This small sample of recommendations show that the nature of recommendations made in practice guides is distinct from reviews of programs. Practice guide recommendations are broader; they often identify topics to be taught and tasks to be accomplished. The level of support claimed for the recommendations is also instructive. As of January 2012, 78 recommendations had been made across the 14 practice guides; 45% of these had minimal support, 33% received moderate support, and 22% were found to have strong support. In two of the practice guides, no recommendations had more than minimal support, and in seven (50% of the practice guides) no recommendations achieved strong support. This very high percentage of recommendations that garner only minimal support and relatively low percentage that enjoy strong support suggests that the team of experts who craft these recommendations are responding to personal experience, logical inferences, and patterns of research findings that are not closely aligned with the results of the strict reviews that are similar to the empirically supported treatment reviews described above. This suggests that the inclusion of greater latitude for judgment and generalization across research findings can result in an importantly different set of conclusions than would be derived from a review that lacks these features.

IES recognizes that even with these requirements for explicit and detailed discussion of how well the research base supports each recommendation, the process of expert judgment may be idiosyncratic. In their description of practice guides, IES notes:

Although the use of practice guides in education is new and the number of examples is limited, this is an important approach to identifying the best available evidence that is relevant to an educational challenge and developing recommendations that may be helpful to practitioners. They include flexibility for expert judgment and also require the discipline of explicit citation of the evidence-base for each recommendation. Because the process of formulating recommendations is flexible and there is room to recommend practices based on indirect evidence, practice guides appear to provide a useful supplement to empirically supported treatment reviews. Practice guides are one example of an alternative means of reviewing the best available evidence and deriving useful recommendations for practice; we expect that there are other alternatives as well. We believe that the most important point is that evidence-based practice is grounded in the best available evidence. Identifying and summarizing the best available evidence is not a simple process that can be accomplished through a single method of reviewing the research. Different methods appear to be appropriate to different questions. The development of a range of review methods that are appropriate for a variety of different kinds of questions and different kinds of research literatures would provide evidence-based practitioners with the most rich, nuanced, and relevant recommendations.

Research on Different Types of Treatments

The basic definition of evidence-based practice requires that practitioners draw on the best available evidence (along with clinical judgment and client values/context) in educational decision-making. This definition does not specify whether the best available evidence is sought to judge the effectiveness of multi-component programs, general strategies, specific techniques, or even broad principles of behavior (cf. Cook & Cook, 2011). In other words, definitions of evidence based practice do not specify the "unit of practice"--a complex multi-component program such as Reading Mastery or Accelerated Reader is large unit of practice and a specific technique such as differentially reinforcing appropriate behavior is a much smaller unit. Empirically supported treatment reviews have almost exclusively focused on evaluating the research support for multi-component programs or treatment packages. WWC (http://ies.ed.gov/ncee/wwc/) has reviewed research on these kinds of treatment packages. Similarly, the Best Evidence Encyclopedia defines its purpose as providing reviews of research proven programs (Best Evidence Encyclopedia, n.d.).

There are good reasons for this focus; when a multi-component program is demonstrated to be effective, there is strong confirmation that the particular arrangement of curriculum, instructional techniques, lesson sequences, and all the other components work together to achieve a complex outcome. In education, the multi-component program is often a very effective unit of practice to evaluate and implement. However, if the use of the best available evidence is limited to reviews to guide selection of the multi-component programs, it will fail to impact the vast majority of educational decisions--those that do not involve selection of such programs and those that involve making decisions about programs for which clear evidence does not exist. Exclusive reliance on evaluation of multi-component treatments will not help educators with these kinds of decisions. As a result, practitioners are left without guidance from the best available evidence when they make these kinds of decisions. Evidence-based practice can be relevant to a much broader range of educational decisions making if it includes a range of "units of practice." Specifically, identifying units of practice that are smaller and simpler than multi-component programs and are supported by the best available evidence could give practitioners additional useful tools for evidence-based practice. Cook and Cook (2011) have referred to these as micro-practices.

We have identified two approaches to identifying units of practice that are simpler than multi-component programs. The first approach is to identify the elements or components that are commonly found in effective treatments. This approach is exemplified by the work of Chorpita and colleagues (Chorpita, Becker, & Daleiden, 2007; Chorpita, Daleiden, & Weisz, 2005)--a group of clinical psychologists working to establish an effective and practitioner-friendly system for disseminating practice recommendations based on the best available evidence. Their concern was that manualized treatments (similar to educational programs) might not be the ideal unit of practice for dissemination. Chorpita and colleagues noted that there are numerous specific manualized treatments that share many common features. Attempts to perform a standard EST review would find a large number of different treatments (distinct manuals) each of which has only a modicum of research support. This would result in conclusions that few of the treatments were supported by sufficient research a conclusion that would be misleading because many of the treatments were highly similar and together families of treatments (i.e., sets of treatments that are highly similar) had strong research support. They argued that identifying the "practice elements" that make up the treatments that have been demonstrated to be successful for a particular problem could make stronger recommendations. Chorpita et al. (2005) defined a practice element as "a discrete clinical technique or strategy (e.g., "time-out," "relaxation") used as part of a larger intervention plan (e.g., a manualized treatment program for youth depression)" (pg. 11).

Chorpita et al. (2007) identified the practice elements that were present in all of the specific treatments shown to be effective for youth with special behavioral and emotional needs. Anxious or avoidant behavior problems was one of seven topics analyzed. The authors identified 145 treatments (specific manuals that had been subjected to empirical research). Of all the specific treatments found to be effective, 82% included exposure, 42% used relaxation, 40% had cognitive elements, 32% used modeling, 29% included psycho-education for the child, 27% prescribed praise/rewards given by the therapist, and so on. The authors noted that because the practice elements were components of complex packages and were not tested independently, the evidence supporting them is only indirect. Nonetheless, they found that practice elements are effective and strategic units for dissemination. Although the evidence supporting practice elements is not the best possible evidence (i.e., direct and unambiguous support from RCTs), it may be the best evidence that is available to support many of the decisions that practitioners face.

The second approach is to directly assess the effects of smaller units of practice through experimental research as empirically supported treatment reviews (Cook & Cook, 2011). Embry and Biglan (2008) and Embry (2004) described the utility of researching and disseminating these kinds of treatment components as a way of broadening the scope of evidence-based practice. They coined the term evidence-based kernels to describe "fundamental units of behavioral influence that appear to underlie effective prevention and treatment for children, adults, and families" (Embry & Biglan, 2008, pg. 75). Embry (2004) described kernels as irreducible units of behavior change technology (active ingredients) that can be incorporated with other kernels into compounds. Kernels may target consequences of behavior, instruction, motivation, or biological functions that influence behavior. Embry and Biglan (2008) provided 52 examples of kernels. Kernels focused on consequences include praise, mystery motivators, public posting of feedback, timeout, and overcorrection. Instructional kernels include cooperative structured peer play, self-monitoring, and paragraph shrinking to write summaries. Kernels aimed at changing motivation include public commitment, motivational interviewing, and media that show the immediate negative social outcomes of problem behavior. Kernels also include interventions that affect behavior primarily through biological functions such as consumption of omega-3 fatty acids, aerobic play, and progressive muscle relaxation. Like Chorpita and colleagues (2005), Embry and Biglan. (2008) argued that compared to more elaborate program packages, kernels can be more easily and cost-effectively implemented, and can be applied more flexibly allowing practitioners to generalize their use to new situations. Unlike Chorpita's (2005) practice elements, Embry and Biglan (2008) argued that kernels must be validated by direct experimental results.

These two approaches to identifying and validating smaller units can be combined. An example is the effective teaching literature (e.g., Brophy & Good, 1986; Rosenshine & Stevens, 1986). This large literature includes both correlational studies that identified specific teaching techniques associated with higher academic growth and experimental studies in which teachers who were taught these techniques became more effective than control teachers. Based on an extensive narrative review of this literature, Rosenshine and Stevens (1986) made numerous recommendations such as: (a) new skills should be introduced by demonstration interspersed with questions to check student understanding, (b) early engagement in new skills should include guided practice featuring a high frequency of teacher questions and overt student practice, (c) student errors should be corrected systematically and they indicate a need for more practice. Similarly, Brophy and Good (1986) listed well-supported techniques for beginning reading instruction such as, "students should receive frequent opportunities to read and respond to questions, and should get clear feedback about the correctness of their performance" (p. 346) and "both progress through the curriculum and pacing within specific activities should be brisk, producing continuous progress achieved with relative ease (small steps, high success rate)" (p. 346). As they stand now, these recommendations are based on narrative reviews of the research, the quality of evidence would be raised if they were supported by practice reviews or empirically supported treatment reviews.

The use of complementary review methods and recognition of empirically supported treatment components would bring many more educational decisions within the scope of evidence-based practice. Even this greatly increased scope would not take advantage of all the evidence that the research base can provide; it would fall short of providing the best available evidence for as many educational decisions as possible. Perhaps the most inclusive strategy for making evidence available to practitioners is the identification and use of principles of learning and behavior. These basic principles represent the distillation of a massive information base on human behavior and they can be applied to solving educational problems when more specific evidence is not available. We define principles, as descriptions of fundamental relations that account for learning and behavior change (e.g., reinforcement, generalization, motivating operations, etc.). These principles have been established through a preponderance of scientific investigations rather than a specific systematic review of literature; therefore we would describe them as "research-based" rather than "empirically supported." They are the most basic and broadly usable tools in the problem-solving toolbox. And they are more than supports to solving problems; they can help practitioners frame problems in ways that lend themselves to effective problem solving. In addition, knowledge of principles can help practitioners learn and maintain implementation of empirically supported techniques, kernels, and programs. These principles do not replace or compete with other methods of deriving practice recommendations from research. Instead, principles complement the other approaches and provide practitioners with useful guides based on scientific research.

Practice elements, kernels, and basic principles are similar in that they provide specific recommendations about important decisions that practitioners make daily -- decisions that are not informed by empirically supported treatment reviews of multi-component programs alone. In the next sections we give specific examples of how these smaller units of practice can be used when a program-level empirically supported treatment does not address the problem.

Selection of Interventions

When program-level empirically supported treatments are not available, smaller units of practice can provide a systematic basis for selecting treatments. A practitioner can recognize the presence or absence of important features (e.g., elements, kernels) in programs. Knowledge of important features can inform the development of evaluation tools (e.g., curriculum checklists) to assist in the selection of programs. Practitioners can use their knowledge of research-based principles to discriminate promising new strategies from ungrounded ones (Wanzek & Vaughn, 2006). For example, a new program that incorporates several effective instructional techniques such as systematic scaffolding, explicit teacher-led instruction, directions for differentiation, and detailed lesson sequence might have greater potential than a program that lacks these features. Decisions about the selection of programs that are based on this kind of evidence may prevent practitioners from being snared by the latest fad practices.

Arranging Important Details and Adapting Interventions

Smaller units of practice like research-based principles can inform the adaption and modification of interventions to suit the needs of specific clients and students. Also, programs cannot specify every important detail about how it is to be implemented. Even well specified programs require practitioners to make numerous decisions that could impact its effectiveness. For example, a preschool teacher might need to provide supplemental vocabulary instruction to several students with limited English skills. A search of relevant databases and websites for empirically supported treatments might yield limited evidence that closely matches the situation. However, related evidence from an empirically supported treatment review of an intervention to teach vocabulary words to older English Language Learners could provide a starting point. Drawing on his or her knowledge of components and principles, the teacher could alter the intervention strategically or combine it with other relevant early childhood approaches to be suitable for his or her classroom.

In another example, a teacher may be implementing an empirically supported treatment that provides clear curriculum and detailed lesson plans, but does not specify a strategy for ensuring that all students are attending to the lesson. The teacher may provide differential reinforcement in the form of attention and praise when the more distractible students are participating appropriately.

Once a treatment is implemented, further adjustments are often necessary in order to maintain an intervention's effectiveness and sustain its use over time. When modifications become necessary, practitioners can rely on their knowledge of effective components and research-based principles to determine which aspects of the intervention can be altered without jeopardizing the effectiveness of the program. Most multi-component programs have features that are essential for effectiveness and other features that can be changed considerably without threatening their integrity. However, in most instances, component analyses that empirically identify critical features have not been conducted. This leaves a great deal up to the practitioner's judgment. For example, if a teacher implements an empirically supported reading curriculum in the standard fashion and one student's progress monitoring results suggest he is not making adequate gains, discarding the entire program is not necessary. Based on her knowledge of effective teaching techniques and research-based principles, she may decide to provide more opportunities for him to practice the new skills or use a more explicit correction procedure. Neither would compromise the essential features of the program.

Understanding and working with the principles in this way may promote effective and sustained implementation of empirically supported treatments (Wanzek & Vaughn, 2006). In their efforts to study variables influencing the sustainability of research-based practices in schools, Klinger, Vaughn, Hughes, and Arguelles (1999) found that teachers who had participated in an intensive, year-long staff development program continued to implement and make adaptations and modifications to fit their style. These teachers demonstrated positive outcomes for their students compared to teachers with limited knowledge of the critical components and principles behind the practice. Likewise, Baker, Gersten, Dimino, and Griffiths (2004) found that knowledge of a practice's underlying principles distinguished between teachers who were high sustainers and teachers characterized as moderate sustainers. The results of these studies suggest that when teachers lack an understanding of research-based principles that allow effective adaptation, interventions may be prematurely discarded and practitioners may conclude that research has little relevance to their practice (Gersten, Vaughn, Deshler, & Schiller, 1997).

Creating Interventions

Teachers are often faced with instructional and behavioral challenges that are so unique that no relevant programs exist or the challenges are small and immediate enough that adopting a whole program is not necessary or efficient. When no relevant program-level empirically supported treatment is available, the next best available evidence may recommend program elements, kernels, and research-based principles. To extend the example above, if a teacher is unable to locate any empirically supported treatment that is relevant to teaching vocabulary to preschool English Language Learners, he or she may need to create a suitable intervention by incorporating components and kernels in a way that is informed by basic principles. The teacher could combine the effective teaching techniques (e.g., active student responding, teaching in small steps, and multiple exemplars) with principles of learning and behavior (e.g., positive reinforcement, discrimination, and generalization) to create a small group word learning intervention in the context of book reading.

Challenges

Including these smaller units of practice as an important part of evidence-based practice has many advantages. Drawing from techniques and principles, practitioners have an infinite number of possible interventions and permeations of interventions available to them. Armed with the fundamentals, practitioners can become more successful problem solvers (Chwalisz, 2003; Hayes, Barlow, & Nelson-Gray, 1999). Considering that children and circumstances are rarely identical to research parameters, knowledge of research-based principles can inform the selection, implementation, and sustainability of evidence-based interventions. In the absence of direct evidence regarding specific interventions, for specific populations, in specific contexts, practitioners can use research-based principles to extrapolate from the extant literature. Nonetheless, this approach has many challenges.

One of the most striking challenges is that incorporating research on smaller units of practice into educational decision-making requires considerable effort. Empirically supported treatments are easily located on clearinghouse websites (e.g., WWC (http://ies.ed.gov/ncee/wwc/) and Best Evidence Encyclopedia (http://www.bestevidence.org/)), but the research supporting smaller units of practice is not currently gathered in such a manner. Practitioners must conduct separate searches of databases for relevant reviews and articles, which is not always practical in an already busy workday. Given the effort and time required to glean useful information, many practitioners may not readily pursue this level of evidence. Another challenge to drawing from evidence of smaller units of practice is the knowledge and skill necessary to translate research into practice. Ideally during preservice training, practitioners learn how to implement smaller sized practice units, their associated research, and how to locate and integrate subsequent research into practice. Unfortunately, quality training in research-based principles, tactics, and components is not commonplace. For instance, in a survey conducted by the National Council on Teacher Quality of 72 teacher education programs (Walsh, Glaser, & Wilcox, 2006), only 15% of them taught all five components of successful reading instruction (National Reading Panel [NRP], 2000) and almost half of the programs taught none of them. Similar results were found for preparation programs in special education (Reschly, Holdheide, Smartt, & Oliver, 2007). Finally, interventions modified for practice based on smaller units of practice, like research-based principles, are not validated through scientific evaluations and as a result, more risk is assumed in their application. Even with a solid understanding of key principles, effective implementation is not easy or certain. More is expected of the practitioner and more assumptions are made through the translation of research to practice. With less certainty of the relation between the intervention and desired outcomes, there is more room for error.

Local Progress Monitoring: The Best Available Evidence

The previous sections of this paper have assumed that the best available evidence is to be found in research articles, and there are many important reasons to emphasize this source of evidence for initial selection or construction of treatments. The various sources of evidence and units of practice that we have discussed are intended to provide a good starting place for intervention. No matter how strong the research support, no program should be assumed to be effective in a particular implementation. Once a treatment has been implemented, local progress monitoring--information on how students are actually progressing during the intervention--becomes the best evidence on how well the treatment is working in the particular context. Because progress-monitoring results reflect the actual implementation of the treatment, it is been described as "practice-based evidence" (Detrich, Keyworth, & States, 2007). Throughout this article, we have emphasized that practitioners must make many decisions in the absence of clear, strong, and specific evidence. In these situations we have argued that evidence-based practice implies that they should be supported with the best evidence that is available even if that evidence is not ideal. In these common situations, the importance of progress monitoring is multiplied further. When the quality and relevance of the best available evidence is limited, progress monitoring and data-based decision making become even more important. One of the strengths of progress monitoring is that no generalization from the research base is required. The characteristics of the student, the setting, the context, and all other relevant dimensions match perfectly. This allows the practitioner to have greater confidence in the data and to make appropriate adjustments. Given the uncertainties of generalizing from research to a particular implementation, it is essential that local progress monitoring be recognized as a critical feature of evidence-based practice. Decision-making cannot stop when an empirically supported treatment is selected. And within the framework of evidence-based practice, the best available evidence should support that decision-making. The best available evidence of the effectiveness of the implementation comes from local progress monitoring.

Conclusions

In this paper we have attempted to sketch a comprehensive vision of how the best available evidence can be identified and made useful for evidence-based practice. We have noted that systematic empirically supported treatment reviews of educational programs are a very powerful means of bringing the best available evidence into educational practice; however, we argue that by themselves, these reviews do not describe the best available evidence for all important educational decisions. First, systematic empirically supported treatment reviews are very effective support for selection among educational treatment packages that have been subjected to high quality research. Other styles of reviews (such as practice guides) may be more effective when the best available research is only indirectly relevant and, perhaps, of lower quality. Together, various styles of systematic reviews can provide practitioners with a fuller and more nuanced summary of the best available evidence than would be possible with any one style. Second, the focus on determining the effectiveness of educational programs and complex treatment packages is important. When a complex treatment is evaluated, we gain specific knowledge of the effectiveness of many components working together. When no well-supported treatment package is relevant to the practitioner's need, information on components or techniques may be the best evidence that is available. Further, principles of behavior provide another level of support for evidence-based practice. These principles summarize large volumes of research and can be used by practitioners in concert with the other types of evidence, and when more specific evidence is not available.

We see these various units of practice working together in a dynamic, fluid fashion based on the specific problem and available research evidence. For instance, an empirically supported treatment would be preferred to assembling a number of components for the same purpose--the program has been tested as a whole and there is evidence that the parts all work together to produce desirable results. A program at this level may be effective for most of the students in a school but some students will require additional supports. Some of these supports may involve making adaptations to the program. If practitioners are well versed in the principles of behavior they can identify why the program is not working for students with specific kinds of needs and what kinds of changes may be needed to increase its effectiveness for these students. Given this clear understanding of the nature of the problem, the practitioner could select a kernel or effective teaching technique that addresses the need. Of course, student progress should be closely monitored to evaluate whether these modifications are having the desired effects. In this way, program-level empirically supported treatments, kernels, techniques and research-based principles can work together to provide interventions based on the best available evidence.

It might be argued that this more complex vision of how the evidence-based practice can be informed by the best available evidence muddies the water and loses the clear focus on a specific type of review for identifying programs that work. We will admit that there is some danger here; however, we believe that this approach so greatly increases the scope of evidence-based practice that the danger associated with a more complex system is acceptable. By relying on this broader, more complex view, evidence-based practice becomes relevant to virtually all educational decisions. In contrast, the narrow view that best available evidence is limited to the results of empirically supported treatment reviews leaves practitioners with no guidance from the research on numerous educational questions.

The choice for educators is not one between making all-important decisions based on extremely high quality and directly relevant empirical support versus accepting lower quality and less direct evidence. High quality and directly relevant empirical support simply does not exist for many important educational decisions. Of course, educators cannot delay their decisions until some future time when extensive evidence is available--they must make decisions every hour of every day. The choice, then, is one between a form of evidence-based practice that is very limited in scope leaving educators with no guidance on many important decisions, and a form that provides guidance on solving a much wider range of educational problems. Our argument is that evidence-based practice will be most beneficial to education if it can help educators integrate the best of the evidence that is available when making these decisions. Therefore, proponents of evidence-based practice in education must find ways to make the best existing evidence available and useful to practitioners. By developing and using additional styles of reviews we can draw important lessons from the research that would not be recognized by empirically supported treatment reviews. By including treatment components and principles as additional "units of practice" that might be empirically supported, we can provide practitioners with additional flexible tools that can be used along with larger treatment packages or in the absence of such packages.

References

American Psychological Association. (2002). Criteria for evaluating treatment guidelines. American Psychologist, 57, 1052-1059.

Baker, S., Gersten, R., Dimino, J. A., & Griffiths, R. (2004). The sustained use of research-based instructional practice: A case study of peer-assisted learning strategies in mathematics. Remedial and Special Education, 25, 5-24.

Best Evidence Encyclopedia. (n.d.). Best evidence encyclopedia. Retrieved from http://www.bestevidence.org/.

Briggs, D. C. (2008). Synthesizing causal inferences. Educational Researcher, 37, 15-22. DOI: 10.3102/0013189X08314286.

Brophy, J., & Good, T. (1986). Teacher behavior and student achievement. In M.C. Wittrock (Ed.), Handbook of research on teaching (3rd ed.) (pp.328-375). New York. NY: Macmillan.

Chwalisz, K. (2003). Evidence-based practice: A framework for twenty-first-century scientist-practitioner training. The Counseling Psychologist, 31, 497-528.

Chorpita, B. F., Becker, K. D., & Daleiden, E. L. (2007). Understanding the common elements of evidence-based practice: Misconceptions and clinical examples. Journal of American Academy of Child Adolescent Psychiatry, 46, 647-652.

Chorpita, B. F., Daleiden, E. L., & Weisz, J. R. (2005). Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Mental Health Services Research, 7, 5-20.

Cook, B. G. & Cook, S. C. ( 2011). Thinking and communicating clearly about evidence-based practices in special education. Retrieved from the Council for Exceptional Children's Division for Research website: https://docs.google.com/viewer?url=http%3A%2F%2Feducation.uoregon.edu%2Fuploads%2FThinking_and_Communicating_Clearly_About_Evidence-based_Practices_in_Special_Education.pdf.

Cook, B. G., Tankersley, M., & Harjusola-Webb, S. H. (2008). Evidence-based special education and professional wisdom: Putting it all together. Intervention in School and Clinic, 44(2), 105-111.

Cook, B., Tankersley, M., & Landrum, T. (Eds.). (2006). Evidence-based practices for re ading, math, writing, and behavior [Special issue]. Exceptional Children, 75(3).

Cook, B. G., Tankersley, M., & Landrum, T. (2009). Determining evidence-based practices in special education. Exceptional Children, 75(3), 365-383.

Detrich, K, Keyworth R., & States, J. (2007). A roadmap to evidence-based education: Building an evidence-based culture. Journal of Evidence-based Practices for Schools, 8, 26-44.

Eddy, D.M., & Hasselblad, V. (1994). Analyzing evidence by the confidence and profile method. In K.A. McCormick, S. R. Moore, & R. A. Siegel (Eds.), Clinical practice guideline and development: Methodology perspectives (AHCPR Publication No. 95-0009). Rockville, Md: Agency for Health Care Policy and Research, Public Health Service, U.S. Department of Health and Human Services.

Embry, D. D., & Biglan, A. (2008). Evidence-based kernels: Fundamental units of behavioral influence. Clinical Child Family Psychology Review, 11, 75-113.

Embry, D. D. (2004). Community-based prevention using simple, low-cost, evidence-based kernels and behavior vaccines. Journal of Community Psychology, 32, 575-591.

Field, M. J., & Lohr, K. N. (1992). Guidelines for clinical practice: From development to use. Washington, D.D.: National Academy Press.

Gentile, J. R., Roden, A. H., & Klein, R. D. (1972). An analysis of variance model for the intrasubject replication design. Journal of Applied Behavior Analysis, 5, 193-198.

Gersten, R., Fuchs, L. S., Compton, D., Coyne, M. Greenwood, C., & Innocenti, M. S. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71, 149-164.

Gersten, R, Vaughn, S., Deshler, D., & Schiller, E. (1997). What we know about using research findings: Implications for

improving special education practice. Journal of Learning Disabilities, 30(5), 466-476.

Hayes, S. C., Barlow, D. H., & Nelson-Gray, R. O. (1999). The scientist practitioner: Research and accountability in the age of managed care. Boston, MA: Allyn and Bacon.

Hoagwood, K., Burns, B. J., & Weisz, J. (2002). A profitable conjunction: From science to service in children's mental health. In B. J. Burns, & K. Hoagwood (Eds.), Community-based interventions for youth with severe emotional disturbances (pp. 327-338). New York, NY: Oxford University Press.

Holland, J. P. (1995). Development of a clinical practice guideline for acute low back pain. Current Opinion in Orthopedics, 6, 63-69.

Homer, R. H., Carr, E. G., Halle, J., Mcgee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165-179.

Institute for Educational Sciences. (n.d.). Publications and Reviews: Practice Guides. Retrieved from http://ies.ed.gov/ncee/wwc/publications_reviews.aspx

Kazdin, A. E. (1976). Statistical analyses for single-case experimental designs. In M. Hersen & D. H. Barlow (Eds.), Single case experimental designs: Strategies for studying behavior change (pp. 265-313). New York, NY: Pergamon Press.

Klinger, J. K., Vaughn, S., Hughes, M. T., & Arguelles, M. E. (1999). Sustaining research-based practices in reading: A 3-year follow-up. Remedial and Special Education, 20, 263-274.

Kratochwill, T. R., & Brody, G. H. (1978). Single subject designs. A perspective on the controversy over employing statistical inference and implications for research and training in behavior modification. Behavior Modification, 2, 291-307.

Kratochwill, T. R., Hitchcock, J., Homer, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2010). Single-case designs technical documentation. Retrieved from What Works Clearinghouse website: http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf.

Lloyd, J. W., Pullen, P. C., Tankersley, M., & Lloyd, P. A. (2006). Critical dimensions of experimental studies and research syntheses that help define effective practices. In B. G. Cook & B. R. Schirmer (Eds.), What is special about special education: The role of evidence-based practices (pp. 136-153). Austin, TX: PRO-ED.

National Autism Center. (2009). National Standards Report: National

Standards Project--Addressing the need for evidence-based practice guidelines for autism spectrum. disorders. Randolph, MA: National Autism Center, Inc.

National Reading Panel. (2000). Report of the National Reading Panel: Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. Washington, DC: National Institute of Child Health and Human Development.

National Association of School Psychologists. (2008). Best practices in School Psychology V. Bethesda, MD: Author.

National Research Council (2001) Educating Children with Autism. Committee on Educational Interventions for Children with Autism. Catherine Lord and James P.McGee, (Eds.) Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

No Child Left Behind, 20 U.S.C. 16301 et seq. (2001).

Odom, S. L., Brantlinger, E., Gersten, R., Homer, R. H., Thompson, B., & Harris, K. R. (2005). Research in special education: Scientific methods and evidence-based practices. Exceptional Children, 71, 137-148.

Reschly, D. J., Holdheide, L. R., Smartt, S. M., & Oliver, S. M. (2007). Evaluation of LBS-1 teacher preparation in inclusive practices, reading, and classroom organization-behavior management. Spring-field, IL: Illinois State Board of Education.

Rosenshine, B., & Stevens, R. (1986). Teaching functions. In M.C. Wittrock (Ed.), Handbook of research on teaching, (3rd ed.) (pp. 376-391). New York, NY: Macmillan.

Schlosser, R. W., Wendt, 0., & Sigafoos, J. (2007). Not all systematic reviews are created equal: considerations for appraisal. Evidence-Based Communication Assessment and Intervention, 1, 138-150.

Sackett, D. L., Straus, S. E., Richardson, W. S., Rosenberg, W., & Haynes, R. B. (2000). Evidence based medicine: How to practice and teach EBM (2nd ed.). London: Churchill Livingstone.

Schriger, D. L. (1995) Training panels in methodology. In K.A. McCormick, S. R. Moore, and R. A. Siegel (Eds.) Clinical Practice Guideline Development: Methodology Perspectives. AHCPR Pub No 95-0009 (pp. 115-121). Rockville, Md: Agency for Health Care Policy and Research.

Shanahan, T., Callison, K., Carriere, C., Duke, N. K., Pearson, P. D., Schatschneider, C., & Torgesen, J. (2010). Improving reading

comprehension in kindergarten through 3rd grade: A practice guide (NCEE 2010-4038). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from whatworks.ed.gov/publications/practiceguides.

Slavin, R. E. (2008). What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37, 5-14. DOI: 10.3102/0013189X08314117.

Wanzek, J., & Vaughn, S. (2006). Bridging the research-to-practice gap: Maintaining the consistent implementation of research-based practices. In B. G. Gook & B. R. Schrimer (Eds.), What is special about special education? Examining the role of evidence-based practices (pp. 165-174). Austin, TX: Pro-Ed.

Walsh, K., Glaser, D., & Wilcox, D. D. (2006). What education schools aren't teaching about reading and what elementary teachers aren't learning. Washington, DC: National Council on Teacher Quality. Retrieved from http://www.nctq.org/p/docs/nctq_read-ing_study_app.pdf.

What Works Clearinghouse. (2011). What Works Clearinghouse Procedures and Standards Handbook (Version 2.1). Retrieved from http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v2_1_standards_handbook.pdf

Woolf, S. H. (1995). An organized analytic framework for practice guideline development: Using the analytic logic as a guide for reviewing evidence, developing recommendations, and explaining the rationale. In K.A. McCormick, S. R. Moore, and R. A. Siegel (Eds.) Clinical Practice Guideline Development: Methodology Perspectives. AHCPR Pub No 95-0009 (pp. 105-113). Rockville, Md: Agency for Health Care Policy and Research.

Timothy A. Slocum Utah State University Trina D. Spencer Northern Arizona University Ronnie Detrich Wing Institute

Correspondence to Timothy A. Slocum, Department of Special Education and Rehabilitation, Utah State University, Logan UT 84322-2865. E-mail: tim.slocum@usu.edu.
John Tukey, The future of data analysis. Annals of
  Mathematical Statistics 33 (1), (1962), page 13.


Strong evidence. Consistent evidence that demonstrates that the
  recommended practice causes improvements and that the effects can be
  generalized to a range of students and settings.

  Moderate evidence. Evidence provides either clear demonstration that
  the practice causes improvement, or strong demonstration of
  generalization, but not both.

  Minimal evidence. The recommendation is based on strong
  findings or theories in related areas, or other indirect
  sources of support. However, the panel cannot identify specific
  research that supports the recommendation and achieves the level
  of moderate evidence.


Table 1

IES Practice Guide Recommendations and levels of support

Practice Guide Recommendation                            Level of
                                                         Support

Improving reading comprehension in kindergarten through
3rd grade

Teach students how to use reading comprehension          Strong
strategies.

Teach students to identify and use the text's            Moderate
organizational structure to comprehend, learn, and
remember content.

Guide students through focused, high-quality discussion  Minimal
on the meaning of text.

Select texts purposetulk' to support comprehension       Minimal
development.

Establish an engaging and motivating context in which to Moderate
teach reading comprehension.
Dropout Prevention

Utilize data systems that support a realistic diagnosis  Minimal
of [he number of students who drop out and that help
identify individual students at high risk of dropping
out.

Assign adult advocates to students at risk of dropping   Moderate
out.

Provide academic support and enrichment to improve       Moderate
academic performance.

Implement programs to improve students' classroom        Minimal
behavior and social skills.

Personalize the learning and instructional process.      Moderate

Provide rigorous and relevant instruction to better      Moderate
engage students in learning and provide the skills
needed to graduate and to serve them after they leave
school.


In some cases research does not provide a clear indication of what
  works, and panelists' interpretation of the existing (but incomplete)
  evidence plays an important role in guiding the recommendations. As a
  result, it is possible that two teams of recognized experts working
  independently to produce a practice guide on the same topic would come
  to very different conclusions. Those who use the guides should
  recognize that the recommendations represent, in effect, the advice of
  consultants. However, ... practice guide authors are nationally
  recognized experts who collectively endorse the recommendations,
  justify their choices with supporting evidence, and face rigorous
  independent peer review of their conclusions. (Shanahan, et al.
  2010, p. 43).
Gale Copyright:
Copyright 2012 Gale, Cengage Learning. All rights reserved.