What is CSR Conversation? Click here to find out.
The CSRQ Center often receives questions regarding the CSRQ Center Reports series. The series of questions presented here have been adapted to answer specific questions that readers may have regarding the latest CSRQ Center report on the evidence of effectiveness and quality of 18 widely adopted middle and high school improvement models. However, most answers apply equally well to the CSRQ Center's two previous reports, on 22 elementary school CSR models and 7 Education Service Providers. Altogether, the Center has now rated nearly 50 of the most widely adopted whole school improvement models in the country.
Question What is Comprehensive School Reform (CSR)?
Response For the past decade, attention has focused increasingly on various strategies to improve schools and student achievement. One approach, comprehensive school reform (CSR), has been tried in thousands of schools nationwide, most of which are high poverty and low performing. This trend is driven by the recognition that school improvement efforts are complex and require a coordinated, systematic approach that addresses every aspect of a school—including curriculum, instruction, governance, scheduling, professional development, assessment, and family and community involvement. Rather than use individual, piecemeal programs or approaches, effective CSR is meant to integrate research-based practices into a unified effort to raise student achievement and achieve other important outcomes such as reduced dropout rates or improved behavior.
Currently, there are several hundred external CSR models (also known as service providers) that districts and schools can turn to support local comprehensive school reform efforts. The U.S. Department of Education provides background information on comprehensive school reform at http://www.ed.gov/programs/compreform/index.html. Additional guidance on schoolwide reform is provided by the Department in its Guidance on Designing Schoolwide Programs (http://www.ed.gov/policy/elsec/guid/designingswpguid.doc).
Question What are CSRQ Center reports?
Response CSRQ Center reports provide consumer friendly reviews and guidance on the effectiveness and quality of widely adopted comprehensive school reform (CSR) models. Although there are hundreds of CSR models offering services, in practice there are less than 100 that are in wide usage throughout the country. The reports are designed to give education consumers the in-depth information they need to make the best possible adoption decisions to meet locally defined needs. Each report provides basic information on the CSR model, including the model's mission and focus, year introduced in schools, grade levels served, number of schools served, and costs as well as ratings on five indicators of quality and effectiveness.
Our reports provide education stakeholders a decision making tool to help them sort through their options regarding the numerous middle and high school improvement choices available to meet local needs. The reviews are intended to clarify options, not to point to or endorse “best buys.” The CSRQ Center’s report on middle and high school CSR models complements our Works in Progress report on secondary school improvement programs issued in January 2005 (http://www.csrq.org/reports.html), by providing a consumer guide to widely adopted CSR models serving middle and high schools.
Question How was this report produced?
Response The CSRQ Center does not conduct field research. That is, we do not go out and do original research on the models reviewed in our reports. Rather we gather and review existing studies and evidence on the effectiveness and quality of CSR models to provide its ratings. The production of this report was guided by the CSRQ Center's Quality Review Tool, or QRT, which provides the criteria and procedures for independent, fair, and credible model reviews. The QRT development process involved several steps. First, CSRQ Center staff developed review frameworks in consultation with some of the nation's most respected education researchers, program evaluators, and school improvement experts. The QRT also drew on prior and current efforts to conduct rigorous research reviews, including standards set by the U.S. Department of Education’s What Works Clearinghouse. Then, the QRT was reviewed, piloted, and revised with the support of the CSRQ Center's Advisory Committee, a nationally respected panel of experts, that includes leading education practitioners, methodologists, and researchers from a variety of fields, including education, sociology, psychology, and economics.
Question What types of evidence do you rate?
Response For each model, the report evaluates the following dimensions of quality:
Whenever possible, the CSRQ Center Reports offer information on model results for specific student groups or specific types of school settings.
Question How does your rating system work?
Response Our rating process is complex and although the same ratings apply across all five categories, the information used to develop the ratings varies. For example, Categories 1-3 combine two elements to provide a single rating:
The strength of the evidence based upon the causal validity of the research designs used to test the model’s impact (e.g., how reliable and credible is it?).
The strength of the reported impact or effect (e.g., does the model raise student achievement a little or a lot?).
Using the QRT (see above), the CSRQ Center applied separate rubrics for each category of effectiveness and quality listed above to arrive at its ratings, which are expressed by a common set of symbols. In general, the rubrics we used result in the following ratings:
VERY STRONG rating is symbolized by a fully shaded circle. This is the “highest” rating provided by the CSRQ Center.
MODERATELY STRONG rating is symbolized by a ¾ shaded circle. This is the “second highest” rating.
MODERATE rating is symbolized by a ½ shaded circle. Models receiving this rating may still have notable evidence of positive outcomes but not as strong as those receiving the ratings above.
LIMITED rating is symbolized by a ¼ shaded circle. This rating indicates that while the CSRQ Center found some evidence of effectiveness, more rigorous research and evidence needs to be conducted on this model to fully support its effectiveness or quality on the category reviewed.ZERO rating is symbolized by a circle with a horizontal slash. This rating means that while we found studies that allowed the CSRQ Center to provide a rating for a category or subcategory, none were of sufficient rigor to be counted as reliable evidence.
NEGATIVE rating is symbolized by a circle with a minus sign. This rating indicates that we found strong evidence of detrimental effects in a given category or subcategory. In practice we did not find any evidence of this kind for any model.
No Rating is symbolized by “NR” in a circle. This rating indicates that the model has no studies (i.e. evidence) available for review in a category or sub-category.
Question What is reviewed in this CSRQ Center report?
Response This CSRQ Center Report on Middle and High School CSR Models provides a scientifically based, consumer-friendly review of the effectiveness and quality of 18 widely adopted middle and high school comprehensive school reform (CSR) models. Together, the reviewed models represent a significant portion of the total number of schools implementing middle and high school CSR models. Each model reviewed serves a minimum of 40 middle and high schools spread over at least 3 states, and is available for adoption throughout most of the country.
Question How did you select which CSR models to rate?
Response The CSRQ Center followed a process through which we visited the Web sites and reviewed publicly available material on more than 30 school improvement models working at the middle and high school level to identify our ultimate sample of 18 CSR models. In the initial screening, we selected each CSR model based on: (1) whether it served a minimum of 40 middle and high schools in at least 3 states, and (2) was available for adoption in almost all states. This yielded 37 comprehensive school reform models. Then, the Center examined whether the CSR model’s design features met the 11 key CSR components identified by the U.S. Department of Education (Department of Education, 2002). The components used by the CSRQ Center for our comprehensiveness rating were governance, technical assistance, classroom practices, professional development, leadership development, benchmarks/assessments, and curriculum. The final list of 18 models was chosen on the basis of its wide implementation (“market share”) and “comprehensiveness.”
Question How many studies did you look at to arrive at your findings?
Response The CSRQ Center looked initially at approximately 1500 documents (studies, articles, and other materials that seemed like they might be relevant for our review), located through an extensive search process, to reduce the number to 197 studies that were considered for this review. Following the QRT process discussed above, the Center eventually identified 42 quantitative studies that could be analyzed to provide ratings in Categories 1-3. Categories 4 and 5 were rated by using other, mainly qualitative information, described in the report. Detailed information to answer this question is included in the “About this Report” and “Methodology” sections of the report.
Question Which CSR models rated best in your review?
Response This CSRQ Center report does not rate “best buys” or promote the adoption of any particular model, even if it has a “very strong” rating. Our goal it to clarify options and provide the in-depth research review and information necessary to make decisions that meet local needs. Decision makers are urged to make “holistic” evaluations that include: (1) the information we present in the five categories of quality, (2) our in-depth profiles, and (3) a consideration of unique variables or characteristics of local settings.
As we point out in the report, educators or policymakers in one district may be more willing to take on the approach of one model over another, even if this means adopting an approach with less evidence of effectiveness. This is their choice. However, they should be aware that they are making a trade-off, and possibly taking a greater gamble on success if they do so. In addition, since implementation is so important to success, decision makers adopting “less proven” models must make a strong commitment to making sure the model works in their school or district. In order to help educators and policymakers make the best decisions possible, our reviews provide extensive background information describing each model.
We believe that schools and students will benefit greatly if decision makers read through these profiles carefully and then use evidence and their sound judgment to weigh their options.
Question What did your report find?
Response The following are some of the most notable findings detailed in the report.
In terms of quantity of available evidence, five models stood out (America’s Choice, First Things First, School Development Program, Success for All-Middle Grades, and Talent Development High School). In contrast, for over half of the models, the CSRQ Center was able to identify only 10 or less studies that seemed to be relevant for our review of their effectiveness in this category.
Overall, in Category 1: Evidence of Positive Effects on Student Achievement. In this critical category we rated:
Five models as Moderate (America’s Choice, First Things First, School Development Program, Success for All-Middle Grades, and Talent Development High School),
Five models as Limited (Expeditionary Learning, KIPP, Middle Start, More Effective Schools, and Project GRAD) and
Eight models as Zero (Accelerated Schools-PLUS, ATLAS Communities, Coalition of Essential Schools, High Schools that Work, Making Middle Grades Work, Modern Red SchoolHouse, Onward to Excellence II, and Turning Points).
As readers review findings in Category 1 they should keep in mind that almost all of the models in the report serve high-poverty students in low-performing schools. Thus, the evidence of effectiveness they present is for success in educating students in highly challenging conditions.
Other findings are that a vast majority of the models reviewed provide moderate to very strong evidence that they can demonstrate a link between research and the model’s design, and equally strong evidence that they provide the services and supports needed by schools to enable successful implementation. Given the importance of implementation to the success of any schoolwide reform, this means that consumers who select models that have “lower” rankings in evidence of effects on student outcomes may still experience success if they implement these models faithfully.
Question You didn’t rank any models as having Very Strong evidence that they have positive effects on student achievement. Does this mean that CSR models represent a weak school improvement approach?
Response This report does not call into question the effectiveness of comprehensive school reform (CSR). Further, the report does not represent a study of the CSR approach as a whole, but rather the effectiveness and quality of selected CSR models. Comprehensive school reform is one of the most widely studied improvement approaches in the history of education and has demonstrated promising evidence of effectiveness in various studies. For example, a 2002 systematic analysis by Dr. Geoffrey Borman and his colleagues of the student achievement outcomes of 29 leading K-12 CSR models—most of which operated in Title 1 schools—reported that “the overall effects of CSR are significant, meaningful, and appear to be greater than the effects of other interventions that have been designed to serve similar purposes and student and school populations” (Borman, Hewes, Overman, & Brown, 2002, p.33).
It is true that none of the models reviewed received a “Very Strong” rating for Category 1. To receive this rating, models must provide considerable evidence indicating positive outcomes based on rigorous research, primarily randomized controlled trials (RCTs, also referred to as experiments) or solid quasi-experimental studies. In general, few RCTs have yet to be conducted on education programs, and very few have been conducted on CSR models. Therefore, it was hard, in this round of reviews, for a model to receive a “Very Strong” rating. Readers should be aware that our standards were deliberately set very high. We felt that in an environment that requires scientifically based research for many program adoption decisions, and features rigorous review standards set by the What Works Clearinghouse, that it is important to give consumers confidence that models in this report were reviewed against stringent standards.
However, several models with a solid body of evidence might have received “higher” ratings, had their outcomes been more consistently positive, or been supported by a few more studies that met our standards. Furthermore, numerous models are paying increased attention to conducting rigorous research to demonstrate their effectiveness.
Question Many of the models in your report are not rated on their evidence of raising student achievement for specific sub-populations such as low income or minority students. Does this mean that they are not effective when they work in Title I or similar schools?
Response No. Readers should not necessarily judge a “No Rating” or a “lower” rating on the category evidence of positive effects on diverse student populations as evidence that the model cannot be effective in Title I schools or other schools with similar student populations. In fact, federally funded CSR models on average serve school populations with a poverty rate of about 70%. Thus, readers may interpret our overall rating on the category of positive overall effects on student achievement as an indicator of their effectiveness in working in challenging settings, such as Title I schools. However, we found that only a few models broke out their results in Title I or other schools in such a way as to permit the Center to provide ratings on model effectiveness for specific subpopulations. Those models that do provide this information are to be commended for providing additional consumer information. All models are encouraged to seek and present this information in future evaluation reports.
Question Why are CSRQ Center reports important to educators and policymakers?
Response If, as a country, we are to realize the potential offered by comprehensive school reform and schoolwide improvement models, decision makers will need help in sorting through the diverse range of information and often competing claims made by researchers and providers about the effectiveness of various comprehensive school reform models. The CSRQ Center provides scientifically based, consumer-friendly reports on CSR model quality and effectiveness and the guidance needed to use this evidence to make effective, locally defined choices.
Question How can educators use this report?
Response To date, educators have had few objective, rigorous, and consumer-friendly sources to turn to when making choices from among the hundreds of CSR models and improvement approaches available for adoption. This report is intended to provide a consumer guide that helps busy educators sort through claims about which approaches could truly meet the needs of students. The CSRQ Center, as a support tool for educators, has sorted through the evidence and reported on models in the form of reviews. Educators, individually or as school improvement teams, are encouraged to use the individual reviews, or the CSR database provided on the CSRQ Center’s Web site to compare and weigh their options. A solid adoption decision is only one step in ensuring effective improvement. Thus, educators are encouraged to conduct school needs assessments prior to investigating their adoption decisions. Once adoption decisions are made, educators must be willing to commit the time and effort to fully implement the improvement approach they have selected.
Educators are encouraged to use systematic approaches for implementing school reform models, such as the process described in the CSRQ Center’s Moving Forward: A Guide for Implementing Comprehensive School Reform & Improvement Strategies. Available free through our Web site, Moving Forward includes tools to assist educators in implementing comprehensive school reform models.
In addition to the CSRQ Center, the Center for Comprehensive School Reform and Improvement (www.csrclearinghouse.org) provides a variety of resources to support effective CSR—from judicious adoption to effective implementation.
Question How can policymakers use this report?
Response It is particularly important in these times of increased accountability and scarce resources for policymakers to encourage the use of school improvement approaches that have demonstrated effectiveness and quality. The CSRQ Center’s reports are not intended to encourage the production of “lists” of approved models, but rather to encourage the use of high quality evidence when making model adoption decisions. Policymakers can contribute to building evidence-based practice in education by insisting that school improvement approaches that are selected are based on solid evidence, and then by providing the support necessary to ensure their full implementation. Research consistently demonstrates that even the most effective improvement approaches may take several years before they demonstrate results, and that effective implementation is the key to getting good results. Policymakers can help schools and districts choose strong approaches and then give them the time and support they need to get results.
Question How did the models included in the report participate in your review process?
Response The CSRQ Center contacted each model at several points in the review process. These contacts were used to verify information that we had gathered from other public sources to ensure that our reviews were as accurate as possible. Our report is absolutely independent of any provider’s views.
As described in the “About this Report” and “Methodology” sections of the report, the CSRQ Center involved the reviewed models at several steps in the production of the report. After developing our own background information and profile of the model based on publicly available sources, we contacted models to inform them of the review and to speak to them informally to verify the information we had compiled. We held conversations with almost all providers. These conversations helped us to identify additional materials to collect, research to review, and individuals to contact for our report.
At the end of the process the CSRQ Center provided all models a draft of their review, and background information on the report so that they could comment on its accuracy. Each model was asked to review the draft and note if the review contained any inaccuracies. The CSRQ Center invited providers to share questions and concerns, and provide documentation for any information they felt needed to be corrected. In addition, the CSRQ Center encouraged interested providers to submit a two-page response letter to their review, to be published along with the report.
Many providers engaged in telephone and email communication with the CSRQ Center in the weeks following receipt of our draft review, and provided the CSRQ Center valuable insight and information to improve our report. The CSRQ Center considered all concerns or suggested edits for inclusion in the final narrative. The letters provided by the models give consumers additional information that they should weigh in making adoption decisions. All letters have been reproduced as submitted to the CSRQ Center and are available as part of our report.
Question How can I find out more about all the other CSR models that are not reviewed in your reports?
Response Some education decision makers may be interested in CSR models, including new or smaller models that have not yet been reviewed by the CSRQ Center Reports. The Center provides a non-evaluative CSR Model Registry (http://www.csrq.org/CSRProgramRegistry.html) on our Web site so that model developers have the opportunity to share their own information about models not included in the CSRQ Center Reports. Consumers can use this registry to further explore and compare their options.
Question The model that is used in my school is not included in your review. What does this mean? Is this a bad model?
Response The fact that only 18 models are included in this review is not meant to indicate that these are the only CSR models worth considering for adoption. As a national “consumer reporting” organization the CSRQ Center tried to review as many widely adopted CSR models as our resources permitted. Given this limitation, our review covered only widely adopted CSR providers. However, the CSRQ Center provides a non-evaluative CSR Model Registry on our Web site (http://www.csrq.org/CSRProgramRegistry.html) so that other model developers have the opportunity to share their own information about models not included in the CSRQ Center Reports. Consumers can use this registry to further explore and compare their options.
If the model used in your school was not reviewed, we encourage you to urge the model to submit information for the Registry. The information provided by the Registry closely parallels the rating categories we have established for our reviews. This will allow you, and others interested in the model you have adopted, to make some comparisons. Finally, readers should remember that a lack of rigorous evidence does not necessarily mean that a model is ineffective or of low quality. However, it does mean that it currently lacks sufficient evidence to support claims of effectiveness that all models, regardless of size, should provide in the future.
Question What is an effect size and why did you choose the range you use to rate model effectiveness?
Response Effect sizes (ESs) are complex calculations, used to standardize measures of the impact of interventions. Generally, ESs are used to estimate gains (+) and losses (-) on achievement or other outcomes, where the differences between the groups being measured are expressed in standard deviations (SDs). ESs are used for two purposes in our report. First, they help us to describe results for individual studies. Second, we use them to calculate average ESs across various studies of the same model. These calculations help us to rate the effectiveness of individual models and to compare the impact of models against one another.
Based on a review of existing literature on ESs for CSR models and in consultation with experts, we set ranges in our ratings for moderate (+0.15 to +0.19), moderately strong (+0.20 to +0.24) and very strong (+0.25 and above) as components of our rating process. Because of differences among study designs and assessments, our determination of ESs can only be considered a rough estimate of impact, allowing comparison among the various models.
Question Why didn’t you look at studies conducted prior to 1980?
Response When conducting systematic evidence reviews that require the identification, collection, and analysis of thousands of study findings, researchers must make important early decisions regarding where and how far back to look for evidence. For each CSR model, researchers searched educational databases (e.g. JSTOR, ERIC, EBSCO, Psychinfo, Sociofile, NWREL, DAI), web-based repositories (e.g. Google, Yahoo, Google Scholar), and two previous large-scale reports on comprehensive school reform (Herman et al., 1999; Borman et al., 2002).
In order to conduct as extensive a search as possible across multiple sources a limit had to be placed in how far back in time we searched for evidence. Based on a review of similar efforts being conducted by the What Works Clearinghouse and in consultation with our advisors, we chose 1980 as the earliest year for studies to qualify for our review.
We felt that this was well within the “industry average” for this work and provided nearly a quarter of a century for each model to demonstrate its effectiveness and quality. Few models have been in existence that long. Some models may have concerns that the 1980 limit excluded key findings on their model, which they consider definitive. In these instances, we encouraged the model providers to submit a letter to us citing these studies so that readers would be able to factor this evidence into their decisions.
Question How do your ratings compare to findings of prior similar studies?
Response In format and approach, this study most closely resembles the Herman, et al, An Educators’ Guide to Schoolwide Reform that was commissioned by leading national education organizations, and issued by the American Institutes for Research (AIR) in 1999. That report reviewed and rated the evidence of (a) positive effect on student achievement and (b) support that the reviewed developers provided to schools. The current report expanded the number of categories of quality and effectiveness reviewed to five, and created new processes and standards to conduct the review. Importantly, it “raised the bar” on the evidence needed to receive the “highest” rating in our Categories 1-3 (see description of categories above). We felt that our standards had to be set at a very high level, in an era that requires scientifically based research, and has the What Works Clearinghouse standards to serve as a guide.
In addition, while our study is not a meta-analysis, we drew important guidance from the systematic review of 29 CSR models produced by Dr. Geoffrey Borman and his colleagues in 2002. Their study served as a strong scientific anchor against which the CSRQ Center conducted its own rating work.
Direct comparisons of these prior studies with our review are difficult for a number of reasons. First, we focused on middle and high CSR models only while the other two looked at a set of K-12 models, only some of which serve middle and high schools. In addition the rules governing which quantitative studies to include differed in each review. As well, our rating framework and ranges differed, making it difficult to equate individual ratings from one study to the next. Finally, since time has elapsed the body of research evidence on models has grown, making it hard to predict the ratings the models we reviewed would have received using the alternative approaches taken by the other two studies.
However, we remind consumers that our report is not about ranking “winners” and “losers” but rather about providing solid information to promote evidence-based decision making. Too much of a focus on which model did “better” or “worse” in one review or another detracts from the effort to find the best models to meet local needs and conditions. In this regard we feel that this report builds on previous ones and provides education decision-makers the best evidence yet available to support informed choices.
Question Don’t positive outcomes for any model depend on implementation?
Response Absolutely. A consistent finding of research on school improvement programs and models—even the most effective ones—is that positive outcomes depend on effective implementation that is carried out by highly committed, adequately trained, and well supported educators.
Many of the models reviewed in this report have solid evidence that they provide the supports necessary for effective implementation. However, it is also true that research evidence indicates that—all other things being equal, including implementation—some programs have demonstrated greater effectiveness than others. In recognition of the fact that implementation can have a powerful impact on outcomes, the CSRQ Center excluded from our analysis of effectiveness those studies in which implementation level did not meet a minimal threshold level. In practice this is difficult since studies do not uniformly measure and present implementation information. To preserve transparency in our decision-making, we carefully documented all decisions regarding inclusion, exclusion, and ratings of studies based on this or any other factor.
Our report helps consumers identify which models currently present the strongest evidence of positive impacts on student achievement and other outcomes. Models presenting strong evidence will change over time, as they improve their operation, and as some demonstrate their effectiveness through rigorous research. In our report, consumers are offered information both about model effectiveness and about support for implementation, so that they can make a wise choice from among alternatives.
CSRQ Center researchers Jennifer Harmon, Jessica Heppen, and Marlene Darwin addressed readers' questions regarding the Elementary School Report on CSR Models released by the CSRQ Center in November 2005, and available on our Web site at http://www.csrq.org/reports.html. This report offers a consumer-friendly review of the effectiveness and quality of 22 widely adopted elementary school comprehensive school reform (CSR) models. The researchers answered questions regarding the methodology and analyses used to yield information about the CSR models that were covered in the Elementary School Report. Click here to read the full biographies of Our Experts.
Question What were the specific components of the Breakthrough to Literacy model that adversely affected its effectiveness on positive student achievement?
Response Unfortunately, there is not enough research evidence on this CSR model for us to answer your question confidently. The CSRQ Center reviewed 10 quantitative studies for the effects of Breakthrough to Literacy on student outcomes. None of these studies met the CSRQ standards for quality of research design. For example, some studies did not take into consideration student outcomes at baseline; other studies did not compare student outcomes to a control group. Therefore, we cannot make valid conclusions on potential positive or negative impacts of this CSR model or its parts. We would like to point out that the effectiveness of a program may vary across school types, student populations, and levels of staff readiness. (The Methodology Section of the report offers a full explanation of the CSRQ Center standards.)
However, we can offer you a suggestion. The CSRQ Center developed materials that may provide you with a process for thinking about how CSR components are aligned with the needs of your school. For instance, we developed a guide and accompanying support materials (Moving Forward: A Guide for Implementing Comprehensive School Reform & Improvement Strategies), to help educators select and implement school reform and improvement models. The Support Materials Packet that accompanies the guide includes worksheets and offers considerations for implementing school reform models around each of the CSR components. These tools and materials can be downloaded free via the CSRQ Web site: http://www.csrq.org/resources.html.
Question Why do schools in high performing districts, or schools with few low-income students, decide to implement CSR Models that are commonly in place in low performing schools with a large population of low-income students?
Response While many schools that adopt Comprehensive School Reform (CSR) models are low-performing and do serve significant numbers of children from low-income families, there are also many mid-to-high performing schools using CSR models that serve children from middle and upper class families. One reason that so many low-performing schools serving low-income students have adopted CSR models is that the federal CSR program provides funding specifically to low-performing Title I schools that serve low-income children. However, many non-Title I, better performing schools have also implemented CSR programs—they have just had to use different funding sources to do so. Some CSR models are specifically designed to provide research-based curriculum, professional development and instructional practices to meet the needs of children at-risk (including children from low-income families who attend low-performing schools). Other models offer research-based curriculum, professional development and instructional practices that may work for a much larger range of children, from many socio-economic backgrounds and many different levels of performance.
In selecting any school improvement model, we suggest that educators examine the research regarding the impact of the model to determine whether the model has had positive impact with students and schools similar to those in your district. Educators may also want to contact the model developer to ask about the results that the model has had in schools that are similar to your own. As educators are considering selecting various CSR Models, it may also be useful to visit schools or districts who have experience in implementing the specific model you are considering. Ask colleagues about the impact of the model on student populations that are similar to those in your own setting.
For additional information, the CSRQ Elementary School Report includes information about the research outcomes of 21 elementary school models. This is a useful resource to start your examination of CSR Models. To learn more about selecting and implementing school improvement models, download the Center's Moving Forward Guide.
Question I would like to better understand how you were able to compare the CSR models in your review given that they vary widely. Some provide schools with a general framework and suggested principles for implementation and others are more directive regarding the suggested curriculum and implementation framework. My question is how can the CSRQ Center compare CSR models that vary so greatly regarding their suggested approach?
Response The CSRQ Center based its rating and reporting framework on the premise that education decision makers want to know whether, regardless of design, a particular model is able to deliver on its stated claims—including that its use helps to raise student achievement or accomplish other important outcomes. We recognize that the models reviewed in the report vary in mission, scope, practices, and services offered, including how prescriptive the developers suggest the model should be implemented. However, we believe that in the end, decision makers are interested in CSR models because they hope to see better outcomes for their students and schools. There is an old Chinese proverb that states, “I don’t care if the cat is black or white, as long as it catches mice.” We took a similar pragmatic approach in our guide, not judging the theoretical value of a particular model, but rather examining its outcome based on existing evidence.
The Center takes no position on whether it is better to adopt a model that offers a tightly scripted curriculum or one that promotes the development of “communities of practice.” Rather we present comparable, reliable evidence and ratings on the effectiveness and quality of each model, helping education consumers to select the approaches that best fit their local needs. CSRQ Center reports provide educators with a decision making tool and the reviews are intended to clarify options, not to point to or endorse “best buys.”
Question Could you tell me why Open Court was not included in the 2005 CSRQ Center Report on Elementary School Comprehensive School Reform Models?
Response Open Court was not included in the report because it is a basal reading textbook series, not a comprehensive school reform (CSR) model. We reviewed widely adopted models that offer a comprehensive set of services that support curricular, professional development, governance, scheduling, family and community relations and other changes needed to improve schools. Rather than use individual, piecemeal programs or approaches, effective CSR is meant to integrate research-based practices into a unified effort to raise student achievement and achieve other important outcomes such as reduced dropout rates or improved behavior. Several of the models reviewed in our report include as part of their overall package curricular components that support reading instruction, but others do not.
In practice, schools that pursue comprehensive school reform may adopt an externally developed model, such as the ones profiled in our report, or build their own from research-based components. It is likely that many elementary schools have used Open Court or other research-based reading/language arts textbooks and materials to create their own comprehensive reform efforts. However, it is beyond the scope of our Center’s work to review individual components, such as reading, math, science, character development, and other programs that may be used to improve schools. Readers may wish to consult the What Works Clearinghouse to find rigorous evidence reviews in these and other areas.
Question The CSRQ Center Report on Elementary School CSR Models is described as providing a review of comprehensive school reform models. However, some of the models analyzed in your report don't appear comprehensive. Additionally, some of the research that you cite as evidence of the model's effectiveness appears to focus on a limited range of academic achievement. Please explain what you mean by comprehensive and how this impacts your review of research related to these models.
Response Comprehensive school reform (CSR) is a systemic approach to school improvement that addresses every aspect of a school, from curriculum to scheduling to management to family and community involvement. Rather than use individual, piecemeal programs, effective CSR integrates research-based practices into one unified program to raise student achievement.
Some schools adopting a CSR approach choose an external CSR model to provide a research-based, replicable set of practices. These external models are meant to be "blueprints" to help a school make improvements in a number of areas. The CSR models are designed based on research and vary in focus, philosophy, and method, but all are intended to help the school raise student achievement. To help schools implement the program, CSR models staff typically provide schools with professional development and hands-on assistance.
Other schools adopting a CSR approach choose to develop their own CSR model. According to the No Child Left Behind and the U.S. Department of Education (see http://www.ed.gov/programs/compreform/2pager.html) a school implementing a CSR approach must address the following 11 components:
The U.S. Department of Education Web site includes an informative slide show outlining the basics of CSR.
Currently, there are several hundred external CSR models (also known as service providers) that districts and schools can turn to support local comprehensive school reform efforts. These models vary in mission, scope, practices and services offered. Although the CSR legislation suggests that a school's CSR program should meet the 11 components of a "comprehensive" program schools often satisfy those components through a combination of CSR models (such as those reviewed in the CSRQ Center report) and other programs or practices that are deemed research-proven. In other words, a CSR model is often part, but not all, of a schools "comprehensive" approach to schoolwide reform. The CSRQ Report profiles 22 of the most widely implemented models. Each model reviewed serves a minimum of 20 elementary schools spread over at least 3 states, and is available for adoption throughout most of the country.
The CSRQ Center does not suggest that these models are all "comprehensive." Instead, the reports are designed to give education consumers the in-depth information they need to make the best possible adoption decisions to meet locally defined needs. Each report provides basic information on the CSR model, including the model's mission and focus, year introduced in schools, grade levels served, number of schools served, and costs as well as ratings on five indicators of quality and effectiveness. By providing ratings about several aspects of a models effectiveness, we aim to provide consumers with a wider set of information about the effectiveness of each model. In addition, each review offers detailed descriptions of the products and services offered by each model to help consumers sort through their options. These reports provides education stakeholders a decision making tool to help them sort through their options regarding the hundreds of elementary school improvement choices available to meet local needs. The reviews are intended to clarify options, not to point to or endorse “best buys.”
Finally, it is important to note that the CSRQ Center does not conduct field research. Rather it gathers and reviews existing studies and evidence on the effectiveness and quality of CSR models to provide its ratings. We hope that by examining each model's evidence in a variety of areas, we will encourage researchers and model developers to conduct high quality studies that examine a model's effectiveness on a range of outcomes and for a variety of specific student populations.
Question What was the methodology you used in the report?
Response The production of this report was guided by the CSRQ Center’s Quality Review Tool (QRT). The QRT provides the criteria and procedures for independent, fair, and credible model reviews. The QRT development process involved several steps. First, CSRQ Center staff developed review frameworks in consultation with some of the nation’s most respected education researchers, program evaluators, and school improvement experts. The QRT also drew on prior and current efforts to conduct rigorous research reviews, including standards set by the U.S. Department of Education’s What Works Clearinghouse. Then, the QRT was reviewed and revised with the help of the CSRQ Center’s Advisory Committee, a nationally respected panel of experts that includes leading education practitioners, methodologists, and researchers from a variety of fields, including education, sociology, psychology, and economics.
Using the QRT, the CSRQ Center applied separate rubrics for each of the five categories of effectiveness and quality to arrive at its ratings, which are expressed by a set of symbols. In general, the rubrics resulted in the following ratings:
A detailed description of the QRT and our methodologies is provided in the Methodology section of the report. Click here to read this section.
Pulitzer Prize-winning former New York Times reporter and Emmy Award-winning producer, Hedrick Smith answered questions about school reform and discussed his upcoming two hour prime time special, Making Schools Work, which aired on PBS on October 5, 2005. Making Schools Work examines various approaches that schools are currently undertaking to raise student achievement, including Comprehensive School Reform models, such as the Comer School Development Program and Success For All. Hedrick Smith is well known for his award winning documentaries and articles which examine diverse and critical cultural and social issues. CSRQ audiences benefited from Hedrick Smiths extensive experience as a correspondent and filmmaker. Click here to read Hedrick Smith's full biography.
Question How do you evaluate videos of student learning that schools produce to show accountability to the local community based on their districts mission statement, vision and goals?
Response We are reporters and not educators, thus, we relied on educational experts to identify what good learning looks like. As reporters what we know how to do well is ask lots of questions:
Why is this good learning?
How is it good?
Does it work for all students - why or why not?
What does poor learning look like?
Which students aren't successful and what do they need to be successful?
In addition we relied not just on what we saw but also on a lot of data and research (we looked at a variety of different test scores over numerous years disaggregated by income and race as well as reports on the effectiveness of the strategies the schools and districts were using).
As for resources, the making schools work Web site at http://www.pbs.org/makingschoolswork offers more reform oriented resources.
WATCH FOR RERUNS OF MAKING SCHOOLS WORK ON PBS (Check your local TV schedule)
On the web at: http://www.pbs.org/makingschoolswork
For DVDS and Videotapes: 1-800-553-7752
Researchers from the CSRQ Center and the American Institutes for Research were also asked this question. The responses can provide readers with a framework for understanding the elements of teaching/learning situations that can be evaluated either through video or real-time observations. Responses included:
Instructional Tasks: An instructional task is “an activity engaged in by teachers and students during classroom instruction that is oriented toward the development of a particular skill, concept, or idea.” (Stein & Lane, 1996, p. 54.) With observational, video, and log data, we examine not only the content focus of instructional tasks but also the number of tasks per lesson, the average time per task, the specific kinds and levels of cognitive demands elicited by each task, and the inter-relatedness of instructional tasks.
Instructional Organization: Instructional organization is the form, social and physical, an instructional task takes. We assess the organization of interaction (e.g., seat work, group work, whole class instruction) for each task or activity and we record the types and uses of available physical materials and tools (e.g., worksheets, manipulatives, technology).
Intellectual Discourse: Classroom intellectual discourse is language during instruction focused on content or student’s understanding of that content. We evaluate the types and cognitive complexity of teacher questions and student responses during an entire lesson. AIR also studies the extent and cognitive complexity of peer interaction during group work.
Instructional Environment: Instructional environment refers to classroom level dimensions of instructional practice assumed to reflect teachers’ philosophical beliefs about mathematics, students, teaching, and learning (Stein et al., 1991). We rate the extent to which assessment is integrated with instruction and the level of classroom thoughtfulness during reading and mathematics instruction.
Question Given the variations in the programs you profile in Making Schools Work (e.g., in whether they have curriculum or not, or their level of structure), what do you think accounts for their uniformly strong outcomes?
Response Despite the variations in strategies, there are several common elements to successful reforms:
First, successful venues demonstrate a rock-solid commitment by the school system, the school, the principal, and the teachers to the belief that ALL CHILDREN can learn and the determination to make this become, all children will learn. No excuses. no explanations. There is also a sense of urgency that they have to catch all the students as early as they can.
Second, positive reform sites promote reaching students where they are and finding their level and adjusting the teaching to reach them. This means finding the hook that will motivate young and teenage students who are drifting through school. This entails working with students at their level rather than from where adults think the students should be. Especially with low-performing students, there is frequently a gap between adult demands and student performance. If misused, this gap can create tension, frustration and a sense of failure. Successful reform strategies turn that bad dynamic around and adults bridge that gap by accepting the student level, gearing teaching to that level and gradually lifting performance. Sometimes this means grouping children with different age groups.
Third, successful settings demonstrate a determination to raise the quality of teaching, either through very clear curriculum and pacing guides, or through increased resources for teacher training and mentoring by master teachers, common planning time among clusters of teachers, or a combination of all of the above. But it inevitably means spending more time and resources on constantly working to improve teaching for all teachers.
Fourth, successful districts and schools provide additional resources to low-performing schools and low-performing students, usually through tutoring, mentoring, or extra class sessions of some kind; often through sending educational teams of specialists into troubled schools to provide additional help in coaching teachers, working with the principal, and helping students.
Fifth, in successful settings, the central mission of the principal is reengineered, from being the building manager in charge of buses, books, facilities and equipment, into the role of instructional leader. This role is clear to acknowledge that teaching and learning are the priority of the school leader. Principals are the main actors in creating change, so their focus has to be on teaching and learning. Successful sites have principals that recognize that others can run the building.
Sixth, there is constant evaluation and reevaluation of performance by teachers, to determine whether students are actually absorbing, understanding, and retaining the material and the learning techniques that are being taught. In other words, using standards not only to check on student progress, but more frequently, to monitor the effectiveness of teaching.
There are other factors, too, but these are central.
Question What are some of the common elements of successful school reform?
Response Most school reform programs, such as Success for All, Comer School Development Program, Knowledge is Power Program and High Schools That Work, have their own strategies and their own profiles. They tackle problem schools from different angles. The track record demonstrates that there are multiple pathways to success, as shown in our two-hour documentary Making Schools Work, due for nationwide broadcast on PBS on Oct. 5.
Success for All uses a highly scripted reading curriculum as its engine for improvement. Comer Process sees a harmonious school culture and holistic child development as the avenue to student gains. The main formula at KIPP - Knowledge is Power Program - is tough-love discipline for at risk kids, plus an extra-long school day and school year. High Schools That Work motivates drifting teen-agers with hands-on learning.
Yet while many flowers bloom, what struck me in three years of off-and-on reporting, was that underneath these different formulas for success, you see common threads, common ingredients of reform.
High Expectations for All students: No idea is more central to success than the rock-solid expectation that all children, not just the fortunate minority, are capable of learning at high levels. "We need to believe that all children can learn," declares Mike Feinberg, co-founder of KIPP. "But then what we need to act on is changing the word 'can' to 'will.' So we need to act on the fact that all children will learn."
Improving Teaching: Finding ways to improve the quality of teaching is the second most universal ingredient of success that we found. Tony Alvarado, former superintendent of New York's district 2, compressed his core philosophy into a succinct mantra: "We have this very simple theory - kids learn from teachers. If the kids need to learn more and more powerfully, then the teachers need to know more and their teaching has to be more powerful."
Both in District 2, and later in San Diego, Alvarado put enormous resources into retraining, coaching, and mentoring teachers. Other reformers, such as Bob Slavin, the architect of the Success for All, seem to stress some other factor, such as a scripted curriculum, but that is merely the method for improving teaching. Slavin, who minces no words in voicing his frustration with well-meaning teachers who do not know how to reach failing readers, seeks nothing less than a revolution in teaching through his program. In Charlotte, former Superintendent Eric Smith took the Slavin approach to a larger scale.
Clear Standards and Regular Testing: Eric Smith's real trademark, however, was another essential ingredient of reform - clear articulation and relentless pursuit of high standards for all children. Under pressure from the state of North Carolina to rescue failing schools and deliver better student performance, Smith use regular testing, at least four times a year, as his vehicle to make certain that students were held to state standards and not left to fall through the cracks.
Time and Buy-In: These two ingredients go together. Our research, our interviews, our on-location reporting made clear time and again that there are no shortcuts and no quick fixes to school reform, and that no reform effort can succeed without buy-in from all the important stakeholders and sufficient time to introduce the reform, train the teachers and new leadership, and then implement the reform.
All the successful models have requirements for buy-in. For example, Success for All requires an 80% vote from teachers in a school before it will introduce its reading program. Other reform models have their own buy-in requirements. District-level reform cannot achieve its potential without a consensus around the reform strategy from the district leadership, the school board, the teachers' union, and a large majority of principals and teachers in a system, as well as activist parents. We saw the contrast between Tony Alvarado's experience in New York's District 2, where he had buy-in from the key constituencies, and in San Diego, where he and Alan Bersin lacked solid buy-in.
Structure: Across the board, effective reforms provide structure. That is a core concept of the school and child development strategy created mainly for elementary schools by Yale University child psychiatrist James Comer, and of the program that KIPP's founders, Mike Feinberg and David Levin, created for middle schools.
But structure can also come from common school-wide and district-wide curriculums and lesson plans, and razor-sharp clarity about the objective of each day's lesson so that struggling students know what they are expected to learn every day. To reformers like Eric Smith in Charlotte and Bob Slavin with Success for All, clarity and focus provide structure.
Resources: Everywhere that we witnessed dramatic student gains in problem schools or neighborhoods, we saw that a commitment of resources was essential to success. It took a fresh input of resources to level the playing field and to put schools in high poverty neighborhoods on a par with affluent suburban schools.
Most low-performing schools are short on resources to start with. The biggest handicap of the weakest schools is that they are generally staffed by the least experienced teachers and principals, those least equipped to educate struggling learners. Quite often, they use watered down curriculums, less challenging textbooks, poorer methods of testing and interpreting tests, less individualized or small group support for failing students.
That was one of Eric Smith's first discoveries on taking over as superintendent in Charlotte in 1996. "We found that the expectations were different for inner city kids versus the suburban," says Smith. "We found that the pace of instruction, the speed with which content was being delivered was different. Totally different expectations."
We found similar stories elsewhere. At Centennial School in Mount Vernon, Washington, Principal Alan McDonald reported spending $100,000 a year for individual 20-minute daily tutoring sessions in reading for about 100 students, as part of the school's Success for All strategy. At KIPP, teachers make about $10,000 a year more than their regular public school counterparts, but they put in longer days, Saturday classes and summer school - all extra time and extra resources to lift students who begin KIPP below grade level. At Corbin High School, special computerized re-teach courses and pre-school morning tutoring are among many extra efforts the school targets at borderline students.
"Children who have education needs, need more time and they need more support and that's more money," explains Susan Agruso, assistant superintendent in Charlotte. "And as a society that's the thing that I hope that we will move towards - recognizing that we can be successful with all children, but we do have to have the resources to do that. "
Question Did you learn anything about how the schools and districts you visited addressed the needs of diverse student populations, such as students who are culturally and linguistically diverse, or those students with disabilities?
Response Yes, we learned a lot about dealing with cultural and linguistic diversity because most of the programs we covered in MAKING SCHOOLS WORK confront that kind of student demographics. For example, the elementary reading program, "Success For All," has a particularly good track record with minority students and students who are English Language Learners. Not only is the basic program well suited to such student populations, but SFA has special strategies, classes and tutoring components geared to them. We saw excellent results with the children of Mexican-American farm laborers and recent immigrants from Russia, Ukraine and Asia.
The KIPP program for middle schools (KIPP+ Knowledge IS Power Program) is also heavily targeted at a culturally diverse and economic hard-hit student population, and it has showed dramatic results, with its methodologies. So I think you would find a lot of material of interest to you.
That is also true of the Comer Process, which is a more comprehensive elementary and middle school reform model, developed and derived especially for low-income neighborhoods and schools, with high components of minority students. The district reforms that we covered in Charlotte, NC and in the lower east side of Manhattan, also demonstrated effectiveness with minority students - Chinese-American, Latino and African-American students.
Nationally recognized researcher, Dr. Libia Gil, a Senior Fellow at the American Institutes for Research, answered questions regarding CSR implementation issues at the district level. Dr. Gil is well known for her expertise on school improvement from an administrative perspective. CSRQ audiences benefited from Dr. Gil’s extensive experience as superintendent of schools for the Chula Vista, California School District. She has authored nationally recognized publications and in 1992, she received the prestigious Harold W. McGraw, Jr. Prize in Education. Click here to read Dr. Gil’s full biography.
Question As a local school administrator, I know that implementing a CSR model requires cooperation between providers and district and school personnel. What recommendations would you offer our team as we put a model in place?
Response These key recommendations are provided with the basic understanding that there is no single best solution for school improvement. Different programs work best for certain students within certain contexts and there are many factors to consider. In considering the current and future direction of CSR, we recognize that sustainable school change must include a systems approach. The focus on system wide impact demands a synchronized change effort in central office functions along with school level change efforts.
Some of the lessons we have learned for successful CSR implementation include:
Additional recommendations offered by Dr. Gill can be found in Putting
the Pieces Together Lessons from Comprehensive School Reform Research,
(2004). Christopher Cross, Editor. Chapter Four: The Past and the Future
of Comprehensive School Reform, Perspectives from a Researcher and Practitioner
by Steven M. Ross and Libia Gill, Pages 160-169.
Nationally recognized researcher, Dr. Steve Ross, answered questions regarding research and school reform. Dr. Ross, Professor and Executive Director of the Center for Research in Educational Policy at the University of Memphis, is well known for his expertise on school improvement and evaluation. He is the author of six textbooks and over 115 journal articles. Click here to read Dr. Ross's full biography.
Topic: Importance of Evaluation & Research
Question How do I impress upon my staff the value of research? Why should they care about evaluation data and CSR models?
Response In a 2003 report, it was estimated that over 400 different CSR models have been adopted by schools nationally! Unfortunately, for the vast majority of models, convincing evidence of their effectiveness from rigorous studies is not available. Clearly, engaging in CSR takes time, effort, and resources for schools. Given the importance and magnitude of the CSR investment, it is obviously quite risky to adopt a model that merely looks good on paper or that some principals or district leaders promote as having “worked at their schools”. Most research-based programs and models work successfully “somewhere”, but schools can’t afford to make the wrong choices for their teachers and students. Only through systematic and rigorously conducted research, can we determine the effectiveness of different models overall and in particular contexts (e.g., such as in rural or urban areas, with certain ethnic or cultural groups, etc.). Rigorous research means being able to attribute achievement gains to the CSR model and not to extraneous factors such as unusually enthusiastic teachers, a new district reading curriculum, smaller class sizes, etc. Although each CSR developer can readily produce “evidence” of their model’s effectiveness, the key question is the quality and validity of the studies that produced such evidence. Principals and teachers generally don’t have the time or research expertise to perform careful evaluations of CSR studies or developer’s portfolios. Fortunately, the CSRQ Center will use its Quality Review Tool to analyze major CSR models and will soon be publishing its findings in the form of a Consumer’s Guide, called CSRQ Reports. Using those reviews and other credible sources such as the Catalog of School Reform Models, produced the Northwest Regional Lab is essential in enabling schools to choose CSR models that (a) are likely to be the best fit for conditions, needs, and budgets; (b) are able to be implemented with high fidelity by teachers and administrators; and most importantly, (c) give the greatest promise of raising student achievement.
Topic: Evaluation & Using Data to Drive School Improvement
Question This looks like an excellent resource for us to begin our preparation for our 5 year SACS review (referring to the CSRQ Works in Progress Report). We are implementing a true school improvement process that will be the model for future years, but what we see developing is lack of necessary info (other than test scores) on our previous 4 years. What kind of position does that place us in, when we don't have a clear enough picture of where we've been but we do know now where we want to be? Is your software something that would help us make the crossover to a real planning process?
Response Research on CSR strongly supports the idea that teaching improvements and strong program implementations do not occur in a vacuum, merely because some new reform model has been selected. The most direct cause of higher student achievement is improving classroom instruction. To improve their effectiveness, however, teachers need quality professional development and the motivation to learn and practice the new instructional strategies and reforms that CSR brings. Research also shows that implementation of CSR is highly correlated with school climate.
Based on this rationale, I suggest that the writer encourage her school(s) to engage in systematic “formative evaluation” to ensure that progress is steadily being made on the precursors to raising student achievement, i.e., positive school climate, teacher buy-in, strong CSR implementation, and teaching improvement. Helpful evaluation tools and suggestions for an overall improvement process may be found at several sources, including my own Center, www.memphis.edu (click on “instruments” or “formative evaluation.”), and the School Improvement Knowledge Base Web site, www.helpforschools.com/sikb/index.shtml, especially element 1, Know your School.
These and similar formative approaches can help to produce a yearly school report on status and needs. The report is then used, along with achievement data, to inform data-driven improvement planning.
P.S. This FL writer may be interested in knowing that coincidentally, but not surprisingly, a school in her state recently submitted such a yearly evaluation report as the primary support material for its SACS review. Not only were they approved, but they were recognized for excellence by SACS at their 2005 meeting in Atlanta.
Editor’s Note: The question asked about software that can be used for school improvement planning. The CSRQ Center does not develop software, however, has compiled variety of school improvement resources at: www.csrq.org/resources.html. The following can be useful:
Data Analysis for Comprehensive School Reform
Author Victoria Bernhardt provides tools and guidance to help schools collect and analyze data to effectively plan for reform. (2004) http://eff.csuchico.edu/books/data_analysis.php
At Your Fingertips - MPR Associates developed this manual to help schools use readily available data, such as graduation and attendance rates, to develop strategic plans for CSR programs. (Updated 2004) http://www.mprinc.com/ps/ayf/index.html
