Voucher and Class-Size Research

by
Alex Molnar and Charles Achilles

Center for Education Research, Analysis, and Innovation
School of Education
University of Wisconsin-Milwaukee
PO Box 413
Milwaukee WI 53201
414-229-2716

October 25, 2000

CERAI-00-28

Voucher and Class-Size Research

By Alex Molnar and CharlesAchilles

Education Week, Oct. 25, 2000

The academic benefits ofeducational vouchers were blockbuster back-to-school news this autumn. Thereport that generated all the media attention, "Test-Score Effects ofSchool Vouchers in Dayton, Ohio; New York City; and Washington, D.C.: Evidence FromRandomized Field Trials," was authored by a team of researchers, includingPaul E. Peterson of Harvard University. Shortly after the voucher report wasreleased, former U.S. Secretary of Labor Robert Reich wrote an opinion piece inThe Wall Street Journal making a case for what he called"progressive vouchers." Referring to the latest voucher study, Mr.Reich asserted: "Evidence mounts that vouchers do work for kids who usethem."

If by "work" Mr.Reich means that voucher programs increase academic achievement, perhaps heshould review the evidence with more care. Virtually all of the data on theachievement impact of publicly funded voucher programs come from the MilwaukeeParental Choice Program and the Cleveland Scholarship and Tutoring Program.Although the Cleveland evaluation is ongoing, the Milwaukee data are now 5years old. Moreover, the 1995 legislation that expanded the Milwaukee programto include religious schools also removed its evaluation component, so no newdata will be available anytime soon. Therefore, most of the evidence about theimpact of voucher programs will, for the foreseeable future, come from theprivately funded voucher programs, such as those in Dayton, New York City, andWashington.

In the August 2000 reportreferred to by Mr. Reich, the Peterson team presented the results from Dayton,New York, and Washington averaged across grade levels. The combined averagedresults from all three cities are also presented. Since averaged results tendto conceal inconsistent findings, they may make the achievement impact reportedappear more generalized than it is.

In this case, even theaveraged results are inconsistent. Considering the results that are significantat the conventional .05 level yields the following information: According tothe Peterson team, simply being offered a voucher makes a significantdifference in reading but not math achievement for African-Americans in NewYork City in year two. In Dayton, the offer produces no significant result ineither year. In Washington, the offer makes a difference in African-Americantest scores in reading and math in both years. Actually switching to avoucher school in New York produces a significant advantage in reading forAfrican-American students in both years, and in math in year one. InWashington, there is an advantage in math in both years, and in reading in yeartwo only. In Dayton, switching to a voucher school produced no achievementadvantage in either year.

Except in Washington, forwhich two grade-level groups (grades 2-5 and grades 6-8) for African-Americanstudents are provided, it is not possible to tell how students in differentgrades performed. The Washington results, however, appear to be driving theaveraged findings for the three cities. More specifically, it seems that theaveraged results are heavily influenced by African-American students in grades2-5 in Washington.

The impression created bythe averaged data was apparently troubling enough for Mr. Peterson's New Yorkresearch partners at Mathematica, a highly respected private research firm, tocause them to issue a separate statement, entitled "Voucher Claims ofSuccess Are Premature in New York City." In their statement, theMathematica researchers reported that students offered a scholarship performedat about the same level as students in the control group. They further notedthat the gains reported for African-American students in New York were largelythe result of the scores of one group of students (who were in the 5th grade in1998-99 and 6th grade in 1999-2000) and, therefore, must be interpreted withcaution. The Mathematica statement pointed out that there was no achievementadvantage for Hispanic students and advised that, since the gains reported wereso concentrated in a single group, "one needs to be very cautious insetting policy based on the overall modest impacts on test scores."

After reviewing the majorresearch reports associated with the academic impact of vouchers, setting asidemethodological qualms, and accepting at face value all findings that achievedthe conventional .05 level of significance, we offer the following advice toparents who are concerned about the academic performance of their children andwho are interested in vouchers:

· In Milwaukee,enroll your child in a voucher school and wait three or four years. Your childmay at that point begin to do better in math and reading (research by thePeterson group), or just in math (research by Princeton University's CynthiaRouse), or in neither subject (research by the University of Wisconsin-Milwaukee'sJohn Witte). A better choice would be to send your child to a Milwaukee publicschool with reduced class size. They outperform voucher schools (Rouse).

· In Cleveland,enroll your child in a voucher school for 4th grade language arts and maybescience, depending on the classroom to which your child is assigned. Becareful, however, to avoid newly created private schools. Students in thoseschools perform significantly less well than students in the Cleveland publicschools in all subjects tested (research by Indiana University's Kim K.Metcalf).

· In Washington, ifyou are an African-American parent, enroll your child in a voucher school ingrades 2-5 (Peterson).

· In New York City,if you are an African-American, enroll your child in a voucher school in 7thgrade for the 2000-01 school year (Mathematica).

Even without consideringthe manifold methodological questions that threaten the validity of manyvoucher findings, the literature on vouchers and academic achievement adds upto an inconsistent gaggle of results that do not offer intelligible guidance toeither parents or policymakers. Unfortunately, as Jay P. Greene previously hasnoted in Education Week (“Rescuing Education Research,"Commentary, April 29, 1998): "Interest groups have learned that they cansuccessfully check research contrary to their goals by producing their ownstudies, no matter how lousy, to sow confusion among policymakers, journalists,and the attentive public about what to believe."

Thus, despite thepreliminary character and inconsistent outcomes of voucher research,well-financed voucher advocates are now sowing confusion by claiming that theachievement evidence is clear, systematic, and compelling. Some compare thepower of the research and the magnitude of the voucher findings to the resultsof class-size research, most often the Tennessee Student-Teacher AchievementRatio, or STAR, experiment. Since education policy is a matter of makingchoices among different policy options, that contention is worth examining.

In general, we can have themost confidence in a body of research that includes studies that are ofsufficient size and scope to provide educationally important and reliableresults. Moreover, current work should support the findings of earlier studies,and the findings should be the subject of rigorous peer review. Using thesecriteria, there is no comparison between class-size and voucher research.

The STAR project, conductedin Tennessee from 1985 to 1989, was a longitudinal, randomized experimentinvolving 11,600 students, grades K-3. STAR was designed to determine theeffect of reducing class size in grades K-3 on student achievement and socialdevelopment. At the end of the experiment, STAR students were followedthroughout their K-12 education careers. They continue to be monitored in theirpostsecondary pursuits. Thus, STAR provides a large-scale, longitudinalassessment of the impact of reducing class size in the primary grades. InitialSTAR results, showing a significant achievement advantage for students in smallclasses, and numerous follow-up and subsidiary studies, have been published inrefereed journals.

The initial STAR data havebeen reanalyzed and the results confirmed by the Princeton University economistAlan Krueger. In a subsequent study, he also found a significant relationshipbetween participation in a STAR small class and the rates at which studentstook college-entrance exams.

Follow-up studies of STARstudents in grades 4-12 have found that they outperformed students who had beenin regular-size classes on tests in all subject areas tested. In fact, studentswho had small classes from kindergarten to grade 3 outperformed regular-classstudents on all tests, every year, and the gap between small and regular-classtest results increased each year. For example, an average STAR studentin a small class for grades K-3 would outperform a regular-class student atgrade 4 by 6.6 months and by 8.7 months at grade 8, fully five years afterleaving the small-class environment.

The magnitude of positivevoucher academic effects is sometimes described as comparable to the total STAReffects, rather than the STAR effects for minority students. For voucherstudies, in which the overwhelming majority of students are African-American,the relevant comparison is to the performance of STAR minority students. Forthose students, the STAR effects were approximately double the total effects.Interestingly, it appears that the voucher students who perform at a higherlevel than students in public schools tend to attend smaller classes in smallerschools.

In addition to STAR, thepositive results of class-size studies in, for example, Wisconsin; California;Fairfax County, Va.; and Burke County, N.C., complement the findings ofpowerful early-intervention studies. The Perry Preschool study followed 123youngsters, and the Abecedarian study followed 109, from preschool intoadulthood. The treatment-group students in these studies have consistentlydemonstrated better educational and social outcomes, such as employment, thantheir control-group peers. When the results of these early-interventionprojects are considered in light of the STAR data on the enduring academicimpact of small classes in grades K-3, all of the findings are strengthened, andpolicy implications can be drawn with considerable confidence.

It is now possible to saywith a high degree of assurance that attendance in a small class is not aone-time inoculation that assures academic success. For long-term, continuingbenefits, children should participate in small classes in the primary grades orpreschool. The small-class experience must be intense-all day, every day, for asufficient duration (that appears to be at least four years). When thiscombination of early intervention, intense treatment, and sufficient durationoccurs, academic and social benefits continue to increase throughout schoolingand extend into young adulthood.

It is no surprise that therecent RAND Corp. study of American education, "Improving StudentAchievement," found strong associations between student achievement andboth higher public prekindergarten participation and reduced pupil-teacherratios in the lower grades.

Some voucher advocates havetaken to supporting their position by asserting that no one has demonstratedthat vouchers harm public schools. In our view, this is not good enough. Goodpublic policy should be constructed using the most powerful data available toactually help the public schools provide a high-quality education to thestudents they serve. When it comes to the evidence on vouchers and the evidenceon class size, the policy choice is clear.

Alex Molnar is a professorof education and the director of the Center for Education, Research, Analysis,and Innovation at the University of Wisconsin-Milwaukee. He is a principalinvestigator of Wisconsin's Student Achievement Guarantee in Education (SAGE)class-size-reduction program. Charles Achilles is a professor of education atEastern Michigan University and was a principal investigator of the TennesseeStudent Teacher Achievement Ratio (STAR) class-size experiment and numerousother class-size studies.