Thursday, October 25, 2007

Media Bias on School Choice

The Milwaukee Journal-Sentinel has rather gleefully jumped on a study from the Wisconsin Policy Research Institute that supposedly shows that school choice doesn’t improve education.
A study being released today suggests that school choice isn’t a powerful tool for driving educational improvement in Milwaukee Public Schools.

But more surprising than the conclusion is the organization issuing the study: the Wisconsin Policy Research Institute, a conservative think tank that has supported school choice for almost two decades, when Milwaukee became the nation’s premier center for trying the idea. The institute is funded in large part by the Milwaukee-based Lynde and Harry Bradley Foundation, an advocate of school choice.

“The report you are reading did not yield the results we had hoped to find,” George Lightbourn, a senior fellow at the institute, wrote in the paper’s first sentence.

“We had expected to find a wellspring of hope that increased parental involvement in the Milwaukee Public Schools would be the key ingredient in improving student performance,” Lightbourn wrote. But “there are realistic limits on the degree to which parental involvement can drive market-based reform in Milwaukee.”
Rick Esenberg has beat us to the punch in critiquing the methodology of this particular study. As he points out, it’s not a study of private school choice, only a study of choice within the public sector.

And further:
First, the study doesn’t really measure what parents in Milwaukee actually do. As the authors say, “the basic approach was to identify the determinants and frequency of parental choice and parental involvement using a national data set, and extrapolate those results to Milwaukee, relying on the particular demographics of the MPS district.” In other words, they take national data on how households with particular demographic characteristics act with respect to choosing schools and becoming involved with the schools and then assume that people with those characteristics in Milwaukee would act the same way. They then look at the characteristics of families in MPS and run the numbers.

That’s a respectable approach, although it would not catch the effects of any local efforts to improve decision-making or increase parental involvement.

Second, it seems less plausible to extrapolate from these data to families who choose voucher schools. The demographics of that group may be different. In addition, opting out of the public schools may itself reflect a higher degree of involvement with a child’s education (it presumably takes more effort) and a population that does that may differ from others with same socioeconomic characteristics.
If social scientists know anything, it is that contextual effects matter, and they often matter a lot. Put people on one social context, and they act one way. Put them in another social context, and (although they may have the same characteristics) they act in an entirely different way.

Thus it is highly questionable to generalize from school districts across the nation that have fewer choices -- and a much shorter history of giving parents a choice -- to Milwaukee.

Then there is the fact that the study insists that parents have to be, essentially, model consumers before choice can work.
Taken together, these three estimates allow one to perform calculations regarding a hypothetical “ideal consumer” in a public school choice system. This consumer would maximize the marketplace pressures on schools, thereby creating the greatest prospects for school reform and student achievement. Such a consumer would:
  • exercise choice, rather than simply enrolling his or her child in the local neighborhood school;
  • consider at least two schools in the choice process, rather than simply choosing a school without assessing the potential costs and benefits of alternatives; and
  • bring performance-based/academic criteria to bear in the choice process.
The estimate of MPS parents meeting all three criteria is just 10 percent. Given this number, it seems unlikely that MPS schools are feeling the pressure of a genuine educational marketplace.
In the first place, loss of ten percent of your students is not trivial. In most industries, a loss of ten percent of market share would set off huge alarm bells in the executive suites.

In the second place, who says that simply putting your kid in the neighborhood school is not an exercise of “choice?” The criterion for “choice” is not that you send your child off to another part of the city to school, rather, it’s that you would if the opportunities there outweighed the advantages of a neighborhood school.

Further, who says you have to consider two or more schools? Rational consumers will often decide to “stand pat” with their current choices if those choices are considered “good enough.” Imagine a parent whose child seems satisfied with his or her current school. The parent sees that the child is doing an appropriate amount of homework, getting decent grades, and doing as well as he or she should be doing on standardized tests.

This parent is perfectly rational to simply “stand pat.”

Finally, who says that parents must “bring performance-based/academic criteria to bear in the choice process?” Choosing a school on the basis of the values taught there (relevant when parents choose a religious school) or extracurricular activities (relevant if a child has a lot of potential in, say, football or swimming or drama) might be the best way to go. Especially if there is no reason, a priori, to suspect you are giving up academic quality in the process.

The Wisconsin Policy Research Institute study can be contrasted with the best academic study on this issue, one that appeared in the American Journal of Political Science.

(In the interests of full disclosure, we have published both in The Wisconsin Interest, published by WPRI, and in the AJPS.)

The scholars who published in the AJPS note that in spite of limited information, parents tend to select schools that match their preferences:
The results in the three analyses presented in Table 3 all point in the same direction-even though levels of objective information held by parents are low, their actual choice of schools reflects their preferences in education. For example, parents who say that high scores are important enroll their children in schools that are more than a quarter of a standard deviation above the district mean in reading scores. Similarly, parents who value safety and those who value diversity also enroll their children in schools that are significantly higher than the average schools in their district on these dimensions. (p. 780)
In short, school choice works, even in the absence of highly informed parents.

The authors of this article further point out:
We demonstrate that the same puzzle we have just documented in school choice-that of matching preferences in the face of low levels of information-has been posed in the study of many private goods markets. Most research into this phenomenon has demonstrated that the solution to the puzzle lies in a small group of consumers who actually gather information about products. Thus, while on average, information about products is low, a small group of buyers in markets tend to be more informed. More importantly, these studies also show that this small percentage of buyers can effectively drive a market toward a competitive outcome. (p. 782)
At least in New York City, were this study was conducted, the informed parents are numerous enough to drive a choice system effectively.

But do we need to look at data from elsewhere? Is there a good analysis of Milwaukee data?

Indeed there is. A study from Rajashri Chakrabarti, an economist at the Federal Reserve Bank of New York, looks specifically at Milwaukee, and reaches the following conclusions:
The Milwaukee voucher program, as implemented in 1990, allowed only non-sectarian private schools to participate in the program. Following a Wisconsin Supreme Court ruling, the program saw a major shift and entered into its second phase, when religious private schools were allowed to participate for the first time in 1998. This led to more than a three-fold increase in the number of private schools and almost a four-fold increase in the number of choice students. Moreover, due to some changes in funding provisions, the revenue loss per student from vouchers increased in the second phase of the program. This paper analyzes the impacts of increase in competition brought about by these changes on public school performance in Milwaukee. Using data from 1987 to 2002, and a difference-in-differences estimation strategy in trends, the paper finds that these changes have led to an improvement of the public schools in the second phase of the program as compared to the first phase. The results are robust to alternative samples and specifications, and survive several sensitivity checks including correcting for mean reversion. The findings imply that voucher design matters and choice of parameters in a voucher program is crucial as far as impacts on public school incentives and performance are concerned.
The article has been accepted for publication by the Journal of Public Economics.

This, of course, makes excellent intuitive sense. Where options for parents are tightly constrained, and where public schools have little to fear, fiscally, from parents “opting out,” there is little reason to expect them to improve their performance.

But when competition becomes a real threat, it also becomes a real prod.

Now, a question.

How much attention did the Milwaukee media pay to the Chakrabarti study?

Searching the archive of the Journal-Sentinel, there is only one mention. It’s a blot entry from Patrick McIlheran pointing how that the study has gotten no attention!

Labels: , , ,


Blogger Dad29 said...

So the question: how MUCH did MPS' 'performance' change?

Seems to me that the JS has run a couple of articles which demonstrate that on standardized testing, MPS' kiddies ain't so hot.

My thought is that the WPRI study held no surprises; overall, MPS' scores were not substantially different--but of course, if one is attempting to make silver from brass, it ain't going to happen.

8:23 AM  
Anonymous George Mitchell said...

The following was sent to the study's author:

October 26, 2007


WPRI’s sponsorship of your recent study, combined with the Journal Sentinel’s misleading article, has generated wide interest in the findings. You can anticipate many questions regarding the methodology you employed and whether it was appropriate for the Milwaukee situation.

For reasons summarized below, I am interested in knowing if you or WPRI submitted your methodology to others for independent review. If so, can you share the responses you received? If the methodology was not reviewed by others, I think it would be advisable for you or WPRI to seek such a review.

You estimate “the extent and nature of public school choice and parental involvement” in Milwaukeee by applying “a variety of demographic variables…correlated with…parental choice and parent involvement...” The variables, derived from a “national data set,” were:

•educational attainment of parents
•race and ethnicity of students
•household composition (single-parent vs. two-parent), and
•mother’s employment status.

The “national data set” consisted of answers to an NHES survey. These answers apparently show, for example, that high educational attainment is correlated positively with parental involvement and more engagement in school choice. Similarly, I assume that two-parent, non-minority families also were positively correlated with parent involvement and engagement in school choice. And so on.

You assume (or “extrapolate”) that MPS parents will behave in the same manner as those who answered the NHES survey. By constructing a demographic profile of MPS parents, you thus believe you can estimate the “extent and nature of public school choice and parental involvement within the Milwaukee Public Schools district.” In the specific case of school choice, you conclude that only 10 percent of MPS parents are discerning choosers.

George Lightbourn’s introduction to your report says, “The report you are reading did not yield the results we had hoped to find.” However, given your methodology, the results were completely predictable — even pre-ordained. Specifically, rather than gathering information about the actual behavior of MPS parents, the study simply assumes that they behave in a manner similar to respondents to a national survey. Given the demographic profile of MPS parents, you knew from the start that you would estimate relatively low levels of school choice and parent involvement. To quote one scholar who has read your report, “[WPRI] simply assumes that parents of a certain demographic profile behave the same way no matter what. In this way, the WPRI study assumes the very thing it claims to prove. The study finds that [parents] are not effective choosers based on data from environments where they have few choices.”

The key question, then, is whether the methodology is appropriate for Milwaukee’s unique situation.

Consider some actual numbers, beginning with your ten percent conclusion. Applied only to MPS enrollment, that yields a group of 9,000 discerning choosers. Applied to all Milwaukee students in tax-supported K-12 education, the total would be 12,000. So, you find that between 9,000 and 12,000 students have discerning parents. Yet:

• There are more than 17,000 students who attend schools in the Milwaukee Parental Choice Program.

• More than 5,000 other students attend independent public charters authorized by the City of Milwaukee and UW-Milwaukee. That’s more than 22,000 students.

• There are thousands — perhaps tens of thousands — of MPS students with parents who have chosen neighborhood schools. Indeed, a specific focus of MPS school selection in recent years has emphasized neighborhood schools. Your report appears to discount all those choices by labeling neighborhood schools as a “default” option.

• Roughly 10,000 other parents use the open enrollment process and the Chapter 220 program to make clear choices about where their students attend school.

• Thousands more parents enroll their children in so-called specialty MPS schools. These parents make clear choices. Many of those schools have waiting lists, yet another measure of choice.

In other words, by any reasonable estimate the number of actual choices made by Milwaukee parents substantially exceeds 9,000-12,000. Even allowing for a subjective definition of what a well-informed choice is, the number is simply much greater than your 10 percent estimate.

This raises the possibility that your methodology is highly inappropriate when it comes to the Milwaukee situation, one that is unique among American cities. Given that Milwaukee parents have so many options — a condition that has existed for more than a decade — it is questionable to assume that their behavior will not differ from other urban parents who don’t have such options.

While there are other assertions in your study that I find debatable, those are overshadowed by the methodology question. As part of the continuing discussion your report will engender, I reiterate that an independent assessment of that methodology is in order. This is all the more so given the existence of peer-reviewed scholarship (such as by Hoxby and Chakrabarti) that is at odds with your conclusions; it is unclear why your report does not acknowledge and discuss their work.


11:18 AM  

Post a Comment

Links to this post:

Create a Link

<< Home