Boston Charter School Study: What “Improved Scores” Look Like

Slate article Do Charter Schools Work? is a good read; I recommend it. But what I wanted to do here is point out some interesting facts in the featured study: Charter Schools and the Road to College Readiness, which is incredibly important in that it’s one of the few studies that focus on high school outcomes.

Relevant quotes:

While the students who were randomly offered a seat at these high schools graduate at about the same rate as those not offered a seat, lottery estimates show that charter enrollment produces gains on Advanced Placement (AP) tests and the SAT. Charter attendance roughly doubles the likelihood that a student sits for an AP exam and increases the share of students who pass AP Calculus. Charter attendance does not increase the likelihood of taking the SAT, but it does boost scores, especially in math. Charter school attendance also increases the pass rate on the exam required for high school graduation in Massachusetts, with especially large effects on the likelihood of qualifying for a state-sponsored college scholarship.

…
Estimates of the effect of charter attendance on AP scores by subject… show a large increase in the likelihood that charter applicants take tests in science, calculus, and history, three of the most commonly taken AP exams. Paralleling charter schools’ large impact on MCAS math scores, the clearest AP score gains are for calculus. Charter attendance boosts the probability of taking the AP calculus test by 21 percentage points, and appears to boost the likelihood of earning a score of at least 2 by nearly 9 points. The corresponding impact on the likelihood of earning a 3 on AP calculus is 7 percentage points, though the estimated increases in the likelihood of scoring 2 or 3 are only marginally statistically significant. Charter attendance increases test-taking in science and US history, with no corresponding impact on scores in these subjects. Charter schools have little effect on English test-taking or scores.

…
Although charter attendance has little effect on the rate at which applicants take the SAT, charter attendance raises the SAT scores that applicants earn on the test. In particular, coding scores as zero for non-takers, charter attendances pushes the SAT composite score (the sum of math, verbal, and writing scores) above the bottom quartile of the state composite score distribution by 11 percentage points. Gains in math contribute most to the shift in composite scores; effects on verbal and writing scores are smaller (the estimated low-end shift in verbal scores is marginally significant). Charter attendance also raises the probability that applicants earn an SAT reasoning score (the sum of math and verbal) above the state median by 13 percentage points, with math again the largest contributor to this gain.

Although charter attendance has smaller effects on verbal and writing scores, the composite SAT score gain is estimated to be a little over 100 points, a large and statistically significant result. The gain here amounts to almost one-third of a standard deviation in the US composite score distribution. The corresponding effect on the SAT reasoning score is 74 points, also a large gain.

I’m torn as to the vantage point from which to consider the SAT score increase. The “average” increase provided by test prep is between 5-10 points. But everyone knows, or should know, that if “average” were “expected”, then test prep wouldn’t have kept Washington Post Company solvent for most of the last 20 years. Most parents would be unhappy with the results (while understanding the law of averages) if their average students only improved 100 points composite, which is an average of 30 points per section. Another way of looking at this increase is to argue that 30 points per section is three times the average achieved by test prep companies. Whoohoo! Except, of course, charters had three years. Test prep companies have about 15 hours.

Of course, there’s this annoying repetition of the test prep canard:

Designed to be challenging for all students, low SAT scores are a special concern for poor and minority students. Gaps in SAT scores by race and socioeconomic status that might be attributable to family background and school quality are further accentuated by the willingness of higher income families to invest heavily in SAT preparation. (see, e.g., Bowen and Bok, 2002)

How hard is it to stop repeating utter pap? Blacks and Hispanics are more likely to use test prep than whites, although East Asians trump everyone. And here’s a table to prove it (table courtesy of Inside Higher Ed, as mentioned in the linked essay):

And until someone can explain the socioeconomic reality of poor whites outscoring high income blacks on the SAT, best not go to that well, either.

Ahem. Back to the study.

Demographics of Boston Public Schools vs. charter students:

So if I’m reading this right, the charter population is way more female, way more black, way less Hispanic and a third as Asian as the lottery losers. The drop in Asians puts the math score increase in perspective, although the study makes clear that the charter group applicants had higher scores than average. (The study ignores any detailed discussion of race.)

And what did they do, these charter schools, to teach their kids?
Taught them for more hours a day, more days in the week, with younger (which in reformspeak means much, much better) teachers and smaller classes (the lower cost per pupil probably reflects fewer special ed and ELL students, but we’ll leave that to the side for now.) Your basic No Excuses model, all the happy tunes that eduformers love to sing.

Ah, here’s the graph I was looking for:

So before I break into my rant, let me say that I was so dumbfounded by these results that I hope I read them incorrectly. I will not be embarrassed if someone points out my error, but relieved. And now, the rant.

That huge increase of 100 points composite in average SAT score translates to an average section score under 500. The Boston Charter students, on average, couldn’t place out of remedial college classes, and wouldn’t even begin to qualify for a competitive university if they weren’t black or Hispanic.

While I often complain about assumptions and conclusions in studies, I have no knowledge of methodology, so I’m going to assume that their study is perfect, and just accept the results.

Ask yourself what those kids got for all those hours of extra time, all those lost Saturdays, all those “Advanced Placement” courses that clearly were nothing more than an exercise in fraud for many students. Improvement on AP outcomes and graduation rates: almost nil. They got a few points bump on the SAT that won’t get them out of remediation, much less exempt them from the “out” of affirmative action. In abilities, they are virtually indistinguishable from the population that tried to get into the charters and failed. What they have, however, are impressive looking transcripts, because charter schools, like majority minority urban schools, are heavily invested in creating the impression of rigor.

That’s how stark, how huge, the achievement gap is, the one that reformers say they can close with school choice. Except, as this study shows, they can’t.

It’s not that the charters did poorly. It’s that the public schools, with far less seat time, no control over their population, and a far smaller percentage of kids who actually want to be educated, did pretty damn well with motivated kids in comparison to the charter schools. They don’t have terribly impressive test scores for the same reason that the charters, with all their extra advantages, don’t have impressive test scores: because black and Hispanic students have low test scores for reasons that have little to do with curriculum, expectations, or teachers.

All you reformers out there, eagerly pumping your fists at the win: The majority of these kids couldn’t break 500 on any section of the SAT. This is what you were fighting for? Or are you going to hold to some measure of consistency in your approach and castigate these charter schools for wasting taxpayer dollars, pretending to hold kids to high standards while failing miserably, fraudulently shoveling kids into AP classes they haven’t a chance of passing, and worst of all, wasting students’ time while not making any meaningful progress in eradicating the achievement gap?

Or maybe, are you going to realize that “failing schools” just have a lot of kids with low test scores because of factors outside everyone’s control?