Boston Charter School Study: What “Improved Scores” Look Like

Slate article Do Charter Schools Work? is a good read; I recommend it. But what I wanted to do here is point out some interesting facts in the featured study: Charter Schools and the Road to College Readiness, which is incredibly important in that it’s one of the few studies that focus on high school outcomes.

Relevant quotes:

While the students who were randomly offered a seat at these high schools graduate at about the same rate as those not offered a seat, lottery estimates show that charter enrollment produces gains on Advanced Placement (AP) tests and the SAT. Charter attendance roughly doubles the likelihood that a student sits for an AP exam and increases the share of students who pass AP Calculus. Charter attendance does not increase the likelihood of taking the SAT, but it does boost scores, especially in math. Charter school attendance also increases the pass rate on the exam required for high school graduation in Massachusetts, with especially large effects on the likelihood of qualifying for a state-sponsored college scholarship.

Estimates of the effect of charter attendance on AP scores by subject… show a large increase in the likelihood that charter applicants take tests in science, calculus, and history, three of the most commonly taken AP exams. Paralleling charter schools’ large impact on MCAS math scores, the clearest AP score gains are for calculus. Charter attendance boosts the probability of taking the AP calculus test by 21 percentage points, and appears to boost the likelihood of earning a score of at least 2 by nearly 9 points. The corresponding impact on the likelihood of earning a 3 on AP calculus is 7 percentage points, though the estimated increases in the likelihood of scoring 2 or 3 are only marginally statistically significant. Charter attendance increases test-taking in science and US history, with no corresponding impact on scores in these subjects. Charter schools have little effect on English test-taking or scores.

Although charter attendance has little effect on the rate at which applicants take the SAT, charter attendance raises the SAT scores that applicants earn on the test. In particular, coding scores as zero for non-takers, charter attendances pushes the SAT composite score (the sum of math, verbal, and writing scores) above the bottom quartile of the state composite score distribution by 11 percentage points. Gains in math contribute most to the shift in composite scores; effects on verbal and writing scores are smaller (the estimated low-end shift in verbal scores is marginally significant). Charter attendance also raises the probability that applicants earn an SAT reasoning score (the sum of math and verbal) above the state median by 13 percentage points, with math again the largest contributor to this gain.

Although charter attendance has smaller effects on verbal and writing scores, the composite SAT score gain is estimated to be a little over 100 points, a large and statistically significant result. The gain here amounts to almost one-third of a standard deviation in the US composite score distribution. The corresponding effect on the SAT reasoning score is 74 points, also a large gain.

I’m torn as to the vantage point from which to consider the SAT score increase. The “average” increase provided by test prep is between 5-10 points. But everyone knows, or should know, that if “average” were “expected”, then test prep wouldn’t have kept Washington Post Company solvent for most of the last 20 years. Most parents would be unhappy with the results (while understanding the law of averages) if their average students only improved 100 points composite, which is an average of 30 points per section. Another way of looking at this increase is to argue that 30 points per section is three times the average achieved by test prep companies. Whoohoo! Except, of course, charters had three years. Test prep companies have about 15 hours.

Of course, there’s this annoying repetition of the test prep canard:

Designed to be challenging for all students, low SAT scores are a special concern for poor and minority students. Gaps in SAT scores by race and socioeconomic status that might be attributable to family background and school quality are further accentuated by the willingness of higher income families to invest heavily in SAT preparation. (see, e.g., Bowen and Bok, 2002)

How hard is it to stop repeating utter pap? Blacks and Hispanics are more likely to use test prep than whites, although East Asians trump everyone. And here’s a table to prove it (table courtesy of Inside Higher Ed, as mentioned in the linked essay):

And until someone can explain the socioeconomic reality of poor whites outscoring high income blacks on the SAT, best not go to that well, either.

Ahem. Back to the study.

Demographics of Boston Public Schools vs. charter students:

So if I’m reading this right, the charter population is way more female, way more black, way less Hispanic and a third as Asian as the lottery losers. The drop in Asians puts the math score increase in perspective, although the study makes clear that the charter group applicants had higher scores than average. (The study ignores any detailed discussion of race.)

And what did they do, these charter schools, to teach their kids? bostonchartermethods
Taught them for more hours a day, more days in the week, with younger (which in reformspeak means much, much better) teachers and smaller classes (the lower cost per pupil probably reflects fewer special ed and ELL students, but we’ll leave that to the side for now.) Your basic No Excuses model, all the happy tunes that eduformers love to sing.

Ah, here’s the graph I was looking for:


So before I break into my rant, let me say that I was so dumbfounded by these results that I hope I read them incorrectly. I will not be embarrassed if someone points out my error, but relieved. And now, the rant.

That huge increase of 100 points composite in average SAT score translates to an average section score under 500. The Boston Charter students, on average, couldn’t place out of remedial college classes, and wouldn’t even begin to qualify for a competitive university if they weren’t black or Hispanic.

While I often complain about assumptions and conclusions in studies, I have no knowledge of methodology, so I’m going to assume that their study is perfect, and just accept the results.

Ask yourself what those kids got for all those hours of extra time, all those lost Saturdays, all those “Advanced Placement” courses that clearly were nothing more than an exercise in fraud for many students. Improvement on AP outcomes and graduation rates: almost nil. They got a few points bump on the SAT that won’t get them out of remediation, much less exempt them from the “out” of affirmative action. In abilities, they are virtually indistinguishable from the population that tried to get into the charters and failed. What they have, however, are impressive looking transcripts, because charter schools, like majority minority urban schools, are heavily invested in creating the impression of rigor.

That’s how stark, how huge, the achievement gap is, the one that reformers say they can close with school choice. Except, as this study shows, they can’t.

It’s not that the charters did poorly. It’s that the public schools, with far less seat time, no control over their population, and a far smaller percentage of kids who actually want to be educated, did pretty damn well with motivated kids in comparison to the charter schools. They don’t have terribly impressive test scores for the same reason that the charters, with all their extra advantages, don’t have impressive test scores: because black and Hispanic students have low test scores for reasons that have little to do with curriculum, expectations, or teachers.

All you reformers out there, eagerly pumping your fists at the win: The majority of these kids couldn’t break 500 on any section of the SAT. This is what you were fighting for? Or are you going to hold to some measure of consistency in your approach and castigate these charter schools for wasting taxpayer dollars, pretending to hold kids to high standards while failing miserably, fraudulently shoveling kids into AP classes they haven’t a chance of passing, and worst of all, wasting students’ time while not making any meaningful progress in eradicating the achievement gap?

Or maybe, are you going to realize that “failing schools” just have a lot of kids with low test scores because of factors outside everyone’s control?

About educationrealist

19 responses to “Boston Charter School Study: What “Improved Scores” Look Like

  • mindweapon

    They’ll keep doing what they do, as long as the Paychex flow.

    There’s a rumor that race realism is going to be “allowed” within the next 1 to 2 years. There was a New York Times article comparing a white family (mom, dad, kids, high investment) with a white woman who had three mixed race kids with a black guy from college who did not stick around, and how hard her life was and deprived her kids were. the white heritage family was taking their white children to swimming lessons and getting them tutoring; the single mom with the three mixed kids coudln’t take the kids to anything.

    The pretense is that the NYT reporter is saying, “look at htis poor single mom. Single moms need MORE of your money, so they can take their kids to swimming lessons like white heritage families!”

    But we read into it, and we suspect a trial balloon. We think the NYT was saying, “look this race mixing thing doesn’t work out so well after all.”

    So we think there are plans in the works to suddenly make a degree of race realism, and therefore white identity, “allowed.” Legalized, ya might say.

    The reason for this is that they want to stop pouring all this money into educational investment of NAM’s, and they realize that they need white competence to continue civilization.

    They can’t afford to continue the civil rights industry, so they want to unwind it. how to do that? Acknowledge that you can test kids at a young age and decide then and there if educational investment is worth it.

    Does it seem crazy or unlikely that they will suddenly allow us to acknowledge racial reality? In the modern age, things change very fast and very radically, and the anti-white race racket is something that is very ripe for take down. Welcome to the Dystopia, where it’s rarely boring.

    • MoscowEast

      It will be the single most astonishing thing I have ever seen in my lifetime if this comes true. Leftists will never admit to being wrong, as intellectually they are vastly superior to everybody else, of course. But what is absolutely obvious to the non-blinkered is now so solidly supported by data that it must be getting harder and harder for them to pretend that all problems are the fault of evil white men, that race is just a construct, that there’s no such thing as intelligence, blah blah blah. Nevertheless, the Left seem to me to be ‘doubling down’ on their denial and I see no signs suggesting imminent change. When it comes to their own children they have no qualms about acting as if they understand reality and they do so ruthlessly. I don’t know whether the hypocrisy of their position is becoming harder for them to sustain socially. Perhaps it is, and that might afford them sufficient incentive to engage in a little race realism. That said, they really love trampling their enemies into the ground with their holier-than-thou rhetoric on race and I just can’t imagine them giving any succour to the enemy.

      • mindweapon

        Yeah, it does sound far-fetched. But one thing we know is that history moves fast nowadays, and we also know that multiculturalism is untenable. We’ll see.

  • Jim

    I’ve no doubt that there are people on the left who are arguing in private for a more realistic approach. But there are also many who are highly fanatical. It is difficult to predict the outcome of these internal struggles. In the mid-twenties many of the ruling Bolsheviks in the Soviet Union had been chastened by the economic failures that had led in the NEP to a partial retreat from socialism. People at high levels including Molotov among others cautioned against collectivizing agriculture and pushing full speed ahead with socialism. They argued for a slower, more pragmatic approach. At first those advocating these more moderate positions seemed to be winning. But then in 1928 under Stalin policy turned sharply to the fanatical left. Agriculture was collectivized at a horrific human and economic cost. The whole system became totally insane.

    So don’t get your hopes up too much.

  • Tom

    When I look at the graphs of SAT scores, the charter school graphs still seem shifted significantly to the right. I realize that the average composite score remains below 500 per section (though I wouldn’t have noticed it if you hadn’t pointed it out), but isn’t the study attempting to measure relative performance? And since the students were randomly chosen, doesn’t that say that the school can take credit for the rightward shift of student performance? I realize that the students spent more days and longer days in class, but there does appear to have been a positive effect. Am I missing something here?

  • ed

    Yeah, I don’t get the glass-half-empty thinking here. Stipulate that most of “the gap” is beyond the ability of schools to fix; given that, these seem like pretty worthwhile results.

    I mean, what’s the downside to charter schools? As far as I can tell, they cost the same or less, they make parents just as happy or happier, and they allow for experiments of different methods that may yield better practices in the long run. They might even put pressure on non-charter public schools to improve over time.

    I mean, if we can shift the distribution of SAT rightward by 30-40 points per section with little or no downside, what are we waiting for?

    • educationrealist

      You are saying that the lost time of kids isn’t a downside? If all it takes is more time in school and bumping unmotivated kids, we can do that at public schools, at scale, for cheaper. Assuming that the kids are fine losing all that time.

      There’s a significant downside to charters, particularly if they continue to grow.

      • ed

        So the main downside you see is that kids have less leisure time? (Not trying to be flippant, just trying to understand your position.)

        According to the Slate article, charters spend less than public schools. So how are we going to increase instructional time in regular public schools for cheaper?

    • educationrealist

      “So the main downside you see is that kids have less leisure time? (Not trying to be flippant, just trying to understand your position.)”

      No. The main downside I see is that the entire premise behind charter schools is based on the failure of public schools. But if all charter schools can do, in three years, is get a mild bump in SAT scores–something that can be achieved with a test prep course—then either charter schools are also a failure, or public schools are doing as well as charters.

      “According to the Slate article, charters spend less than public schools. So how are we going to increase instructional time in regular public schools for cheaper?”

      Test prep courses are cheap, and they haven’t achieved much a test prep course couldn’t. And yes, the cost of increasing public school capacity from X to X + Y is cheaper than costs of a charter school.

      However, I think the larger issue is this: your questions show you really don’t understand why this mild improvement is a problem, given the hard sell. So all th picayune questions aren’t really relevant. The first question, in your first comment, is really just a neon sign saying you’re either a charter advocate or largely clueless. That’s fine, but try to avoid drawing any conclusions about what you think I think. You’re almost certainly wrong.

      • ed

        Wow. I’m genuinely wondering what I said to provoke such a rude, dismissive response. I think I’m mostly on your side and I generally like your blog.

        I could respond to some of your points, but I guess it’s hard to have a productive discussion online, so I won’t bother.

      • wildiris

        Ed, I think I know where he is coming from. Both of my kids go to a charter school. The only reason we do this is because the regular public schools here are so bad(*).

        If the public school system was as good as it was when I was young, then there would be no need for charter school alternatives. In fact, my public-school K-12 education (back in the 50’s and 60’s) was as good as some of the high-priced college-prep schools are nowadays.

        In other words, charter schools only look better in comparison to the mess the public school system is in. But in reality, charter schools don’t do any better than a well run regular public school would anyway.

        (*)We live in a farming community that is 70-80% Hispanic. Because of so many ESL learners, the schools spend an inordinate amount of time in bi-lingual mode . We also have a significant Oaxacan population here that in many cases are not even literate in Spanish to start with. Then there is the Nortenos and Surenos street gang presence on the high school campuses.

      • educationrealist

        ” But in reality, charter schools don’t do any better than a well run regular public school would anyway.”

        Exactly. So change public schools. Don’t pretend charter schools are better.

  • Bill

    There are pretty big effects in the tails, though. Looking at the math score, ask what %age of the kids are above 600. I’m doing the integrals with my eyeballs, so there is error, but it looks to me like ~14% in charter and ~6% in traditional are above 600. 600 math is a decent score for a college-bound student still, right? The absolute numbers for reading are something like 4% above 600 for charter and 2% above 600 for traditional.

    Charter something like doubles the number of college-ready students. Of course, you are torturing the other 95% of the charter students in order to deliver benefits to the top 5%. Perhaps there is some way to identify freshmen who are in the top 5% and only let them into charters, since charters seem like a good deal for them. Maybe use IQ tests plus grades or teacher evaluations. Actually, if you believe in peer effects, you would probably get even better performance if you let the charters be more selective, so maybe you should let in the top 10%.

    • wildiris

      Bill, another thing that should be noted about this study was that it was not based on a truly random sample to begin with. According to the study, “By comparing randomly-admitted lottery winners and losers,…” If this was their sample space then there was already a self-selection process going on before a student even became part of the study. The bump in the high end that you see is most likely due to the fact that higher achieving students would, on average, be the ones most likely to apply for the chance to go to a charter school in the first place. And these kids are the same small cohort that would also most benefit from having a less disruptive and more supportive learning environment than the one to be found at the mainstream school they came from.

  • Soddy

    Charters are more orderly than conventional public schools. Behavior tolerated in conventional public schools is generally not tolerated in charters. That improves the experience of education. “Race realism” is nothing more than excuses and whining by old white men of lower intelligence levels. You need order and discipline for successful education

  • Eugene

    Bill, We can’t skim. It would be a form of tracking. Though tracking is occurring any way in any high school that has honors and AP courses. It is leveling without say “We are tracking.” I think there is a lot to be said for tracking as long as there is mobility between tracks (might take some kids longer to get through school, but there would be a lot less teaching to the middle). The public school system is warped in that way. They (well me – because I teach in public school) are very worried about the bottom. Bring the bottom students up to the middle. There are a couple of problems with that thinking. The greatest of which is mathematical (which makes very clear how math stupid (or math blind) ed reformers, administration, and politicians are), if everybody improves the bottom kids will still be at the bottom and that does not necessarily mean they will know more. You can’t bring kids up to the middle, because then it won’t be the middle any more. the gap will still exist. The second is that very few teachers teach just upper level or just lower level kids; it tends to be mixed to promote equity, therefore it has been my observation that teachers (not all but enough to matter) struggle so much in the weak classes (behavior, increased planning for more differentiation) that they are not quite as rigorous as they should be for the upper kids because, “oh, they will be fine, because they are motivated.”

    These are purely observations, I have no data. I no good at finding data on the internet since most of it is either raw (my stat skills are rusty and time is fleeting) or manipulated that it is hard to tell what the truth is.

    Ed, on a totally separate note. Could the achievement gap between black and white be explained by breeding? Or why do African Americans tend to be more athletic? Could two hundred years of slave trading and picking the strongest and healthiest (intelligence was not an important trait to slave owners) effected the genetic pool so much that it continues to hamper the current generation?

  • Sia

    Oh, for… how about improved learning?

  • Charters: The Center Won’t Hold | educationrealist

    […] charters actually address, while bragging about improving test scores, which they don’t (in any meaningful […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: