The false god of elementary school test scores

Rocketship Academy wants to go national. Rocket Academy is a hybrid charter school chain that focuses solely on getting low income Hispanic elementary school students to proficiency. (Note: Larry Cuban has some excellent observations from his visit to a Rocketship Academy.)

First things first: I’ve checked the numbers every way I can think of, and Rocketship’s numbers are solid. They don’t have huge attrition problems that I can see. They are, in fact, getting 60% or higher proficiency in most test categories, and the bulk of their students are Hispanic, many of them not proficient in English. Of course, that brings up an interesting question–if they are proficient on the ELA tests, why aren’t they considered proficient in English? But I digress.

The larger point is this: getting high test scores on California’s elementary school math tests ain’t all that much to get worked up about. Here’s some data from the 2011 California Standards test in math:

I used two standards, because the NCLB obsession with “Proficient and higher” is, to me, moronic. I prefer Basic or higher. The blue line is the percentage of all California in grades 2-9 scoring Basic or higher in General Math, the red is the percentage of same scoring Proficient or higher.

So it gets a bit tricky here, because after 6th grade, the entry to algebra varies. In order to simplify it slightly, I’m ignoring the seventh grade algebra track (call it “accelerated advanced” path), which is about 40,000 students this year, fewer in previous years.

I combined 8th and 9th grade students in General Math and attached that result to the red and blue lines.

Then I separated two groups–the ones who took algebra in 8th grade, and the ones who took algebra later than that. The first group are those who entered algebra in 8th grade, passed it, and continued on the “average advanced” course path, culminating with Calculus senior year. The second group are those who took algebra for the first, second, or third time in high school and then continued on. For each group, I calculated percentages for Basic + and Proficient +.

Notes:

1. Through grade 6, the scores represent all students. In grade 7 and 8/9, general math scores reflect only those students who haven’t moved onto algebra. That’s probably why the proficiency levels drop to 50% and lower for the last two groups.In other words, the green and purple lines represent the advanced track students–most, but not all of the the strong algebra and higher math students. The turquoise and orange lines represent the weaker students taking algebra and higher.
2. Roughly 80% of all students test at Basic or higher from second through sixth grade.
3. Over 70% of the strong studentws test at Basic or higher from algebra through “summative math” (taken for all subjects after Algebra 2).
4. The percentage of students testing proficient from second through sixth grade starts at 65%, rises slightly, and then drops steadily.
5. In no course do more than 50% of the strong students in algebra and higher achieve a score of Proficient or higher.
6. In no course do more than 50% of the weaker students in algebra or higher achieve a score of Basic or higher.

So the chart reveals that all California second through sixth graders, high and low ability, averaged higher scores on their tested subject than the strongest high school students did.

I used 2011 scores, and I may have made a minor error here or there, but the fall off has been in the scores for several years now, and it’s easy enough to check.

What could cause this? Why are California’s elementary school students doing so phenomenally well, and then fall apart when they get to high school? Let’s go through the usual culprits.

California’s high school math teachers suck.–Well, in that case, there’s not much point in demanding higher standards for math teachers, because California’s high school math teachers have had to pass a rigorous content knowledge test for over 20 years. California’s elementary school teachers have to pass a much easier test–which is much harder than anything they had to pass before 2001. In other words, try again.

The teachers aren’t covering the fundamentals! So when the students get to algebra, they aren’t prepared.–But hang on. Elementary school kids, the ones being taught the fundamentals, are getting good test scores. What evidence do you have that they aren’t being taught properly?

Well, they’re only getting good test scores because the tests are too easy!—dingdingding! This is a distinct possibility. Perhaps the elementary tests aren’t challenging enough. Having looked at the tests, I’m a big believer in this one. I think California’s elementary math tests, through seventh grade, are far less challenging to the tested elementary school population than are the general math and specific subject tests are to the older kids. (On the other hand, the NAEP scores show this same dropoff.)

However, while that might explain the disparity between the slower track math student achievement and elementary school, it doesn’t adequately address why the students in the “average advanced” track aren’t achieving more than 50% proficiency, does it?

Trigonometry is harder than memorizing math facts–We should take to heart the Wise Words of Barbie. Math achievement will fall off as the courses get more challenging. Students who excelled at their times tables and easily grasped fractions might still struggle with complex numbers or combinatorics.

So if you ask me—and no one does. Hell, no one has even really noticed the fall-off—it’s a combination of test design and subject difficulty.

Whatever the reason, the test score falloff has enormous implications for those who are banking on Rocketship Academy, KIPP, and all those other “proven” charters that focus exclusively on elementary school children.

Elementary school test scores are false gods. We have no evidence that kids who had to work longer school days simply to achieve proficiency in fifth grade reading and math will be, er, “shovel ready” for algebra and Hamlet. KIPP’s College Completion Report made no mention of its college students SAT scores, or indeed made any mention of demonstrated ability (e.g., AP tests), and color me a cynic, but I’m thinking they’d have mentioned both if the numbers were anything other than dismal.

So let’s assume that those Rocketship scores are solid (and I do). So what? How will they do in high school? Where’s the follow through? Everyone is banking on the belief that we can “catch them early”. Get kids competent and engaged while they are young, and it all falls into place.

Fine. Just let me know when the test scores back up that lovely vision.

Added in January 2014: Well, hey now. Growing Pains for Rocketship’s Blended-Learning Juggernaut.

Alas, it seems that Rocketship’s scores are declining, their model doesn’t scale, they are making decisions based on cost rather than learning outcomes and, my FAVORITE part:

Lynn Liao, Rocketship’s chief programs officer, said the organization has also received troubling feedback on how students educated under the original blended learning model fare in middle school.

“Anecdotal reports were coming in that our students were strongly proficient, knew the basics, and they were good rule-followers,” Ms. Liao said. “But getting more independence and discretion over time, they struggled with that a lot more.”

That graven image gets you every time, doesn’t it?