Education: No Iron Triangle

I came from the corporate world, which invented the project management triangle. (“Fast, Good, Cheap: Pick Two.”)

Education has no triangle.

Money, of course, doesn’t work. Just ask Kansas City. Or Roland Fryer, who learned that kids would read more books for money but couldn’t seem to produce higher test scores for cash. Increased teacher salaries, merit pay, reduced class size are all suggestions that either don’t have any impact or have a limited impact….sometimes. Maybe. But not in any linear, scalable pattern.

“Good”? Don’t make me laugh. We don’t have a consensus on what it means. Most education reformers use the word “quality” exclusively to mean higher test scores. Teachers do not. Nor do parents, as Rahm Emanuel, Cami Anderson, Adrian Fenty and Michelle Rhee have learned. Common Core supporters have had similar moments of revelation.

So until we agree on what “good” is, what a “high quality education” means, we can’t even pretend that quality is a vertex of education’s triangle, even if it existed. We could save a whole lot of wasted dollars if people could just grasp that fact.

Time is an odd one. We never use the word directly, but clearly, politicians, many parents, and education reformers of all stripes believe we can educate “faster”. Until sixty years ago, calculus was an upper level college course. Once the high school movement began, fewer than 3% of students nationwide took trigonometry, between 10-20% took geometry, and the high point for algebra was 57%–over one hundred years ago–then declining to 25%. (Cite.) One of the little noted achievements of the New Math movement was to alter the math curriculum and make high school calculus a possibility. At first, just kids with interest and ability took that path. Then someone noticed that success in algebra I predicted college readiness and everyone got all cargo cult about it. By the turn of the century, if not earlier, more of our kids were taking advanced math in high school than at any point in our history.

And that was before kids started taking algebra in seventh grade. Sophomores take now take honors pre-calculus so they can get a second year of AP calculus in before graduation. Common Core has gone further and pushed algebra 2 down into algebra I.

Yet 17 year old NAEP scores have been basically stagnant for the same amount of time our high school students have been first encouraged, then required, to take three or more years of advanced math.

Not only do we try to educate kids faster, we measure their gain or loss by time. Poor kids of uneducated parents lose two months learning over the summer. CREDO, source of all those charter studies, refers to additional days of learning. Everyone comparing our results to Singapore always mentions the calendar, how much earlier their kids start working with advanced math. These same people also point out that Singapore has a longer school year. Longer school years don’t appear to work reliably either.

Except maybe KIPP, whose success is mostly likely due to extended school hours. KIPP focuses on middle school and has not really been scrutinized at the high school level. Scrutiny would reveal that the program doesn’t turn out stellar candidates, and while more KIPP alumni complete college than the average low income black or Hispanic student, the numbers are reasonable but not extraordinary when compared against motivated students in the same category who attended traditional schools. Particularly given the additional support and instruction hours the KIPP kids get.

So KIPP’s “success” actually adds weight to the NAEP scores as evidence that time–like money and quality–doesn’t respond to the project management constraints.

Kids learn what they have the capacity to learn. Spending more instruction hours will–well, may–help kids learn more of what they are capable of learning in fewer school years. But the NAEP scores and all sorts of other evidence says that learning more early doesn’t lead to increased capacity later. And so, we’ve moved 1979 first grader readiness rules to preschool with considerable success, but that success hasn’t given us any traction in increasing college readiness at the other end of childhood. Quite the contrary.

I probably don’t have much of a point. I was actually thinking about the increasing graduation rates. It’ll be a while until part 2. I’m swamped at work, moving again, writing some longer pieces, and really would like to post some math curriculum rather than detangle my mullings.

But the triangle thing is important. Really.

Take note: under 1000 words. Hey, I have to do it every year or so.

NAEP TUDA Scores—Detroit isn’t Boston

So everyone is a-twitter over NAEP TUDA (Trial Urban District Assessment) scores. For those who aren’t familiar with The Nation’s Report Card, the “gold standard” of academic achievement metrics, it samples performance rather than test every student. For most of its history, NAEP only provided data at the state level. But some number of years ago, NAEP began sampling at the district level, first by invitation and then accepting some volunteers.

I don’t know that anyone has ever stated this directly, but the cities selected suggest that NAEP and its owners are awfully interested in better tracking “urban” achievement, and by “urban” I mean black or Hispanic.

I’m not a big fan of NAEP but everyone else is, so I try to read up, which is how I came across Andy Smarick‘s condemnation of Detroit, Milwaukee, and Cleveland: “we should all hang our heads in shame if we don’t dramatically intervene in these districts.”

Yeah, yeah. But I was pleased that Smarick presented total black proficiency, rather than overall proficiency levels. Alas, my takeaway was all wrong: where Smarick saw grounds for a federal takeover, I was largely encouraged. Once you control for race, Detroit looks a lot better. Bad, sure, but only a seventh as bad as Boston.

So I tweeted this to Andy Smarick, but told him that he couldn’t really wring his hands until he sorted for race AND poverty.

He responded “you’re wrong. I sorted by race and Detroit still looks appalling.”

He just scooted right by the second attribute, didn’t he?

Once I’d pointed this out, I got curious about the impact that poverty had on black test scores. Ironic, really, given my never-ending emphasis on low ability, as opposed to low income. But hey, I never said low income doesn’t matter, particularly when evaluating an economically diverse group.

But I began to wonder: how much does poverty matter, once you control for race? For that matter, how do you find the poverty levels for a school district?

Well, it’s been a while since I did data. I like other people to do it and then pick holes. But I was curious, and so went off and did data.

Seventeen days later, I emerged, blinking, with an answer to the second question, at least.

It’s hard to know how to describe what I did during those days, much less put it into an essay. I don’t want to attempt any sophisticated analysis—I’m not a social scientist, and I’m not trying to establish anything certain about the impact of poverty on test scores, an area that’s been studied by people with far better grades than I ever managed. But at the same time, I don’t think most of the educational policy folk dig down into poverty or race statistics at the district level. So it seemed like it might be worthwhile to describe what I did, and what the data looks like. If nothing else, the layperson might not know what’s involved.

If my experience is any guide, it’s hard finding poverty rates for children by race. You can get children in poverty, race in poverty, but not children by race in poverty. And then it appears to be impossible to find enrolled children in a school district—not just who live in it, which is tough enough—by poverty. And then, of course, poverty by enrollment by race.

First, I looked up the poverty data here (can’t provide direct links to each city).

But this is overall poverty by race, not child poverty by race, and it’s not at the district level, which is particularly important for some of the county data. However, I’m grateful to that site because it led me to American Community Survey Factfinder, which organizes data by all kinds of geographic entities—including school districts—and all kinds of topics–including poverty—on all sorts of groups and individuals—including race. Not that this is news to data geeks, which I am not, so I had to wander around for a while before I stumbled on it.

Anyway. I ran report 1701 for the districts in question. If I understand googledocs, you can save yourself the trouble of running it yourself. But since the report is hard to read, I’ll translate. Here are the overall district black poverty rates for the NAEP testing regions:

Again, these are for the districts, not the cities.

(Am I the only one who’s surprised at how relatively low the poverty rates are for New York and DC? Call me naïve for not realizing that the Post and the Times are provincial papers. Here I thought they focused on their local schools because of their inordinately high poverty rates, not their convenient locations. Kidding. Kind of.)

But these rates are for all blacks in the district, not black children. Happily, the ACS also provides data on poverty by age and race, although you have to add and divide in order to get a rate. But I did that so you don’t have to–although lord knows, my attention to detail isn’t great so it should probably be double or triple checked. So here, for each district, are the poverty rates for black children from 5-17:

In both cases, Boston and New York have poverty rates a little over half those of the cities with the highest poverty rates—and isn’t it coincidental that the four cities with the lowest black NAEP scores have the highest black poverty rates? Weird how that works.

But the NAEP scores and the district data don’t include charter or private schools in the zone, and this impacts enrollment rates differently. So back to ACS to find data on age and gender, and more combining and calculating, with the same caveats about my lamentable attention to detail. This gave me the total number of school age kids in the district. Then I had to find the actual district enrollment data, most of which is in another census report (relevant page here) for the largest school districts. The smaller districts, I just went to the website.

Results:

Another caveat–some of these data points are from different years so again, some fuzziness. All within the last three or four years, though.

So this leads into another interesting question: the districts don’t report poverty anywhere I can find (although I think some of them have the data as part of their Title I metrics) and in any event, they never report it by race. I have the number and percent of poor black children in the region, but how many of them attend district schools?

So to take Cleveland, for example, the total 5-17 district population was 67,284. But the enrolled population was 40871, or 60.7% of the district population.

According to ACS, 22,445 poor black children age 5-17 live in the district, and I want an approximation of the black and overall poverty rates for the district schools. How do I apportion poverty? I do not know the actual poverty rate for the district’s black kids. I saw three possibilities:

1. I could use the black child poverty rate for the residents of the Cleveland district (ACS ratio of poor black children to ACS total black children). That would assume (I think) that the poor black children were evenly distributed over district and non-district schools.
2. I could have take the enrollment rate and multiplied that by the poor black children in ACS—and then use that to calculate the percentage of poor kids from blacks enrolled.
3. I could assign all the black children in poverty (according to ACS) to the black children enrolled in the district (using district given percentage of black children enrolled).

Well, the middle method is way too complicated and hurts my head. Plus, it didn’t really seem all that different from the first method; both assume poor black kids would be just as likely to attend a charter or private school than they would their local district school. The third method assumes the opposite—that kids in poverty would never attend private or charter schools. This method would probably overstate the poverty rates.

So here are poverty levels calculated by methods 1 and 3–ACS vs assigning all the poor black students to the district. In most cases, the differences were minor. I highlight the districts that have greater than 10 percentage points difference.

Again, is it just a coincidence that the schools with the lowest enrollment rates and the widest range of potential poverty rates have some of the lowest NAEP scores?

Finally, after all this massaging, I had some data to run regression analysis on. But I want to do that in a later post. Here, I want to focus on the fact that gathering this data was ridiculously complicated and required a fair amount of manual entry and calculations.

If I didn’t take the long way round, I suspect this effort is why researchers use the National Student Lunch Program (“free and reduced lunch”) as a poverty proxy.

The problem is that the poverty proxy sucks, and we need to stop using it.

Schools and districts have noticed that researchers use National School Lunch enrollment numbers as a proxy for poverty, and it’s also a primary criterion for Title I allocations. So it’s hard not to wonder about Boston’s motives when the district decides to give all kids free lunches regardless of income level, and whether it’s really about “awkward socio-economic divides” and “invasive questions”. The higher the average income of a district’s “poor” kids, the easier it is to game the NCLB requirements, for example.

Others use the poverty proxy to compare academic outcomes and argue for their preferred policy, particularly on the reform side of things. For example, charter school research uses the proxy when “proving” they do a “great job educating poor kids” when in fact they might just be skimming the not-quite-as-poor kids and patting themselves on the back. We can’t really tell. And of course, the NAEP uses the poverty proxy as well, and then everyone uses it to compare the performance of “poor” kids. See for example, this analysis by Jill Barshlay, highlighted by Alexander Russo (with Paul Bruno chiming in to object to FRL as poverty proxy). Bruce Baker does a lot of work with this.

To see exactly how untrustworthy the “poverty proxy is”, consider the NAEP TUDA results broken down by participation in the NSLP.

Look at all the cities that have no scores for blacks who aren’t eligible for free or reduced lunch: Boston, Cleveland, Dallas, Fresno, Hillsborough County, Los Angeles, Philadelphia, and San Diego. These cities apparently have no blacks with income levels higher than 180% of poverty. Detroit can drum up non-poor blacks, but Hillsborough County, Boston, Dallas, and Philadelphia can’t? That seems highly unlikely, given the poverty levels outlined above. Far more likely that the near-universal poverty proxy includes a whole bunch of kids who aren’t actually poor.

In any event, the feds, after giving free lunches to everyone, decided that NSLP participation levels are pretty meaningless for deciding income levels “…because many schools now automatically enroll everyone”.

I find this news slightly cheering, as it suggests that I’m not the only one having a hard time identifying the actually poor. Surely this article would have mentioned any easier source?

So. If someone can come back and say “Ed, you moron. This is all in a table, which I will now conveniently link in to show you how thoroughly you wasted seventeen days”, I will feel silly, but less cynical about education policy wonks hyping their notions. Maybe they do know more than I do. But it’s at least pretty likely that no one is looking at actual district poverty rates by race when fulminating about academic achievement, because what I did wasn’t easy.

Andy Smarick, at any rate, wasn’t paying any attention to poverty rates. And he should be. Because Detroit isn’t Boston.

This post is long enough, so I’ll save my actual analysis data for a later post. Not too much later, I hope, since I put a whole bunch of work into it.

Algebra and the Pointlessness of The Whole Damn Thing

The whole algebra debate kicked off by Hacker’s algebra essay has…..well, if not depressed me, then at least enervated me.

A recap:

Hacker:

We shouldn’t make everyone take algebra. No one needs algebra anyway; we never really use it. Statistics would be much more useful. Algebra is the primary obstacle to high school success; millions of kids are failing because they can’t manage this course. If we just allowed students to have an easier time in high school, more of them would graduate successfully and go on to college.

Outraged Opposition:

Algebra is essential to college success and “real life” and one of many obstacles to high school success. No one is happy with the current state of affairs, but it’s clear that kids aren’t learning algebra because their teachers suck, particularly in elementary school. We need to teach math better in the lower grades, rather than lower our standards. Besides, the corollary to “not everyone should take algebra” is “some people should take algebra” and just how are you planning to divide up those teams? (Examples: Dan Willingham, Dropout Nation)

Judicious Analysis:

Sigh. Guys, this is really a debate about tracking, you know? And no one wants to go there. While it’s true that algebra really isn’t necessary for college, colleges use success in advanced math as a convenient sorting mechanism. Besides, once we say algebra isn’t necessary, where do we stop? Literature? Biology? Chemistry? But without doubt, Hacker is right in part. Did I say that no one wants to go there? Or just hint it really, really loudly?
Examples: Dana Goldstein, Justin Baeder Iand II.

Voldemort Support:

Well, of course not everyone should take algebra, trig, or calculus. Or advanced literature. Or science. Not everyone has the cognitive ability or the interest. We should have a richer and more flexible curriculum, allowing anyone with the interest to take whatever classes they like with the understanding that not all choices lead to college and that outcomes probably won’t have the racial distributions we’d all prefer to see. Oh, and while we’re at it, we should be reviewing our immigration policies because it’s pretty clear that our country doesn’t need cheap labor right now.

Hacker, Outraged Opposition and Judicious Analysis to Voldemort Support:

SHUT UP, RACIST!

So really, what else is left to say? The Judicious Analysis essays I linked above were the strongest by far, particularly Justin Baeder II.

Instead, I’m going to revisit a chart I updated from the last time I posted it:

These are California’s math scores by grade and subject, the percentage scoring basic/proficient or higher on the CST. Algebra entry points differ, so the two higher (and slightly longer) of the four short lines are the percentages of “advanced” students with those scores—those who took algebra in 7th or 8th grade. The lower, shortest lines represent the scores of students who began algebra in 9th grade.

Notice that advanced students don’t match the performance of the entire elementary school population through 5th grade. Notice, too, that the percentage of advanced students scoring proficient or higher is just around half of the population. When I just considered algebra students who began in 8th grade (see link above), the percentage never tops 50. Notice that around 40% of the kids who started algebra in 9th grade achieved basic or higher.

NAEP scores show the same thing—4th grade math scores have risen, while 12th grade scores stay flat. In fact, Daniel Willingham, who declares above that we’re doing a bad job at teaching elementary math, was considerably more sanguine about teacher quality back in December, citing the improved elementary school math performance shown in the NAEP. So the strong elementary school performance, coupled with a huge dropoff in advanced math, is not unique to California.

These numbers, on the surface, don’t support the conventional wisdom about math performance: namely, that elementary school teachers need improvement and that the seeds of our students’ failure in higher math starts in the lower grades. Elementary students are doing quite well. It’s only in advanced math, when the teachers are much more knowledgeable, with higher SAT scores and tougher credentialling tests, that student performance starts to decline dramatically.

What these numbers do suggest is that as math gets harder, fewer and fewer students achieve mastery, or anything near it. . What they suggest, really, is that math knowledge doesn’t advance in a linear fashion. Shocking news, I know. We have all forgotten the Great Wisdom of Barbie.

Break it down by race and the percentages vary, but not the pattern. I skipped Asians, because California tracks Asians by subcategory, and life’s too short. I’m going to go right out on a limb and predict that Asians did a bit better than whites.

(Note: I know it’s weird that in all cases, 9th graders in general math have nearly the same percentages as 9th graders in algebra, but it’s easily confirmed: whites, blacks, Hispanics).

Whites in the standard math track perform as well as advanced math blacks and just a bit worse than advanced track Hispanics. Sixty to seventy percent of blacks and Hispanics on the standard track fail to achieve a “basic” score.

Some people are wondering how poverty affects these results, I’m sure. Let’s check.

Hey! Look at that! The achievement gap disappears!

Just kidding. This chart shows the results of blacks and Hispanics who are NOT economically disadvantaged and whites who ARE economically disadvantaged. You can see it on the legend.

So that’s how to make the achievement gap disappear: compare low income whites to middle class or higher blacks and Hispanics and hey, presto.

And that’s all the charts for today. I’m not detail-oriented, and massaged this all in Excel. You can do your own noodling here. Let me know if I made any major errors. The 2012 results should be out in a couple weeks.

Anyway. With numbers like these, it’s hard not to just see this entire debate as insanely pointless. In California, at least, tens of thousands of high school kids are sitting in math classes that they don’t understand, feeling useless, understanding deep in their bones that education has nothing to offer them. Meanwhile, well-meaning people who have never spent an hour of their lives trying to explain advanced math concepts to the lower to middle section of the cognitive scale pontificate about teacher ability, statistics vs. algebra, college for everyone, and other useless fantasies that they are allowed to engage in because until our low performers represent the wide diversity of our country to perfection, no one’s going to ruin a career by pointing out that this a pipe dream. And of course, while they’re engaging in these fantasies, they’ll blame teachers, or poverty, or curriculum, or parents, or the kids, for the fact that their dreams aren’t reality.

If we could just get whites and Asians to do a lot worse, no one would argue about the absurdity of sending everyone to college.

Until then, everyone will divert themselves by engaging in this debate—which, like many kids stuck in the hell of unfair expectations, will go nowhere.

Fundamentally Flawed

Mike Petrilli is puzzled by NAEP scores:

One of the great mysteries of modern-day school reform is why we’re seeing such strong progress (in math at least, especially among our lowest-performing students) at the elementary and middle school levels, but not in high school.

How is this not completely predictable?

Elementary school education focuses on fundamental skills and knowledge, increasing difficulty each year in a linear fashion. Each year, the students add to their existing base of knowledge. Start with adding and subtracting single digit integers, move to adding and subtracting two and three digit numbers. Eventually move into multiplication, then into division. With division comes the notion of non-integers, and so onto understanding fractions and decimals, then operations with fractions and decimals. Likewise, geometry starts with understanding shapes, then moving onto perimeter and area and then volume. Reading begins with decoding, and imperfectly expands vocabulary, voice complexity (irony, unreliable narrators, etc), while ideally adding content knowledge.

This is why elementary education is always assessed at “grade level”, because the curriculum assigns a particular amount of knowledge and achievement to be demonstrated at each grade. Principals looking to improve outcomes talk obsessively about how many students they can get to “grade level”, or better still, “above grade level”. Even the great content guru E. D. Hirsch talks about “what your Nth grader should know”–but only through 6th grade.

The rate at which students move through the “grade level” varies. Some students can effortlessly score at or above “grade level” without doing a bit of homework and watching 8 hours of TV a day. Other students will rarely achieve grade level without dramatic changes in instruction method and hours in school. Still others will never achieve “grade level”. These students are not distributed proportionately by either race or income, the two categories we monitor progress by because we don’t want to monitor by cognitive ability.

Because the rate of progress varies, and because the knowledge requirements are fundamental, education “reform” can achieve results. Students with poor reading skills can get up to their grade level or even beyond, if they are motivated and taught for longer hours in a few key subjects. But we don’t really know that they are actually achieving academic success or readiness for more demanding material. All we know for sure is they are acquiring that fundamental knowledge of the elementary school curriculum at a faster pace—and of course, in many happy cases, getting a higher level of that fundamental knowledge than they would otherwise acquire.

But then comes high school, where E. D. Hirsch has no advice, where no one talks about grade level, and where the knowledge transferred is anything but fundamental.

The cognitive demands of high school are not a linear step up from 8th grade. In fact, they so far outstrip elementary and middle school expectations that our insistence on ignoring this fact is just downright mindboggling. Kids whose teachers took 9 hours a day, chants, threats, exhortations, or College DAys to keep at or slightly above grade school level reading and math are, sadly, not at all ready to succeed at algebra, trigonometry, Shakespeare, and chemistry.

Of course, many students aren’t even getting to 8th grade level by 8th grade, meaning they are starting the onslaught of high school without even the bare minimum of fundamental knowledge. Given a decent and realistic high school education, some of these students could get those fundamentals by their senior year, but we won’t allow them, instead demanding they spend time in subjects they can’t understand.

“College prep” high school level curriculum, once the option of a few select students with the interest and ability to move to college, is now demanded of all students in the absurd notion that expectations–and, of course, good teachers—are all these students ever needed to achieve. When they don’t achieve despite expectations, who is left to blame but the teachers?

Genuine college prep work is simply beyond the cognitive ability of our lowest ability students. It doesn’t matter if their elementary school teachers got them to grade level on time, or even if they learned more as a “below level proficiency” student in fourth grade.

Why is this so hard to understand? You want to see better performance by our 12th graders? Make the test easier. That’s why the improvement is noticeable in the earlier grades. The 12th grade test doesn’t acknowledge the existence of students who aren’t capable of 8th grade complexity.

No one’s at fault. Our education system didn’t fail. The only problem is that everyone’s not smart enough for high school, much less college prep.

Now, eduformers and progressives both may find this idea offensive or unpleasant. They may just disagree. But to leave it off the list of possibilities entirely bespeaks a certain blindness that, sadly, pervades all levels of our educational policy debate.