NAEP TUDA Scores—Detroit isn’t Boston

So everyone is a-twitter over NAEP TUDA (Trial Urban District Assessment) scores. For those who aren’t familiar with The Nation’s Report Card, the “gold standard” of academic achievement metrics, it samples performance rather than test every student. For most of its history, NAEP only provided data at the state level. But some number of years ago, NAEP began sampling at the district level, first by invitation and then accepting some volunteers.

I don’t know that anyone has ever stated this directly, but the cities selected suggest that NAEP and its owners are awfully interested in better tracking “urban” achievement, and by “urban” I mean black or Hispanic.

I’m not a big fan of NAEP but everyone else is, so I try to read up, which is how I came across Andy Smarick‘s condemnation of Detroit, Milwaukee, and Cleveland: “we should all hang our heads in shame if we don’t dramatically intervene in these districts.”

Yeah, yeah. But I was pleased that Smarick presented total black proficiency, rather than overall proficiency levels. Alas, my takeaway was all wrong: where Smarick saw grounds for a federal takeover, I was largely encouraged. Once you control for race, Detroit looks a lot better. Bad, sure, but only a seventh as bad as Boston.

So I tweeted this to Andy Smarick, but told him that he couldn’t really wring his hands until he sorted for race AND poverty.

He responded “you’re wrong. I sorted by race and Detroit still looks appalling.”

He just scooted right by the second attribute, didn’t he?

Once I’d pointed this out, I got curious about the impact that poverty had on black test scores. Ironic, really, given my never-ending emphasis on low ability, as opposed to low income. But hey, I never said low income doesn’t matter, particularly when evaluating an economically diverse group.

But I began to wonder: how much does poverty matter, once you control for race? For that matter, how do you find the poverty levels for a school district?

Well, it’s been a while since I did data. I like other people to do it and then pick holes. But I was curious, and so went off and did data.

Seventeen days later, I emerged, blinking, with an answer to the second question, at least.

It’s hard to know how to describe what I did during those days, much less put it into an essay. I don’t want to attempt any sophisticated analysis—I’m not a social scientist, and I’m not trying to establish anything certain about the impact of poverty on test scores, an area that’s been studied by people with far better grades than I ever managed. But at the same time, I don’t think most of the educational policy folk dig down into poverty or race statistics at the district level. So it seemed like it might be worthwhile to describe what I did, and what the data looks like. If nothing else, the layperson might not know what’s involved.

If my experience is any guide, it’s hard finding poverty rates for children by race. You can get children in poverty, race in poverty, but not children by race in poverty. And then it appears to be impossible to find enrolled children in a school district—not just who live in it, which is tough enough—by poverty. And then, of course, poverty by enrollment by race.

First, I looked up the poverty data here (can’t provide direct links to each city).

But this is overall poverty by race, not child poverty by race, and it’s not at the district level, which is particularly important for some of the county data. However, I’m grateful to that site because it led me to American Community Survey Factfinder, which organizes data by all kinds of geographic entities—including school districts—and all kinds of topics–including poverty—on all sorts of groups and individuals—including race. Not that this is news to data geeks, which I am not, so I had to wander around for a while before I stumbled on it.

Anyway. I ran report 1701 for the districts in question. If I understand googledocs, you can save yourself the trouble of running it yourself. But since the report is hard to read, I’ll translate. Here are the overall district black poverty rates for the NAEP testing regions:

Again, these are for the districts, not the cities.

(Am I the only one who’s surprised at how relatively low the poverty rates are for New York and DC? Call me naïve for not realizing that the Post and the Times are provincial papers. Here I thought they focused on their local schools because of their inordinately high poverty rates, not their convenient locations. Kidding. Kind of.)

But these rates are for all blacks in the district, not black children. Happily, the ACS also provides data on poverty by age and race, although you have to add and divide in order to get a rate. But I did that so you don’t have to–although lord knows, my attention to detail isn’t great so it should probably be double or triple checked. So here, for each district, are the poverty rates for black children from 5-17:

In both cases, Boston and New York have poverty rates a little over half those of the cities with the highest poverty rates—and isn’t it coincidental that the four cities with the lowest black NAEP scores have the highest black poverty rates? Weird how that works.

But the NAEP scores and the district data don’t include charter or private schools in the zone, and this impacts enrollment rates differently. So back to ACS to find data on age and gender, and more combining and calculating, with the same caveats about my lamentable attention to detail. This gave me the total number of school age kids in the district. Then I had to find the actual district enrollment data, most of which is in another census report (relevant page here) for the largest school districts. The smaller districts, I just went to the website.

Results:

Another caveat–some of these data points are from different years so again, some fuzziness. All within the last three or four years, though.

So this leads into another interesting question: the districts don’t report poverty anywhere I can find (although I think some of them have the data as part of their Title I metrics) and in any event, they never report it by race. I have the number and percent of poor black children in the region, but how many of them attend district schools?

So to take Cleveland, for example, the total 5-17 district population was 67,284. But the enrolled population was 40871, or 60.7% of the district population.

According to ACS, 22,445 poor black children age 5-17 live in the district, and I want an approximation of the black and overall poverty rates for the district schools. How do I apportion poverty? I do not know the actual poverty rate for the district’s black kids. I saw three possibilities:

1. I could use the black child poverty rate for the residents of the Cleveland district (ACS ratio of poor black children to ACS total black children). That would assume (I think) that the poor black children were evenly distributed over district and non-district schools.
2. I could have take the enrollment rate and multiplied that by the poor black children in ACS—and then use that to calculate the percentage of poor kids from blacks enrolled.
3. I could assign all the black children in poverty (according to ACS) to the black children enrolled in the district (using district given percentage of black children enrolled).

Well, the middle method is way too complicated and hurts my head. Plus, it didn’t really seem all that different from the first method; both assume poor black kids would be just as likely to attend a charter or private school than they would their local district school. The third method assumes the opposite—that kids in poverty would never attend private or charter schools. This method would probably overstate the poverty rates.

So here are poverty levels calculated by methods 1 and 3–ACS vs assigning all the poor black students to the district. In most cases, the differences were minor. I highlight the districts that have greater than 10 percentage points difference.

Again, is it just a coincidence that the schools with the lowest enrollment rates and the widest range of potential poverty rates have some of the lowest NAEP scores?

Finally, after all this massaging, I had some data to run regression analysis on. But I want to do that in a later post. Here, I want to focus on the fact that gathering this data was ridiculously complicated and required a fair amount of manual entry and calculations.

If I didn’t take the long way round, I suspect this effort is why researchers use the National Student Lunch Program (“free and reduced lunch”) as a poverty proxy.

The problem is that the poverty proxy sucks, and we need to stop using it.

Schools and districts have noticed that researchers use National School Lunch enrollment numbers as a proxy for poverty, and it’s also a primary criterion for Title I allocations. So it’s hard not to wonder about Boston’s motives when the district decides to give all kids free lunches regardless of income level, and whether it’s really about “awkward socio-economic divides” and “invasive questions”. The higher the average income of a district’s “poor” kids, the easier it is to game the NCLB requirements, for example.

Others use the poverty proxy to compare academic outcomes and argue for their preferred policy, particularly on the reform side of things. For example, charter school research uses the proxy when “proving” they do a “great job educating poor kids” when in fact they might just be skimming the not-quite-as-poor kids and patting themselves on the back. We can’t really tell. And of course, the NAEP uses the poverty proxy as well, and then everyone uses it to compare the performance of “poor” kids. See for example, this analysis by Jill Barshlay, highlighted by Alexander Russo (with Paul Bruno chiming in to object to FRL as poverty proxy). Bruce Baker does a lot of work with this.

To see exactly how untrustworthy the “poverty proxy is”, consider the NAEP TUDA results broken down by participation in the NSLP.

Look at all the cities that have no scores for blacks who aren’t eligible for free or reduced lunch: Boston, Cleveland, Dallas, Fresno, Hillsborough County, Los Angeles, Philadelphia, and San Diego. These cities apparently have no blacks with income levels higher than 180% of poverty. Detroit can drum up non-poor blacks, but Hillsborough County, Boston, Dallas, and Philadelphia can’t? That seems highly unlikely, given the poverty levels outlined above. Far more likely that the near-universal poverty proxy includes a whole bunch of kids who aren’t actually poor.

In any event, the feds, after giving free lunches to everyone, decided that NSLP participation levels are pretty meaningless for deciding income levels “…because many schools now automatically enroll everyone”.

I find this news slightly cheering, as it suggests that I’m not the only one having a hard time identifying the actually poor. Surely this article would have mentioned any easier source?

So. If someone can come back and say “Ed, you moron. This is all in a table, which I will now conveniently link in to show you how thoroughly you wasted seventeen days”, I will feel silly, but less cynical about education policy wonks hyping their notions. Maybe they do know more than I do. But it’s at least pretty likely that no one is looking at actual district poverty rates by race when fulminating about academic achievement, because what I did wasn’t easy.

Andy Smarick, at any rate, wasn’t paying any attention to poverty rates. And he should be. Because Detroit isn’t Boston.

This post is long enough, so I’ll save my actual analysis data for a later post. Not too much later, I hope, since I put a whole bunch of work into it.

Escaping Poverty

Bryan Caplan asks: “Suppose a 15-year-old from a poor family in the First World asked you an earnest question: ‘What can I do to escape poverty?’ How would you answer?”

I doubt he wants an answer from a teacher/test prep instructor/tutor, but what the heck:

Caplan doesn’t indicate the cognitive ability or race of the poor 15-year-old. Strangely enough, it doesn’t matter too much until the last few steps in the process. So here’s what I’d tell the kid:

1. Cut your family loose. I don’t mean you have to abandon them, or hate them, but their needs are secondary to yours. If they’re making demands, you have to say “No”. All the time. No, you can’t stay home to babysit because your little sister is sick. No, you can’t go pick your father up at work at 2 in the morning. No, you can’t drop your niece and nephew off at school and be late to class. No, you can’t miss a morning of school to drive your mother to the utility company to help her tell a sob story that gets the power turned back on until she has money to pay the bills. No, you can’t work extra shifts just because the family’s broke. No, you can’t lose an entire weekend to visiting your dad/brother/sister/grandfather in jail. I don’t care if your parents are bums or hardworking joes. They made their lives, and if you want a chance of getting out and making your family’s life better, you don’t get sucked in by their problems. If your parents share your goals, then they’re already making this happen. Otherwise, they are millstones round your neck.
   

2. If you live in a city or suburb: within a ten mile radius of your school, there are fifteen to twenty organizations dedicated to helping at risk youth. You are at risk. Go check them out and pick the best one. If your school has an AVID program, sign up for that. There is a bunch of do-gooder money funding a whole host of programs that will give you, for free, everything you need to prepare for college. They will give you daily snacks, mentors, tutoring support, monitoring, care, test prep, college visits, free college admissions tests, and anything else you need. All you have to do is show up. Reporters will periodically feature one of these organizations as if they are unique or their services are rare and surprising. They are neither. Counsellors may not even know of their existence. You must find these places. If you live in a rural area, I can’t be as helpful here, but I suspect your school will be much more knowledgeable about existing support than suburban and urban schools are, and may even be more involved in coordinating these programs. So start with your school. Ask your church. Consult the phone book. If you end up having to do without this support, be certain that it wasn’t out there waiting for you to show up. And worst case, every single fee you can think of has a waiver form and you will certainly qualify.
   

3. Stay away from anyone your age who doesn’t share your goals.
   

4. Stay away from anything illegal: drugs, boosting cars, sex with anyone outside the approved age range, whatever. I’ve lived a clean life; I have no idea what the temptations are. Avoid. If you ignore this advice, memorize these words: “I WANT A LAWYER. NOW.” While screwing up on this point is dangerous, it’s not necessarily fatal. I know a Hispanic kid who graduated from high school while in jail (boosting cars); he then went to a junior college and graduated as valedictorian and went to Columbia. No, I’m not making this up. I tutored him for his SATS when he was in his second year of community college. Yes, he’s an exception.
   

5. Don’t get pregnant. Don’t get anyone pregnant. Don’t pretend that you aren’t your own worst enemy if you ignore this advice. I have no happy anecdotes for this rule. Jail has less of an opportunity cost than a kid.
   

6. Get good grades. Most teachers grade on effort, not ability. Use this if you need to, which means you can get good grades simply by doing your homework and making the teacher happy. If you get a teacher who grades on ability, take the opportunity as a valuable benchmark. Are you doing well? Your abilities are strong. Are you in danger of failing? Buckle down and take the opportunity to improve to the best of your capabilities. That opportunity will be worth the grade hit. Grades are an area in which your mentoring organization can help. A lot. They are designed around helping you get good grades. Use them.
   

7. Don’t believe the people who tell you that you need X years of math or Y years of English to get to college. Race determines your transcript and test requirements. If you’re white or Asian, then you need an impressive transcript and decent test scores, no matter how poor you are. If you’re black or Hispanic, you’ve got a decent shot at the best schools in the country if you have SAT scores of 550 or higher per section, and a decent GPA (say 3.0 or higher). Blacks and Hispanics who can read, write, calculate at a second-year algebra level, and care enough about school to have a 3.0 GPA are an exceptionally rare commodity (about 10% of blacks, 20% of Hispanics).

   But what if you can’t hit that ability mark? What if you aren’t very intellectual, work hard but don’t do very well on tests, can’t score above 500 on any section of the SAT, despite all your test prep? All is not lost. Whatever you do, don’t lie to yourself about your abilities, and don’t let anyone else lie to you. If you are a low income black or Hispanic kid, many people are uninterested in your actual abilities. You are a statistic they can use to brag about their commitment to diversity. That’s fine. Use their self-interest to your advantage. But if you can’t break 500 on any section of the SAT, then college is going to present a considerable challenge. Don’t compound that challenge by choosing a college where your degree would be a case of overt fraud. Start thinking in terms of training, not academics. Find the best jobs you can, and build good working relationships. Put more priority on acquiring basic skills, and find the classes that will help you do that. Tap into your support group mentioned above, tell them your goals. This doesn’t mean college isn’t an option, but it’s important to keep your goals realistic. If you are a low income white or Asian kid with little interest or ability in academics, no one will lie to you, and no one is interested in helping you because you represent the wrong sort of diversity. However, the advice remains the same. And for all races, if your skills aren’t too low, don’t forget the military.

   Remember that colleges only use grades for admission. Once you’re in, they give you placement tests and grades don’t matter at all. This is great news for high ability kids who screwed around in high school; bad news for low ability kids who worked hard. Remediation has derailed a number of dreams. Be prepared, know what to expect, and minimize your need for it by taking advantage of every minute of your free high school education. And remember: no matter how bad your school is, it has teachers there who can teach motivated kids. Be one of the kids and find those teachers.

8. Do not overpay for college. Set your goals based on the advice I’ve given here, as well as the advice of those you trust. Get a job to offset expenses. To the extent possible, find jobs that look good on a resume. A secretarial job looks better than a stint at Subway; a tutoring job looks better than a custodial one. Bank your money; if it’s at all possible to accept an unpaid internship that looks good on a resume, you want the option. If you’re studying for a trade, learn everything you can about the job opportunities: from your college, from seminars, from employers in the field. Try to know what you can expect and what sort of positions you want. But if you don’t know what you want, then don’t drift. Find a job, even if it’s not perfect, and see what happens.

If you’ve managed to achieve everything up to that point, you will have escaped poverty. How and by how much are yet to be determined, but you’re on your way.

It’s too easy to say “Get a good support system, go to school, don’t get knocked up or locked up, go to college.” All are optimal, most are necessary, but they sure aren’t sufficient if you don’t understand the game and jump through the right hoops. I’ve tried here to point out some hoops. Good luck.