Spring 2013: These students aren’t really prepared, either.

I’m teaching Geometry and Algebra II again, so I gave the same assessment and got these results, with the beginning scores from the previous semester:

I’m teaching two algebra II classes, but their numbers were pretty close to identical—one class had the larger range and a lower mode—so I combined them.

The geometry averages are significantly lower than the fall freshmen only class, which isn’t surprising. Kids who move onto geometry from 8th grade algebra are more likely to be stronger math students, although (key plot point) in many schools, the difference between moving on and staying back in algebra come down to behavior, not math ability. At my last school, kids who didn’t score Proficient or Advanced had to take Algebra in 9th grade. I’d have included Basic kids in the “move-on” list as well. But sophomores who not only can’t factor or graph a line, but struggle with simple substition ought not to be in second year algebra. They should repeat algebra I freshman year, go onto geometry, and then take algebra II in junior year—at which point, they’d still be very weak in algebra, of course, but some would have benefited from that second year of first year.

Wait, what was my point? Oh, yeah–this geometry class class is 10-12, so the students took one or more years of high school algebra. Some of them will have just goofed around and flunked algebra despite perfectly adequate to good skills, but a good number will also be genuinely weak at math.

On the other hand, a number of them really enjoyed my first activity: visualizing intersecting planes, graphing 3-D points. I got far more samples from this class. I’ll put those in another post, also the precalc assessment.

I don’t know if my readers (I have an audience! whoo!) understand my intent in publishing these assessment results. In no way am I complaining about my students.

My point in a huge nutshell: how can math teachers be assessed on “value-added” when the testing instrument will not measure what the students needed to learn? Last semester, my students made tremendous gains in first year algebra knowledge. They also learned geometry and second year algebra, but over half my students in both classes will test Below Basic or Far Below Basic–just as they did the year before. My evaluation will faithfully record that my students made no progress—that they tested BB or FBB the year before, and test the same (or worse) now. I will get no credit for the huge gains they made in pre-algebra and algebra competency, because educational policy doesn’t recognize the existence of kids taking second year algebra despite being barely functional in pre-algebra.

The reformers’ response:

1) These kids just had bad teachers who didn’t teach them anything, and in the Brave New World of Reform, these bad teachers won’t be able to ruin students’ lives;

2) These bad teachers just shuffled students who hadn’t learned onto the next class, and in the Brave New World of Reform, kids who can’t do the work won’t pass the class.

My response:

1) Well, truthfully, I think this response is moronic. But more politely, this answer requires willful belief in a delusional myth.

2) Fail 50-60% of kids who are forced to take math classes against their will? Seriously? This answer requires a willful refusal to think things through. Most high schools require a student to take and pass three years of math for graduation. Fail a kid just once, and the margin for error disappears. Fail twice and the kid can’t graduate. And in many states, the sequence must start with algebra—pre-algebra at best. So we are supposed to teach all students, regardless of ability, three years of increasingly abstract math and fail them if they don’t achieve basic proficiency. If, god save us, the country was ever stupid enough to go down this reformer path, the resulting bloodbath would end the policy in a year. We’re not talking the occasional malcontent, but over half of a graduating class in some schools—overwhelmingly, this policy impacts black and Hispanic students. But it’s okay. We’re just doing it for their own good, right? Await the disparate impact lawsuits—or, more likely, federal investigation and oversight.

Reformers faithfully hold out this hope: bad teachers are creating lazy students who could do the work but just don’t want to. Oh, yeah, and if we catch them in elementary school, they’ll be fine in high school.

It is to weep.

Hey, under 1000 words!

Algebra 1 Growth in Geometry and Algebra II

Last September, I wrote about my classes and the pre-algebra/Algebra 1 assessment results.

My school covers a year of instruction in a semester, so we just finished the first “year” of courses. I start with new students and four preps on Monday. Last week, I gave them the same assessment to see if they’d improved.

Unfortunately, the hard drive on my school computer got wiped in a re-imaging. This shouldn’t have been a problem, because I shouldn’t have had any data on the hard drive, except I never got put on the network. Happily, I use Dropbox for all my curriculum development, so an entire year’s worth of intellectual property wasn’t obliterated. I only lost the original assessment results, which I had accidentally stored on the school hard drive. I should have entered the scores in the school grading system (with a 0 weight, since they don’t count towards the grade) but only did that for geometry, the only class I can directly compare results with.

My algebra II class, though, was incredibly stable. I only lost three students, one of whom got a perfect score—which the only new addition to the class also got, so balance maintained. The other two students who left got around 10-15 wrong, so were squarely in the average at the time. I feel pretty comfortable that the original scores didn’t change substantially. My geometry class did have some major additions and removals, but since I had their scores I could recalculate.

	Mean	Median	Mode	Range
Original	just above 10	9.5	7	22
Recalculated	just below 10 (9.8)	8	7	22

I didn’t have the Math Support scores, and enough students didn’t take the second test that comparisons would be pointless.

One confession: Two Algebra II students, the weakest two in the class, who did no work, scored 23 and 24 wrong, which was 11 more than the next lowest score. Their scores added an entire point to the average wrong, increased the range by 14 points, and you know, I just said bye and stopped them from distorting the results the other 32 kids. (I don’t remember exactly, but the original A2 tests had five or six 20+ wrong scores.)

So here’s the original September graph and the new graph of January:

The geometry class was bimodal: 0 and 10. Excel refused to acknowledge this and I wasn’t sure how to force it. The 10s, as a group, were pretty consistent—only one of them improved by more than a point. The perfect scores ranged from 8 wrong to 2 wrong on the first test.

In short, they learned a lot of first year algebra, and that’s because I spent quite a bit of time teaching them first year algebra. In Algebra II, I did it with data modeling, which was a much more sophisticated approach than what they’d had before, but it was still first year algebra. In geometry, I minimize certain standards (proofs, circles, solid shapes) in favor of applied geometry problems with lots of algebra.

And for all that improvement, a still distressing number of students answered x² + 12 when asked what the product of (x+3) and (x+4) was, including two students who got an A in the class. I beat this into their heads, and STILL some of them forget that.

Some folks are going to draw exactly the wrong impression. “See?” these misguided souls will say, nodding wisely. “Our kids just aren’t being taught properly in early grades. Better standards, better teachers, this problems’s fixed! Until then, this poor teacher has to make up the slack.” In short, these poor fools still believe in the myth that they’ve never been taught.

When in fact, they were taught. Including by me—and I don’t mean the “hey, by the way, don’t forget the middle term in binomial multiplication”, but “you are clubbing orphan seals and making baby Jesus cry when you forget the middle term” while banging myself on the head with a whiteboard. And some of them just forgot anyway.

I don’t know how my kids will do on their state tests, but it’s safe to say that the geometry and second year algebra I exposed them to was considerably less than it would have been had their assessment scores at the beginning of class been the ones they got at the end of class. And because no one wants to acknowledge the huge deficit half or more of each class has in advanced high school math, high schools won’t be able to teach the kids the skills they need in the classes they need—namely, prealgebra for a year, “first year” algebra for two years, and then maybe some geometry and second year algebra. If they do okay on the earlier stuff.

Instead, high schools are forced to pretend that transcripts reflect reality, that all kids in geometry classes are capable of passing a pre-algebra test, much less an algebra one test. Meanwhile, reformers won’t know that I improved my kids’ basic algebra skills whilst still teaching them a lot of geometry/algebra II, because the tests they’ll insist on judging me with will assume a) that the kids had that earlier material mastered or b) that I could just catch them up quickly because after all, the only problem was the kids’ earlier teachers had never taught them.

Teaching Students with Utilitarian Spectacles

In my last post, commenter AllaninPortland said, of my Math Support students, “Their brains are wired a little too literally for modern life.”

James Flynn, of the Flynn Effect:

A century ago, people mostly used their minds to manipulate the concrete world for advantage. They wore what I call “utilitarian spectacles.” Our minds now tend toward logical analysis of abstract symbols—what I call “scientific spectacles.” Today we tend to classify things rather than to be obsessed with their differences. We take the hypothetical seriously and easily discern symbolic relationships.

Yesterday I gave my math support kids a handout on single step equations similar to the one in the link.

“Oh, I know how to do this,” said Dewayne. “Just subtract six from both sides.”

“You could do that,” I said. “But here’s what I want people to try. I want everyone to read the first equation as a sentence. What is it saying?”

“Some number added to six gets fourteen,” came from Andy.

“Excellent!”

“You mean, you don’t want us to subtract, add, do things to get x by itself?” asked Jose.

“That’s called ‘isolation’. You are ‘isolating’ x, getting it all by itself as you put it. Who knows how to do that?” Over half the class raised their hands. “Great. You can do that if you want to, but I’d like you to try seeing each equation just as Andy described it. Put the equation you see into words. This will help make it real, and will often give you the answer right away. For example, what number do I add to six to get 14?”

“Eight.” chorused most of the room.

“There you go. Now, remember, what did I say a fraction was?”

“Division.”

“So instead of saying ‘x over 5′, you’re going to say….”

“X divided by 5″ came back a number of students.

“Off you go.”

This worked for most of the students, but one student, Gerry, sat at the back of the room drawing, as he often does. After watching him do no work for 10 minutes, I called him up front. (Normally, I am wandering the room, but every so often I call them up for conversations instead.)

“So you aren’t working.”

“Yeah. I can’t do this.”

“Remember yesterday, when we were doing those PEMDAS problems? You were on fire!”

“Yeah, but it didn’t have the letters in it. I can do math when it doesn’t have letters. And yesterday, when you showed us how to just draw pictures for the word problems? That was cool. I think I can do those now.”

“You need to look at these problems from a different part of your brain.”

“A different what?”

“This is a really, really easy problem. Way easier than the math problems you solved in your head yesterday. But you don’t see this as the same kind of problem, so we have to fool your brain.”

“How do we do that?”

“Read the first problem aloud.”

“X + 6 = 14. This is when you have to do stuff to both sides, right? I can’t do that.”

“Read it again. But instead of saying x, say ‘what’.”

“Say ‘what’?”

“Yep.”

“You crazy.”

“Definitely. Try it.”

“What plus 6 = 14? 8.”

“There you go.”

He was sitting in one of my wheeled chairs, pushing it back and forth with his feet. This stopped him cold.

“Eight’s the answer? Holy sh**.”

“Try another. Without the language.”

“What minus 3 = 7. That’s nine…no, 10. Ten? Really? No f**k….no way.”

“And this one?”

“Oh, that’s a fraction. I can’t do those.”

“What did I tell you fractions were?”

“Division. Oh. What divided by 5 is 9? Forty five? No way?”

“So. I want to see you do this whole handout, 1-26, and every time you see an x, call it ‘what’. Remember to sketch out subtraction questions on a numberline and think about direction.”

“Okay. Man, I can’t believe this.”

Fifteen minutes later, Gerry was done with the entire set. Only three minor errors, all involving negative numbers.

“I feel like a math genius,” he said with a wry grin.

I sat down next to him. “It’s like I said. We have to ask your brain a different question. So instead of tuning me out, next time I come up with some goofy idea using pictures or tiles or different words, give it a shot. And tell me if it works to give your brain the right question. Some of my ideas will work, some won’t. And some things, we won’t be able to fool your brain to answer a different way. But you know a lot more math than you think you do. You just have to figure out how to ask the question in a way your brain understands.”

Back to Flynn:

A greater pool of those capable of understanding abstractions, more contact with people who enjoy playing with ideas, the enhancement of leisure—all of these developments have benefited society. And they have come about without upgrading the human brain genetically or physiologically. Our mental abilities have grown, simply enough, through a wider acquaintance with the world’s possibilities.

But not everyone is capable of understanding abstractions to the same degree. Some people do better learning the names of capitals and Presidents and the planets in the solar system. They’d learn confidence and competence through interesting, concrete math word problems and situations, and enjoy reading and writing about specific historic events, news, or scientific inventions that helped society. Instead, we shovel them into algebra, chemistry, and literature analysis and make them feel stupid.

Students’ names have been changed. They are all awesome kids. Do not say mean things about them in the comments, which I can control, or other blogs, which I cannot.

The Sinister Assumption Fueling KIPP Skeptics?

Stuart Buck on KIPP critics:

It’s unwitting, to be sure; most of the critics haven’t thought through the logical implications of what they’re saying, and they would sincerely deny being racist in their thoughts or intentions. But even granting their personal good will, what they are saying is full of racially problematic implications. These KIPP critics are effectively saying that poor minority children are incapable of genuinely learning anything more than they already do. If poor minority children seem to be learning more, it can’t really be true; there must be some more sinister explanation for what’s going on.
…..
Now here’s the key point: If selection and attrition is what explains KIPP’s good results, then that logically means that several hundred extra hours a year being instructed in reading, math, music, art, etc. do NOT explain KIPP’s good results. But wait a minute: what does that really mean?
….
Nothing less than this: several hundred hours a years instructing kids doesn’t actually make much difference. Recall that KIPP’s critics say that if KIPP’s students seem to be learning more, it must be an artifact of how KIPP selects kids and then pushes out the low-performers. In saying that, KIPP’s critics are implying, however unwittingly, that no amount of effort or study could possibly get poor urban minorities to learn anything more.

Okay, let me be clear that I am not speaking for any other KIPP critic. While I don’t talk much about KIPP, I am certainly one who thinks their results are due to attrition, creaming, and the benefits that accrue from a homogenous and motivated population.

But yeah. In a nutshell, I’m saying this:

IF you take low ability kids (of any race or income) and IF you select for motivation in the parents, at least, and IF you remove the misbehaving or otherwise highly dysfunctional kids who don’t share their parents’ motivation, and IF you enforce strict behavioral indoctrination in middle class mores and IF you give them hundreds of hours more education a year and IF they are in middle school and IF they are simply being asked to catch up with the material that middle to high ability kids learned fairly effortlessly—that is, elementary reading and math skills…..

…then they will have a slightly better test scores than similarly motivated low ability kids stuck in classes with the misbehavers and highly dysfunctional kids and fewer hours of seat time and less behavioral indoctrination into middle class mores, but their underlying abilities will still be weak and just as far behind their higher ability peers as they were before KIPP.

I’ve written before, improving elementary school or middle school scores is a false god when it comes to improving actual high school outcomes. Children who need tons of hours to get up to grade level fundamentally differ from those reading at or above grade level from kindergarten on, and this difference matters increasingly as school gets harder. High school isn’t the linear steps through increased difficulty that occurs in grades K-8, but a much different and far more difficult animal, now that we make everyone take college prep classes. There’s no evidence that KIPP students are learning more or closing the gap in high school, and call me cynical but I’m really, really sure we’d be hearing about it if they were. KIPP is not transforming low ability kids into high ability kids, or even mid-level ability kids.

I am comfortable asserting that hours and hours of additional education time does nothing to change underlying ability. I’m not a racist, nor am I a nihilist who believes outcomes are set from birth. I do, however, hold the view that academic outcomes are determined in large part by cognitive ability. The reason scores are low in high poverty, high minority schools is primarily due to the fact that the students’ abilities are low to begin with, not because they enter school with a fixable deficit that just needs time to fill, and not because they fall behind thanks to poor teachers or misbehaving peers.

That doesn’t mean we can’t improve outcomes, particularly in high school, when we do a great deal of harm by trying to teach kids what they can’t learn and refusing to teach them what they can learn. And it doesn’t mean we couldn’t tremendously improve elementary school outcomes in numbers, if not individual demonstrated ability, by allowing public schools to do what KIPP does—namely, limit classes to motivated kids of similar ability.

Paul Bruno, another KIPP skeptic (whose views in no way should be confused with mine), thinks it’s wrong to dismiss KIPP achievements, because they show that public schools for low income kids simply need much more money. I disagree. What KIPP “success” shows is the importance of well-behaved, homogeneous classes.

So here’s my preferred takeaway from KIPP and other successful charter schools:

Since it’s evident that much of these schools’ success stories come from their ability to control and limit the population, why are we still hamstringing public schools? Here’s a thought: how about KIPP schools take those really, really tough kids and only those kids? Misbehave too often in public schools and off you go to a KIPP bootcamp, where they will drill you with slogans and do their best to indoctrinate you into middle class behavior and after a while you’ll behave because please, god, anything to get back to the nicer public schools! You could also create KIPP schools for special ed kids–put the special ed kids with cognitive issues and learning disabilities in their own, smaller schools. Meanwhile, public schools could extend the school day a bit, help the kids catch up as much as possible while still making school fun. While the average test score might not improve much, this approach would keep a lot of kids engaged in school through elementary school instead of lost, bored, or acting out in chaotic classes disrupted by a few unmanageable or extremely low ability kids.

See, that would scale a lot better. Instead, we set up small schools for what is actually the majority of all low income students—reasonably well-behaved, of low to middle ability and, with no one around to lead them astray, willing to give school a shot. Only a few kids get into these schools, while the rest of them are stuck in schools where just a few misbehavers make class impossible and really low ability kids take up a lot of addtional teacher time. Crazy, that’s what it is. But what I just laid out is completely unworkable from an ideological standpoint, and as I just explained in an earlier post, school policy is set by ideology and politics, not educational validity. To say nothing of the fact that KIPP doesn’t want to teach “those” kids.

Anyway. The reality is that yes, a low ability kid, regardless of income or race, will not, on average, become a high or mid ability kid simply because he spends a lot of seat time working his butt off in a KIPP school. Sorry Stuart.

SAT Prep for the Ultra-Rich, And Everyone Else

Whenever I read about SAT tutors charging in the hundreds of dollars, I’m curious. I know they exist, but I also know that I’m pretty damn good, and I’m not charging three figures per hour (close, though!). So I always read them closely to see if, in fact, these test prep tutors are super fab in a way that I’m not.

At the heart of all test prep stories lies the reporter’s implicit rebuke: See what rich people are doing for their kids? See the disadvantage that the regular folks operate under? You can’t afford those rates! You’re stuck with Kaplan or cheaper, cut-rate tutors! And that’s if you’re white. Blacks and Hispanics can’t even get that much. Privilege. It sucks.

And so the emphasis on the cost of the tutors, rather than any clear-eyed assessment of what, exactly, these tutors are doing that justifies an hourly rate usually reserved for low-end lawyers, never mind the fact that these stories are always about the SAT, when in fact the ACT is taken by as many kids as the SAT. The stories serve up propaganda more than they provide an accurate picture of test prep.

I’ve written before about the persistence of test prep delusions. Reality, summarized: blacks and Hispanics use test prep more than whites, Asians use it more than anyone. Rich parents are better off buying their kids’ way into college than obsessing about the last few points. Test prep doesn’t artificially inflate ability.

So what, in fact, is the difference between Lisa Rattray, test prep coach charging $300/hour; me, charging just short of 3 figures; and a class at Kaplan/Princeton/other SAT test prep schools?

Nothing much. Test prep coaches can work for a company or on their own. The only difference is their own preferences for customer acquisition. Tutors and instructors with a low risk tolerance just sign on with a company. Independent operators, comfortable with generating their own business, then pick their markets based on their own tolerance. My customers sit comfortably in the high income bracket, say $500K to $5 million yearly income, although I’ve worked with a couple Fortune 500 families. Lisa Rattray and Joshua Brown, the featured tutors, clearly work with families a couple notches up the income ladder from mine.

None of this has anything to do with quality of instruction. Test prep is a sales and marketing game. The research is clear: most kids improve at least a little, quite a few kids improve a lot, a very few kids stay put or, heaven forfend, get worse.

Obviously, instructor quality influences results a bit, but only rarely change a kid from one category (mild improvement) to another (major improvement). Remember, all test prep instructors have high test scores, and they’re all excellent at understanding how the test works. So they make career decisions based on their tolerance for sales and marketing, not the quality of their services. I know of some amazingly god-awful tutors who charge more than I do, having learned of them from their furious ex-clients who assumed a relationship between price and quality. These tutors have websites, business cards, offered their own prepared test materials, saw students in their rented space, and often accepted credit card deposits. I have none of these accoutrements, show up at my clients’ houses, usually but not always on time, and take checks. Every so often I get a client who whips out a wad of bills and pays me $500 in cash, which I find a tad unnerving.

I’m just as good now as I was at Kaplan (in fact, I privately tutored my own students while at Kaplan, tutoring theirs), but I only got paid $24/hour for Kaplan work, which charged about $125/hour for my services. Kaplan will (at least, when I worked there) boost a teacher’s hourly rate to $50/hour if they get 80% or more “perfect” customer ratings. Instructors who convinced their students that to respond to the online survey and give them excellent ratings got more money. This is independent of actual improvement. A customer who doesn’t improve at all but felt reassured and valued by her instructor could give straight 5s (or 1s, whatever the highest rating is). A customer who sees a 300 point improvement might not fill in the survey at all. Their research showed that customers who give their instructors perfect ratings gave awesome word of mouth and that was worth rewarding. Nothing else was. Asian cram schools pay instructors based on the students who sign up, with a premium for those who sign up specifically for that instructor. See? Sales and marketing.

Test prep companies, long castigated as the luxury option of the wealthy, have been the first choice of the middle class for a decade or more. For the reasons I’ve outlined, any parent can find excellent instructors in all the test prep companies: Kaplan, Princeton Review, Asian cram schools. They won’t brag about it, though, because these companies are about the brand. Kaplan doesn’t want word getting out that Joe Dokes is a great Kaplan instructor; it wants everyone to be happy with Kaplan. No one is “Princeton Review’s star tutor” for very long, because Princeton doesn’t like it and at that point, the most risk-averse instructor probably has enough word of mouth fame to go independent.

I’ve often advised my students to consider a class. The structure helps. Some of my kids don’t do any work unless I’m there, so what I end up doing is sitting there playing Spider on my android on my client’s dime while the kid works problems, rather than reviewing a bunch of work to move forward. I’m pretty sure Lisa and Joshua would celebrate this, going to the parent and pointing out how much they are helping. I have better things to do and other clients to see. So I tell the parents to fork out an extra thousand for a class, make sure the kid goes, and then we review the completed work. The student gets more hours, more focus and, usually, higher scores, regardless of the quality of the second instructor.

I’m not saying Lisa and Joshua are wrong, mercenary, or irresponsible. They just play to a different clientele, and a huge chunk of their ability to do so rests on their desire to sell an image. That’s fine. That’s just not me. Besides, Josh forks out $15K of his profit for a rental each summer. Lisa gets constant text messages from anxious parents. Also not me.

So you’re a white, middle class or higher parent with a teenager, worried about SAT scores. What do you do? Here are some guidelines. Recognize that GPA or parental income smacks down test scores without breaking a sweat. If Johnny doesn’t have a GPA of 3.8 or higher, elite universities are out of the question unless his parents are alumni or rich/connected enough to make it worth the school’s while.

If Sally qualifies on GPA, has a top-tier transcript (5 or more AP classes) and wants to go to a top 10 school, test scores should be 700 or higher per section. If they’re at that point, don’t waste your time or money or stress. At that point, the deciding factors aren’t scores but other intangibles, including the possibility that the admissions directors toss a pile of applications in the air and see which ones travel the farthest.

If Jesse is looking for a top 20 or 30 school, the GPA/transcript requirements are the same, but looking at the CDS of these schools, realistically a 650 or higher per section will do the trick. It might be worth boosting the test scores to low 700s, but if Jesse is a terrible tester, then don’t break the bank. One of the schools will probably come through.

If Sammy has a lower GPA (3.3 to 3.8) but excellent test scores (high 600s or higher per section) , then look to the schools in the middle–say, from 40 to 60. It’s actually worth spending money to maximize Sammy’s scores, because these mid-tier schools often get a lot of high effort hard workers with mediocre test scores. Not only will Sammy look good, but he might get some money. (By the way, if you’ve got a Sammy whose grades are much lower than his abilities, you should still push him into the hardest classes, even if he and the counsellors cavil. If your Sammy is like most of them, he’s going to get Bs and Cs regardless, so he may as well get them in AP classes and get some college credit from the AP tests. And the transcript will signal better, as well.)

The biggest bang for the test prep buck lies not in making kids competitive for admissions, but to help them test out of remediation at local universities. So if Austin has a 3.0 GPA, works hard but tests poorly, then find out the SAT cut score at his university. If he’s not above that point, then spend the money to get him there, and emphasize the importance of this effort to his college goals.

If your kid is already testing at 650 or higher, either send her to an Asian cram school (they will be the only white kid there, for the most part, but the instruction will be excellent) or invest in a tutor. The average white kid class at Kaplan or Princeton might have an instructor who can finetune for their issues, but probably won’t.

Otherwise, start with a class and supplement with a tutor if you can afford it. Ask around for good instructors, or ask the test prep company how long the instructor has been teaching. Turnover in test prep instructors is something like 75%; the 25% who stay long term do so because they’re good. As for the tutor, I hope I’ve convinced everyone that price isn’t an issue in determining quality. I would ask around for someone like me, because our ability to get a high rate without the sales and marketing suggests we must be, in fact, pretty good. And there’s always someone like me around. Otherwise, I’d go with the private tutoring options at a test prep company, with interviews.

As I said, these rules are for middle class or higher white kids. Only 6% of blacks and Hispanics get above 600 on any section of the SAT–in fact, the emphasis on GPA came about in large part to bypass the unpleasant reality of the score gap. There are only around 300 black students that get higher than 700 on two sections of the SAT. That’s barely enough blacks for one top ten school. Rules are very different. The main reason for blacks and Hispanics to take test prep is to get their scores above the remediation number. Middle class or higher Asians face much higher standards because universities know their (or their parents’) dedication to getting good grades and good test scores is more than a tad unnatural and probably overstates their value to the campus. Athletes and artists of note play by different rules. Poor whites and poor Asians have it really, really tough.

What this means, of course, is that the kids in the Hamptons are probably already scoring 700 or higher per section and are, consequently, wasting their time. But what the hell, they’re doing the economy some good. Or maybe some of them are Asian.

Note: I wrote this focusing on the SAT but it all applies to the ACT as well, and the ACT is a much better test. I wrote about the ACT here.

The false god of elementary school test scores

Rocketship Academy wants to go national. Rocket Academy is a hybrid charter school chain that focuses solely on getting low income Hispanic elementary school students to proficiency. (Note: Larry Cuban has some excellent observations from his visit to a Rocketship Academy.)

First things first: I’ve checked the numbers every way I can think of, and Rocketship’s numbers are solid. They don’t have huge attrition problems that I can see. They are, in fact, getting 60% or higher proficiency in most test categories, and the bulk of their students are Hispanic, many of them not proficient in English. Of course, that brings up an interesting question–if they are proficient on the ELA tests, why aren’t they considered proficient in English? But I digress.

The larger point is this: getting high test scores on California’s elementary school math tests ain’t all that much to get worked up about. Here’s some data from the 2011 California Standards test in math:

I used two standards, because the NCLB obsession with “Proficient and higher” is, to me, moronic. I prefer Basic or higher. The blue line is the percentage of all California in grades 2-9 scoring Basic or higher in General Math, the red is the percentage of same scoring Proficient or higher.

So it gets a bit tricky here, because after 6th grade, the entry to algebra varies. In order to simplify it slightly, I’m ignoring the seventh grade algebra track (call it “accelerated advanced” path), which is about 40,000 students this year, fewer in previous years.

I combined 8th and 9th grade students in General Math and attached that result to the red and blue lines.

Then I separated two groups–the ones who took algebra in 8th grade, and the ones who took algebra later than that. The first group are those who entered algebra in 8th grade, passed it, and continued on the “average advanced” course path, culminating with Calculus senior year. The second group are those who took algebra for the first, second, or third time in high school and then continued on. For each group, I calculated percentages for Basic + and Proficient +.

Notes:

1. Through grade 6, the scores represent all students. In grade 7 and 8/9, general math scores reflect only those students who haven’t moved onto algebra. That’s probably why the proficiency levels drop to 50% and lower for the last two groups.In other words, the green and purple lines represent the advanced track students–most, but not all of the the strong algebra and higher math students. The turquoise and orange lines represent the weaker students taking algebra and higher.
2. Roughly 80% of all students test at Basic or higher from second through sixth grade.
3. Over 70% of the strong studentws test at Basic or higher from algebra through “summative math” (taken for all subjects after Algebra 2).
4. The percentage of students testing proficient from second through sixth grade starts at 65%, rises slightly, and then drops steadily.
5. In no course do more than 50% of the strong students in algebra and higher achieve a score of Proficient or higher.
6. In no course do more than 50% of the weaker students in algebra or higher achieve a score of Basic or higher.

So the chart reveals that all California second through sixth graders, high and low ability, averaged higher scores on their tested subject than the strongest high school students did.

I used 2011 scores, and I may have made a minor error here or there, but the fall off has been in the scores for several years now, and it’s easy enough to check.

What could cause this? Why are California’s elementary school students doing so phenomenally well, and then fall apart when they get to high school? Let’s go through the usual culprits.

California’s high school math teachers suck.–Well, in that case, there’s not much point in demanding higher standards for math teachers, because California’s high school math teachers have had to pass a rigorous content knowledge test for over 20 years. California’s elementary school teachers have to pass a much easier test–which is much harder than anything they had to pass before 2001. In other words, try again.

The teachers aren’t covering the fundamentals! So when the students get to algebra, they aren’t prepared.–But hang on. Elementary school kids, the ones being taught the fundamentals, are getting good test scores. What evidence do you have that they aren’t being taught properly?

Well, they’re only getting good test scores because the tests are too easy!—dingdingding! This is a distinct possibility. Perhaps the elementary tests aren’t challenging enough. Having looked at the tests, I’m a big believer in this one. I think California’s elementary math tests, through seventh grade, are far less challenging to the tested elementary school population than are the general math and specific subject tests are to the older kids. (On the other hand, the NAEP scores show this same dropoff.)

However, while that might explain the disparity between the slower track math student achievement and elementary school, it doesn’t adequately address why the students in the “average advanced” track aren’t achieving more than 50% proficiency, does it?

Trigonometry is harder than memorizing math facts–We should take to heart the Wise Words of Barbie. Math achievement will fall off as the courses get more challenging. Students who excelled at their times tables and easily grasped fractions might still struggle with complex numbers or combinatorics.

So if you ask me—and no one does. Hell, no one has even really noticed the fall-off—it’s a combination of test design and subject difficulty.

Whatever the reason, the test score falloff has enormous implications for those who are banking on Rocketship Academy, KIPP, and all those other “proven” charters that focus exclusively on elementary school children.

Elementary school test scores are false gods. We have no evidence that kids who had to work longer school days simply to achieve proficiency in fifth grade reading and math will be, er, “shovel ready” for algebra and Hamlet. KIPP’s College Completion Report made no mention of its college students SAT scores, or indeed made any mention of demonstrated ability (e.g., AP tests), and color me a cynic, but I’m thinking they’d have mentioned both if the numbers were anything other than dismal.

So let’s assume that those Rocketship scores are solid (and I do). So what? How will they do in high school? Where’s the follow through? Everyone is banking on the belief that we can “catch them early”. Get kids competent and engaged while they are young, and it all falls into place.

Fine. Just let me know when the test scores back up that lovely vision.

Black teachers, teacher quality, and education reform, revisited

In the interest of focus, I left a few things off my original post.

First, and most importantly: teaching is insanely complicated and non-linear. For me, it’s a free-form high-wire act on a daily basis, interspersed with bouts of intense mental activity outside of class as I try to figure out the best way to explain complicated subjects to kids who don’t want to be there and often don’t have the requisite skills to master the material. But for others, it’s a highly structured daily routine in which each lesson is planned out months in advance. I know many math teachers who have each problem worked out before they teach it (in many cases problems they’d done for years); I often make up my problems as I write them on the board, so I can carefully calibrate the difficulty level based on the students in that class. I know history teachers who have tremendous difficulty lecturing without index cards and can’t casually lecture on any topic in their curriculum without preparation; I can go from complete unfamiliarity to ready to talk in 2 hours.

Many non-teachers visualize their ideal teacher as someone more like me, except a lot younger–smart, fluid, an expert on anything a student might want to inquire about, and particularly an expert in the subject taught. Highly educated elites, in particular, have romantic notions of their little snowflakes being taught by a bright Harvard graduate who went to a top 50 school and wants to help the next generation be as enamored by learning as she is.

Given the tough times I have finding jobs, I encourage all those who hold these romantic notions to find the evidence that teacher IQ and general intelligence is dispositive in successful teaching. (It must be said, however, that principals don’t like teachers that are too smart. Or too old.)

Much as I’d like a world that makes it easier for me to find a job, though, I think the reality is much less comforting to those who want “smarter” teachers. Certainly, research has provided little comfort. When the best news available tells us that teachers in the 95% percentile get very small improvements over the very worst bottom-dwellers, and any improvement at all is considered great news because it’s so hard to find any teacher quality criteria that show any increase at all….well, it’s just possible that being smart ain’t all that.

I really understand the intuitive belief that smarter teachers are better teachers, because I used to hold the same notion, when I was a suburban parent. But that belief was shaken even before I became a teacher and realized that teachers weren’t complete morons and, furthermore, that the demonstrated knowledge requirements for teachers took a sharp increase after NCLB, particularly for elementary school teachers. Huge. So big that, if teacher competency were a factor, we should have seen some improvement in teacher outcomes. Instead, recent research shows, again, that experience boosts performance a bit and new teachers are still weakest of all. I’m open to having my mind changed on that one because no research has specifically tested on this point. But again, the boosts in demonstrated ability were huge in many states, and shouldn’t we have seen some improvement in performance?

But what we got for sure were far fewer black and Hispanic teachers.

Notice that the fraud ring involved existing teachers, teachers who were probably caught in the NCLB net and forced to re-qualify for existing positions. As always, ETS explains this with a helpful graphic (I’ve combined text and image from page 16. ETS has excellent data. That’s why everyone trying to push an educational policy ignores it.)

In 2001, NCLB required teachers to be fully credentialed, forcing many existing teachers with emergency credentials to pass a Praxis test. Many black teachers couldn’t. Hence, the fraud. This wasn’t simply a case of wannabe teachers, but actual teachers, teachers with jobs. What if they were considered competent teachers?

There’s a really interesting study idea: test the outcomes of the teachers who committed fraud, compare them to white and black teachers who passed the test legally. What if they do well?

I’m not fuming at the double standard revealed by the reformers who scream about “unqualified teachers” but duck and cover when black teachers are found to have committed fraud. I am, however, annoyed that reformers are constantly promoting a lie, or at least a fantasy unsupported by research, and they don’t even have the balls to hold consistently to that position. Instead, they wilt and run away from the Clarence Mumford case. They never seem to commit, exactly, to the qualifications teachers should have, and how the current tests fall short.

Why? Because failure to commit to a line in the sand allows them to skate on two points. First, the minute they draw that line, they will be ferociously questioned about the impact their standards will have on black and Hispanic teachers. Race is an area that reformers are absolutely determined to avoid, unless it’s an opportunity to call teachers racist for the uneven performance results, of course.

Furthermore, the minute they draw that competency line, they will be forced to confront the fact that, as I’ve said with some frequency, research doesn’t support their claims. It’s hard to argue for changes that will further obliterate the population of black and Hispanic teachers when you can’t prove being smart makes that much difference—and that the teacher’s race seems to matter.

So instead, reformers prate endlessly about incompetent, mediocre teachers who aren’t anything but white, of course, talking cheap about improving standards while fleeing in terror from the tiniest suggestion that raising teacher test scores will disparately impact black and Hispanic teachers.

It’s too bad, because if they stood up for their beliefs, we could have a meaningful discussion about where, exactly, the line is for teacher competency. One reasonable interpretation of the research thus far is that we are well above the line needed. Bad news for reformers, if so.

I keep wondering about one other possibility, which might explain why a low or failing score on a basic skills tests could nonetheless belong to an effective elementary school teacher.

Maybe they’re underperforming.

My years in test prep have shown me a number of oddities, including more than a few African American kids (by far, the smallest percentage of my demographic), who have a terrible time reading and thinking “on their feet” (that is, a general skills test), and can’t think abstractly at all, yet have very strong demonstrated abilities that they’ve internalized.

Two different African American girls have had exceptional writing skills with a strong demonstrated vocabulary (both got perfect scores on their essay), but struggled to break 500 on the SAT verbal. One of them got a 4 on the AP US History test. More than one boy (including some Hispanics) can do relatively complex math word problems beautifully but, given a simple equation, can’t isolate x. I had one kid who could not solve 4x -3 = 21, but if you asked him what number you could multiply by 4 and subtract 3, and get 21, he’d say “6″ before I’d worked it out myself.

I’ve had more than a couple kids ask me why they were being tested on history when they’d never studied it in school. It took me a while to realize they thought the questions on the reading passage were questions they were supposed to know offhand. Their reading scores shot up (from say, the 3rd percentile to the 35th or 40th) when I explained that the big chunk of text on the right had the answers to the questions. They had no idea. And although they did better, they still complained that they wanted to be tested on “what they know” rather than “learn new stuff on the test”. Yes, that sort of thinking is completely alien to me and yes, it’s still pretty common.

In other words, I wonder if maybe crystallized vs. fluid intelligence impacts test scores on the bottom half of the bell curve. This might explain why relatively low skilled people have difficulty showing that knowledge on the test, but can be effective classroom teachers to young kids.

So I’m not as ready as I was five years ago to say that people who can’t pass a basic skills test don’t, in fact, have sufficient mastery of basic skills.

I’m not excusing the fraud. But it infuriates me that everyone’s ignoring it, because the conversations it would kick up are conversations we need to have—and, of course, they’re conversations we’re afraid to have.

What’s the difference between the SAT and the ACT?

I couldn’t find anything terribly wrong with this Ed Week article. But it didn’t offer anything terribly useful, either,so I thought I’d offer up some facts that might do some good.

Historically, the ACT was the test for the Midwest and South, and the SAT was the test for the coasts, but after the 2005 SAT changes, the ACT’s test population caught up. Both tests are given to around 1.6 million students.

Test Content

The ACT tests the same fact base as the SAT. It’s about 20 minutes shorter than the SAT, although it has far more questions and four sections:”

• English: 45 minutes, 5 passages of 15 questions.
• Math: 60 minutes, 60 questions.
• Reading: 35 minutes, 4 passages of 10 questions.
• Science: 35 minutes, 7 passages of 4-8 questions (40 total).

The ACT section times are brutal, which is why the ACT benchmarks purporting to report on college readiness should be taken with a healthy dose of salt. In my view, they dramatically underreport the reading, science, and (to a lesser extent) math ability of the lower to mid-range “college” students (keeping in mind that these kids shouldn’t be in college anyway, but that’s a different story).

Each section is scored on a scale of 1-36. The sections are then averaged for a Composite score, which is every bit as useless, really, as the SAT total. Colleges use the section scores far more than is generally known for placement in or out of remediation.

How do you convert ACT scores to SAT?

The University of California used to offer a direct conversion. One sign of the ACT’s growing popularity is that both tests are now converted to a “UC score”.
Roughly, a 21 on any section is the ability equivalent of a 500 on the SAT, a 26 is a 600, and a 31 a 700. However, a one to one combination isn’t possible, with 4 ACT sections and 3 SAT sections.

The UC conversion adds two-thirds of the math/reading/science total to the English/writing combined score. This weights the converted score towards English–rather unfairly, in my view, but not enough to do serious damage.

Which is more closely aligned to school curriculum

Both test knowledge and abilities that students should have mastered in school; the ACT doesn’t directly test science, but content knowledge will make the questions more familiar. The ACT also tests slightly more math: trigonometry, analytic geometry (circle and ellipse equations), and the occasional matrix question. Neither tests specific content knowledge in history, science, or English; for some reason, people say the ACT does. They are wrong.

Which test should students take?

Most students will score in roughly the same percentile on each test. However, some students have strong preferences for the ACT.

Low to mid-tier students are almost always better off with the ACT, something that I wish more do-gooder organizations understood. Much of the SAT’s difficulty is front-loaded–a big challenge in many questions is simply figuring out what the question is. The ACT actually tests more material but its questions are more straightforward. Any student who prefers the concrete to the abstract should consider the ACT, and most low to mid ability students will have a preference for the concrete. However, see the caveat below regarding reading abilities.

Students with SAT section scores in the high 600s/low 700s should always check out the ACT. The 2005 SAT changes reduced the number of questions in each section by 10%, and the cuts were primarily from the higher-difficulty questions. Many students in the mentioned range are every bit as bright as those getting 760+ scores, but are less detail-oriented, and usually make a few unforced errors. They used to make up the difference with their performance on the really difficult problems. Fewer difficult problems, slightly lower scores. (I am nearly certain that the reduced number of questions caused the decline noted when the SAT was changed in 2005.)

The ACT has far more questions than the SAT–215 to 171–and has no “guessing penalty”, which gives high ability students who make the occasional unforced error a significant advantage. To give an example: my son took the old SAT as an early junior and got 690 M, 660 V. I expected him to get high 600s, low 700s on the new one, which he took in March 2005. He received 630s across the board. After working on his accuracy, he took it again and received a 690,690, 670, or 2050.

His ACT scores were English 34, Math 34, Reading 36 (a perfect score), Science 29, which in SAT terms is high 700s across the board, or a 2250 using the UC conversion. At his performance level, that’s a huge boost. I have other anecdotal evidence, but they aren’t my kids so I can’t discuss specifics. Without question, all high ability kids should take both to see if they have a preference.

If taking both, which prep class should I take?

High ability students: take the SAT prep course. First, there are exponentially more SAT classes than ACT, even now. Asians, the primary consumers of test prep courses, don’t seem to take the ACT much (at least around here). Another major consumer, schools offering classes for their own students, also seem ignorant of the ACT.

Moreover, moving from the SAT to the ACT is far more organic than the other way round; the SAT has far more tricks and tidbits that a good test prep teacher can help with. Practicing for the ACT is little more than learning how to work fifty times faster on everything or, if that’s not possible, devising a strategy for getting as much done as possible. Did I mention the brutal timing requirements of the ACT? Oh, well, it bears repeating.

Low to mid-ability students: anyone planning a class aimed to low income, low ability students should select the ACT. Students with weaker abilities will receive more useful instruction, as it has fewer test-specific tricks and the test prep instructor will spend more time on content.

Who Shouldn’t Take the ACT?

The ACT is reading intensive–three of the four tests involve reading comprehension and two of those sections have (here it is for the third time) brutal time requirements. Students whose reading skills are significantly out of alignment with their other abilities (e.g., dyslexia, reading LDs), may want to stick with the SAT.

The problem with fraudulent grades

A while back I wrote a rant about homework and grades and the impact that the former has on the latter. In that case, I was primarily referring to the many students who fail or get low grades despite reasonable demonstrated ability.

But the flip side of that, particularly in urban schools and majority-minority charters, are the students who get As for little more than working hard. Some of them find out in college, like Darryl Robinson, who got a full ride to Georgetown despite weak academic skills. Some of them find out when they sign up for SAT test prep, like Angela Lopez.

For the past 8 years, I’ve taught a spring ACT prep class to underprivileged kids. Certain clear patterns emerge. Assume that the students in these next paragraphs are Hispanic.

On average, the best-prepared pool of students in my class, year after year, come from the comprehensive high schools, whose demographics are usually 65-35 Hispanic to white. These students are usually in Calculus or Pre-Calc, and in AP or IB programs. Their skills are in line with their transcripts. The weaker comprehensive high school students in my class have transcripts with GPAs below 2.0, and are in Geometry or even Algebra. Their skills are usually stronger than their transcripts.

The students from two charter schools fill out the middle and bottom tiers of my classes—with the occasional rock star as an exception. One charter is a well-regarded chain. The kids’ transcripts aren’t outright lies; most of them are in Geometry or Algebra II as juniors. They’re on the low end of the achievement spectrum, and have abilities equivalent with the low-end students from the comprehensive schools, but with better GPAs.

The other charter, sponsored by the local Super Prestigious University, routinely provides the least-prepared, weakest students in the class, with a shocking disconnect between abilities, transcripts and grades. As a rule, the strongest kids from this school are barely average in my class, while the weakest kids are clearly outclassed by the low-performing comprehensive school students, even though on paper, the charter school students are 2-3 years ahead in math and science.

Yet the kids from the university charter school are just as likely to be accepted to the top universities as the top kids from the comprehensive schools, despite huge gaps in demonstrated ability. Needless to say, they are far more likely to gain access to good schools than the kids from the corporate charter or the lower-ability kids from the comprehensive schools, even though the latter have equivalent or better abilities.

Lying gets the job done. Majority minority schools, either charter or comprehensive, frequently deliver watered-down courses and grade more on effort and “social justice” than ability*. Their students, armed with fraudulent transcripts, get into decent or excellent schools and then are shocked to learn how little they know. That’s what happened to both Darryl and Angela; Angela’s adviser, Pablo, might have made his students feel better, but that reassurance isn’t helping her reading scores.

Some of the students I run into from these schools are like Angela—at worst, mildly discombobulated by the realization that, far from being top students, they’re distressingly close to the bottom. A few minutes with a cheerleader like Pablo is enough to banish all fears. They don’t improve much, but are convinced it won’t matter. They embrace the lie. I worry they are mostly doomed at college.

A few students have known they’ve been lied to, but are willing to accept the lie to improve their prospects. Aware that their transcripts and grades are a scam, they’re secretly terrified at what they will learn when taking the ACT or the SAT, since those tests won’t lie about their abilities. And yet they step up to reality, grit their teeth at the early practice scores and work through their fear to get the best score they can. These kids, I would slit a vein for. Happily, they almost always improve modestly, and their delight at these relatively small bumps is one of the great rewards of the job. While not one of them has ever been a super-star, they were all accepted into state schools with realistic goals and a realistic shot of making it through remedial classes.

But every so often I get a student who didn’t know it was a lie and can’t accept the rationalizations that worked for Angela. These are the ones that give me nightmares. I’d rather Angela’s smug acceptance of her inferior abilities than the despair of a kid who suddenly has to face the fact that his entire self-image as a star student has been a lie. The three students I’ve seen in this category all gave up on trying to improve their scores. All of them went on to college, one of them probably graduated. But emotionally, the discovery of their true abilities just wrecked them and it was painful to watch.

Some people read this story and think that, with a proper education, these kids could have really been ready for college. I disagree. The Super Prestigious University charter has, on one notable occasion, turned out a genuinely high-achiever—an illegal immigrant who had been in this country only 4 years when I met him, who got a 1900 on the SAT and a 27 composite on the ACT. Now, he came to the school with strong record of real achievement, but the school taught him advanced math and improved his reading and writing skills. Why? Because he was a bright kid who could take advantage of the education.

When you read of the Darryls and Angelas, it’s best to assume that their weak skills are a product of cognitive abilities that just aren’t up to more. The problem isn’t that they were capable of more, but that their teachers and schools lied to them. That may not always be the case—If Darryl isn’t exaggerating, then he’s managing calculus and biology at the college level. I’d want to know his SAT scores to be sure. But most of the kids in these cases are of below average intellect, hard workers who were fed a fraud.

But of course, it’s much easier to blame the teachers.

One more thing to chew on: The Angelas and Darryls will often be pulled through school by virtue of a huge, expensive support system, similar to the one that is helping Dasmine Cathay, a mediocre, illiterate football player who will quite possibly graduate from his Tennessee state college. This support system just perpetuates the fraud through college, further devaluing the high school degree.

But if you think the support system that hauls low-level kids to a college degree is the right idea, remember this: the kids from comprehensive high schools that I described above, the ones with stronger skills and better test scores but a terrible GPA, (thanks to teachers who, like the charters, grade on homework but have far less compliance) will probably end up in community college if they’re lucky. They will see little in the way of a support system and few people will call them to make sure they get to class.

I am not convinced that we should be doling out support so unevenly, and I am certain that students should not benefit from greater support simply because their schools were willing and able to lie about their abilities.

*While I’m picking on majority-minority schools here, a similar problem is found in suburban progressive charters, which mouth platitudes about diversity and social justice but are primarily interested in boosting white middle-achiever college prospects by inflated transcripts and grades. The parents of these students are a big part of their donation base.

Still progressive. But smarter with it.

Yong Zhao, author and education professor, is asked to predict the education landscape after 5 years of Common Core:

Question: What will be different five years from now if the current plans go forward?

Yong Zhao: It’s always dangerous to predict the future. But if history is any indication, judging from the accomplishment of NCLB and Race-to-the Top, I would say that five years from now, American education will still be said to be broken and obsolete. We will find out that the Common Core Standards, after billions of dollars, millions of hours of teacher time, and numerous PD sessions, alignment task forces, is not the cure to American’s education ill. Worse yet, we will likely have most of nation’s schools teaching to the common tests aligned with the Common Core. As a result, we will see a further narrowing of the curriculum and educational experiences. Whatever innovative teaching that has not been completely lost in the schools may finally be gone. And then we will have a nation of students, teachers, and schools who are compliant with the Common Core Standards, but we may not have much else left.

He then goes on to argue that minorities do poorly because they need more government support, that tests aren’t accurate evaluators (really, what does it mean to be “good” at something?), and that students really should write about what they feeeeeeeeeel instead of what they think. In short, he’s a touchy-feely progressive who has all sorts of loopy ideas.

But hey. He’s right about the Common Core standards.

One of my great surprises of the past year or so is the emergence of progressives as, dare I say, realists on achievement, in contrast to the eduformers’ edging ever-nearer to a role as totalitarian dreamers.