False Positives

I quit writing about tests. And test prep.  Five, six, years ago? I still taught test prep until this year, always giving in to my old employer’s pleas to teach his Saturday classes. But I largely quit the SAT after the last changes, focusing on the ACT. I still love tests, still enjoy coaching kids for the big day.

Explaining why has been a task I’ve avoided for several years, as the doubt is hard to put into words. 

It was an APUSH review course, the last one I taught, I think. Class hadn’t started for the day, but one of my five students was sitting there highlighting notes. She was a tiny little thing, perky and eager but not intellectually remarkable and it was March of what would have been her junior year.

“This is my last test prep course. I’ve taken the SAT for the fourth time, took AP Calculus BC last year, and I’m all done.”

“Yay! How’d you do on the SAT?”

“2400,” she said, casually. “I got 2000 the first time, but I spent the whole summer in two prep courses, plus over Christmas.”

Boom.

Like I said, she was….ordinary. Bright, sure. But her APUSH essays were predictable, regurgitating the key points she’d read in the prep material–pedestrian grammar, too many commas. Her lexile level was unimpressive. Nothing terrible. I gave her some tips. 

This girl had placed in the 99th percentile for the SAT but couldn’t write a grammatically complex sentence, much less an interesting one. Couldn’t come up with interesting ways to use data (graphs, statistics). Couldn’t accurately use the words she’d memorized and didn’t understand their nuance in reading text

She was a false positive.

I’ve known a lot of high scoring students of every ethnicity over the years–and by high scoring, I mean 1400-1600 on the 1600 SAT, and 2200-2400 on the 10 years with the three tests. 5s on all AP tests, 700+ on all Subject tests. Until that conversation, I would have said kids had high test scores were without exception tremendously impressive kids: usually creative, solid to great writing, opinionated, spotted patterns, knew history, knew the underlying theory of anything that interested them. I could see the difference, I’d say, between these kids and those slightly lower on the score scale–the 1200s, the kids who were well rounded with solid skills who were sometimes as impressive, sometimes not, sometimes a swot, sometimes a bright kid who didn’t see much point in striving.

Every time saying it, though, I’d push back memories of a few kids who’d casually mentioned a 5 score, or a 1600 or 2400, that took me aback. That particular kid who didn’t seem all that remarkable for such a high score. But in all these cases, I was only relying on gut instinct and besides, disappointingly high IQ folks exist.   For every Steven Hawking there’s a Ron Hoeflin. Or a Marilyn vos Savant, telling us whether or not larks are happy.  Surely the test would sometimes capture intellect that just wasn’t there in the creative original ways I looked for. Or hey, maybe some of those kids were stretching the truth.

But here, I had my own experience of her work and her scores were easily confirmable, as my employer kept track (her name was on the “2400 list”, the length of which was another shock to my prior understanding). She got a perfect score despite being a banal teen who couldn’t write or think in ways worthy of that score.

Since that first real awareness, I’ve met other kids with top 1% test scores who are similarly…unimpressive.  98+ percentile SAT scores, eight 5 AP scores, and a 4.5 GPA with no intellectual depth, no ability to make connections, or even to use their knowledge to do anything but pick the correct letter on the multiple choice test or regurgitate the correct answer for a teacher. Some I could confirm their high scores, others I just trusted my gut, now that I’d validated instinct. These are kids with certainly decent brains, but not unusually so.  No shame in that.  But no originality, not even the kind I’d expect from their actual abilities. No interest in anything but achieving high scores, without any interest in what that meant.

It probably won’t come as a shock to learn that all the kids with scores much higher than demonstrated ability were born somewhere in east Asia, that they all spent months and months learning how to take the test, taking practice tests, endlessly prepping.

The inverse doesn’t hold. I know dozens, possibly hundreds, of exceptional Asian immigrants with extraordinary brains and the requisite intellectual depth and heft I would expect from their profile of perfect SAT scores and AP Honors status. But when I am shocked at a test score that is much higher than demonstrated ability, the owner of that score is Chinese or Korean of recent vintage. 

I don’t know whether American kids (of any race) could achieve similar scores if they swotted away endlessly. Maybe some of them are. But my sample size of all races is pretty high, and I’ve not seen it.  On the other hand, I’m certain that very few American kids would find this a worthwhile goal. 

Brief aside: when I taught ELL, I had a kid who was supposedly 18. That’s what his birth certificate said, although there’s a lot of visa fraud in Chinese immigrants, so who knows. He didn’t look a day older than fourteen. And he had very little interest in speaking or learning English. Maybe he was just shy, like Taio, although I’d test him every so often by offering him chocolate or asking him about his beloved bike and he showed no sign of comprehension. But then he’d ace multiple choice reading passages. Without reading the passage. He had no idea what the words meant, but he’d pick the right A, B, or C, every time. I mentioned this to the senior ELL teacher, a Chinese American, and she snorted, “It’s in our genes.”

I don’t think she was kidding but the thing is, I don’t much care how it happens. If American kids are doing this, then it changes not a whit about my unhappiness. It’s not a skill I want to see transferred to the general teenage American population. (That said, the college admissions scandal makes it pretty clear that, as I’ve said many times, rich parents are buying or bribing their way in, not prepping. And unsurprisingly, it appears that Chinese parents were the biggest part of his business.)

Now, before everyone cites data that I probably know better than they do, let me dispatch with the obvious. Many people think test prep doesn’t work at all. That was never my opinion  When people asked me if test prep “worked”, I’d always say the same thing: depends on the kid. “Average score improvement” is a useless metric; some kids don’t improve, some improve a bit, some improve a huge amount. Why not pay to see if your kid improves a lot? But I also felt strongly that test prep couldn’t distort measured ability to beyond actual ability, and I no longer believe that.

But I didn’t believe what critics at the time said, that test prep worked…..too well. I didn’t believe that false positives were a real problem. And the terrible thing is–at least to me–is that I still believe normal test prep is a good thing. Distortion of ability, however, is not.

As the push to de-emphasize tests came, as test-entry high schools came under attack, as colleges turn to grades only–a change I find horrifying–I could no longer join the opposition because the opposition focused their fire almost exclusively on their dismay at the end of meritocracy and the concomitant discrimination against Asian immigrants. I oppose the discrimination, but I no longer really believe the tests we have reliably reveal merit to a granular degree. The changes I want to see in the admissions process would almost certainly reduce Asian headcount not by design, but by acknowledging that specific test scores aren’t as important.

I have other topics I’ve been holding off discussing:

• why I support an end to test-based high schools in its current form
• why we still need tests
• how the SAT changes made all this worse
• how the emphasis on grades for the past 20 years has exacerbated this insanity
• why we need to stop using hard work as a proxy for merit

But I needed to try, at least,  to express how my feelings have changed. This is a start. It’s probably badly written, but as you all know, I’ve been trying to write more even if the thoughts aren’t fully baked, so bear with me.

 

 

Why Chris Hayes Fails

Chris Hayes has a book to sell and guilt to expunge. The poor lad feels guilty that he benefited from the Evil Mostly White Meritocracy:

But the problem with my alma mater is that over time, the mechanisms of meritocracy have broken down. In 1995, when I was a student at Hunter, the student body was 12 percent black and 6 percent Hispanic. Not coincidentally, there was no test-prep industry for the Hunter entrance exam. That’s no longer the case. Now, so-called cram schools like Elite Academy in Queens can charge thousands of dollars for after-school and weekend courses where sixth graders memorize vocabulary words and learn advanced math. Meanwhile, in the wealthier precincts of Manhattan, parents can hire $90-an-hour private tutors for one-on-one sessions with their children.

By 2009, Hunter’s demographics were radically different—just 3 percent black and 1 percent Hispanic, according to the New York Times. With the rise of a sophisticated and expensive test-preparation industry, the means of selecting entrants to Hunter has grown less independent of the social and economic hierarchies in New York at large. The pyramid of merit has come to mirror the pyramid of wealth and cultural capital.

Here, Hayes is relying on the cheapest and most meretricious of the education myths: the rich have the ability to improve their test scores, SAT or otherwise, through expensive test prep, while the low income blacks and Hispanics do not. The higher scores are not genuine, and thus the acceptance is not truly meritocratic.

There’s just one tiny glitch in this mythology:

Blacks and Hispanics are more likely to use test prep than whites. Cite, cite, and oh look, this cite has a table:

Use of Test-Prep Courses and Gains, by Race and Ethnicity

Group	% Taking Test-Prep Course	Post-Course Gain in Points on SAT
East Asian American	30%	68.8
Other Asian	15%	23.8
White	10%	12.3
Black	16%	14.9
Hispanic	11%	24.6

The idea that blacks and Hispanics don’t have access to test prep is some sort of delusion that all the reality in the universe can’t shake out of progressives.

Within a ten mile radius of my home, at least 10 organizations are dedicated to providing free test prep, college admissions advice, and academic support to low income, first generation college blacks and Hispanics. Double the radius and the count will be in the dozens, if not hundreds–as it probably is anywhere in America. Any low-income black or Hispanic who wants SAT/ACT test prep and thinks he or she can’t afford it is the victim of criminally ignorant high school advisors–and the facts suggest that this isn’t a big problem.

Low income whites are a different story; few charitable organizations are dedicated to improving their test scores. Of course, given that low income whites trounce high income blacks on the SAT (Cite, cite, and
cite), I guess maybe organizations figure there’s no point making the gap worse? But of course, the very fact that poor whites outscore wealthy blacks pretty much kills whatever remained of Hayes’ theory about the test score advantage of the rich and powerful.

Furthermore, as Steve Sailer and commenters to Hayes’ article point out, Hayes complete ignores another reality: the huge shift in Hunter College High School demographics isn’t so much from low income to high income, but from whites to Asians.

If you read of a school that’s suddenly moved to elite status or seen a dramatic rise in test scores (e.g., AIPCS), or heard that a test prep process has gotten out of control, it’s a sure thing that it’s become “an Asian school”, as we call them in my area. Once a school “goes Asian”, hitting a tipping point of about 40%, it’s a short step to 60-80%. Check out the top-scoring comprehensive high schools by SAT average, and the highest ones will be “asian schools”. They end up Asian because of white flight. It’s not that whites don’t like Asians, but their kids will lose access to AP/honors courses and get lower GPAs—not because they have lower abilities, but because the white parents haven’t managed to convince their kids that the world will end of they don’t get straight As. Donations, as a rule, decline with this demographic change, which is why wealthy school districts get more than a little annoyed when their schools are at risk of “going Asian”, and come up with all sorts of odd rules to discourage it (giving up class ranking or limiting AP grade bumps).

Hayes engages in yet another fiction (and that’s just in this excerpt!): that through test prep, the rich are distorting their abilities. The poor and the rich have similar abilities in a purely meritocratic world but thanks to test prep, the rich are making themselves look smarter, even though it’s a mirage.

Clearly, that can’t be true, or rich blacks would have higher test scores.

But here I will bring in personal experience in test prep. For the past nine years, I’ve been preparing students for the SAT, the ACT, the Subject tests (Math, Histories, English Lit), the high school admissions tests (HSPT, ISEE, SSAT), and all grad school tests except the MCAT (although this last not as much as I used to). I do this both through private instruction institutions (Kaplan in the past, an SAT academy now) and private tutoring (with rates in line with those in tony Manhattan, apparently). I work with Asians of all income levels, wealthy and upper income whites (as well as middle income whites in my Kaplan days), low income Hispanics, and low income African Americans.

In other words, unlike many people who yammer on about test prep, I actually have some experience preparing people of all races and all demographics for all sorts of tests, and will draw upon that experience to assert this as fact: test prep primarily helps people use their existing abilities more effectively. With some people, the bump is huge, with others it’s minimal, with still others, non-existent. In only a very few cases are students actually distorting their abilities by improving their test scores, but rather showing their abilities in the best possible light.

Is it possible to game the test, to prep so much that the score is a blatant misrepresentation? Yes, but it’s rare. The people who are most likely to do this are not the rich of any color, who can buy their way into whatever school they want. And it’s not low income blacks or Hispanics, who I’ve coached and seen huge increases that still only bring the majority of the kids to just below national averages. It’s certainly not middle-class or low income whites, who are clearly the least likely to even use test prep.

No, the students who might be actually distorting their abilities through test prep would most likely be Asian. (Please note that this statement is only assuming such distortion is possible.) I work at an Asian SAT “cram school”, teaching book clubs and math enrichment. Their parents call it “SAT school”, even though the kids are rising freshmen and sophomores for book club, and rising seventh and eighth graders for geometry, because as far as the parents are concerned, the kids are doing this as part of a five year program to improve their SAT scores. Junior summer, they are in SAT boot camp: 20 hours a week (plus a test) for 10 weeks in the summer, and then Saturday school until the test.

The kids I’m working with, dozens of hours per year, aren’t distorting their abilities, but going through all that work for the last 10 or 20 points possible of their score range. That’s leaving aside the Korean cram schools, which somehow enable kids with limited English skills to score an 800 on the SAT reading section. Now that, I would argue, is distortion.

Unfortunately for Hayes, though, these Asians aren’t rich. Wrong again.

Hayes is correct about one thing, though: the elites are locking out the hoi polloi from highest-level institutions. But it takes a real ignorance to pretend that the rich are doing this because of over-reliance on test scores or test prep, as opposed to buying their way in, using their powerful networks to only hire from the “right” schools, and the fuzzy math of the “holistic” evaluation process. Give me test scores any day.