SAT’s Competitive Advantage

Asians taking the SAT: 196,030 out of 1,640,047, or about 11.9%.
(No response: 62,603)

Asians taking the ACT: 71,677 out of 1,799,243, or about 3.9%
(No response: 110, 305)

I can’t tell if these numbers are US-based only, or including overseas testers. The ACT has a small international market. The SAT dominates, particularly in the Asian countries. 50,000 Chinese visit a Hong Kong testing site in just one year. While I couldn’t find totals for Korea, 900 students got their scores cancelled in 2007, clearly just a fraction of the total testers. In India, the College Board has 32 test centers (up from 20 just four years ago), the ACT has 60 testers.

So if the ratio above, in the tests’ national reports, are indeed just for the US, then the overall Asian preference is even greater than the 2.75:1 brand preference for Asian American testing. Since I can’t find any verification of this, let’s assume the ratio is overall preference.

In either case, the numbers are stark. The SAT is already losing out to the ACT. Why would it make changes that, if accurately described, would make the test less attractive to Asians, who have largely rejected the ACT?

Before I get to that: outside of the knowledgeable education reporters, the coverage has been nonsense. Contrast this relatively solid piece by NYTimes reporter Tamar Lewin, (whose twitter tagline should be “not half as ignorant and less than a quarter as irritating as Motoko Rich”) with this obsequious David Coleman profile by author Todd Balf. Discussing his work, Balf described the “checks” that restrain the College Board from using its power, :

Either Coleman does something about [the dissatisfaction with the SAT], or at the college level more and more schools will go test optional. Coleman wouldn’t say that the reforms are in response to places like Wake Forest, but it is hard to ignore.

Sigh. A few thousand colleges or university systems use the SAT as a first round placement test: University of Georgia, University of California, University of Michigan, University of Nevada, University of Wisconsin, a whole bunch of community college systems, and oh, yeah, the largest university system in the country. Any number of private universities do, as well, including Baylor, Columbia, and Duke.

So the SAT and ACT allow many schools to avoid the hassle and expense of placing hundreds of thousands of students each year.

But yeah, the College Board is all worked up about Wake Forest going exam optional.

People who try to prognosticate the changes are mostly being silly. I like Peter Wood, but his rant here doesn’t add up. Like most non-experts, Wood completely ignores the existence of the ACT, which has the same “biases” that Wood thinks the SAT is trying to fix. The College Board can’t “dumb down” the test. If blacks suddenly started scoring at the 80th percentile on the SAT, but were still scoring at the 35th on the ACT, people would notice.

So leaving aside what people who don’t really understand things think, and leaving my random thoughts about certain changes for another day, and then pointing out that I liked this analysis by Eduwonk, not usually a favorite of mine, my big question is still why would the College Board deliberately screw over Asians, the customer base with the strongest preference for its product?

The Chinese test prep companies are less than thrilled about the changes, which they believe will prevent them from teaching students to game the test . I am asserting this as fact: many Chinese international students openly acknowledge they don’t have the underlying ability their SAT scores denote.

Not much outrage from the Koreans, but then they just buy early versions of the test and distribute it to paying customers. Obviously, this method works regardless of underlying test modifications. Cheating is so rampant in South Korea that again, I am asserting as fact that many of these testers lack the underlying knowledge that their SAT scores supposedly indicate.

I doubt either country is exclusive in its approach. Presumably, the Chinese cheat and the Koreans game.

Indians seem cautiously optimistic—hey, they mostly do speak English, right?

I consider the SAT changes completely irrelevant. The interesting question to me is, assuming no one changes their behavior, how will this all play out?

The one thing I feel pretty confident of: faith in Coleman is wholly unjustified. Leaving aside the value of his Common Core work, note the lack of follow-through. Typical consultant, he jumped in long enough to tell people what to do, then left before all the errors in his ideas are revealed through disastrous implementation results.

In the same way, Coleman’s never run a major business, and has no idea about the political realities of helming the most visible, if not the most popular, college admissions and placement test. He has a story, but no plan. Meanwhile, boy, he’s happy to diss the test his predecessors pushed—and he makes free with insulting his competitor, too.

But for all Coleman trashes his predecessor, Gaspard Caperton, the ex-Appalachian governor has big shoes that I very much doubt Coleman can fill. Complain all you like about the writing section, it was pure genius as a business decision. On Caperton’s watch, the College Board dumbed down an existing test that a couple hundred thousand kids took, made it cheaper to grade, and forced a million kids to pay a higher price to take it as part of the “new” SAT, even though lots of colleges don’t use the section score—and let everyone think that the UC forced the change. The company made millions in a single year. At his behest, the College Board abandoned the claim that test prep doesn’t work, and sold its own product, which had to be taken off the Amazon bestseller list because everyone found it too upsetting. And that’s just the SAT’s profit center. Caperton also presided over the Advanced Placement’s stunning growth, achieved by taxpayers shoveling money into the CB’s coffers for entirely unqualified kids to take tests in order that Jay Mathews put their school on his Index.

In contrast, Coleman has just made the essay optional and harder to grade, ordered a complete redesign of the test with a stable competitor ready and willing to pick up the doubters, and pissed off his most dedicated customer segment. I wouldn’t be surprised if he declared that his purer company won’t profit from the filthy lucre of test prep, having turned that function over to Khan Academy (which long before its College Board arrangement has neglected to provide free services for the ACT). God knows what Coleman has planned for the AP suite.

Coleman could have some grand plan that I can’t anticipate (other than, as Steve Sailer puts it, all students should Be Like Me.) But reasonable people can and should wonder if he is wholly ignorant of the SAT’s market position, or if he actually believes that his Common Core curriculum is going to increase student ability and end the achievement gap. From there, it’s easy to postulate a scenario in which the College Board spends a fortune redesigning the test in a vacuum, not discovering until field tests that the changes either widen or narrow the achievement gap unforgiveably. Worse still, what if they make the test too hard? All of these possible outcomes wouldsend testers and their parents running into the arms of the Midwestern Mama, all safety and security.

In such an event, I trust the remainder of the College Board regroups before rollout. That’s been done before: the 2005 essay was originally supposed to be 50% of the writing score, but worrisome field tests led to the essay getting just a third of the section weight. A complete backout, cancel, reset is not impossible. The ETS cancelled its first set of GRE changes, despite a significant investment. Like the ETS, the College Board has a known quantity of a test to fall back on—just fire Coleman and move on.

I find that outcome very possible. But suppose that the SAT redesign works as promised. The achievement gap remains intact, the test is as described, the scores among American students are roughly compatible with the old SAT and the current ACT.

So then back to the big question: why, exactly, are Asians favoring the current SAT so strongly, and what will they do in the face of an altered test? The usual reason offered for their brand loyalty is the Asians aren’t aware of the ACT.

They figure out to game the SAT, they buy advanced copies of the SAT so their kids can cheat, they’re aware of and concerned about SAT changes, but somehow they’ve never heard of the SAT’s competition. Yeah. Not buying.

That leaves a few possibilities. First, the Asian test prep companies just don’t see the point in spending the money and time cracking a second test given their certainty on the first. Second, they haven’t actually cracked the test, aren’t really gaming it, but in fact are all using the Korean method of buying the test. All the coverage on the gaming is just sham. So they either can’t or haven’t yet paid to achieve the same penetration of the ACT’s secrets.

Third, and this is the fun one, they’ve tried cracking the ACT and can’t.

Eighteen months ago, I thought the possibility that Asians were artificially inflating their scores was theoretically possible, but unlikely. Over the next year, my experiences and additional research has changed my view. I now think it likely that both here and overseas, a decent percentage of Asian testers are either cheating or gaming the SAT and AP tests—-or both. I’ve written up some of this but not all, and I’m not expecting anyone to just take my word for it.

But when I start from that premise, and look at the SAT as a test that can be gamed, I see loads of potential.

Naturally, a purchased test is still the best guarantee—certainty being so much more reliable—but Big Data could identify many patterns. A few years back, you could see open discussion of an LA based test prep company with a largely Asian clientele that promised a reduced (300 or fewer words) vocabulary list, that was supposedly built by on analyzing previous tests and predicting a rotation cycle for the words. Or maybe they just had an in at the College Board, as the comment suggests.

Maybe they’ve analyzed each type of reading comprehension question and noticed a pattern beyond “pick C”, but rather a particular type of wording that is associated with a correct answer. This is something all bright people can do, of course, but it’s hard to teach as a system without mountains of data. The SAT’s grammar questions have all sorts of patterns that could likewise be broken down and systematized. Obviously, they’ve collected the essay prompts and found certain theme patterns that they use to teach kids to memorize essays verbatim. It goes without saying that the students didn’t write the essays. Or maybe they just had prior knowledge.

But assume, for the moment, that it was gaming, not cheating. I’m not an expert on Big Data or psychometrics, but I’m knowledgeable about both the SAT and the ACT, and the latter doesn’t lend itself to that sort of patterning. English, Reading, and Science test on passages with aggressive time requirements. While their questions do have patterns, the patterns are heavily reliant on context. You have to understand the text at least slightly in order to find the correct answer for the particular type of question. At least, that’s how it seems to me. (But then, I looked at the memorization necessary to game the SAT and thought it unlikely, so what do I know?)

Imagine that College Board successfully screws over the barely-English-speaking Asian market with their new test. Suppose also that both companies are ethical and no one is selling advance data on the test (which, in fact, I do assume). Suppose, in other words, that Asians with extremely limited English skills, here and abroad, are no longer able to misrepresent their abilities with SAT tests.

Is that really what colleges want?

Balf’s right about one thing–the colleges are the ultimate test customer. Public universities have dramatically increased their foreign admits, most of them Chinese, because they can charge them out of state fees for all four years.

They admit these students already knowing that the Chinese applications are largely fraudulent, and are well aware of the gaming and cheating, since they end up with students who can’t speak English and cheat here, too.

All this knowledge hasn’t slowed down their Asian international student admission.

Does this sound like a customer base that really wants to lock out non-English speaking Asians?

No matter which of these scenarios play out, I don’t see how the ACT doesn’t benefit. Why would any student opt for a complete unknown when they can prepare for the ACT using test prep materials and experts who’ve been working with the same test for years? I expect test prep companies to step up their offerings. If they don’t, then I throw up my hands and declare confusion.

This all strikes me as considerably more relevant than pipe dreams about the end of test prep, much less the achievement gap. And much more interesting.