SAT’s Competitive Advantage

Asians taking the SAT: 196,030 out of 1,640,047, or about 11.9%.
(No response: 62,603)

Asians taking the ACT: 71,677 out of 1,799,243, or about 3.9%
(No response: 110, 305)

I can’t tell if these numbers are US-based only, or including overseas testers. The ACT has a small international market. The SAT dominates, particularly in the Asian countries. 50,000 Chinese visit a Hong Kong testing site in just one year. While I couldn’t find totals for Korea, 900 students got their scores cancelled in 2007, clearly just a fraction of the total testers. In India, the College Board has 32 test centers (up from 20 just four years ago), the ACT has 60 testers.

So if the ratio above, in the tests’ national reports, are indeed just for the US, then the overall Asian preference is even greater than the 2.75:1 brand preference for Asian American testing. Since I can’t find any verification of this, let’s assume the ratio is overall preference.

In either case, the numbers are stark. The SAT is already losing out to the ACT. Why would it make changes that, if accurately described, would make the test less attractive to Asians, who have largely rejected the ACT?

Before I get to that: outside of the knowledgeable education reporters, the coverage has been nonsense. Contrast this relatively solid piece by NYTimes reporter Tamar Lewin, (whose twitter tagline should be “not half as ignorant and less than a quarter as irritating as Motoko Rich”) with this obsequious David Coleman profile by author Todd Balf. Discussing his work, Balf described the “checks” that restrain the College Board from using its power, :

Either Coleman does something about [the dissatisfaction with the SAT], or at the college level more and more schools will go test optional. Coleman wouldn’t say that the reforms are in response to places like Wake Forest, but it is hard to ignore.

Sigh. A few thousand colleges or university systems use the SAT as a first round placement test: University of Georgia, University of California, University of Michigan, University of Nevada, University of Wisconsin, a whole bunch of community college systems, and oh, yeah, the largest university system in the country. Any number of private universities do, as well, including Baylor, Columbia, and Duke.

So the SAT and ACT allow many schools to avoid the hassle and expense of placing hundreds of thousands of students each year.

But yeah, the College Board is all worked up about Wake Forest going exam optional.

People who try to prognosticate the changes are mostly being silly. I like Peter Wood, but his rant here doesn’t add up. Like most non-experts, Wood completely ignores the existence of the ACT, which has the same “biases” that Wood thinks the SAT is trying to fix. The College Board can’t “dumb down” the test. If blacks suddenly started scoring at the 80th percentile on the SAT, but were still scoring at the 35th on the ACT, people would notice.

So leaving aside what people who don’t really understand things think, and leaving my random thoughts about certain changes for another day, and then pointing out that I liked this analysis by Eduwonk, not usually a favorite of mine, my big question is still why would the College Board deliberately screw over Asians, the customer base with the strongest preference for its product?

The Chinese test prep companies are less than thrilled about the changes, which they believe will prevent them from teaching students to game the test . I am asserting this as fact: many Chinese international students openly acknowledge they don’t have the underlying ability their SAT scores denote.

Not much outrage from the Koreans, but then they just buy early versions of the test and distribute it to paying customers. Obviously, this method works regardless of underlying test modifications. Cheating is so rampant in South Korea that again, I am asserting as fact that many of these testers lack the underlying knowledge that their SAT scores supposedly indicate.

I doubt either country is exclusive in its approach. Presumably, the Chinese cheat and the Koreans game.

Indians seem cautiously optimistic—hey, they mostly do speak English, right?

I consider the SAT changes completely irrelevant. The interesting question to me is, assuming no one changes their behavior, how will this all play out?

The one thing I feel pretty confident of: faith in Coleman is wholly unjustified. Leaving aside the value of his Common Core work, note the lack of follow-through. Typical consultant, he jumped in long enough to tell people what to do, then left before all the errors in his ideas are revealed through disastrous implementation results.

In the same way, Coleman’s never run a major business, and has no idea about the political realities of helming the most visible, if not the most popular, college admissions and placement test. He has a story, but no plan. Meanwhile, boy, he’s happy to diss the test his predecessors pushed—and he makes free with insulting his competitor, too.

But for all Coleman trashes his predecessor, Gaspard Caperton, the ex-Appalachian governor has big shoes that I very much doubt Coleman can fill. Complain all you like about the writing section, it was pure genius as a business decision. On Caperton’s watch, the College Board dumbed down an existing test that a couple hundred thousand kids took, made it cheaper to grade, and forced a million kids to pay a higher price to take it as part of the “new” SAT, even though lots of colleges don’t use the section score—and let everyone think that the UC forced the change. The company made millions in a single year. At his behest, the College Board abandoned the claim that test prep doesn’t work, and sold its own product, which had to be taken off the Amazon bestseller list because everyone found it too upsetting. And that’s just the SAT’s profit center. Caperton also presided over the Advanced Placement’s stunning growth, achieved by taxpayers shoveling money into the CB’s coffers for entirely unqualified kids to take tests in order that Jay Mathews put their school on his Index.

In contrast, Coleman has just made the essay optional and harder to grade, ordered a complete redesign of the test with a stable competitor ready and willing to pick up the doubters, and pissed off his most dedicated customer segment. I wouldn’t be surprised if he declared that his purer company won’t profit from the filthy lucre of test prep, having turned that function over to Khan Academy (which long before its College Board arrangement has neglected to provide free services for the ACT). God knows what Coleman has planned for the AP suite.

Coleman could have some grand plan that I can’t anticipate (other than, as Steve Sailer puts it, all students should Be Like Me.) But reasonable people can and should wonder if he is wholly ignorant of the SAT’s market position, or if he actually believes that his Common Core curriculum is going to increase student ability and end the achievement gap. From there, it’s easy to postulate a scenario in which the College Board spends a fortune redesigning the test in a vacuum, not discovering until field tests that the changes either widen or narrow the achievement gap unforgiveably. Worse still, what if they make the test too hard? All of these possible outcomes wouldsend testers and their parents running into the arms of the Midwestern Mama, all safety and security.

In such an event, I trust the remainder of the College Board regroups before rollout. That’s been done before: the 2005 essay was originally supposed to be 50% of the writing score, but worrisome field tests led to the essay getting just a third of the section weight. A complete backout, cancel, reset is not impossible. The ETS cancelled its first set of GRE changes, despite a significant investment. Like the ETS, the College Board has a known quantity of a test to fall back on—just fire Coleman and move on.

I find that outcome very possible. But suppose that the SAT redesign works as promised. The achievement gap remains intact, the test is as described, the scores among American students are roughly compatible with the old SAT and the current ACT.

So then back to the big question: why, exactly, are Asians favoring the current SAT so strongly, and what will they do in the face of an altered test? The usual reason offered for their brand loyalty is the Asians aren’t aware of the ACT.

They figure out to game the SAT, they buy advanced copies of the SAT so their kids can cheat, they’re aware of and concerned about SAT changes, but somehow they’ve never heard of the SAT’s competition. Yeah. Not buying.

That leaves a few possibilities. First, the Asian test prep companies just don’t see the point in spending the money and time cracking a second test given their certainty on the first. Second, they haven’t actually cracked the test, aren’t really gaming it, but in fact are all using the Korean method of buying the test. All the coverage on the gaming is just sham. So they either can’t or haven’t yet paid to achieve the same penetration of the ACT’s secrets.

Third, and this is the fun one, they’ve tried cracking the ACT and can’t.

Eighteen months ago, I thought the possibility that Asians were artificially inflating their scores was theoretically possible, but unlikely. Over the next year, my experiences and additional research has changed my view. I now think it likely that both here and overseas, a decent percentage of Asian testers are either cheating or gaming the SAT and AP tests—-or both. I’ve written up some of this but not all, and I’m not expecting anyone to just take my word for it.

But when I start from that premise, and look at the SAT as a test that can be gamed, I see loads of potential.

Naturally, a purchased test is still the best guarantee—certainty being so much more reliable—but Big Data could identify many patterns. A few years back, you could see open discussion of an LA based test prep company with a largely Asian clientele that promised a reduced (300 or fewer words) vocabulary list, that was supposedly built by on analyzing previous tests and predicting a rotation cycle for the words. Or maybe they just had an in at the College Board, as the comment suggests.

Maybe they’ve analyzed each type of reading comprehension question and noticed a pattern beyond “pick C”, but rather a particular type of wording that is associated with a correct answer. This is something all bright people can do, of course, but it’s hard to teach as a system without mountains of data. The SAT’s grammar questions have all sorts of patterns that could likewise be broken down and systematized. Obviously, they’ve collected the essay prompts and found certain theme patterns that they use to teach kids to memorize essays verbatim. It goes without saying that the students didn’t write the essays. Or maybe they just had prior knowledge.

But assume, for the moment, that it was gaming, not cheating. I’m not an expert on Big Data or psychometrics, but I’m knowledgeable about both the SAT and the ACT, and the latter doesn’t lend itself to that sort of patterning. English, Reading, and Science test on passages with aggressive time requirements. While their questions do have patterns, the patterns are heavily reliant on context. You have to understand the text at least slightly in order to find the correct answer for the particular type of question. At least, that’s how it seems to me. (But then, I looked at the memorization necessary to game the SAT and thought it unlikely, so what do I know?)

Imagine that College Board successfully screws over the barely-English-speaking Asian market with their new test. Suppose also that both companies are ethical and no one is selling advance data on the test (which, in fact, I do assume). Suppose, in other words, that Asians with extremely limited English skills, here and abroad, are no longer able to misrepresent their abilities with SAT tests.

Is that really what colleges want?

Balf’s right about one thing–the colleges are the ultimate test customer. Public universities have dramatically increased their foreign admits, most of them Chinese, because they can charge them out of state fees for all four years.

They admit these students already knowing that the Chinese applications are largely fraudulent, and are well aware of the gaming and cheating, since they end up with students who can’t speak English and cheat here, too.

All this knowledge hasn’t slowed down their Asian international student admission.

Does this sound like a customer base that really wants to lock out non-English speaking Asians?

No matter which of these scenarios play out, I don’t see how the ACT doesn’t benefit. Why would any student opt for a complete unknown when they can prepare for the ACT using test prep materials and experts who’ve been working with the same test for years? I expect test prep companies to step up their offerings. If they don’t, then I throw up my hands and declare confusion.

This all strikes me as considerably more relevant than pipe dreams about the end of test prep, much less the achievement gap. And much more interesting.

Advertisements

About educationrealist


26 responses to “SAT’s Competitive Advantage

  • malcolmthecynic

    Okay, I’m trying to follow you here (BTW, awesome blog), so excuse the dumb question.

    What do you actually think of all this? Do you support it because it will make cheating/gaming the test harder? Or do you oppose it because the changes are ridiculous?

    Excuse my slowness, but I honestly can’t tell.

    (I’m working under the old theory that if you have a question other people almost certainly have the same question.)

  • educationrealist

    I try to describe what is, not what I want. To a fault.

    If I knew that Coleman was deliberately turning his back on the gaming mess and trying to make a good test, to hell with market share, I’d be in favor. I’m pretty sure he’s too ignorant to be doing that, so I just see one of two things: a potential train wreck, which the College Board itself (not Coleman) will avert if necessary by cancelling, or an irrelevant change. The larger question is the degree to which the CB understand that Asians are (apparently) using the test to lie. I don’t know the answer, and so I can’t really tell you what will happen next.

    The changes themselves are irrelevant. The SAT hasn’t been a decent test in a while, so if it dies entirely, leaving the ACT behind, I wouldn’t be unhappy.

    https://educationrealist.wordpress.com/2013/11/30/not-why-this-just-why-not-that/

    Here’s my overall wish list, at the end.

  • BAF

    How about reserving our universities for Americans, you know the secondary, forgotten, indigenous people who founded the country? The first were Native Americans. They don’t cheat. Let the Chinese, Japanese, and other foreigners worry about their own tests in their own countries.

    • Hattie

      I do sometimes wonder why, if the Asians are so smart, they can’t create their own world class institutions.

      • Sisyphean

        If your primary drive is to acquire a credential then whether you’re actually learning anything useful is irrelevant. What is important is making the credential as hard to acquire as possible, solely to limit the number and raise the prestige/remuneration. Those who seek credentials and those who seek useful knowledge both can say a degree ‘means something’ but be referring to totally different things.

        ~S

      • SC

        Isn’t the University of Tokyo a world class institution?

    • Marc

      Incidentally, the Asians didn’t want relations with the US and wanted to be left alone. The US actually forced relations with Japan and China and opened them up.

      This was also the case with education. One of the early major programs for Chinese students to study in the US was the Boxer Indemnity Scholarship Program. The Chinese had to pay the US indemnities for the Boxer Rebellion, and ended up overpaying the amount the US had demanded. The Chinese wanted the overpaid amount to be returned and used within China, but the US refused and demanded that it be used for scholarships for Chinese students to study in the US:

      https://en.wikipedia.org/wiki/Boxer_Indemnity_Scholarship_Program

      • BAF

        Marc,

        Typical of our government to engage itself in actions detrimental to the rights of deserving Americans.

        “The US actually forced relations with Japan and China and opened them up.”

        In alliance with elites foaming at the mouth for diversity, not to mention, full paying students.

      • Janon

        The European powers were also involved in opening up China and Japan. More than half a century before Commodore Perry and the Convention of Kanagawa, the British government sent Lord Macartney to the court of Emperor Qianlong with the goal of easing restrictions on British traders and establishing a permanent British embassy there..

  • Anthony

    What do you think of Sailer’s idea that Coleman’s changes are a revolt against English teachers, among other things?

    • educationrealist

      Hey, I’m sorry I missed this. I don’t think so. I mean, Coleman may hate English teachers, but English teachers these days aren’t big on vocabulary, nor are they fans of the Writing test. So if anything, they’d be in favor of the changes.

  • franklindmadoff

    Excluding more blatant and rampant cheating in some Asian countries, which I suspect there is something to, it seems to me that the idea that the (domestic) asians must either “game” the tests en masse or actually have the implied general intelligence that would normally correlate with these results is something of a false dichotomy. If domestic Asians as a group are much more likely to work (very) hard throughout school and study hard for these particular standardized tests it seems very likely to me that they would simply do systematically better at any given level of individual intelligence.

    In other words, the tests may be systematically over-predicting their actual cognitive abilities in some pretty significant ways even though they are not truly gaming it. Do not get me wrong. I am a believer in general intelligence (g) and I believe that cognitively loaded tests have real significant predictive validity for actual intelligence (especially when the n is large and randomly sampled), but it seems to me the claims that such tests are effectively immune to dramatic differences sustained behavioral differences are highly improbable (especially at the upper end of the ability distribution). These differences may not necessarily show themselves in differential validity tests (especially Freshman year GPA as reported by the college board) since they also capture willingness to work hard in the past too, which probably correlates reasonably well to freshman year at least, but I suspect they are still there. Moreover, I would not entirely dismiss the possibility that different groups may have different sorts of strengths in intelligence and that asians may have a form of it that shines especially in the more limited domain that they test for….

    I would be very interested in seeing how well different groups do in challenging upper level courses, as compared to their standardized test scores, as Steve Hsu et al. did for the general population at UO. Likewise, it’d be interesting to see if IQ vs SAT correlates differently for Asians vs Whites. Are you aware of any data like this?

    • DensityDuck

      “If domestic Asians as a group are much more likely to work (very) hard throughout school and study hard for these particular standardized tests it seems very likely to me that they would simply do systematically better at any given level of individual intelligence.”

      The issue is that they study, but they don’t study to gain the knowledge and cognitive ability the test is meant to exercise; they study how to take the test.

      It’s the difference between someone who learns to play football and someone who learns how to do football practice drills. The point of, e.g., running through those tires is not to be the best tire-runner, the point is to develop agility and speed.

    • educationrealist

      I agree with Density Duck, but I also think this statement is much truer than I would have thought 18 months ago:

      “but it seems to me the claims that such tests are effectively immune to dramatic differences sustained behavioral differences are highly improbable ”

      If I understand what you’re saying, you are saying that the tests *can* be studied for in ways that we had never anticipated?

      Yes. So you’re saying that it means yes, they’re smart. DD is saying no, that the idea of studying to this level is not what we expect these tests to denote. That is, the tests are proxies. If you can study to that extent, they aren’t proxies anymore.

      And this is what I had no idea was happening, because I work at a legit hagwon that doesn’t do this sort of thing. But what I started noticing was that all my kids had Advanced or Proficient on their state tests, despite incredibly weak vocabularies, and that got me thinking.

      • DensityDuck

        In math-nerd terms, ER, it’s like you and I are saying that the test is NP-complete; that it’s possible for someone to create a “test-taking algorithm” and that this algorithm will return a correct solution set without performing the activity the test is supposed to require. franklindmadoff is saying that it is *not* NP-complete; that the only way to get the correct solution set for the test is to do the activity the test requires you to do.

  • DensityDuck

    “In the same way, Coleman’s never run a major business, and has no idea about the political realities of helming the most visible, if not the most popular, college admissions and placement test. He has a story, but no plan. Meanwhile, boy, he’s happy to diss the test his predecessors pushed—and he makes free with insulting his competitor, too.”

    Change “Coleman” to “Obama” and the SAT bit to, well, everything, and you have a description of the current Presidential administration.

    “Shovel Ready” was clearly the idea of someone who had never had to negotiate a government contract from the supplier side.

  • 200 Posts | educationrealist

    […] SAT’s Competitive Advantage—I like this one because NO ONE is talking about the issues I raised. Some day soon, I will look prescient. It’s the Tests, Zitbrains!—entry #7 in my railing about the impact of credential tests on URM teacher population. […]

  • College Confidential and Brain Dumping the SAT | educationrealist

    […] wonder Asians have such a strong preference for the SAT. The credulous press tends to believe in the super tutors of Asia, but they’re much more […]

  • The SAT is Corrupt. No One Wants to Know. | educationrealist

    […] weren’t worth the hassle for other than US students living abroad (as indeed, the ACT seems to have done for years). Either way, a crackdown on testing security would seriously compromise Chinese and Korean […]

  • Ian Malcolm on Eva Moskowitz | educationrealist

    […] or telling the kids the answers, or we’d hear about it. Possibly they are engaging in the Chinese variety of test […]

  • Evaluating the New PSAT: Reading and Writing | educationrealist

    […] I predicted in my essay on the SAT’s competitive advantage, the College Board has released a test that has much in common with the ACT. I did not predict that […]

  • Evaluating the New PSAT: Math | educationrealist

    […] Key point: The SAT and the ACT’s ability to reflect a full range of abilities is an unacknowledged attribute of these tests. Many colleges use these tests as placement proxies, including many, if not most or all, of the public university systems. […]

  • Braindumping the PSAT: A Few Questions for David Coleman | educationrealist

    […] too poor to pay for the actual tests from organized Chinese crime rings, while the ACT has almost no international market. But you don’t care about market advantage, right? You’re “non-profit”. In […]

  • The SAT is Corrupt: Reuters Version | educationrealist

    […] mention that the SAT dominates in the international market; my own research confirms that the ACT’s international presence is minimal. My understanding is that the SAT only […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: