The SAT is Corrupt: Reuters Version

Dear Renee Dudley, Steve Stecklow, Alexandra Harney and Irene Jay Liu,

As someone who has studied and written extensively about the SAT corruption and the fraud delivery system known as Asian test prep , I congratulate you on the thorough job exposing the College Board’s open courting of corruption for profit in the overseas SAT market.

I’m not a reporter, just a teacher who does pretty good research, but for what it’s worth I think you did an outstanding job on many aspects of the story. I offer these suggestions (and one correction) with some disagreement, but little criticism (and lord knows, I can criticize).

The College Board Was Aware Long Before 2013
Your article strongly suggests time and again that the College Board learned that Asian test prep companies had obtained the tests by May 2013. In fact, the College Board knew that the test had been corrupted by January 2007, when it cancelled the scores of all 900 South Korean testers. It was common knowledge at the time that the College Board recycled old tests. The AP story doesn’t mention hagwons, and shows a touching faith in the CB’s assurance that kids couldn’t possibly benefit from seeing the test again. However, the College Confidential forums were very clear that the source of the test was a hagwon, not some “accidental exposure”. I worked for Kaplan and a major US hagwon at the time (only taught SAT at Kaplan) and my hagwon boss told me all about the methods Korean test prep companies used to get the “held back” tests.

So the College Board has known for at least nine years and probably longer that Asian international testers were cheating.

More Context on Cheaters

Your reporting begins with Xingyuan Ding, a Chinese national now attending ULA, who scored an 800 on the SAT Critical Reading section. Ding claims that “about half” of the answers on the reading section were in his “jijing”, or answer key.

That’s highly unlikely. While I understand the need for objectivity, some context would be useful. An 800 on the reading section is a 99th percentile score, meaning 99% of the testers receive a lower score. Even among Asians, it’s a 98th percentile score. So how likely is it that a Chinese national got that score?

Your reporting also mentions Linfeng Liu, another Chinese student who tested in Hong Kong, who says that she was “helped” by recognizing five—just five!—vocabulary questions, which enabled her, she said, to focus more on reading comprehension. Her overall score isn’t mentioned, but again, how likely is it that she cheated “just a little bit”?

Information that would have helped give context. Were these interviews conducted in Chinese or in English? How fluent was Ding, the 800 SAT critical reading guy, in English? How were these students doing in their classes?

Another related issue: In your FAQ, you mention repeatedly that just seeing the questions, not the answers, would still constitute a major advantage. That’s true. However, the story gives the impression that seeing the questions provides the primary, even sole, advantage to the students. This is almost certainly not true, and while that statement may be too strong, I think you tilt too far the other way. These are Chinese and Korean nationals with very weak English. A preview of the reading passages isn’t going to give them a big advantage. The answers do. Whether they’ve memorized long sequences or just have them handily tucked into a cell phone, the Chinese kids you interviewed almost certainly had the answers.

Cheating: It’s Not Just In China Anymore
Your reporting revealed that Asian test prep companies sent employees over here:

Sanli, the Chinese test-prep chain, says it sent 11 teachers to the United States to collect information on the redesigned exam. They debriefed 40 Sanli students studying at U.S. high schools who took the new SAT as they exited test centers, according to Wu, the general manager. Sanli presented its findings at a seminar at a Shanghai hotel.

(emphasis mine)

Heavens, that’s interesting. I did not know this. So are these Sanli students “Asian Americans”–born here, or immigrant children of long-time residents–or are they “Asian nationals” over here on F1 visas? As you probably know, Chinese students are flooding US public high schools whenever they can and US Christian private high schools when they can’t (due to the 1-year restriction), with our Beijing embassy eagerly courting more. Between 2009 and 2014, the number of F1 high school visas grew from 1,700 to 80,000, the vast majority of them Chinese or South Korean.

So when you report that Asian test prep companies are using “their” students to gain SAT knowledge to enable cheating, are they F-1 students on visas? Or are they Asian Americans who enroll with these test prep companies?

Of course, the College Board is not reusing national tests–at least, not often—which raises another question: what advantages do the Asian test prep companies offer Asians living in the US?

Your story hints at possibilities here:

Eight of the 10 existing Mathematics Level II subject tests were compromised – three in their entirety and five in part, including two exams that had never been given anywhere, the PowerPoint shows. Ten of the 13 Biology exams were also compromised in whole or in part, including one totally new test.

I’ve written on this before as well; Asian test prep companies have certainly used corrupt school administrators to gain early access to tests.

So the story probably goes beyond the international cheating–as if that’s not enough. It’s pretty clear that Asian test prep companies are getting early versions of tests. Given the College Board’s lax procedures, the leak may be from within the company.

What About the ACT?

You mention that the SAT dominates in the international market; my own research confirms that the ACT’s international presence is minimal. My understanding is that the SAT only recycles tests for the international sitting.

So what does the ACT do? Does the ACT recycle tests? What is the relative size of the two markets? If the SAT is the only test with a significant international presence, could it possibly be due to the fact that international testers have access to early copies? The overwhelming bulk of international testers are Asian or Middle Eastern, all countries with culture of cheating that Americans can barely comprehend.

Or perhaps there’s another explanation. But getting the ACT’s procedure would be a useful comparison point. Your story acknowledges the fact that ACT is the national leader, but apparently SAT’s touchstone status causes reporters to forget that and ignore the market king’s methods. (OK, that was a tad snarky.)

David Coleman Stonewalls

You know how reporters say [so and so] refused repeated requests to comment? You apparently only asked David Coleman to comment once.

David Coleman is a celebrity in the world of education reform. He is celebrated, rightly or wrongly, for Common Core standards. He took over the helm at the College Board in 2012–that fact alone should be mentioned, should it not? Presumably he was present, along with other “senior College Board staff” at the meeting with the Power Point slides in June 2013.

It’s not really your fault. But it’s over a day since your story came out, and not a peep from the College Board, much less Coleman. Couldn’t you have at least said “repeated requests”? Or did you only ask once?

National Interests

But like all reported stories I’ve seen on Asian cheating of the SAT, there’s no connection to the larger interests involved.

As your story mentions, many public and private universities are recruiting foreign students who are mostly from China and South Korea, even though the students are cheating on applications and tests, lying about their grades and resumes. Keep in mind that universities get tax breaks and other federal funding and public universities were chartered to serve the educational needs of their states.

Meanwhile, the SAT is moving outside its old beat as a college admissions test into a high school graduation test. Several states have committed use the SAT as a graduation requirement. Several states have switched from the ACT, which focuses on American students, to the SAT, which manifestly does not.

This isn’t just an issue for worried parents of college applicants. The College Board encourages and benefits from international criminal racketeering organizations that engage in immigration and mail fraud while enabling colleges to pretend they are accepting qualified applicants when in fact the colleges know full well their applicants lied. It collects money from multiple state contracts for a test product they can’t be bothered to spend money protecting from those organized criminal enterprises. State and private universities knowingly consume a fraudulent information product in order to fatten their coffers, all the while benefiting from tax-exempt status at both the federal and state interest.

Should the College Board be allowed to sell state contracts given its knowing participation in organized crime? Should our tax dollars be spent on universities if they are no longer acting in the public interest? Reasonable people can undoubtedly disagree on these questions. But surely they should at least be raised.

I want to emphasize again how pleased I am at your story and that while I had the above quibbles, overall it really was an excellent, thoroughly reported piece. I hope Reuters pushes it harder; you guys should be on TV, talking about this. Feel free to use my “national interest” take when you’re on the air.

All the best,

ER

The Prima Donna Rock Star Tester Treatment

I met with her the first time last Sunday a week before the SAT, mother looking on, and the conversation went something like this.

“I want to specialize in one test. Which one should I take?”

“Yeah, okay, back up a bit. You took SAT test prep over the summer, right?”

“Yeah, but I knew everything they told me. It didn’t help.”

“What’s your course load?” (she goes to a 50% Asian school.)

” I’m taking a history honors class now, but it’s my first. Precalc for math.”

“And your GPA? What colleges are you considering? ”

Shrug. “3.8 or so. Colleges, I have no idea. But what I want to know is, should I specialize in the ACT or the SAT? And should I take the old one or the new one?”

“Do you have a target SAT score?”

“2000. What’s the equivalent in ACT? But I really think I should take the old SAT and be done. ”

“Your last practice test was a 1400.” She winced. “Even if all colleges take the old SAT for 2016 admissions–something I find unlikely despite assurances to the contrary–I’m not sure how you can find the time to focus on improvement between now and January, the last sitting of the old test. Besides, why the hurry?”

She waved dismissively. “I want to be done with all this. I hate the SAT. Maybe I should specialize in the ACT. I don’t want to learn the new SAT.”

“Yeah, we’re back to this whole ‘pick a test’ thing. Let’s discuss something touchier. Are you frustrated by the difference between your school performance and your test performance?”

She got very still. “Yes.”

“When I see an academic profile significantly higher than a test score, the student usually mentions it first. I’ve met many kids, a lot of them girls, with a profile like yours. They’ll tell me that they really just want to improve, to get their score into a respectable range, and that they haven’t had good luck with test prep so far. I didn’t hear any of that from you. Instead it’s ‘gotta pick a test’, need a 2000′ despite no college plans, without any acknowledgment of what must be a very disappointing practice history.”

I said all this as delicately as possible, but she was already surreptitiously wiping away tears.

” I don’t see your mom behind this. You’re causing your own pressure but are also very resistant to making more effort or exploring options.”

She started nodding before I finished, and her mom handed her a Kleenex. “I just think I’m wasting my time.”

“So let’s start there. Do you have trouble with school tests? No? How about your state tests? So it’s not a general testing problem, just big standardized tests. Is it nerves?”

She laughed, sadly. “No. My big problem is motivation.”

I snorfed involuntarily, and she looked up in shock. “Sorry. I’m not at all laughing at you. Just the idea that the kid I see in front of me barking orders like an executive suffers from motivation problems.”

The mother demurred here. “Well, her GPA is only a 3.8.”

“Forgive me, but you’re Chinese and prone to distortion on this point.” They’re American enough to laugh. ” I see an articulate, bright, driven girl who appears to have an intellect that I would put conservatively three or four hundred points above this practice score. You are using that intellect in school. I don’t see an obvious motivation issue.”

“No, not in school. Not studying. When I’m testing–you know, like the practice tests? I lose all motivation.”

Well, hey now.

“Tell me if any of this is familiar: The test begins and you’re working away, feeling good. Then you run into a problem that you don’t know how to solve and suddenly, as you try to figure the problem out, everything seems pointless. You give up, make a guess, go on to the next problem. Except now you aren’t sure what to do with this one, either. Suddenly, nothing matters. You simply stop caring. I see by your face that I’m not off-base.”

“How did you know?”

“I’ve seen it before. I describe it as a sort of stress reaction.¹”

” I’m not nervous at all.”

” You should be so lucky. Jitters don’t usually affect performance. You get bored by stress. What happens, best I can tell after hearing many students describe the feeling, is that your brain shuts down to avoid feeling stress.”

My first case was a short, slight blond boy back before the SAT changes, so before 2005. I was going through his practice test explaining the missed problems, and he’d finish my sentences. That is, he knew how to do many of the problems he’d gotten incorrect on the test.

So why the high error count, I asked.

It was after I got bored, he replied. Once the boredom hit, he’d start to randomly bubble. I was aghast. He may as well have told me he sucked dead chickens’ eyeballs for candy, so incomprehensible was his behavior.

“So what you have to start doing, have to understand, is that you are a testing prima donna.”

“A prima donna?”

“You know how movie stars always order off-menu? Because they’re just too special for the pre-arranged menu that the rest of us use. Or the ballerinas or opera stars who simply refuse to be rushed, because they are artists. Or rock stars, the kind who make huge demands for their hotel rooms sometimes—Van Halen famously demanded brown M&Ms be removed from the candy bowl (yes, I know they had another reason, but her parents are never going to let her listen to Van Halen, so I’m safe). You need to be a prima donna rock star tester.”

“How?”

“Take two SAT sections daily, from the blue book. Use deadly serious test conditions. No music. No interruptions. No stopping the clock. No laying on the floor or on your bed. Sit at a table, door shut, start the timer.”

“That’s not even an hour.”

“And when the timer starts, I want you to take two minutes, at least, to go through the test and cherrypick. Circle the problems you’ll deign to do.”

“Um. What?”

“In math, pick and choose your problems. Circle the good ones. ‘This one, I shall do. This one, pah!’ Spit upon it. If you don’t instantly vibe to the question, avert your eyes and scratch an X next to that problem, which clearly must be for peasants and other little people. Can you do that?”

She giggled. “Really? What about reading?”

” Skip anything with long paragraphs that looks less desirable than root canal. You like sentence completions?”

“Yes!”

“Do them first, then evaluate each reading passage to determine whether or not Her Majesty–that’s you–is interested. Which part of the writing section do you like best, the paragraph at the end?”

“How do you know this?”

“Do those six questions at the end first. Then go back to the front. The second–I mean the second—you find a long sentence you can’t instantly decipher, that question OFFENDS you. Turn up your nose. Move on.”

“So that’s all I want for the week. Two sections. Vary the subject. Every night. Take them like a rock star looking at candy bowls to make sure there are no…oh, look there’s a brown M&M. Skip it.”

“But I might only want to do four or five questions a section.”

“Great. Do those. Then, oh, hey. You’ve still got 20 minutes to kill. What’ll help pass the time? Let’s look at the other questions to see if they hold any interest. You are a movie star stuck in Podunk, in search of decent dim sum.”

“But the whole thing is a lie. The problems I can’t do aren’t stupid.”

“Sure, but we need to fake out your psyche. You have a fragile testing temperament that must be coddled and swathed in protective coating.”

The mom was a bit stunned, but accepting. “So none of the strategies she learned in test prep?”

“Mom, they didn’t work anyway. But what if I don’t have enough time to go back and do the problems that bored me?”

“Then you will have spent a whole test section working on problems you can do. How is that worse?”

“But if I try to read the long passages, I know I will get bored.”

“Well, I have some ideas for that later, but for now, read the passages that meet with your approval, and do the questions. Then for the rest, amuse yourself with the peasant passages. Do the vocabulary questions. The ones with line numbers. Don’t read them if they bore you. Normally, you understand, I wouldn’t suggest this.”

“So practice that all week. Eat pizza, chocolate, noodles, sesame balls with red bean paste, whatever your favorite food is Friday night. Saturday, have a good breakfast and visualize rejecting all those peasant problems.”

“What if I get bored anyway?”

“That’s a very real possibility. At the first moment you identify boredom, put your pencil down. Take a breath. Remind yourself that while it’s scary, this boredom is a valuable opportunity to practice dealing with it. That it only feels like boredom. Do not give up. Do not let yourself randomly bubble. If you feel done and can’t fight off the boredom, put your head down and take a nap. Otherwise, go back to the test and look for test questions that pique your curiosity.”

“But you said I didn’t have to read the passages.”

“Sure. But don’t randomly bubble, or give up. Estimate. Eliminate known wrong answers. Guess based on the context. But if you can’t kick off the boredom and feel hopeless, take a rest until the next section.”

“And here’s the important part: under no conditions are you to worry about your score. You’re not there for the score. You’re there to practice being a rock star who picks and chooses her projects. We’ll do scores later, if you like.”

“That’s okay. I don’t think I’m going to improve now, so at least I might know why.”

“It’s helpful just to know what the problem is,” her mother agreed.

They actually smiled as I left, both noticeably less anxious than they were when I arrived.

Note: she’s a junior, and has no reason whatsoever to take the SAT in October. I tried to talk the mom out of that, but she was determined to keep the date. Ideally, I wouldn’t send a student to try out this method on a live test, but that was the only option.

Will it work, this refusal to tolerate brown M&Ms and uninviting questions? Typically, yes, although since I’ve cut back on tutoring I haven’t run into the prima donna tester in several years. The cases I remember always saw an instant boost of 100-150 points the first time they took the test in rock star mode. In every case, they were also mentally exhausted afterwards. They’d never worked the entire test before, having mentally checked out. Prima donnas are fixable. The ones who go into a fugue state, not so much. Fortunately, that’s even rarer.

I started to make a larger point, but it’s too complicated and, since returning this August I’ve vowed to post more. I had too many ideas piling up that just weren’t…perfect, and so I kept putting them off, even though each idea had more than enough for a post. Time for me to limit scope and bite off achievable chunks. Otherwise I’ll think I’m bored and don’t care when really I’m stressed out….hey. Good thing I don’t get like this for tests.

So don’t read too much into this beyond an interesting behavior that I’ve learned to treat. Don’t apply it to policy. Do I think some people underperform their abilities on tests? Yes, I do. Do I think that tests can be gamed by people whose essential intelligence is high on mimicry and memory, giving the impression of skills they don’t actually have? Yes, I do. Do I think tests are mostly accurate? Yes, for most people. It’s a big ol’ world out there. Many cases exist simultaneously.

Meanwhile, I hope all you testers out there did well yesterday. And if you know any fragile testing temperaments, give this strategy a try.

**********************************
¹ While writing this piece, I googled and learned that researchers call it stress, too.

Evaluating the New PSAT: Reading and Writing

The College Board has released a new practice PSAT, which gives us a lot of info on the new SAT. This essay focuses on the reading and writing sections.

As I predicted in my essay on the SAT’s competitive advantage, the College Board has released a test that has much in common with the ACT. I did not predict that the homage would go so far as test plagiarism.

This is a pretty technical piece, but not in the psychometric sense. I’m writing this as a long-time coach of the SAT and, more importantly, the ACT, trying to convey the changes as I see them from that viewpoint.

For comparison, I used these two sample ACT, this practice SAT (old version), and this old PSAT.

Reading

The old SAT had a reading word count of about 2800 words, broken up into eight passages. Four passages were very short, just 100 words each. The longest was 800 words. The PSAT reading count was around 2000 words in six passages. This word count is reading passages only; the SAT has 19 sentence completions to the PSAT’s 13.

So SAT testers had 70 minutes to complete 19 sentence completions and 47 questions over eight passages of 2800 words total. PSAT testers had 50 minutes to complete 13 sentence and 27 questions over six passages of 2000 words total.

The ACT has always had 4 passages averaging 750 words, giving the tester 35 minutes to complete 40 questions (ten for each passage). No sentence completions.

Comparisons are difficult, but if you figure about 45 seconds per sentence completion, you can deduct that from the total time and come up with two rough metrics comparing reading passages only: minutes per question and words per question (on average, how many words is the tester reading to answer the questions).

Metric	Old SAT	Old PSAT	ACT	New PSAT
Word Count	2800	2000	3000	3200
Passage Count	8	6	4	5
Passage Length	100-850	100-850	750	500-800
MPQ	1.18	1.49	1.14	1.27
WPQ	59.57	74.07	75	69.21

I’ve read a lot of assertions that the new SAT reading text is more complex, but my brief Lexile analysis on random passages in the same category (humanities, science) showed the same range of difficulty and sentence lengths for old SAT, current ACT, and old and new PSAT. Someone with more time and tools than I have should do an indepth analysis.

Question types are much the same as the old format: inference, function, vocabulary in context, main idea. The new PSAT requires the occasional figure analysis, which the College Board will undoubtedly flaunt as unprecedented. However, the College Board doesn’t have an entire Science section, which is where the ACT assesses a reader’s ability to evaluate data and text.

Sentence completions are gone, completely. In passage length and overall reading demands, the new PSAT is remarkably similar in structure and word length to the ACT. This suggests that the SAT is going to be even longer? I don’t see how, given the time constraints.

tl;dr: The new PSAT reading section looks very similar to the current ACT reading test in structure and reading demands. The paired passage and the questions types are the only holdover from the old SAT/PSAT structure. The only new feature is actually a cobbled up homage to the ACT science test in the form of occasional table or graph analysis.

Writing

I am so flummoxed by the overt plagiarism in this section that I seriously wonder if the test I have isn’t a fake, designed to flush out leaks within the College Board. This can’t be serious.

The old PSAT/SAT format consisted of three question types: Sentence Improvements, Identifying Sentence Error, and Paragraph Improvements. The first two question types presented a single sentence. In the first case, the student would identify a correct (or improved) version or say that the given version was best (option A). In the ISEs, the student had to read the sentence cold with no alternatives and indicate which if any underlined word or phrase was erroneous (much, much more difficult, option E was no change). In Paragraph Improvements, the reader had to answer grammar or rhetoric questions about a given passage. All questions had five options.

The ACT English section is five passages running down the left hand side of the page, with underlined words or phrases. As the tester goes along, he or she stops at each underlined section and looks to the right for a question. Some questions are simple grammar checks. Others ask about logic or writing choices—is the right transition used, is the passage redundant, what would provide the most relevant detail. Each passage has 15 questions, for a total of 75 questions in 45 minutes (9 minutes per passage, or 36 seconds per question). The tester has four choices and the “No Change” option is always A.

The new PSAT/SAT Writing/Language section is four passages running down the left hand side of the page, with underlined words or phrases. As the tester goes along, he or she stops at each underlined section and looks to the right for a question. Some questions are simple grammar checks. Others ask about logic or writing choices—is the right transition used, is the passage redundant, what would provide the most relevant detail. Each passage has 11 questions, for a total of 44 questions in 35 minutes (about 8.75 minutes per passage or 47 seconds a question). The tester has four choices and the “No Change” option is always A.

Oh, did I forget? Sometimes the tester has to analyze a graph.

The College Board appears to have simply stolen not only the structure, but various common question types that the ACT has used for years—as long as I’ve been coaching the test, which is coming on for twelve years this May.

I’ll give some samples, but this isn’t a random thing. The entire look and feel of the ACT English test has been copied wholesale—I’ll add “in my opinion” but don’t know how anyone could see this differently.

Writing Objective:

• Best Conclusion: ACT vs. New PSAT
• Add Best Detail: ACT vs. New PSAT
• Best Transitioning for Paragraph:ACT vs. PSAT

Style and Logic:

• Logical Placement: ACT vs. New PSAT
• Check for Conciseness/Redundancy: ACT vs. New PSAT
• Check for Formal Style: ACT vs. New PSAT
• Logically Correct Transition (cause, contrast, continuation): ACT vs. New PSAT
• Writer Choice: ACT vs. New PSAT

Grammar/Punctuation:

• Dash vs. Semicolon: ACT vs. New PSAT
• The apostrophe/plural check: ACT vs. New PSAT

tl;dr: The College Board ripped off the ACT English test. I don’t really understand copyright law, much less plagiarism. But if the American College Test company is not considering legal action, I’d love to know why.

The PSAT reading and writing sections don’t ramp up dramatically in difficulty. Timing, yes. But the vocabulary load appears to be similar.

The College Board and the poorly informed reporters will make much of the data analysis questions, but I hope to see any such claims addressed in the context of the ACT’s considerably more challenging data analysis section. The ACT should change the name; the “Science” section only uses science contexts to test data analysis. All the College Board has done is add a few questions and figures. Weak tea compared to the ACT.

As I predicted, The College Board has definitely chosen to make the test more difficult for gaming. I’ve been slowly untangling the process by which someone who can barely speak English is able to get a high SAT verbal and writing score, and what little I know suggests that all the current methods will have to be tossed. Moving to longer passages with less time will reward strong readers, not people who are deciphering every word and comparing it to a memory bank. And the sentence completions, which I quite liked, were likely being gamed by non-English speakers.

In writing, leaving the plagiarism issue aside for more knowledgeable folk, the move to passage-based writing tests will reward English speakers with lower ability levels and should hurt anyone with no English skills trying to game the test. That can only be a good thing.

Of course, that brings up my larger business question that I addressed in the competitive advantage piece: given that Asians show a strong preference for the SAT over the ACT, why would Coleman decide to kill the golden goose? But I’ll put big picture considerations aside for now.

Here’s my evaluation of the math section.

SAT’s Competitive Advantage

Asians taking the SAT: 196,030 out of 1,640,047, or about 11.9%.
(No response: 62,603)

Asians taking the ACT: 71,677 out of 1,799,243, or about 3.9%
(No response: 110, 305)

I can’t tell if these numbers are US-based only, or including overseas testers. The ACT has a small international market. The SAT dominates, particularly in the Asian countries. 50,000 Chinese visit a Hong Kong testing site in just one year. While I couldn’t find totals for Korea, 900 students got their scores cancelled in 2007, clearly just a fraction of the total testers. In India, the College Board has 32 test centers (up from 20 just four years ago), the ACT has 60 testers.

So if the ratio above, in the tests’ national reports, are indeed just for the US, then the overall Asian preference is even greater than the 2.75:1 brand preference for Asian American testing. Since I can’t find any verification of this, let’s assume the ratio is overall preference.

In either case, the numbers are stark. The SAT is already losing out to the ACT. Why would it make changes that, if accurately described, would make the test less attractive to Asians, who have largely rejected the ACT?

Before I get to that: outside of the knowledgeable education reporters, the coverage has been nonsense. Contrast this relatively solid piece by NYTimes reporter Tamar Lewin, (whose twitter tagline should be “not half as ignorant and less than a quarter as irritating as Motoko Rich”) with this obsequious David Coleman profile by author Todd Balf. Discussing his work, Balf described the “checks” that restrain the College Board from using its power, :

Either Coleman does something about [the dissatisfaction with the SAT], or at the college level more and more schools will go test optional. Coleman wouldn’t say that the reforms are in response to places like Wake Forest, but it is hard to ignore.

Sigh. A few thousand colleges or university systems use the SAT as a first round placement test: University of Georgia, University of California, University of Michigan, University of Nevada, University of Wisconsin, a whole bunch of community college systems, and oh, yeah, the largest university system in the country. Any number of private universities do, as well, including Baylor, Columbia, and Duke.

So the SAT and ACT allow many schools to avoid the hassle and expense of placing hundreds of thousands of students each year.

But yeah, the College Board is all worked up about Wake Forest going exam optional.

People who try to prognosticate the changes are mostly being silly. I like Peter Wood, but his rant here doesn’t add up. Like most non-experts, Wood completely ignores the existence of the ACT, which has the same “biases” that Wood thinks the SAT is trying to fix. The College Board can’t “dumb down” the test. If blacks suddenly started scoring at the 80th percentile on the SAT, but were still scoring at the 35th on the ACT, people would notice.

So leaving aside what people who don’t really understand things think, and leaving my random thoughts about certain changes for another day, and then pointing out that I liked this analysis by Eduwonk, not usually a favorite of mine, my big question is still why would the College Board deliberately screw over Asians, the customer base with the strongest preference for its product?

The Chinese test prep companies are less than thrilled about the changes, which they believe will prevent them from teaching students to game the test . I am asserting this as fact: many Chinese international students openly acknowledge they don’t have the underlying ability their SAT scores denote.

Not much outrage from the Koreans, but then they just buy early versions of the test and distribute it to paying customers. Obviously, this method works regardless of underlying test modifications. Cheating is so rampant in South Korea that again, I am asserting as fact that many of these testers lack the underlying knowledge that their SAT scores supposedly indicate.

I doubt either country is exclusive in its approach. Presumably, the Chinese cheat and the Koreans game.

Indians seem cautiously optimistic—hey, they mostly do speak English, right?

I consider the SAT changes completely irrelevant. The interesting question to me is, assuming no one changes their behavior, how will this all play out?

The one thing I feel pretty confident of: faith in Coleman is wholly unjustified. Leaving aside the value of his Common Core work, note the lack of follow-through. Typical consultant, he jumped in long enough to tell people what to do, then left before all the errors in his ideas are revealed through disastrous implementation results.

In the same way, Coleman’s never run a major business, and has no idea about the political realities of helming the most visible, if not the most popular, college admissions and placement test. He has a story, but no plan. Meanwhile, boy, he’s happy to diss the test his predecessors pushed—and he makes free with insulting his competitor, too.

But for all Coleman trashes his predecessor, Gaspard Caperton, the ex-Appalachian governor has big shoes that I very much doubt Coleman can fill. Complain all you like about the writing section, it was pure genius as a business decision. On Caperton’s watch, the College Board dumbed down an existing test that a couple hundred thousand kids took, made it cheaper to grade, and forced a million kids to pay a higher price to take it as part of the “new” SAT, even though lots of colleges don’t use the section score—and let everyone think that the UC forced the change. The company made millions in a single year. At his behest, the College Board abandoned the claim that test prep doesn’t work, and sold its own product, which had to be taken off the Amazon bestseller list because everyone found it too upsetting. And that’s just the SAT’s profit center. Caperton also presided over the Advanced Placement’s stunning growth, achieved by taxpayers shoveling money into the CB’s coffers for entirely unqualified kids to take tests in order that Jay Mathews put their school on his Index.

In contrast, Coleman has just made the essay optional and harder to grade, ordered a complete redesign of the test with a stable competitor ready and willing to pick up the doubters, and pissed off his most dedicated customer segment. I wouldn’t be surprised if he declared that his purer company won’t profit from the filthy lucre of test prep, having turned that function over to Khan Academy (which long before its College Board arrangement has neglected to provide free services for the ACT). God knows what Coleman has planned for the AP suite.

Coleman could have some grand plan that I can’t anticipate (other than, as Steve Sailer puts it, all students should Be Like Me.) But reasonable people can and should wonder if he is wholly ignorant of the SAT’s market position, or if he actually believes that his Common Core curriculum is going to increase student ability and end the achievement gap. From there, it’s easy to postulate a scenario in which the College Board spends a fortune redesigning the test in a vacuum, not discovering until field tests that the changes either widen or narrow the achievement gap unforgiveably. Worse still, what if they make the test too hard? All of these possible outcomes wouldsend testers and their parents running into the arms of the Midwestern Mama, all safety and security.

In such an event, I trust the remainder of the College Board regroups before rollout. That’s been done before: the 2005 essay was originally supposed to be 50% of the writing score, but worrisome field tests led to the essay getting just a third of the section weight. A complete backout, cancel, reset is not impossible. The ETS cancelled its first set of GRE changes, despite a significant investment. Like the ETS, the College Board has a known quantity of a test to fall back on—just fire Coleman and move on.

I find that outcome very possible. But suppose that the SAT redesign works as promised. The achievement gap remains intact, the test is as described, the scores among American students are roughly compatible with the old SAT and the current ACT.

So then back to the big question: why, exactly, are Asians favoring the current SAT so strongly, and what will they do in the face of an altered test? The usual reason offered for their brand loyalty is the Asians aren’t aware of the ACT.

They figure out to game the SAT, they buy advanced copies of the SAT so their kids can cheat, they’re aware of and concerned about SAT changes, but somehow they’ve never heard of the SAT’s competition. Yeah. Not buying.

That leaves a few possibilities. First, the Asian test prep companies just don’t see the point in spending the money and time cracking a second test given their certainty on the first. Second, they haven’t actually cracked the test, aren’t really gaming it, but in fact are all using the Korean method of buying the test. All the coverage on the gaming is just sham. So they either can’t or haven’t yet paid to achieve the same penetration of the ACT’s secrets.

Third, and this is the fun one, they’ve tried cracking the ACT and can’t.

Eighteen months ago, I thought the possibility that Asians were artificially inflating their scores was theoretically possible, but unlikely. Over the next year, my experiences and additional research has changed my view. I now think it likely that both here and overseas, a decent percentage of Asian testers are either cheating or gaming the SAT and AP tests—-or both. I’ve written up some of this but not all, and I’m not expecting anyone to just take my word for it.

But when I start from that premise, and look at the SAT as a test that can be gamed, I see loads of potential.

Naturally, a purchased test is still the best guarantee—certainty being so much more reliable—but Big Data could identify many patterns. A few years back, you could see open discussion of an LA based test prep company with a largely Asian clientele that promised a reduced (300 or fewer words) vocabulary list, that was supposedly built by on analyzing previous tests and predicting a rotation cycle for the words. Or maybe they just had an in at the College Board, as the comment suggests.

Maybe they’ve analyzed each type of reading comprehension question and noticed a pattern beyond “pick C”, but rather a particular type of wording that is associated with a correct answer. This is something all bright people can do, of course, but it’s hard to teach as a system without mountains of data. The SAT’s grammar questions have all sorts of patterns that could likewise be broken down and systematized. Obviously, they’ve collected the essay prompts and found certain theme patterns that they use to teach kids to memorize essays verbatim. It goes without saying that the students didn’t write the essays. Or maybe they just had prior knowledge.

But assume, for the moment, that it was gaming, not cheating. I’m not an expert on Big Data or psychometrics, but I’m knowledgeable about both the SAT and the ACT, and the latter doesn’t lend itself to that sort of patterning. English, Reading, and Science test on passages with aggressive time requirements. While their questions do have patterns, the patterns are heavily reliant on context. You have to understand the text at least slightly in order to find the correct answer for the particular type of question. At least, that’s how it seems to me. (But then, I looked at the memorization necessary to game the SAT and thought it unlikely, so what do I know?)

Imagine that College Board successfully screws over the barely-English-speaking Asian market with their new test. Suppose also that both companies are ethical and no one is selling advance data on the test (which, in fact, I do assume). Suppose, in other words, that Asians with extremely limited English skills, here and abroad, are no longer able to misrepresent their abilities with SAT tests.

Is that really what colleges want?

Balf’s right about one thing–the colleges are the ultimate test customer. Public universities have dramatically increased their foreign admits, most of them Chinese, because they can charge them out of state fees for all four years.

They admit these students already knowing that the Chinese applications are largely fraudulent, and are well aware of the gaming and cheating, since they end up with students who can’t speak English and cheat here, too.

All this knowledge hasn’t slowed down their Asian international student admission.

Does this sound like a customer base that really wants to lock out non-English speaking Asians?

No matter which of these scenarios play out, I don’t see how the ACT doesn’t benefit. Why would any student opt for a complete unknown when they can prepare for the ACT using test prep materials and experts who’ve been working with the same test for years? I expect test prep companies to step up their offerings. If they don’t, then I throw up my hands and declare confusion.

This all strikes me as considerably more relevant than pipe dreams about the end of test prep, much less the achievement gap. And much more interesting.

SAT Prep for the Ultra-Rich, And Everyone Else

Whenever I read about SAT tutors charging in the hundreds of dollars, I’m curious. I know they exist, but I also know that I’m pretty damn good, and I’m not charging three figures per hour (close, though!). So I always read them closely to see if, in fact, these test prep tutors are super fab in a way that I’m not.

At the heart of all test prep stories lies the reporter’s implicit rebuke: See what rich people are doing for their kids? See the disadvantage that the regular folks operate under? You can’t afford those rates! You’re stuck with Kaplan or cheaper, cut-rate tutors! And that’s if you’re white. Blacks and Hispanics can’t even get that much. Privilege. It sucks.

And so the emphasis on the cost of the tutors, rather than any clear-eyed assessment of what, exactly, these tutors are doing that justifies an hourly rate usually reserved for low-end lawyers, never mind the fact that these stories are always about the SAT, when in fact the ACT is taken by as many kids as the SAT. The stories serve up propaganda more than they provide an accurate picture of test prep.

I’ve written before about the persistence of test prep delusions. Reality, summarized: blacks and Hispanics use test prep more than whites, Asians use it more than anyone. Rich parents are better off buying their kids’ way into college than obsessing about the last few points. Test prep doesn’t artificially inflate ability.

So what, in fact, is the difference between Lisa Rattray, test prep coach charging $300/hour; me, charging just short of 3 figures; and a class at Kaplan/Princeton/other SAT test prep schools?

Nothing much. Test prep coaches can work for a company or on their own. The only difference is their own preferences for customer acquisition. Tutors and instructors with a low risk tolerance just sign on with a company. Independent operators, comfortable with generating their own business, then pick their markets based on their own tolerance. My customers sit comfortably in the high income bracket, say $500K to $5 million yearly income, although I’ve worked with a couple Fortune 500 families. Lisa Rattray and Joshua Brown, the featured tutors, clearly work with families a couple notches up the income ladder from mine.

None of this has anything to do with quality of instruction. Test prep is a sales and marketing game. The research is clear: most kids improve at least a little, quite a few kids improve a lot, a very few kids stay put or, heaven forfend, get worse.

Obviously, instructor quality influences results a bit, but only rarely change a kid from one category (mild improvement) to another (major improvement). Remember, all test prep instructors have high test scores, and they’re all excellent at understanding how the test works. So they make career decisions based on their tolerance for sales and marketing, not the quality of their services. I know of some amazingly god-awful tutors who charge more than I do, having learned of them from their furious ex-clients who assumed a relationship between price and quality. These tutors have websites, business cards, offered their own prepared test materials, saw students in their rented space, and often accepted credit card deposits. I have none of these accoutrements, show up at my clients’ houses, usually but not always on time, and take checks. Every so often I get a client who whips out a wad of bills and pays me $500 in cash, which I find a tad unnerving.

I’m just as good now as I was at Kaplan (in fact, I privately tutored my own students while at Kaplan, tutoring theirs), but I only got paid $24/hour for Kaplan work, which charged about $125/hour for my services. Kaplan will (at least, when I worked there) boost a teacher’s hourly rate to $50/hour if they get 80% or more “perfect” customer ratings. Instructors who convinced their students that to respond to the online survey and give them excellent ratings got more money. This is independent of actual improvement. A customer who doesn’t improve at all but felt reassured and valued by her instructor could give straight 5s (or 1s, whatever the highest rating is). A customer who sees a 300 point improvement might not fill in the survey at all. Their research showed that customers who give their instructors perfect ratings gave awesome word of mouth and that was worth rewarding. Nothing else was. Asian cram schools pay instructors based on the students who sign up, with a premium for those who sign up specifically for that instructor. See? Sales and marketing.

Test prep companies, long castigated as the luxury option of the wealthy, have been the first choice of the middle class for a decade or more. For the reasons I’ve outlined, any parent can find excellent instructors in all the test prep companies: Kaplan, Princeton Review, Asian cram schools. They won’t brag about it, though, because these companies are about the brand. Kaplan doesn’t want word getting out that Joe Dokes is a great Kaplan instructor; it wants everyone to be happy with Kaplan. No one is “Princeton Review’s star tutor” for very long, because Princeton doesn’t like it and at that point, the most risk-averse instructor probably has enough word of mouth fame to go independent.

I’ve often advised my students to consider a class. The structure helps. Some of my kids don’t do any work unless I’m there, so what I end up doing is sitting there playing Spider on my android on my client’s dime while the kid works problems, rather than reviewing a bunch of work to move forward. I’m pretty sure Lisa and Joshua would celebrate this, going to the parent and pointing out how much they are helping. I have better things to do and other clients to see. So I tell the parents to fork out an extra thousand for a class, make sure the kid goes, and then we review the completed work. The student gets more hours, more focus and, usually, higher scores, regardless of the quality of the second instructor.

I’m not saying Lisa and Joshua are wrong, mercenary, or irresponsible. They just play to a different clientele, and a huge chunk of their ability to do so rests on their desire to sell an image. That’s fine. That’s just not me. Besides, Josh forks out $15K of his profit for a rental each summer. Lisa gets constant text messages from anxious parents. Also not me.

So you’re a white, middle class or higher parent with a teenager, worried about SAT scores. What do you do? Here are some guidelines. Recognize that GPA or parental income smacks down test scores without breaking a sweat. If Johnny doesn’t have a GPA of 3.8 or higher, elite universities are out of the question unless his parents are alumni or rich/connected enough to make it worth the school’s while.

If Sally qualifies on GPA, has a top-tier transcript (5 or more AP classes) and wants to go to a top 10 school, test scores should be 700 or higher per section. If they’re at that point, don’t waste your time or money or stress. At that point, the deciding factors aren’t scores but other intangibles, including the possibility that the admissions directors toss a pile of applications in the air and see which ones travel the farthest.

If Jesse is looking for a top 20 or 30 school, the GPA/transcript requirements are the same, but looking at the CDS of these schools, realistically a 650 or higher per section will do the trick. It might be worth boosting the test scores to low 700s, but if Jesse is a terrible tester, then don’t break the bank. One of the schools will probably come through.

If Sammy has a lower GPA (3.3 to 3.8) but excellent test scores (high 600s or higher per section) , then look to the schools in the middle–say, from 40 to 60. It’s actually worth spending money to maximize Sammy’s scores, because these mid-tier schools often get a lot of high effort hard workers with mediocre test scores. Not only will Sammy look good, but he might get some money. (By the way, if you’ve got a Sammy whose grades are much lower than his abilities, you should still push him into the hardest classes, even if he and the counsellors cavil. If your Sammy is like most of them, he’s going to get Bs and Cs regardless, so he may as well get them in AP classes and get some college credit from the AP tests. And the transcript will signal better, as well.)

The biggest bang for the test prep buck lies not in making kids competitive for admissions, but to help them test out of remediation at local universities. So if Austin has a 3.0 GPA, works hard but tests poorly, then find out the SAT cut score at his university. If he’s not above that point, then spend the money to get him there, and emphasize the importance of this effort to his college goals.

If your kid is already testing at 650 or higher, either send her to an Asian cram school (they will be the only white kid there, for the most part, but the instruction will be excellent) or invest in a tutor. The average white kid class at Kaplan or Princeton might have an instructor who can finetune for their issues, but probably won’t.

Otherwise, start with a class and supplement with a tutor if you can afford it. Ask around for good instructors, or ask the test prep company how long the instructor has been teaching. Turnover in test prep instructors is something like 75%; the 25% who stay long term do so because they’re good. As for the tutor, I hope I’ve convinced everyone that price isn’t an issue in determining quality. I would ask around for someone like me, because our ability to get a high rate without the sales and marketing suggests we must be, in fact, pretty good. And there’s always someone like me around. Otherwise, I’d go with the private tutoring options at a test prep company, with interviews.

As I said, these rules are for middle class or higher white kids. Only 6% of blacks and Hispanics get above 600 on any section of the SAT–in fact, the emphasis on GPA came about in large part to bypass the unpleasant reality of the score gap. There are only around 300 black students that get higher than 700 on two sections of the SAT. That’s barely enough blacks for one top ten school. Rules are very different. The main reason for blacks and Hispanics to take test prep is to get their scores above the remediation number. Middle class or higher Asians face much higher standards because universities know their (or their parents’) dedication to getting good grades and good test scores is more than a tad unnatural and probably overstates their value to the campus. Athletes and artists of note play by different rules. Poor whites and poor Asians have it really, really tough.

What this means, of course, is that the kids in the Hamptons are probably already scoring 700 or higher per section and are, consequently, wasting their time. But what the hell, they’re doing the economy some good. Or maybe some of them are Asian.

Note: I wrote this focusing on the SAT but it all applies to the ACT as well, and the ACT is a much better test. I wrote about the ACT here.

What’s the difference between the SAT and the ACT?

I couldn’t find anything terribly wrong with this Ed Week article. But it didn’t offer anything terribly useful, either,so I thought I’d offer up some facts that might do some good.

Historically, the ACT was the test for the Midwest and South, and the SAT was the test for the coasts, but after the 2005 SAT changes, the ACT’s test population caught up. Both tests are given to around 1.6 million students.

Test Content

The ACT tests the same fact base as the SAT. It’s about 20 minutes shorter than the SAT, although it has far more questions and four sections:”

• English: 45 minutes, 5 passages of 15 questions.
• Math: 60 minutes, 60 questions.
• Reading: 35 minutes, 4 passages of 10 questions.
• Science: 35 minutes, 7 passages of 4-8 questions (40 total).

The ACT section times are brutal, which is why the ACT benchmarks purporting to report on college readiness should be taken with a healthy dose of salt. In my view, they dramatically underreport the reading, science, and (to a lesser extent) math ability of the lower to mid-range “college” students (keeping in mind that these kids shouldn’t be in college anyway, but that’s a different story).

Each section is scored on a scale of 1-36. The sections are then averaged for a Composite score, which is every bit as useless, really, as the SAT total. Colleges use the section scores far more than is generally known for placement in or out of remediation.

How do you convert ACT scores to SAT?

The University of California used to offer a direct conversion. One sign of the ACT’s growing popularity is that both tests are now converted to a “UC score”.
Roughly, a 21 on any section is the ability equivalent of a 500 on the SAT, a 26 is a 600, and a 31 a 700. However, a one to one combination isn’t possible, with 4 ACT sections and 3 SAT sections.

The UC conversion adds two-thirds of the math/reading/science total to the English/writing combined score. This weights the converted score towards English–rather unfairly, in my view, but not enough to do serious damage.

Which is more closely aligned to school curriculum

Both test knowledge and abilities that students should have mastered in school; the ACT doesn’t directly test science, but content knowledge will make the questions more familiar. The ACT also tests slightly more math: trigonometry, analytic geometry (circle and ellipse equations), and the occasional matrix question. Neither tests specific content knowledge in history, science, or English; for some reason, people say the ACT does. They are wrong.

Which test should students take?

Most students will score in roughly the same percentile on each test. However, some students have strong preferences for the ACT.

Low to mid-tier students are almost always better off with the ACT, something that I wish more do-gooder organizations understood. Much of the SAT’s difficulty is front-loaded–a big challenge in many questions is simply figuring out what the question is. The ACT actually tests more material but its questions are more straightforward. Any student who prefers the concrete to the abstract should consider the ACT, and most low to mid ability students will have a preference for the concrete. However, see the caveat below regarding reading abilities.

Students with SAT section scores in the high 600s/low 700s should always check out the ACT. The 2005 SAT changes reduced the number of questions in each section by 10%, and the cuts were primarily from the higher-difficulty questions. Many students in the mentioned range are every bit as bright as those getting 760+ scores, but are less detail-oriented, and usually make a few unforced errors. They used to make up the difference with their performance on the really difficult problems. Fewer difficult problems, slightly lower scores. (I am nearly certain that the reduced number of questions caused the decline noted when the SAT was changed in 2005.)

The ACT has far more questions than the SAT–215 to 171–and has no “guessing penalty”, which gives high ability students who make the occasional unforced error a significant advantage. To give an example: my son took the old SAT as an early junior and got 690 M, 660 V. I expected him to get high 600s, low 700s on the new one, which he took in March 2005. He received 630s across the board. After working on his accuracy, he took it again and received a 690,690, 670, or 2050.

His ACT scores were English 34, Math 34, Reading 36 (a perfect score), Science 29, which in SAT terms is high 700s across the board, or a 2250 using the UC conversion. At his performance level, that’s a huge boost. I have other anecdotal evidence, but they aren’t my kids so I can’t discuss specifics. Without question, all high ability kids should take both to see if they have a preference.

If taking both, which prep class should I take?

High ability students: take the SAT prep course. First, there are exponentially more SAT classes than ACT, even now. Asians, the primary consumers of test prep courses, don’t seem to take the ACT much (at least around here). Another major consumer, schools offering classes for their own students, also seem ignorant of the ACT.

Moreover, moving from the SAT to the ACT is far more organic than the other way round; the SAT has far more tricks and tidbits that a good test prep teacher can help with. Practicing for the ACT is little more than learning how to work fifty times faster on everything or, if that’s not possible, devising a strategy for getting as much done as possible. Did I mention the brutal timing requirements of the ACT? Oh, well, it bears repeating.

Low to mid-ability students: anyone planning a class aimed to low income, low ability students should select the ACT. Students with weaker abilities will receive more useful instruction, as it has fewer test-specific tricks and the test prep instructor will spend more time on content.

Who Shouldn’t Take the ACT?

The ACT is reading intensive–three of the four tests involve reading comprehension and two of those sections have (here it is for the third time) brutal time requirements. Students whose reading skills are significantly out of alignment with their other abilities (e.g., dyslexia, reading LDs), may want to stick with the SAT.