The SAT is Corrupt: Reuters Version

Dear Renee Dudley, Steve Stecklow, Alexandra Harney and Irene Jay Liu,

As someone who has studied and written extensively about the SAT corruption and the fraud delivery system known as Asian test prep , I congratulate you on the thorough job exposing the College Board’s open courting of corruption for profit in the overseas SAT market.

I’m not a reporter, just a teacher who does pretty good research, but for what it’s worth I think you did an outstanding job on many aspects of the story. I offer these suggestions (and one correction) with some disagreement, but little criticism (and lord knows, I can criticize).

The College Board Was Aware Long Before 2013
Your article strongly suggests time and again that the College Board learned that Asian test prep companies had obtained the tests by May 2013. In fact, the College Board knew that the test had been corrupted by January 2007, when it cancelled the scores of all 900 South Korean testers. It was common knowledge at the time that the College Board recycled old tests. The AP story doesn’t mention hagwons, and shows a touching faith in the CB’s assurance that kids couldn’t possibly benefit from seeing the test again. However, the College Confidential forums were very clear that the source of the test was a hagwon, not some “accidental exposure”. I worked for Kaplan and a major US hagwon at the time (only taught SAT at Kaplan) and my hagwon boss told me all about the methods Korean test prep companies used to get the “held back” tests.

So the College Board has known for at least nine years and probably longer that Asian international testers were cheating.

More Context on Cheaters

Your reporting begins with Xingyuan Ding, a Chinese national now attending ULA, who scored an 800 on the SAT Critical Reading section. Ding claims that “about half” of the answers on the reading section were in his “jijing”, or answer key.

That’s highly unlikely. While I understand the need for objectivity, some context would be useful. An 800 on the reading section is a 99th percentile score, meaning 99% of the testers receive a lower score. Even among Asians, it’s a 98th percentile score. So how likely is it that a Chinese national got that score?

Your reporting also mentions Linfeng Liu, another Chinese student who tested in Hong Kong, who says that she was “helped” by recognizing five—just five!—vocabulary questions, which enabled her, she said, to focus more on reading comprehension. Her overall score isn’t mentioned, but again, how likely is it that she cheated “just a little bit”?

Information that would have helped give context. Were these interviews conducted in Chinese or in English? How fluent was Ding, the 800 SAT critical reading guy, in English? How were these students doing in their classes?

Another related issue: In your FAQ, you mention repeatedly that just seeing the questions, not the answers, would still constitute a major advantage. That’s true. However, the story gives the impression that seeing the questions provides the primary, even sole, advantage to the students. This is almost certainly not true, and while that statement may be too strong, I think you tilt too far the other way. These are Chinese and Korean nationals with very weak English. A preview of the reading passages isn’t going to give them a big advantage. The answers do. Whether they’ve memorized long sequences or just have them handily tucked into a cell phone, the Chinese kids you interviewed almost certainly had the answers.

Cheating: It’s Not Just In China Anymore
Your reporting revealed that Asian test prep companies sent employees over here:

Sanli, the Chinese test-prep chain, says it sent 11 teachers to the United States to collect information on the redesigned exam. They debriefed 40 Sanli students studying at U.S. high schools who took the new SAT as they exited test centers, according to Wu, the general manager. Sanli presented its findings at a seminar at a Shanghai hotel.

(emphasis mine)

Heavens, that’s interesting. I did not know this. So are these Sanli students “Asian Americans”–born here, or immigrant children of long-time residents–or are they “Asian nationals” over here on F1 visas? As you probably know, Chinese students are flooding US public high schools whenever they can and US Christian private high schools when they can’t (due to the 1-year restriction), with our Beijing embassy eagerly courting more. Between 2009 and 2014, the number of F1 high school visas grew from 1,700 to 80,000, the vast majority of them Chinese or South Korean.

So when you report that Asian test prep companies are using “their” students to gain SAT knowledge to enable cheating, are they F-1 students on visas? Or are they Asian Americans who enroll with these test prep companies?

Of course, the College Board is not reusing national tests–at least, not often—which raises another question: what advantages do the Asian test prep companies offer Asians living in the US?

Your story hints at possibilities here:

Eight of the 10 existing Mathematics Level II subject tests were compromised – three in their entirety and five in part, including two exams that had never been given anywhere, the PowerPoint shows. Ten of the 13 Biology exams were also compromised in whole or in part, including one totally new test.

I’ve written on this before as well; Asian test prep companies have certainly used corrupt school administrators to gain early access to tests.

So the story probably goes beyond the international cheating–as if that’s not enough. It’s pretty clear that Asian test prep companies are getting early versions of tests. Given the College Board’s lax procedures, the leak may be from within the company.

What About the ACT?

You mention that the SAT dominates in the international market; my own research confirms that the ACT’s international presence is minimal. My understanding is that the SAT only recycles tests for the international sitting.

So what does the ACT do? Does the ACT recycle tests? What is the relative size of the two markets? If the SAT is the only test with a significant international presence, could it possibly be due to the fact that international testers have access to early copies? The overwhelming bulk of international testers are Asian or Middle Eastern, all countries with culture of cheating that Americans can barely comprehend.

Or perhaps there’s another explanation. But getting the ACT’s procedure would be a useful comparison point. Your story acknowledges the fact that ACT is the national leader, but apparently SAT’s touchstone status causes reporters to forget that and ignore the market king’s methods. (OK, that was a tad snarky.)

David Coleman Stonewalls

You know how reporters say [so and so] refused repeated requests to comment? You apparently only asked David Coleman to comment once.

David Coleman is a celebrity in the world of education reform. He is celebrated, rightly or wrongly, for Common Core standards. He took over the helm at the College Board in 2012–that fact alone should be mentioned, should it not? Presumably he was present, along with other “senior College Board staff” at the meeting with the Power Point slides in June 2013.

It’s not really your fault. But it’s over a day since your story came out, and not a peep from the College Board, much less Coleman. Couldn’t you have at least said “repeated requests”? Or did you only ask once?

National Interests

But like all reported stories I’ve seen on Asian cheating of the SAT, there’s no connection to the larger interests involved.

As your story mentions, many public and private universities are recruiting foreign students who are mostly from China and South Korea, even though the students are cheating on applications and tests, lying about their grades and resumes. Keep in mind that universities get tax breaks and other federal funding and public universities were chartered to serve the educational needs of their states.

Meanwhile, the SAT is moving outside its old beat as a college admissions test into a high school graduation test. Several states have committed use the SAT as a graduation requirement. Several states have switched from the ACT, which focuses on American students, to the SAT, which manifestly does not.

This isn’t just an issue for worried parents of college applicants. The College Board encourages and benefits from international criminal racketeering organizations that engage in immigration and mail fraud while enabling colleges to pretend they are accepting qualified applicants when in fact the colleges know full well their applicants lied. It collects money from multiple state contracts for a test product they can’t be bothered to spend money protecting from those organized criminal enterprises. State and private universities knowingly consume a fraudulent information product in order to fatten their coffers, all the while benefiting from tax-exempt status at both the federal and state interest.

Should the College Board be allowed to sell state contracts given its knowing participation in organized crime? Should our tax dollars be spent on universities if they are no longer acting in the public interest? Reasonable people can undoubtedly disagree on these questions. But surely they should at least be raised.

I want to emphasize again how pleased I am at your story and that while I had the above quibbles, overall it really was an excellent, thoroughly reported piece. I hope Reuters pushes it harder; you guys should be on TV, talking about this. Feel free to use my “national interest” take when you’re on the air.

All the best,

ER

What You Probably Don’t Know About the Gaokao

When I first read  Brook Larmer ‘s profile of 18-year-old Yang and his family inside Chinese test prep factory,  I was slightly skeptical of Larmer’s narrative regarding Yang, the young lad who exceeds his dreams on the intensely competitive gaokao.

Why was I skeptical? Well, while the novice might find Larmer’s article emotionally draining, anyone with even a rudimentary understanding of Chinese academic culture will notice a huge, gaping hole.

I noticed the hole, which led me to an observation, which led me to seek a better understanding of how the gaokao works, which led to my somewhat horrified understanding that the gaokao’s reality is almost  exactly the opposite of its reputation, at least as presented by the American media.

The hole: In a story dedicated to students preparing for the National Higher Education Entrance Examination (aka the gaokao) Larmer never once mentions cheating. This would be a problematic oversight in any event, but particularly so given the outcome of Yang’s efforts.

When Larmer returned to the town for his second visit, the day before the gaokao, Yang’s scores, which had been dropping, had not improved. As a result, Yang had kicked out his mom and brought his grandfather to live with him in Maotanchang for the last few weeks of prep. While Larmer drove into town with Yang’s parents, the grandfather refused to let Larmer accompany the family to the test site. Grandpa was afraid the family might “get in trouble” for talking to a reporter, according to “someone”.

Yang does exceptionally well, given his fears—“his scores far surpassed his recent practice tests”. Sadly, his friend Cao tanks because he “had a panic attack”.

Yang’s scores were considerably beyond what his recent performance had predicted. Yet it apparently never once occurred to Larmer that perhaps Yang and Grandpa prudently got the New York Times reporter out of the way before they paid for a guaranteed result. Maybe Yang wanted more aid than could be provided with “‘brain-rejuvenating’ tea”. Maybe Gramps didn’t want Larmer to see Yang wired up for sound. Or maybe the old man had really put in some money and paid for a double.

It’s possible, I guess, that Yang just had a great day and exceeded his wildest dreams.  But Larmer’s failure to mention the possibility is journalistic malpractice.

When I realized that Larmer hadn’t mentioned cheating, I read the piece again, thinking I must have missed it. Nope. But that second readthrough led to an observation.

I got curious—just curious, nothing skeptical at this point—about the school’s gender restriction on teachers. Was that just for cram schools? What was the gender distribution of Chinese teachers?

I couldn’t find anything. No confirmation that the teacher were all male, no comprehensive source on cram schools, no readily available data on Maotanchang. I couldn’t find anything at all about the school’s business practices online. So I went back to Larmer’s paper to look for a source for that fact—and nothing.

And so, the observation: In his description of the school’s interior and practices, Larmer doesn’t mention interviews with school representatives, other journalism, or a Big Book of Facts on Chinese Cram Schools.

The earliest detailed description of Maotanchang online appears to be this August 2013 article in China Youth Daily, a Beijing paper, which created quite a furor in China and largely ignored here because we can’t read Chinese. Rachel Lu, senior editor at Foreign Policy magazine, restated some key points for those folks who don’t read Chinese, which is nice of her, because what idiot would copy and paste the Chinese piece into Google Translate?

Yeah, well, I’m an idiot. I won’t bore people with the extended version, but a lot of the details that Larmer didn’t seem to personally witness show up in the Chinese story: same school official quoting management theory, teachers using bullhorns, Maotanchang’s 1939 origins, bus license plates ending in 8, burning incense at the town’s sacred tree, teacher dismissals for low scores.

The excitement over the China Youth Daily article generated more interest, like Exam Boot Camp, also written in August 2013, happily in English, which profiled a female student and her mother who provide data points like higher prices for lower scoring students ,lack of electrical outlets, and surveillance cameras in the classroom.

Am I accusing Larmer of lifting tidbits from these other stories? Well, let’s say instead that I’d like to know where he got the information.

Reading through these stories looking for sources led me to all sorts of “new things” to learn about the gaokao. These “new things” are readily available online; in fact, anyone can find most of the information in the Wikipedia entry. But I’ve never  read any of these not-in-fact new things but well-established facts explicitly laid out by any major media outlet (although now that I know, I can see hints). I don’t know why. I can’t even begin to see how any reporter wouldn’t trumpet these facts to the world, narrative or no.

China’s supposedly meritocratic test is a fraud.

To begin with, Larmer, like just about any other reporter discussing the gaokao, describes it as a “grueling test, which is administered every June over two or three days (depending on the province), is the lone criterion for admission to Chinese universities.”

Wrong. The test score is, technically, the sole criterion for admission. But in China, the test score and the test performance aren’t the same thing.

Testers get additional points literally added to their scores for a number of attributes. China’s 55 ethnic minorities (non-Han) get a boost of up to 30 points , although the specific number varies by province. Athletic and musical certifications appear to be in flux, but still giving some students more points, even though the list of certification sports culled from 70 to 17. Children whose parents died in the military and Chinese living overseas get extra points, and recently the government announced point boosts for morality.

Remember when the University of Michigan used to give students 20 points if they were black, and 12 points if they had a perfect SAT score? Well, imagine those points were just added into the SAT/ACT score. That’s what the Chinese do.

But even after the extra points are allotted, test scores aren’t relevant until the tester’s residence has been factored in. Larmer: “The university quota system also skews sharply against rural students, who are allocated far fewer admissions spots than their urban peers.”

I first understood this to mean that colleges used the same cut scores for everyone, but just accepted fewer rural students, without grasping the implications: city kids have lower cut scores than rural kids.

Xu Peng, the only Maotanchong student to make the cut off score for Tsinghua, where the “minimum score for students from Anhui province taking the science exam was 641.”

Two years earlier, the cutoff score for Tsinghua for a Beijing student was somewhere under 584.

Rachel Lu again:” the lowest qualifying score for a Beijing-based test-taker may be vastly lower than the score required from a student taking the examination in Henan or Jiangsu. [rural provinces]. ”

A joke goes:

Of course, don’t make the mistake, as I did, of thinking the cut scores mean the same thing for each student.

Curious about the nature of the studying/memorization the students do (another vague area for Larmer’s piece), I tried to find more information on the gaokao content. The actual gaokao essay questions are usually published each year and they’re….well, insane.

When I finally did find an an actual math question (note: I only linked to the actual chart, which has disappeared. But it was a simple graph.)

it seemed surprisingly easy and then, I realized that it was only for the Beijing test:

Then I went back to the essay questions and it sunk in: the essay questions differed by city.

The gaokao isn’t the same test in every province. Many provinces develop their own custom test and just call it the gaokao.

At which point, I threw up my hands and mentally howled at Larmer, my current proxy for the mainstream American press: you didn’t think this worth mentioning? Or didn’t you know?

If all this is true, then the wealthier province universities use a lower cut score for their residents. But just to be sure, some provinces make an easier test for their residents, so that the rural kids are taking a harder test on which they have to get a higher score. Please, please, please tell me I’m misunderstanding this.

Consider Larmer’s story again in light of this new information. Larmer can’t say definitively who had the best performance without ascertaining whether Yang or Cao got extra points. Both Yang and Cao might both have outscored many students who were admitted to top-tier universities. Cao may or may not have “panicked”, and may not have even done poorly, in an absolute sense. None of this context is provided.

In my last story about Chinese academic fraud, I pointed out that so much money was involved that few people have any incentive to fix the corruption. All the people bellyaching about the American test prep industry should pause for a moment to think about the size of the gaokao enterprise. The original China Youth Daily story focused on Maotanchang’s economic transformation, something Larmer also mentions. Parents are paying small fortunes for tutoring, for cheating devices, for impersonators, for bribes for certificates. All of these services have their own inventory supply chains and personnel. Turn the gaokao into a meritocratic test and what happens to a small but non-trivial chunk of the Chinese economy?

But I’m just stunned at how much worse the Chinese fraud is than I’d ever imagined.

Sure, well-connected parents could probably bribe their kids into college. Sure, urban kids who had better schools that operated longer with educated teachers would likely learn more than those stuck with “substitutes”. Sure, the content was probably absurd and has little relationship to actual knowledge. Sure, the tests were little more than a memory capacity game, with students memorizing essays as well as facts that had no real meaning to them. Without question the testers were engaging in rampant cheating.

But not once had I considered that the test difficulty varied by province, that some kids got affirmative action or athletic points added directly to their score, and worst of all, that a kid from Outer Nowhere who scored a 650 would have no chance at a college that accepted a kid from Beijing with a 500.

Once again, I am distressed to realize that my cynical skepticism has been woefully inadequate to the occasion.

The gaokao isn’t a meritocracy. Millions of kids who live in the wrong province are getting screwed by a test whose great claim to fame is that it will reward applicants strictly by merit. And of course, the more kids who apply to college, the more cut scores and test difficulty will increase–but only for those students from those wrong provinces. Meanwhile, the kids from the “right” provinces have a (relatively) easy time.

In this context, the 2013 gaokao cheating riot takes on a whole new light. If you really want to feel sad, consider the possibility that Yang’s friend, Cao, now working as a migrant, might have scored higher on a harder test than a rich kid in Shanghai.

By the way, could someone alert Ron Unz?

*Note: in the comments, someone who understands this is (bizarrely, to me) fussed over my use of the “rural/urban” paradigm. I was using the same construct that Brooke Larmer and others have. The commenter seems to think it makes a difference. My point is simpler, and I don’t think obscured for non-Chinese readers. But I caution anyone that I’m utterly unfamiliar with Chinese geography.

The SAT is Corrupt. No One Wants to Know.

“We got a recycled test, BTW. US March 2014.”.

This was posted on the College Confidential site, very early in the morning on December 6, the test date for the international SAT.

Did you get it?

Get what?

I mean how do you know it was a recycled Marhc test? Do you have the March Us test?

Oh, no. I just typed in one of the math questions from today’s test and the March US 2014 forum popped right up.

And of course, the March 2014 test thread has all the answers spelled out. The kids (assuming it’s kids) build a Google doc in which they compile all the questions and answers.

This is a pattern that goes on for every SAT, both domestic and international. The kids clearly are using technology during the test. They acknowledge storing answers on their calculators, but don’t explain what allows them to remember all the sentence completions, reading questions and even whole passages verbatim, much less post their entire essay online. Presumably, they are using their phones to capture the images?

They create a google doc, in which they recreate as many of the questions as can be remembered (in many cases, all) and then they chew over the answers. By the end of the collaboration, they have largely recreated the test. They used to post links to openly with any request. But recently the College Confidential moderators, aware that their site is being exposed as a cheating venue, have cracked down on requests for the link, while banning anyone who links to the document.

So floating out there somewhere in the Internet are copies of the actual test, which many hagwons put out (and pull them down because hey, no sense letting people have them for free), as well as the results of concentrated braindumping by hundreds of testers.

For international students, “studying for the SAT” doesn’t mean increasing math and vocabulary skills, but rather memorizing the answers of as many tests as possible.

And those are just the kids that aren’t paying for the answers.

The wealthy but not super-rich parents who want a more structured approach pay cram schools–be they hagwons, jukus or buxiban–to provide kids with all the recycled tests and memorize every question. No, not learn the subject. Memorize. As described here, cram schools provide a “key king”, a compilation of all the answer sequences for sections, using all the potential international tests. They know which ones will be recycled because the CB “withholds” these tests.

Of course, the super-rich parents don’t want to fuss their kids with all that memorizing. Cram schools have obtained copies of all the potential international tests by paying testers to photograph them. Then they pay someone to take the SAT in the earliest time zone for the International, and disseminate the news via text to all the testers. They just copy the answers from the pictures. Using phones. Which they have told the proctors they don’t have, of course.

I don’t know exactly how all this works—for example, are the cram schools offering tiered pricing for key kings vs. phoned in answers? Do different cram schools have different offerings? I’ve read through the documented process provided by Bob Schaeffer of FairTest (a guy I don’t often agree with), and it seems very credible. He’s also provided a transcript of an offer to provide answers to the test. Valerie Strauss got on the record accounts of this process from two international administrators, Ffiona Rees and Joachim Ekstrom.

Every so often Alexander Russo complains that Valerie Strauss shouldn’t do straight education reporting, given her open advocacy against reform.

Great. So where’s all the other hard reporting on this topic? The New York Times, whose public editor Margaret Sullivan just encouraged to “to enlighten citizens, hold powerful people and institutions accountable and maybe even make the world a better place”, bleeds for the poor Korean and Chinese testers anxious for their scores and concerned they’ll be tarred with the same brush. Everyone else just spits out the College Board press release–if they mention it at all. While most news outlets reported the October cancellation, few other than Strauss reported that the November and December international tests scores were delayed as well.

At the same time Strauss reported the College Board is stonewalling any inquiries as to how many kids were cheating, how many scores were cancelled, or what it was doing to prevent further corruption, an actual Post “reporter”, Anna Fifield, regurgitates a promotional ad for a Korean SAT equivalent coach.*

Well, you can understand why. The millionaire Korean test prep coach-called-a-teacher story is one of the woefully underreported stories of the 21st century. I mean, we only had one promo put out by the Wall Street Journal the year before, and another glowing testimonial CBS a few months later (even mentioning the tops in performance, bottom in happiness poll). But really, only one or two a year of these stories have been coming out since 2005.

So you can see why the Post felt another story on a Korean test prep instructor making millions required immediate exposure, if not anything approaching investigation or reporting.

These stories are catnip to reporters who get all their education facts from The Big Book Of Middlebrow Education Shibboleths. First, unlike our cookie cutter teacher tenure system, Korean teachers work in a real meritocracy where kids and their parents reward excellence with cash. Take that, teachers!

Then, unlike American moms and dads, Korean parents care about their kids and put billions into their education. Take that, parents!

And oy, the faith Anna shows in her subjects. Cha is a “top-ranked math teacher” who “says” he earns a “cool $8 million last year.” Cha says he’s been teaching for 20 years, but refuses to give his age and there’s no mention of the topic or school he attended for his PhD, or if he ever got one. But he’s got a really popular video, so he must be great!

Some outlets are less adulatory. The Financial Times points out that the Korean government is cracking down on hagwon fees and operating hours, and preventing them from pre-teaching topics. Megastudy, the company in the 2005 story linked in above, just went up for sale because of those government changes. Michael Horn of the Christiansen Institute is doing no small part to alert people to the madness of the Korean system. The New York Times, despite its tears for the Korean and Chinese testers, has done its fair share to report on the endemic cheating in Chinese college applications.

But when it comes to the College Board and the SAT, everyone seems to be hands off the international market. At what point will it occur to reporters to seriously investigate whether a large chunk of the money spent on cram schools is not for instruction, but for “prior knowledge” cheating? When will they ask the Korean cram school instructors if they are fronts for an organized criminal conspiracy, if the money they get is not for tutoring, but for efficient delivery of test answers on test day? And how many of those test days are run by the College Board?

People think “well, sure, there’s some cheating, but so what? Some kids cheat.” Yeah, like I’d be writing this if it were a few dozen, or even a few hundred kids. Asian immigrants cheating on major tests in this country is in the high hundreds a year. Maybe more. In China and Korea? I suspect it’s beyond our comprehension, us ethical ‘murricans.

One of the depressing things about the past three years is that I start looking into things more closely. I never really trusted the media, mind you, but I did assume that journalists skewed stories because of bias. I fondly imagined, silly me, that journalists wanted to investigate real wrongdoing. Yes. Laugh at my foolish innocence.

Consider what would be disrupted if public American pressure forced the College Board to end endemic international student cheating. First, the CB would lose millions but weep no tears, it’s a non-profit company. hahahahah! Yeah, that makes me laugh, too.

But public universities increasingly rely on international student fees and the pretense that they are qualified to do college work. After all, the thinking goes, we accept a lot of Americans who aren’t prepared for college work—may as well take in some kids who pay full freight. Private schools, too, appreciate the well-heeled Chinese students who don’t expect tuition discounts.

So suppose public pressure forces the College Board to use brand new tests for the overseas market, require all international testing to be done at US international schools, use different tests at different locations. The College Board might decide that the international market profits weren’t worth the hassle for other than US students living abroad (as indeed, the ACT seems to have done for years). Either way, a crackdown on testing security would seriously compromise Chinese and Korean students’ ability to lie about their college readiness and English skills.

A wide swath of public universities would either have to forego those delightful international fees or simply waive the SAT requirement, but without those inflated test scores it will be tough to justify letting in these kids over the huge chunk of white and Asian Americans who are actually qualified. No foreign students, more begging for money from state legislatures. Private universities would have a difficult time bragging about their elite international students without the SAT scores to back thing up.

Plus, hell, we changed the source country for zombies because we didn’t want to piss off China. Three years ago, the College Board wanted to open up mainland China as a market. 95% of the SAT testers in Hong Kong are Chinese. Stop all that money flowing around? People are going to be annoyed.

At this point, I start to feel too conspiratorial, and go back to figuring that reporters just don’t care. I’ve got a lot of respect for education policy reporters—the Edweek reporters are excellent on most topics—and most reporters do a good job some of the time.

But the SAT is basically corrupt in the international market. I’ve already written about test and grade corruption among recent Asian immigrants over here, particularly in regards to the Advanced Placement tests and grades.

Yet no one seems to really care. Sure, people disapprove of the SAT, but for all the wrong reasons: it’s racist, it’s nothing more than an income test, it reinforces privilege, it has no relationship to actual ability. None of these proffered reasons for hating the SAT have any relationship to reality. But that the SAT is this huge money funnel, taking money from states and parents and shoveling it directly or indirectly into the College Board, universities, and the companies who have essentially broken the test? Eh. Whatever.

The people who are hurt by this: middle and lower middle class whites and Asian Americans. So naturally, who gives a damn?

enlighten citizens, hold powerful people and institutions accountable and maybe even make the world a better place

Sigh. Happy New Year.

*****************************
*In the comments, an actual SAT prep coach making millions–no, really, he assures us, millions!–simply by being a fabulous coach with stupendous methods is insulted that I insinuated that the Washington Post story was on an SAT prep coach, rather than the Korean equivalent of the SAT. I knew that, but at one point referred to the guy as a SAT prep coach. I fixed the text.

College Confidential and Brain Dumping the SAT

SAT Scores Delayed for Asian International Students

The above is the official story put out by the Washington Post, which is far more informative than any other outlet I could find. However, Valerie Strauss put some other information in two blog entries:

On Oct. 8, 2014 — days before the Oct. 11 administration of the SAT — the National Center for Fair & Open Testing received an anonymous tip about cheating that included what the sender claimed to be a copy of the December 2013 SAT that was supposedly going to be administered at international sites Oct. 11. This was reported by Bob Schaeffer, public education director of the center, a nonprofit dedicated to ending the abuse of standardized tests commonly known as FairTest. He said FairTest tried to confirm the claims but could not.

According to Schaeffer, SAT tests given at international sites are “almost always” repeats of exams administered previously in the United States but not publicly released.

Students began to think that the October 2014 international version of the SAT was identical to the December 2013 U.S. version by Googling some vocabulary words and passage topics and finding that the 2013 test was the one that came up in discussions threads on “collegeconfidential.com,” according to Schaeffer. It is not yet clear, however, whether the two tests are identical.

I’ll have more to say about the media coverage, but I got distracted by reading up on College Confidential. I’ve always been skeeved by the forum, but that’s because I’m usually researching the test threads which are almost certainly populated by Asians and Asian Americans. No doubt the forums have other purposes; I hear parents frequent them. Little has been written about the forum;the NY Times wrote a feature about it that seems out of date. Quantcast shows that Asians represent 13% of the users, considerably above average. 18-24 is the largest age group, 45-54 is second. So it’s clearly not just used for college tests.

Anyway, I read the college confidential thread, which was opened back in early November for the December test, but from page 4 to page 70 is nothing but brain dumps. The posters make reference to Tiny Chat, a conferencing chat room, and google docs, where they are clearly compiling a list of all the answers. Many posters are putting down all the answers they can remember, in specific detail. One poster lists all the math answers by section (page 57, 58, page 59):

A few weeks later, a new thread is opened for the December international test, held on December 7th—and posted so early that the thread date was December 6th (the forums on US time, I assume). In response to the creator’s query, another poster announces that the December international test was a reissue of the June 2012 test, and for good measure gives a table:

JAN 2013- MARCH 2010
MAY 2013- JUNE 2009
JUNE 2013- MARCH 2012
OCTOBER 2013- MARCH 2013
NOVEMBER 2013- JUNE 2011
DECEMBER 2013 JUNE 2012

One thread asked about the December 7 international test

The poster is then sent to the June 12th thread, where again, all the answers are put down. One person (poster name largeblackman. I am deeply skeptical) posts reading section answers.

These are the only two months I checked.

Someone reading this going to say “I did this back when I took the SAT. Chewed over everything I remembered with my friends, worried if we didn’t get the same answers.” Well, no. You didn’t do this. Some of the posters are going into shocking detail. They have question numbers, letter answers. A good chunk of the posters were clearly coordinating the creation of a complete document with all the questions and answers.

They were braindumping, an activity that Microsoft spends a lot of time and energy preventing, but the College Board seems to actively encourage by reusing old tests for international students.

No wonder Asians have such a strong preference for the SAT. The credulous press tends to believe in the super tutors of Asia, but they’re much more likely to be New Oriental “prep” methods revisited. Steal the test, then memorize everything on it. GMAT had similar issues.

Valerie Strauss quotes the head of an international school who caught a cheater: This is certainly organized crime.

I suppose it’s possible that all these posts at College Confidential are just 17-year-olds pranking each other. I find that unlikely. More probably, the posters in question aren’t all 17, but adults who are paid to go in and take the tests while photographing or at least memorizing as much of the test as is possible. Or at the very least, the posters are actual high school students coordinating information illegally. Certainly, someone should at least investigate: ask the owners to provide the IP addresses, actually read the threads, ask the posters to produce the google docs they mention, find the actual names of people who participated.

But universities want the Chinese money, and College Board wants the test fees, and the FBI has to keep watch on Ferguson so that Holder can admonish the grand jury when Darren Wilson isn’t indicted. Who has the time or inclination to investigate a possible organized criminal enterprise that’s corrupting our educational institutions?

SAT’s Competitive Advantage

Asians taking the SAT: 196,030 out of 1,640,047, or about 11.9%.
(No response: 62,603)

Asians taking the ACT: 71,677 out of 1,799,243, or about 3.9%
(No response: 110, 305)

I can’t tell if these numbers are US-based only, or including overseas testers. The ACT has a small international market. The SAT dominates, particularly in the Asian countries. 50,000 Chinese visit a Hong Kong testing site in just one year. While I couldn’t find totals for Korea, 900 students got their scores cancelled in 2007, clearly just a fraction of the total testers. In India, the College Board has 32 test centers (up from 20 just four years ago), the ACT has 60 testers.

So if the ratio above, in the tests’ national reports, are indeed just for the US, then the overall Asian preference is even greater than the 2.75:1 brand preference for Asian American testing. Since I can’t find any verification of this, let’s assume the ratio is overall preference.

In either case, the numbers are stark. The SAT is already losing out to the ACT. Why would it make changes that, if accurately described, would make the test less attractive to Asians, who have largely rejected the ACT?

Before I get to that: outside of the knowledgeable education reporters, the coverage has been nonsense. Contrast this relatively solid piece by NYTimes reporter Tamar Lewin, (whose twitter tagline should be “not half as ignorant and less than a quarter as irritating as Motoko Rich”) with this obsequious David Coleman profile by author Todd Balf. Discussing his work, Balf described the “checks” that restrain the College Board from using its power, :

Either Coleman does something about [the dissatisfaction with the SAT], or at the college level more and more schools will go test optional. Coleman wouldn’t say that the reforms are in response to places like Wake Forest, but it is hard to ignore.

Sigh. A few thousand colleges or university systems use the SAT as a first round placement test: University of Georgia, University of California, University of Michigan, University of Nevada, University of Wisconsin, a whole bunch of community college systems, and oh, yeah, the largest university system in the country. Any number of private universities do, as well, including Baylor, Columbia, and Duke.

So the SAT and ACT allow many schools to avoid the hassle and expense of placing hundreds of thousands of students each year.

But yeah, the College Board is all worked up about Wake Forest going exam optional.

People who try to prognosticate the changes are mostly being silly. I like Peter Wood, but his rant here doesn’t add up. Like most non-experts, Wood completely ignores the existence of the ACT, which has the same “biases” that Wood thinks the SAT is trying to fix. The College Board can’t “dumb down” the test. If blacks suddenly started scoring at the 80th percentile on the SAT, but were still scoring at the 35th on the ACT, people would notice.

So leaving aside what people who don’t really understand things think, and leaving my random thoughts about certain changes for another day, and then pointing out that I liked this analysis by Eduwonk, not usually a favorite of mine, my big question is still why would the College Board deliberately screw over Asians, the customer base with the strongest preference for its product?

The Chinese test prep companies are less than thrilled about the changes, which they believe will prevent them from teaching students to game the test . I am asserting this as fact: many Chinese international students openly acknowledge they don’t have the underlying ability their SAT scores denote.

Not much outrage from the Koreans, but then they just buy early versions of the test and distribute it to paying customers. Obviously, this method works regardless of underlying test modifications. Cheating is so rampant in South Korea that again, I am asserting as fact that many of these testers lack the underlying knowledge that their SAT scores supposedly indicate.

I doubt either country is exclusive in its approach. Presumably, the Chinese cheat and the Koreans game.

Indians seem cautiously optimistic—hey, they mostly do speak English, right?

I consider the SAT changes completely irrelevant. The interesting question to me is, assuming no one changes their behavior, how will this all play out?

The one thing I feel pretty confident of: faith in Coleman is wholly unjustified. Leaving aside the value of his Common Core work, note the lack of follow-through. Typical consultant, he jumped in long enough to tell people what to do, then left before all the errors in his ideas are revealed through disastrous implementation results.

In the same way, Coleman’s never run a major business, and has no idea about the political realities of helming the most visible, if not the most popular, college admissions and placement test. He has a story, but no plan. Meanwhile, boy, he’s happy to diss the test his predecessors pushed—and he makes free with insulting his competitor, too.

But for all Coleman trashes his predecessor, Gaspard Caperton, the ex-Appalachian governor has big shoes that I very much doubt Coleman can fill. Complain all you like about the writing section, it was pure genius as a business decision. On Caperton’s watch, the College Board dumbed down an existing test that a couple hundred thousand kids took, made it cheaper to grade, and forced a million kids to pay a higher price to take it as part of the “new” SAT, even though lots of colleges don’t use the section score—and let everyone think that the UC forced the change. The company made millions in a single year. At his behest, the College Board abandoned the claim that test prep doesn’t work, and sold its own product, which had to be taken off the Amazon bestseller list because everyone found it too upsetting. And that’s just the SAT’s profit center. Caperton also presided over the Advanced Placement’s stunning growth, achieved by taxpayers shoveling money into the CB’s coffers for entirely unqualified kids to take tests in order that Jay Mathews put their school on his Index.

In contrast, Coleman has just made the essay optional and harder to grade, ordered a complete redesign of the test with a stable competitor ready and willing to pick up the doubters, and pissed off his most dedicated customer segment. I wouldn’t be surprised if he declared that his purer company won’t profit from the filthy lucre of test prep, having turned that function over to Khan Academy (which long before its College Board arrangement has neglected to provide free services for the ACT). God knows what Coleman has planned for the AP suite.

Coleman could have some grand plan that I can’t anticipate (other than, as Steve Sailer puts it, all students should Be Like Me.) But reasonable people can and should wonder if he is wholly ignorant of the SAT’s market position, or if he actually believes that his Common Core curriculum is going to increase student ability and end the achievement gap. From there, it’s easy to postulate a scenario in which the College Board spends a fortune redesigning the test in a vacuum, not discovering until field tests that the changes either widen or narrow the achievement gap unforgiveably. Worse still, what if they make the test too hard? All of these possible outcomes wouldsend testers and their parents running into the arms of the Midwestern Mama, all safety and security.

In such an event, I trust the remainder of the College Board regroups before rollout. That’s been done before: the 2005 essay was originally supposed to be 50% of the writing score, but worrisome field tests led to the essay getting just a third of the section weight. A complete backout, cancel, reset is not impossible. The ETS cancelled its first set of GRE changes, despite a significant investment. Like the ETS, the College Board has a known quantity of a test to fall back on—just fire Coleman and move on.

I find that outcome very possible. But suppose that the SAT redesign works as promised. The achievement gap remains intact, the test is as described, the scores among American students are roughly compatible with the old SAT and the current ACT.

So then back to the big question: why, exactly, are Asians favoring the current SAT so strongly, and what will they do in the face of an altered test? The usual reason offered for their brand loyalty is the Asians aren’t aware of the ACT.

They figure out to game the SAT, they buy advanced copies of the SAT so their kids can cheat, they’re aware of and concerned about SAT changes, but somehow they’ve never heard of the SAT’s competition. Yeah. Not buying.

That leaves a few possibilities. First, the Asian test prep companies just don’t see the point in spending the money and time cracking a second test given their certainty on the first. Second, they haven’t actually cracked the test, aren’t really gaming it, but in fact are all using the Korean method of buying the test. All the coverage on the gaming is just sham. So they either can’t or haven’t yet paid to achieve the same penetration of the ACT’s secrets.

Third, and this is the fun one, they’ve tried cracking the ACT and can’t.

Eighteen months ago, I thought the possibility that Asians were artificially inflating their scores was theoretically possible, but unlikely. Over the next year, my experiences and additional research has changed my view. I now think it likely that both here and overseas, a decent percentage of Asian testers are either cheating or gaming the SAT and AP tests—-or both. I’ve written up some of this but not all, and I’m not expecting anyone to just take my word for it.

But when I start from that premise, and look at the SAT as a test that can be gamed, I see loads of potential.

Naturally, a purchased test is still the best guarantee—certainty being so much more reliable—but Big Data could identify many patterns. A few years back, you could see open discussion of an LA based test prep company with a largely Asian clientele that promised a reduced (300 or fewer words) vocabulary list, that was supposedly built by on analyzing previous tests and predicting a rotation cycle for the words. Or maybe they just had an in at the College Board, as the comment suggests.

Maybe they’ve analyzed each type of reading comprehension question and noticed a pattern beyond “pick C”, but rather a particular type of wording that is associated with a correct answer. This is something all bright people can do, of course, but it’s hard to teach as a system without mountains of data. The SAT’s grammar questions have all sorts of patterns that could likewise be broken down and systematized. Obviously, they’ve collected the essay prompts and found certain theme patterns that they use to teach kids to memorize essays verbatim. It goes without saying that the students didn’t write the essays. Or maybe they just had prior knowledge.

But assume, for the moment, that it was gaming, not cheating. I’m not an expert on Big Data or psychometrics, but I’m knowledgeable about both the SAT and the ACT, and the latter doesn’t lend itself to that sort of patterning. English, Reading, and Science test on passages with aggressive time requirements. While their questions do have patterns, the patterns are heavily reliant on context. You have to understand the text at least slightly in order to find the correct answer for the particular type of question. At least, that’s how it seems to me. (But then, I looked at the memorization necessary to game the SAT and thought it unlikely, so what do I know?)

Imagine that College Board successfully screws over the barely-English-speaking Asian market with their new test. Suppose also that both companies are ethical and no one is selling advance data on the test (which, in fact, I do assume). Suppose, in other words, that Asians with extremely limited English skills, here and abroad, are no longer able to misrepresent their abilities with SAT tests.

Is that really what colleges want?

Balf’s right about one thing–the colleges are the ultimate test customer. Public universities have dramatically increased their foreign admits, most of them Chinese, because they can charge them out of state fees for all four years.

They admit these students already knowing that the Chinese applications are largely fraudulent, and are well aware of the gaming and cheating, since they end up with students who can’t speak English and cheat here, too.

All this knowledge hasn’t slowed down their Asian international student admission.

Does this sound like a customer base that really wants to lock out non-English speaking Asians?

No matter which of these scenarios play out, I don’t see how the ACT doesn’t benefit. Why would any student opt for a complete unknown when they can prepare for the ACT using test prep materials and experts who’ve been working with the same test for years? I expect test prep companies to step up their offerings. If they don’t, then I throw up my hands and declare confusion.

This all strikes me as considerably more relevant than pipe dreams about the end of test prep, much less the achievement gap. And much more interesting.

Why I teach enrichment, reasons #1138 and beyond

I teach summer school because compared to regular school it’s like free money.

But then, I teach enrichment classes in reading and composition, as well as PSAT prep, to kids whose parents make them show up and are too compliant to rebel–that is, I teach Asian first generation immigrants (Chinese, Korean, Indian, and the occasiohnal Vietnamese). I tease them about their compliance mercilessly, and they are, for the most part, fascinated by a teacher who tells them they should go home and watch more TV.

Brief anecdotes:

• I gave them an essay question, “Should school grades be determined entirely by an end of year test?” All but two of the 18 students took positions emphatically in the negative, and all gave the same reason: kids would just goof around and cram at the end.

  I pointed out the implicit value system inherent in their answers: kids only care about grades, not about learning. I said tht this was a big problem, as I saw it, with their attitude towards school and it was something they might want to think about. However, I said, regardless of their opinion, they could make their essay much stronger by attributing this unattractive behavior to other students. “Regrettable as it is to consider, many students have no interest in learning, and are only interested in getting a good grade.” Now they have distanced themselves from the value system, while still holding onto their position and their argument—which had obvious merits.

  And I reminded them that my course had no grade. So why were they in the class? What was their purpose? I hoped they would focus on my big three: better thinkers, better writers, richer vocabularies.

• Every year, we get a day off for the Fourth of July. Every year, I ask my class what the holiday celebrates. Every year, at least one kid doesn’t know. Every year, at least one of the kids who does know doesn’t know the year. And every year, that kid got an A in 8th grade US History. Every year, I point out that this is a symptom of a kid who wants to get an A, not learn about American history, and that all of the kids in this class have similar gaps—stunning ignorance in subjects where they learned to get the A and never thought about it again.

  And what I truly love about this class is that every year, this lecture makes a huge impression. The kids understand my point. They understand that I value knowledge, not grades, and for the first time, they realize the utter emptiness of an A achieved without actually learning anything.

• Another “every year” conversation:
  “What courses are you signed up for?”

  Typical rising sophomore answer: “Calc A/B (or Math Analysis), Honors Chem, Spanish III, World History, English II.”

  “Why not Honors English or WHAP (World History AP)?”

  “It’s too hard.”

  Please remember that, the next time you hear about how Asians take tough classes. They don’t take tough classes. They take classes that everyone else (read, white people) find tough. The classes they find tough, they skip.

• It is not at all uncommon to find exceptional writers. This is not encouraged by their parents.
• One of my few classroom rituals is “do your two”. Students are required to read two detailed columns four times a week. The column must either be an opinion piece or informational reporting. It can be on any topic: sports, movies, politics, news. They have to report back. I then intersperse my opinions, ensure that they’ve read it accurately, and understood the purpose. Over the course of the summer, I observe the kids picking up more general content knowledge (the purpose of the exercise), but they also start building on each other. So one kid reports on the Euro status, and another kid sees an article two days later that they click on because of the first kid’s report. It’s a very useful exercise.

  Today, one of my students discussed a new study showing that most people don’t actually listen very well. (I’ll find the link later) This generated a fun discussion, culminating in my announcement that I liked to think my students listened to me. Was I wrong?

  Joey started suddenly, looked up, and said, “Uh, what?”

  Little punk.