The Takeaway from the TFA Study

Jersey Jazzman’s list of cautions mirrors my own on the TFA study. I would add that the TFA skew towards middle school (in the study) makes a big difference; TFAers push testing excitement heavily in middle school, something that’s very tough to do with high schoolers.

I had an interesting twitter exchange with Morgan Polikoff who, can I just say, makes me feel both ancient and unproductive, about the “significance” of the TFA improvement margin. He explained that hey, that’s what researchers have always used. Well, yeah. I know that. I’m not questioning the study’s use of that particular metric, I’m questioning the value of that particular metric.

Recently, the august NY Times declared that “Guesses and Hype Give Way to Data in Study of Education” but alas, all this knowledge “has another hurdle to clear: Most educators, including principals and superintendents and curriculum supervisors, do not know the data exist, much less what they mean.”

Y’all are going to tell us about it, right? While us teachers are supposed to just go “Wow, thanks! We’ll go put this in action!” (Now I sound like a typical teacher who values practice over data, which I actually don’t, and I find this very annoying.)

But in fact, teachers are actually reading more research than at any time in the past, if only because reformers are presenting us all as a bunch of incompetent buffoons whose results aren’t just equaled, but improved, by smarter, ambitious, desirable college graduates who are in it for the resume improvement. It’s just slightly possible, isn’t it, that when a bunch of teachers become familiar with “research practice”—created by people who usually have very little teaching experience or have been told to ignore that experience in favor of “research practice”—they might question that practice?

Note to those who want to start “educating” teachers on research: expect pushback if you try to sell “significant” improvement that translates to something like a couple questions more on a test. Here’s a small sample of what math teachers will want to know on this particular study: did the TFAers do better at teaching linear equations, factoring quadratics? Were the additional correct questions random, or specific to a content area? Was the improvement consistent over each subject taught, or specific to one subject? Did the classrooms have similar behavior referral rates? Mind you, we’ll see the “significance” as irrelevant either way, but at least we’ll know whether it’s “real”.

But accept the results as “significant” and here’s the big takeaway, hinted at by Dara Zeehandelaar of Fordham:

Both TFA and Teaching Fellows have less experience than their peers, are less likely to be minorities, more likely to have graduated from more selective colleges, less likely to be math majors but more likely to score higher on tests of math content. However, only years of experience and, for high school, math content knowledge were associated with higher student achievement. These findings add to the glut of research indicating that traditional certification programs could benefit from greater selectivity, indeed from a radical overhaul.

Dara is celebrating a future teaching force with fewer blacks and Hispanics. Me, I look at it from the opposite side of the mirror, where it reads something like this:

CAEP, the ed school accreditation organization, is setting new standards that include higher candidate selection benchmarks. Selective ed schools have been left some wiggle room, in that the cohort, not each individual candidate, has to have an average GPA over 3.0 and average test scores in the top third of a nationally normed achievement test. However, it will annihilate predominantly black/Hispanic teacher schools, which include no small amount of public universities. Only 6% of African Americans and 10% of Hispanics get over 600 on each section of the SAT, which is roughly the top 30% nationwide.

So if CAEP doesn’t blink, or ed schools don’t get creative, we will soon have almost no black and Hispanic teachers, since blacks and Hispanics who get over 600 on each section of the SAT go off to become doctors and lawyers and Wall Street hedge fund managers. While the test score requirements will almost certainly be loosened or eliminated as the import becomes clear, our nation will see a lot fewer black and Hispanic teachers. We’ve already cut into the supply back during the NCLB changes, and people are already scratching their heads about these “missing” minority teachers, as Mokoto Rich of the New York Times has termed this, blithely ignoring the reality right in front of her. If the CAEP standards have the intended effect, we’ll lose even more.

But that’s okay, right? Because sure, we’ll lose in the current generation, but the next generation, taught by those newer, brilliant teachers who really care about their kids and aren’t just doing it because it’s some pesky job will raise achievement! Blacks and Hispanics will score the same as whites and Asians! Stricter drug laws will eliminate addiction! Tougher gun laws will prevent Newtowns! Dogs and cats will live together in peace and harmony! Hell, Israel and Palestine will straighten things out.

So the TFA study gives us a preview of that brave new world. TFAers were three times as likely to be white; the control group of teachers had seven times as many blacks. And lo! you see the predictable one standard deviation difference in test score means between the groups. The TFA group is exactly the
highly qualified, selective crew of new penny bright teachers everyone says they want.

And you get .07 of a standard deviation difference in student outcomes.

Of course, we already knew that, since it was an example of what the clearinghouse on education research knows doesn’t work:

For example, Michael Garet, the vice president of the American Institutes for Research, a behavioral and social science research group, led a study that instructed seventh-grade math teachers in a summer institute, helping them understand the math they teach — like why, when dividing fractions, do you invert and multiply?

The teachers’ knowledge of math improved, but student achievement did not.

“The professional development had many features people think it should have — it was sustained over time, it involved opportunities to practice, it involved all the teachers in the school,” Dr. Garet said. “But the results were disappointing.”

So maybe we can add the TFA study to the clearinghouse as additional evidence that past a certain point (unknown), increased teacher ability doesn’t result in improved student achievement. Or are we going to still pretend that .07 of a standard deviation is 2.6 months of instruction, that, as Jerseyman says, the increase is actually “practically” instead of “statistically”, significant?

Both reformers and progressives push “improve teacher quality” as an easy mantra that really doesn’t have much basis in fact. However, reformers go farther. Reformers look at the existing state of affairs and see obvious failure, failure so manifest that it’s a simple matter to fix. Low test scores? Give them teachers who care. Smarter teachers. Higher standards. Over the past decade, their enthusiasm has been blunted a tad by the realization that at best, their “obvious” improvements, if you squint really hard and pretend peer environment is irrelevant, will improve outcomes a squidge around the edges. But still, they keep coming back for more. And so, they push this study as evidence that TFA works, not realizing that the study foretells the lackluster improvement they’ll see at the expense of a virtually closed career path for blacks and Hispanics.

That’s the takeway of the TFA study.

Can someone mention this to CAEP?