The Takeaway from the TFA Study

Jersey Jazzman’s list of cautions mirrors my own on the TFA study. I would add that the TFA skew towards middle school (in the study) makes a big difference; TFAers push testing excitement heavily in middle school, something that’s very tough to do with high schoolers.

I had an interesting twitter exchange with Morgan Polikoff who, can I just say, makes me feel both ancient and unproductive, about the “significance” of the TFA improvement margin. He explained that hey, that’s what researchers have always used. Well, yeah. I know that. I’m not questioning the study’s use of that particular metric, I’m questioning the value of that particular metric.

Recently, the august NY Times declared that “Guesses and Hype Give Way to Data in Study of Education” but alas, all this knowledge “has another hurdle to clear: Most educators, including principals and superintendents and curriculum supervisors, do not know the data exist, much less what they mean.”

Y’all are going to tell us about it, right? While us teachers are supposed to just go “Wow, thanks! We’ll go put this in action!” (Now I sound like a typical teacher who values practice over data, which I actually don’t, and I find this very annoying.)

But in fact, teachers are actually reading more research than at any time in the past, if only because reformers are presenting us all as a bunch of incompetent buffoons whose results aren’t just equaled, but improved, by smarter, ambitious, desirable college graduates who are in it for the resume improvement. It’s just slightly possible, isn’t it, that when a bunch of teachers become familiar with “research practice”—created by people who usually have very little teaching experience or have been told to ignore that experience in favor of “research practice”—they might question that practice?

Note to those who want to start “educating” teachers on research: expect pushback if you try to sell “significant” improvement that translates to something like a couple questions more on a test. Here’s a small sample of what math teachers will want to know on this particular study: did the TFAers do better at teaching linear equations, factoring quadratics? Were the additional correct questions random, or specific to a content area? Was the improvement consistent over each subject taught, or specific to one subject? Did the classrooms have similar behavior referral rates? Mind you, we’ll see the “significance” as irrelevant either way, but at least we’ll know whether it’s “real”.

But accept the results as “significant” and here’s the big takeaway, hinted at by Dara Zeehandelaar of Fordham:

Both TFA and Teaching Fellows have less experience than their peers, are less likely to be minorities, more likely to have graduated from more selective colleges, less likely to be math majors but more likely to score higher on tests of math content. However, only years of experience and, for high school, math content knowledge were associated with higher student achievement. These findings add to the glut of research indicating that traditional certification programs could benefit from greater selectivity, indeed from a radical overhaul.

Dara is celebrating a future teaching force with fewer blacks and Hispanics. Me, I look at it from the opposite side of the mirror, where it reads something like this:

CAEP, the ed school accreditation organization, is setting new standards that include higher candidate selection benchmarks. Selective ed schools have been left some wiggle room, in that the cohort, not each individual candidate, has to have an average GPA over 3.0 and average test scores in the top third of a nationally normed achievement test. However, it will annihilate predominantly black/Hispanic teacher schools, which include no small amount of public universities. Only 6% of African Americans and 10% of Hispanics get over 600 on each section of the SAT, which is roughly the top 30% nationwide.

So if CAEP doesn’t blink, or ed schools don’t get creative, we will soon have almost no black and Hispanic teachers, since blacks and Hispanics who get over 600 on each section of the SAT go off to become doctors and lawyers and Wall Street hedge fund managers. While the test score requirements will almost certainly be loosened or eliminated as the import becomes clear, our nation will see a lot fewer black and Hispanic teachers. We’ve already cut into the supply back during the NCLB changes, and people are already scratching their heads about these “missing” minority teachers, as Mokoto Rich of the New York Times has termed this, blithely ignoring the reality right in front of her. If the CAEP standards have the intended effect, we’ll lose even more.

But that’s okay, right? Because sure, we’ll lose in the current generation, but the next generation, taught by those newer, brilliant teachers who really care about their kids and aren’t just doing it because it’s some pesky job will raise achievement! Blacks and Hispanics will score the same as whites and Asians! Stricter drug laws will eliminate addiction! Tougher gun laws will prevent Newtowns! Dogs and cats will live together in peace and harmony! Hell, Israel and Palestine will straighten things out.

So the TFA study gives us a preview of that brave new world. TFAers were three times as likely to be white; the control group of teachers had seven times as many blacks. And lo! you see the predictable one standard deviation difference in test score means between the groups. The TFA group is exactly the
highly qualified, selective crew of new penny bright teachers everyone says they want.

And you get .07 of a standard deviation difference in student outcomes.

Of course, we already knew that, since it was an example of what the clearinghouse on education research knows doesn’t work:

For example, Michael Garet, the vice president of the American Institutes for Research, a behavioral and social science research group, led a study that instructed seventh-grade math teachers in a summer institute, helping them understand the math they teach — like why, when dividing fractions, do you invert and multiply?

The teachers’ knowledge of math improved, but student achievement did not.

“The professional development had many features people think it should have — it was sustained over time, it involved opportunities to practice, it involved all the teachers in the school,” Dr. Garet said. “But the results were disappointing.”

So maybe we can add the TFA study to the clearinghouse as additional evidence that past a certain point (unknown), increased teacher ability doesn’t result in improved student achievement. Or are we going to still pretend that .07 of a standard deviation is 2.6 months of instruction, that, as Jerseyman says, the increase is actually “practically” instead of “statistically”, significant?

Both reformers and progressives push “improve teacher quality” as an easy mantra that really doesn’t have much basis in fact. However, reformers go farther. Reformers look at the existing state of affairs and see obvious failure, failure so manifest that it’s a simple matter to fix. Low test scores? Give them teachers who care. Smarter teachers. Higher standards. Over the past decade, their enthusiasm has been blunted a tad by the realization that at best, their “obvious” improvements, if you squint really hard and pretend peer environment is irrelevant, will improve outcomes a squidge around the edges. But still, they keep coming back for more. And so, they push this study as evidence that TFA works, not realizing that the study foretells the lackluster improvement they’ll see at the expense of a virtually closed career path for blacks and Hispanics.

That’s the takeway of the TFA study.

Can someone mention this to CAEP?

About educationrealist

14 responses to “The Takeaway from the TFA Study

  • Hattie

    How much does the race of teachers matter? Maybe I just haven’t been socialised in American racial niceties, but why would it be such a tragedy to have fewer minority teachers?

    • Hattie

      “…why would it be such a tragedy to have fewer minority teachers?”

      I’m not saying it would be a good thing, either. I’m just not sure why it matters either way. Does it matter?

    • Anthony

      In terms of educational results, it won’t really matter much – the kids will, or won’t, learn their material whether their teachers are black or white. But the political cost would be huge, and looking at the results ER is mentioning, not worth it.

      There are two ways in which it might actually affect educational outcomes: by reinforcing the idea already common among young blacks that doing well in school is a “white thing”, and in the case of Hispanic teachers, by making it harder for kids less than proficient in English to understand the material. I don’t know if either of these have been researched, and I suspect the effects would be even less than the 0.07 SD that ER tells us about for TFA.

  • Cara

    I get your concern. I will just say that in my discussions with people in the alt cert universe, they seem pretty well aware of the need to diversify.

    As for the research itself, of course the effect sizes are small. That’s because there are no easy fixes when it comes to student achievement. In my view, the rigor of the research matters, and random assignment renders the results more credible – as does the fact that this isn’t the first study to indicate that TFA teachers outperform others when it comes to math.

    In an ideal universe, we’d work toward making staying the profession more desirable. But I suspect there are no easy fixes there either.

    • educationrealist

      It took me a while to figure out what you meant by staying in the profession, until I realized you think I’m complaining about TFA. Look, I don’t much care one way or the other about TFA. The difference between TFAers and the graduates of elite ed school programs, such as the one I went to, is non-existent, and the idea that TFA training has something to do with it is ludicrous.

      The issue is simply whether anyone would look at these numbers and see it as “outperformance”.

      That’s because there are no easy fixes when it comes to student achievement.

      That is not at all what people think when they yammer on about smarter teachers being needed. Clearly, they think that there are easy fixes. And if there aren’t easy fixes–and I agree–then who the hell cares about .07 sd of improvement? It’s pointless to anyone who actually teaches kids, and tragic to see policy wonks talking about it as if it matters.

      they seem pretty well aware of the need to diversify.

      They won’t be able to diversify. There are two possible reasons for that miniscule difference. First, and the most likely, that TFA teachers harp endlessly on test scores and get the kids to care slightly more. I suspect that’s it.

      The other possibility is that a HUGE difference in cognitive means for the group translates to a tiny difference in achievement outcomes. If that’s the case, then as Anthony says, the political cost isn’t worth that miniscule outcome, unless anyone’s a fool who believes it’s possible that outcomes that tiny accrue. Moreover, if the difference is due to cognitive ability in groups, then diversification will wipe out that advantage. Or there will be far fewer black lawyers, doctors, and hedge fund managers.

  • Cara

    I wasn’t just talking about TFA when I referenced staying in the profession. Turnover is a pretty big issue for schools serving low-income students regardless of how teachers were prepared, and it’s directly related to the point about no easy fixes. If by those who “yammer on about needing smarter teachers” you are referring to people who argue that we can just fire the least effective teachers and that alone will improve the overall quality of the teacher workforce, then yes – that strikes me as an overly simplistic solution. But I count myself among the many who yammer on about improving teacher quality not as an easy solution, but as an ongoing cycle in which we should invest significant resources.

    As for the point about diversifying – I guess I’m just less cynical. I certainly don’t think that diversifying will wipe out the advantage.

    • educationrealist

      If by those who “yammer on about needing smarter teachers” you are referring to people who argue that we can just fire the least effective teachers

      No, I’m talking about those who not just argue, but have, increased the test score burden to require teachers to score in the top third on a nationally normed test. That’s what CAEP has done. TFA has already done that. While I suspect they are committing some affirmative action to bring in some blacks and Hispanics, it’s clear that they are primarily pulling people from the top third when it comes to whites and Asians. And they’re not dropping their standards much, or they’d have more blacks and Hispanics.

      It’s not cynical to point out that very few blacks and Hispanics will be in the top third. It’s simply a fact.

      The TFA study has just shown exactly how much “improving teacher quality” by EXCLUDING LOW SCORING TEACHERS is worth. And it’s not much.

  • not too late

    Teacher quality matters. This is true. However, above a certain threshold of competency, it matters very little. Classic diminishing return situation. The current certification requirements are adequate. A woman of average intelligence in fact can teach kindergarteners to read and count and identify the city and state in which they live. Ironically for TFA types, the students who might really benefit from TFA teachers are the GT students especially in smaller towns and districts that don’t attract top teachers.

  • CitizensArrest

    Basically what educationrealist is talking about is the difference between performance and effectiveness as discussed here. If anyone wants to know what race matters, just ask the kids for starters.

  • CitizensArrest

    “WHY” race matters.

  • Dara Zeehandelaar

    Just to clarify: I did not write, nor conduct, the TFA study. It can be found here:
    And as a former member of an alternative certification program myself (DC Teaching Fellows), I by no means celebrate a less-diverse teaching workforce. In fact, just the opposite! I was pointing out that the reason TFA teachers might be marked as more effective than their peers was not because TFA itself provided any better preparation than a traditional path to certification, but rather because TFA was selective in who it chose. (The same can be said about any selective program — does Harvard law school create great lawyers, or does it only admit people who were going to be good lawyers anyway?) There is constant criticism that traditional teacher certification programs are not choosing the best applicants — and by “best,” I mean not equipped with adequate content knowledge. The point is that students might benefit if traditional certification programs were more selective in who they admitted (for example, by admitting students with higher GRE scores, higher undergraduate GPAs, undergraduate or even graduate coursework in the content area in which they want to teach, professional experience, etc. — the same things that TFA looks for). “Selectivity” in this case has nothing to do with race.

    All of that aside, in my own teaching experience I found that students benefited from high-quality teaching, period.

  • educationrealist

    I did not say you wrote the study, nor that you were in any way involved in it.

    You said, “These findings add to the glut of research indicating that traditional certification programs could benefit from greater selectivity, indeed from a radical overhaul. ” immediately after saying the “better” teaching force was “less likely to be minority”. Not the most felicitous choice of words, but there it is. You appeared to say that teaching would benefit if there were fewer minorities. Now, I understand you will say no, no, we can have selectivity *and* minorities, but you’re wrong.

    Oh, wait–you already did say that! ” “Selectivity” in this case has nothing to do with race.”

    Yes, alas, it does. In SAT terms, pulling from the top 30% of scorers would mean students with SAT section scores of 600 or higher. That would leave out all but 6% of blacks and 10% of Hispanics.

    Besides, ed school selectivity is irrelevant. Anyone they select will have to pass the credential test.

  • Teacher Quality Report: Lacking a Certain Quality | educationrealist

    […] we need fewer black and Hispanic teachers. And, as the recent TFA study’s big takeaway shows, all you get for largely eradicating black and Hispanic teachers is, maybe, .07 of a standard […]

  • EH

    I didn’t hear of TFA until I read an article in the Atlantic about their supposed data-driven focus. The representative for TFA’s statistical spin was a guy I had gone to high school with, 8 classes together, I think, with most of those being the smallest classes in the high school. We were pretty much in the same social circle, too and I considered him a friend. I just don’t know how I could talk to him after looking into the alleged evidence for TFA effectiveness, and even more the PC pap extruded in their so-called training materials. His position depends on not only knowing the reality of what the statistics say about ability and the ineffectiveness of interventions, but consciously denying and covering up that reality.

    The whole tone of TFA is so pitched to the establishment denial of reality that it makes me sick. When the co-CEO (or whatever her title is) Villanueva Beard, wrote an especially gag-inducing paste of puffery and saccharine psy-ops “How I Define the Status Quo”, I wrote a bit of a screed in the comments, which strangely do not seem to be accessible now.

    It’s amazing how they push the flawed Mathematica study when the orders of magnitude larger NYC study (which corrected for everything short of sunspots) found that there was no advantage to TFA teachers. Without the over-correction the TFA teachers would look worse.

    The most interesting finding of that study was that if teacher performance is rated by increase in student test scores, then 1 standard deviation better than average teacher corresponds to 0.2 standard deviation better student test scores. This implies that there are maybe 1 or 2 teachers in the US capable of raising student test scores by a single standard deviation. (5s.d. is ~1 in 3.5M) And there is no reason to believe that person would do better with bad students than good students, so even if we made a clone army out of that teacher, it still wouldn’t close the achievement gap.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: