Tag Archives: education research

The Takeaway from the TFA Study

Jersey Jazzman’s list of cautions mirrors my own on the TFA study. I would add that the TFA skew towards middle school (in the study) makes a big difference; TFAers push testing excitement heavily in middle school, something that’s very tough to do with high schoolers.

I had an interesting twitter exchange with Morgan Polikoff who, can I just say, makes me feel both ancient and unproductive, about the “significance” of the TFA improvement margin. He explained that hey, that’s what researchers have always used. Well, yeah. I know that. I’m not questioning the study’s use of that particular metric, I’m questioning the value of that particular metric.

Recently, the august NY Times declared that “Guesses and Hype Give Way to Data in Study of Education” but alas, all this knowledge “has another hurdle to clear: Most educators, including principals and superintendents and curriculum supervisors, do not know the data exist, much less what they mean.”

Y’all are going to tell us about it, right? While us teachers are supposed to just go “Wow, thanks! We’ll go put this in action!” (Now I sound like a typical teacher who values practice over data, which I actually don’t, and I find this very annoying.)

But in fact, teachers are actually reading more research than at any time in the past, if only because reformers are presenting us all as a bunch of incompetent buffoons whose results aren’t just equaled, but improved, by smarter, ambitious, desirable college graduates who are in it for the resume improvement. It’s just slightly possible, isn’t it, that when a bunch of teachers become familiar with “research practice”—created by people who usually have very little teaching experience or have been told to ignore that experience in favor of “research practice”—they might question that practice?

Note to those who want to start “educating” teachers on research: expect pushback if you try to sell “significant” improvement that translates to something like a couple questions more on a test. Here’s a small sample of what math teachers will want to know on this particular study: did the TFAers do better at teaching linear equations, factoring quadratics? Were the additional correct questions random, or specific to a content area? Was the improvement consistent over each subject taught, or specific to one subject? Did the classrooms have similar behavior referral rates? Mind you, we’ll see the “significance” as irrelevant either way, but at least we’ll know whether it’s “real”.

But accept the results as “significant” and here’s the big takeaway, hinted at by Dara Zeehandelaar of Fordham:

Both TFA and Teaching Fellows have less experience than their peers, are less likely to be minorities, more likely to have graduated from more selective colleges, less likely to be math majors but more likely to score higher on tests of math content. However, only years of experience and, for high school, math content knowledge were associated with higher student achievement. These findings add to the glut of research indicating that traditional certification programs could benefit from greater selectivity, indeed from a radical overhaul.

Dara is celebrating a future teaching force with fewer blacks and Hispanics. Me, I look at it from the opposite side of the mirror, where it reads something like this:

CAEP, the ed school accreditation organization, is setting new standards that include higher candidate selection benchmarks. Selective ed schools have been left some wiggle room, in that the cohort, not each individual candidate, has to have an average GPA over 3.0 and average test scores in the top third of a nationally normed achievement test. However, it will annihilate predominantly black/Hispanic teacher schools, which include no small amount of public universities. Only 6% of African Americans and 10% of Hispanics get over 600 on each section of the SAT, which is roughly the top 30% nationwide.

So if CAEP doesn’t blink, or ed schools don’t get creative, we will soon have almost no black and Hispanic teachers, since blacks and Hispanics who get over 600 on each section of the SAT go off to become doctors and lawyers and Wall Street hedge fund managers. While the test score requirements will almost certainly be loosened or eliminated as the import becomes clear, our nation will see a lot fewer black and Hispanic teachers. We’ve already cut into the supply back during the NCLB changes, and people are already scratching their heads about these “missing” minority teachers, as Mokoto Rich of the New York Times has termed this, blithely ignoring the reality right in front of her. If the CAEP standards have the intended effect, we’ll lose even more.

But that’s okay, right? Because sure, we’ll lose in the current generation, but the next generation, taught by those newer, brilliant teachers who really care about their kids and aren’t just doing it because it’s some pesky job will raise achievement! Blacks and Hispanics will score the same as whites and Asians! Stricter drug laws will eliminate addiction! Tougher gun laws will prevent Newtowns! Dogs and cats will live together in peace and harmony! Hell, Israel and Palestine will straighten things out.

So the TFA study gives us a preview of that brave new world. TFAers were three times as likely to be white; the control group of teachers had seven times as many blacks. And lo! you see the predictable one standard deviation difference in test score means between the groups. The TFA group is exactly the
highly qualified, selective crew of new penny bright teachers everyone says they want.

And you get .07 of a standard deviation difference in student outcomes.

Of course, we already knew that, since it was an example of what the clearinghouse on education research knows doesn’t work:

For example, Michael Garet, the vice president of the American Institutes for Research, a behavioral and social science research group, led a study that instructed seventh-grade math teachers in a summer institute, helping them understand the math they teach — like why, when dividing fractions, do you invert and multiply?

The teachers’ knowledge of math improved, but student achievement did not.

“The professional development had many features people think it should have — it was sustained over time, it involved opportunities to practice, it involved all the teachers in the school,” Dr. Garet said. “But the results were disappointing.”

So maybe we can add the TFA study to the clearinghouse as additional evidence that past a certain point (unknown), increased teacher ability doesn’t result in improved student achievement. Or are we going to still pretend that .07 of a standard deviation is 2.6 months of instruction, that, as Jerseyman says, the increase is actually “practically” instead of “statistically”, significant?

Both reformers and progressives push “improve teacher quality” as an easy mantra that really doesn’t have much basis in fact. However, reformers go farther. Reformers look at the existing state of affairs and see obvious failure, failure so manifest that it’s a simple matter to fix. Low test scores? Give them teachers who care. Smarter teachers. Higher standards. Over the past decade, their enthusiasm has been blunted a tad by the realization that at best, their “obvious” improvements, if you squint really hard and pretend peer environment is irrelevant, will improve outcomes a squidge around the edges. But still, they keep coming back for more. And so, they push this study as evidence that TFA works, not realizing that the study foretells the lackluster improvement they’ll see at the expense of a virtually closed career path for blacks and Hispanics.

That’s the takeway of the TFA study.

Can someone mention this to CAEP?

Algebra Terrors

A day or so before the school year began, I went to an empty classroom that had a supply cabinet. This classroom was way better than mine. It was 3 or 4 feet wider, had shelving and a smart board. Now I didn’t care much about the smart board, but all smartboards have document cameras, which my room does not.

“Hey. This is a great room. Who gets it?”

“I guess the other new teacher.”

“When’s he coming?”

“I don’t think he’s even hired yet. You know, you should ask for this room! You’re here first.”

“Yeah, I think I will. It can’t hurt.”

So off I went to find an administrator, and the first one I I ran into was the AVP of discipline and scheduling. (As a sidenote, the conversation recorded below is the one of only two I’ve ever had with him.)

“Hi. Please don’t view this as a complaint of any sort, but I really like to teach with document cameras, and I notice that room 1170E has a smartboard. Hank (not his real name) suggested I ask if I could switch rooms?”

“To 1170E? Oh, yes, that’s for Ramon. I suppose I could switch rooms, but we’d need to change schedules as well. You see, we got the Promethean smartboard funding as part of our algebra initiative, and we committed to give those boards to teachers teaching Algebra I at least 60% of the time. If you’re interested….”

You know in Terminator 2, when Linda Hamilton has just finally broken out of her padded cell, broken Earl Boen’s arm, beaten the crap out of three security guards and is waiting for the elevator? Freedom is there, baby. She can taste it. She can get her son, escape to Mexico, stop the machines, save the world. All she needs is the ding of the elevator door.


….and out of the elevator steps

Algebra I Arnold

“I was told that you had expressed a strong preference to teach geometry and intermediate algebra. But I’m always happy to find interested algebra I teachers….”

“No, no, sir, no really. It’s fine. I do have a very strong preference to teach geometry and intermediate algebra, you were correctly informed, and I am happy with my current room. It’s fantastic. I can deal without a camera, it’s fine.”

“You’re sure?” Clearly, this man is an evil sadist. “You really do seem to like the document camera, and we prefer that the rooms go to teachers who will use them…”

No! No! I’ll stop! The machines can win! Take my son! Just don’t make me go back!!!


It was just a bad scare. I’m teaching geometry and algebra 2. Well, Math Support, but even though the kids are weaker, I’d rather teach Math Support than Algebra I.

Math teachers think this story is very funny.

In retrospect, my second year of teaching was my most brutal, thanks to my schedule of all algebra I, all the time. I learned a lot. I never want to go back. Oh, sure, I’d like to teach one class of Algebra I, particularly to see if my data modeling lessons they work as well in algebra I as algebra II. But I do not want to be an “algebra I specialist”, and never, ever, EVER want to devote anything more than a class a year to algebra I. I’ve said it many times, but I’m always ready to bore folks: high school algebra I classes should convince anyone—from loopy liberal progressive to anti-teacher union tenure hating eduformer—that our educational policy is twisted and broken beyond all recovery.

So why bring it up now? Because until I saw this article, I’d forgotten the very worst part: A Double Dose of Algebra (ht: Joanne Jacobs).

Yes, I didn’t just teach straight algebra I classes. I taught a double class of Algebra Intervention. Let’s switch T2 characters for just a moment, shall we?

This is what happens when I’m reminded of that intervention class without time to prepare myself.

What is Algebra Intervention, or “double dose algebra”? Well, it’s this brilliant strategy of identifying kids who are really weak in math and increasing their hours of torture.

The best study of this approach, by Takako Nomi and Elaine Allensworth, examined the short-term impact of such a policy in the Chicago Public Schools (CPS), where double-dose algebra was implemented in 2003. …. Nomi and Allensworth reported no improvement in 9th-grade algebra failure rates as a result of this intervention, a disappointing result for CPS. The time frame of their study did not, however, allow them to explore longer-run outcomes of even greater importance to students, parents, and policymakers. (emphasis mine)

So double dose algebra didn’t work. Did that stop them? Hahahahah! Of course not! They just commissioned another study! One that would allow them to explore “outcomes of even greater importance” to students, like “will I make an extra $50,000 a year to compensate me for the time I spent in this tortuous hell?”

Using data that track students from 8th grade through college enrollment, we analyze the effect of this innovative policy by comparing the outcomes for students just above and just below the double-dose threshold. These two groups of students are nearly identical in terms of academic skills and other characteristics, but differ in the extent to which they were exposed to this new approach to algebra. Comparing the two groups thus provides unusually rigorous evidence on the policy’s impact.

Wait. You checked the kids just below and just above the threshold? So you only compared the strongest intervention students with the weakest regular students? Well, golly. Did you, perchance, check how the weakest regular students did compared to the weakest intervention students? Was it substantially different from the gap between the strongest intervention and weakest intervention?

The benefits of double-dose algebra were largest for students with decent math skills” but below-average reading skills, perhaps because the intervention focused on written expression of mathematical concepts.

Guys, half of all regular high school algebra students can’t add fractions or work with negative numbers—that is, they do not have decent math skills. So what the hell is relevant about progress made by intervention students with “decent math skills”?

With the new policy, CPS offered teachers of double-dose algebra two specific curricula called Agile Mind and Cognitive Tutor, stand-alone lesson plans they could use, and three professional development workshops each year, where teachers were given suggestions about how to take advantage of the extra instructional time.

Eight days of PD. EIGHT DAYS! In three plus years of teaching, I’ve taken 1.5 days off for being sick. In one year of teaching algebra and algebra intervention, I was required to leave the classroom for 8 days. The PD was utterly useless. The lunches with the other math teachers, good—lots of conversations, sharing of lessons, venting, and so on. We would do better to just give us money and an extra half hour every month for lunch.

CPS also strongly advised schools to schedule their algebra support courses in three specific ways. First, double-dose algebra students should have the same teacher for their two periods of algebra. Second, the two algebra periods should be offered consecutively. Third, double-dose students should take the algebra support class with the same students who are in their regular algebra class. Most schools followed these recommendations in the initial year. In the second year, schools began to object to the scheduling difficulties of assigning the same teacher to both periods, so CPS removed that recommendation.

It wasn’t just the schools that objected, I’m betting. I taught intervention the first year it was offered by the school. Of the three intervention teachers, one (a TFAer) turned in her resignation in January purely because she felt beaten down by intervention. Another teacher, an algebra specialist, a near-phlegmatically calm Type B, burst into tears when she met with the principal to make absolutely sure she wasn’t given an intervention class the next year.

I was the third. I never complained. I was under continual pressure because I wouldn’t tolerate three kids who were deliberately disrupting the class. The administration hinted I was racist, that I was exaggerating their behavior, and only relented on the pressure when my induction adviser witnessed a middling incident of blatant misbehavior and blew a gasket when the AVP of discipline shrugged it off until he learned that she’d seen it. Admin got a long letter from her and started to make the kids’ lives hell.

The following year, the school dropped the requirement for consecutive periods and allowed two teachers to split the course, rather than requiring the same teacher to do both sections. That same year, the intervention teachers got called into a room by the district and were given a blistering come to jesus meeting in which they were informed that their pass rates better go way, way up until they were as good as the pass rates from last year, which were clearly a goal to be attained. Of course, that last year, when they were dumping all that pressure on me, they never said “Yeah, too many referrals but hey, your pass rate is awesome. You’re only failing two kids, who never show up. Great job!”

This year, the school has dropped the requirement that the students all be in the same class. Hey. That sounds familiar, doesn’t it?

The pressure on the teachers is tremendous. So the schools try to find a way to pay lip service to the method—we’re offering intervention for our weak students!—without all their teachers quitting on them simultaneously. Intervention is brutal on teachers.

The recommendation that students take the two classes with the same set of peers increased tracking by skill level. All of these factors were likely to, if anything, improve student outcomes. We will also show, however, that the increased tracking by skill placed double-dose students among substantially lower-skilled classmates than non-double-dose students, which could have hurt student outcomes.

In addition to the strain on teachers, intervention is a huge hassle for administration and has an unintended consequence that escapes the notice of people who haven’t talked to an AVP responsible for the master schedule. But the reality is that a group of kids who must take two classes back to back end up taking most, if not all, of their classes together.

Say a school has 10 freshmen English classes but only three of them are double block remedial and (please note, this will come up again) many of the kids who take double block algebra also require double block English. The intervention freshmen are in periods 3 and 4 for algebra intervention, and the only other double block English class they can take is 5 and 6, leaving periods 1 and 2 open. Only one freshman PE class available in period 1, so science (bio or general) has to go in period 2. All done. So all the kids in algebra intervention periods 3 and 4 who are also in double block English take all their classes together. For every intervention class, some 20-30 underachieving, low incentive kids are moving through their entire day together, in non-remedial and remedial classes both. Of course, since most intervention kids are weak in all their subjects, this means that their classes have a disproportionately high number of low achievers—all of whom spend their entire day together, socializing. Or planning ways to wreak havoc. The troublemakers in my class arranged signals that they would use to disrupt classes—all their classes. They’d pick a code word, and whenever the teacher said that word, they’d all start laughing loudly, or squeaking their shoes, or sneezing.

I’m a big fan of tracking. I am vehemently opposed to taking a group of low achieving kids who are already buddies, already with next to no investment in school, already really annoyed at having to take a double dose of math—and give them every single class together, so they can reinforce each other in noncompliance and have an entire school day to socialize.

And then this section, which caused more flashbacks:

Overall, 55 percent of CPS students scored below the 50th percentile and thus should have been assigned to double-dose algebra, but only 42 percent were actually assigned to the support class. In addition, some students took double-dose algebra, even though they scored above the cutoff on the exam.

You’re thinking, wait. Some of the weak kids didn’t get intervention, and some of the strong kids did? That’s a weird fluke, isn’t it?

And so, another anecdote.

My strongest intervention kids had taken Algebra I the year before. Each of these six kids had scored higher on their state test than my average score for all my non-intervention algebra students. Yes, you read that right. Six of my intervention kids were good enough for the top half of my non-intervention algebra class. Not just better than my worst. Better than HALF the 100 students in my non-intervention classes. Two of them had actually achieved Basic on the previous year’s test. I got them to bring in their test scores and show them to the AVP of Instruction, demanding they be put into normal algebra (leaving me out of it, of course). One of them was put into my regular algebra class, and got an A-. The other four missed Basic by just a few points and despite my asking on their behalf, were required to take intervention. This despite the fact that I had over a dozen non-intervention freshmen who’d scored Below Basic or Far Below Basic. None of it mattered.

I was so foolish as to write the AVP of Instruction saying randomly, casually, something like “Hey, okay, so I can’t move strong students out. But so long as I’m teaching an intervention class for really really weak students, could I move some of my weak non-intervention class IN? Some of them are even Resource (sped) students, so they could substitute the intervention class for their guided studies class, so it wouldn’t create a scheduling disaster (sped kids get a study hall). Here’s a list.”


Three weeks later, as God is my witness, the AVP of Instruction sends me a note, “I’m moving Fred McInery [not his real name] into your intervention class. He is a weak math student who needs more support.”

I look up Fred. He is a sophomore. I am very excited, because I am a moron, and send her a note. “Hey, great! We’re putting sophomores in intervention now? Could we revisit my list? I really think it will help these extremely weak students succeed in math.”



That story is an amusing flashback, even if it is crazy-making. Here is a horrible one:

We shall call her Denise. She is a doll. She was in my intervention class and had extremely weak skills and was the propaganda child for intervention, the one that everyone is thinking of when they propose it, because she worked her ass off and actually became better at math. She was not “just below the cutoff point”, either, but an FBB child who surreptitiously counted on her fingers to add 4 + 2. But conceptually, she got it. In the first semester, she did so poorly she was one of my contract students. She improved dramatically on her test, missing Basic by just a point. She had the third highest state test score of my intervention students, and passed my class.

Not only didn’t the school move her on to Geometry, but they put her into intervention again. (Yes. Now they had a sophomore intervention class.) When Denise told me this, I went quietly berserk and emailed the AVP of Instruction. It is not the same AVP. This one is worse.

Keep in mind, this second year at the same school, I am teaching Geometry, thank all the gods, and have two of my last year’s intervention kids taking my class even though they received slightly lower state test scores than Denise. Five others of my kids have also moved on to geometry with lower state test scores. Denise and three others were kept behind. I email the AVP of Instruction—a different one, as last year’s AVP has been promoted to principal of another school. This AVP is much worse–and point out these facts. I do not point out that I can discern no organizing principle behind this decision, that I suspect a very disorganized AV principal behind it. I am very polite; hey, this is just some oversight? Want to make sure it gets cleared up.

I write three notes, all very polite, and finally, a month after school starts, Denise gets moved….to a regular Algebra class. I gnash my teeth, but Denise is thrilled and thanks me profusely.

I see Denise at the year-end, and ask how she’s doing. “Great. But I failed the first semester of geometry, so I’ll have to go to summer school.”

“What? They put you in geometry?”

“Yeah, they said you advised it. But they didn’t move me until, like, November, so I failed. But that’s okay. I did good second semester, and I’m going to pass it over the summer.”

I guess it worked out okay, ultimately. But had she been put in the geometry class originally, she’d have had her summer.

Double-dosing had an immediate impact on student performance in algebra, increasing the proportion of students earning at least a B by 9.4 percentage points, or more than 65 percent. It did not have a significant impact on passing rates in 9th-grade algebra, however, or in geometry (usually taken the next year). Double-dosed students were, however, substantially more likely to pass trigonometry, a course typically taken in 11th grade. The mean GPA across all math courses taken after freshman year increased by 0.14 grade points on a 4.0 scale.

(emphasis mine)

Clearly, most students did not do all that well. As the study acknowledges, the low-achieving students did not benefit at all from the intervention; the students most likely to benefit were those who just missed the cutoff. More on that later.

Here’s what the study doesn’t make clear: many high school algebra students never make it to trig. They take it twice in high school, then take geometry twice. Or they take algebra once, geometry twice, and algebra 2 without trig (that’s the class I teach). So are they only counting the students who made it to trig?

The more meaningful stat would be the percentage of double-dosed kids who made it to trig vs. the non-double-dosed kids who achieved same. Reading this passage, the study appears to be saying that all the kids made it to trig and hahahahahaha, no. Not happening.

And since that’s not happening, then who, exactly, is being compared in the GPA? All the kids, or just the ones that made it to trigonometry? Presumably, just that set, because otherwise, the GPA number isn’t worth much. Hey, the double dose kids who flunked algebra twice and made through geometry by their senior year had a GPA .14 points higher than the single dose kids who made it through trig. Whoo and hoo.

It is important to note that many of these results are much stronger for students with weaker reading skills, as measured by their 8th-grade reading scores. For example, double-dosing raised the ACT scores of students with below-average reading scores by 0.22 standard deviations but raised above-average readers’ ACT scores by only 0.09 standard deviations. The overall impact of double-dosing on college enrollment is almost entirely due to its 13-percentage-point impact on below-average readers (see Figure 3). This unexpected pattern may reflect the intervention’s focus on reading and writing skills in the context of learning algebra.

(emphasis mine)

Oh, yes. That’s what we do in these algebra intervention classes. We focus on reading and writing! We’re given a bunch of kids who add 8 to 6 on their fingers, and we figure their struggle comes from not being able to read the word problems. So we put up a word wall and teach them five new terms and suddenly their reading skills skyrocket wildly.

Or—and this is just a wild, random, thought—perhaps my last school isn’t the only school in which Set A = {names of students taking double dose algebra} and Set B = {names of students taking double dose English} and is a Venn diagram in the two circles largely overlap?

You say oh, don’t be silly, ER. Of course they’d account for the possibility that the double dose algebra kids are also getting a double dose of reading intervention! And then not mention it! And I say, you don’t read much educational research, do you?

Because keep in mind the conclusion of this research:

As a whole, these results imply that the double-dose policy greatly improved freshman algebra grades for the higher-achieving double-dosed students, but had relatively little impact on passing rates for the lower-achieving students.

Apart from that, Mrs. Lincoln, how’d you like the friggin’ play?

Look. None of my outraged noise makes any sense at all if you don’t realize that, in the world of high school math, the kids who benefited, according to this study, kids achieving just below the passing standard, are WAY ABOVE AVERAGE for that population, particularly in a Title I school. Intervention exists because these schools have dozens, if not hundreds, of algebra students who have taken the course three times and still score Far Below Basic. It does not exist to help kids just below the 50% mark in math get better scores in reading, marginally higher grades and ACT scores, and better Trig scores—if they get to trig, which the normal intervention kid does not.

What people fondly imagine algebra intervention to do is this: kids are just a little behind, you know? They just need some extra time learning integer operations and fractions. They didn’t learn it the FIRST FIFTEEN TIMES they were taught it, so all they really need is another hour or so a day and they’ll be right up there with the rest of them, all right? And if they aren’t, well, it’s those damn teachers who just don’t want to work with “those kids”, and we’ll just have to find more teachers who really, really care about these kids who just need a few hours more help than the others. (Yes. This is the myth of “They’ve never been taught…..”)

Meanwhile, forty percent of the freshman class comes in having taken algebra once and scored far below basic or barely below basic, and are randomly assigned to double block or no double block using a dartboard, from what I can see. The teachers are dealing with the same lack of basic skills in both double and single block algebra, and rapidly realize (if they didn’t know already) that the kids who don’t know integer operations and fractions have this gap because they aren’t terribly bright. They can’t come up with an intervention vs. non-intervention approach, because some kids in the intervention class don’t need support while some kids in the non-intervention class do. But in the non-intervention classes, the teachers only have to deal with 3-8 kids with low skills, while in the intervention classes it’s 14-15 out of 20. So the only thing different about the intervention classes is monstrously bad behavior and more time in hell.

All this, mind you, so that we can do research that reveals no real improvement in outcomes.

But I’m out of it, baby. It’s enough to make me believe in god. Death to algebra intervention.

But the nightmares, they won’t stop until it’s destroyed!


Get every new post delivered to your Inbox.

Join 749 other followers