Direct Instruction Miracle? The Lewis Lemon Case

Back in February, I observed that Direct Instruction hadn’t failed because of teacher disdain, as Robert Pondiscio charged, but because educational leadership from superintendents on up through US senators had not only refused to support the program, but in some cases actively and systemically ripped out implementations that seemed to be successful.  As I said at the time, I considered it a real mystery.

So I dug deep into Zig Engelmann’s book, Teaching Needy Kids in Our Backwards System looking for an example of a success story I could independently verify. He goes on at some length about both San Diego and Baltimore schools, but not in any way I could pin down DI vs. Non-DI. Quite often Engelmann would cite a newspaper article as evidence.

The Ventura County Star carried an article on March 15, 2003, titled “Effective Reading Program Must Go”, which indicated that the only school in Ventura County and one of 109 in the state to receive a citation for achieving exemplary progress was forced to drop their [DI] program.

No mention of the school. Or the superintendent. Or the degree to which test scores increased. One gets the distinct impression that Zig….well, read about it in the paper. He uses virtually identical wording in this Edweek article. California keeps detailed, easily searchable records; with the name of the school, the specifics of the decision could be confirmed.

The exception is Lewis Lemon.

But then, as happened consistently through the decades, a new superintendent came in and demanded Lewis Lemon abandon Direct Instruction and adopt “balanced literacy”. The principal, Tiffany Parker, flatly refused. She was accused of cheating, so the students were tested again. They passed the new test.  The district actually had to give back Reading First federal funds because they dropped DI. Apparently, the district blamed the principal for informing the feds that the district was no longer using the program, and Parker was demoted. She sued, and settled.  If you google this, you’ll find a New York Times article, a New York Sun article, this Heartland article, and references to many Rockford Register Star articles by Carrie Watters, who from what I can tell now works at an Arizona paper. The Rockford Register Star archive goes back before 2003, and has other articles by Carrie Watters, but contains none of the many stories Watters did on the Lewis Lemon controversy. Even Joanne Jacobs’ blog entry on Rockford, “They Messed with Success” has disappeared, which had me really freaked until I learned she was had temporarily moved her archives.

The story made national headlines when the superintendent decided, in 2005, to force Lewis Lemon off of DI into “balanced literacy”, while the strong student performance of 2003 only got covered locally. The New York Times piece went into a bit more depth, mentioning that the fifth graders didn’t do as well. Also, the reports weren’t always specific. Was it math or reading? Or both? So I went digging for data.

If contemporaneous reports are scarce, the contemporaneous data survives. Illinois test scores comparing Lewis Lemon performance to the district and state by race are all online. Here they are, and please check my results:

2001–the year they implemented DI
2002
2003
2004
2005–the year they were forced to discontinue DI
2006

So now I’m going to go all graph heavy, like Spotted Toad. First up: the top level comparison of Lewis Lemon scores to the district and state level for both 3rd and 5th grade (reading down).

Note the big ol’ spike in 2003. Lewis Lemon 3rd graders had far more students meet or exceed standards than the district or state averages in that year. But then, third graders at Lewis Lemon outscored the district average  every year both before and after DI was implemented in  Notice, too, that the fifth grade scores aren’t nearly as impressive, but that both math and English show a spike in 2005, which is the year the third graders of 2003 would have been in fifth grade.  Then again, fifth graders had much better reading scores in 2002 than in any time after DI implementation.

Lewis Lemon officials blame the lower fifth grade reading scores on a less than perfect implementation, and students who were further behind. On the surface, this seems unlikely, given the high scores in 2002. But there may be other causes.

So I disaggregated, first by race. Here are black and white 3rd grade reading score scores compared at the school, district, and state level, broken down by achievement category.

Up first, 3rd grade blacks:

Now remember, this is Lemon Lewis black students compared to other black students only (not the entire population). Blue is Lemon Lewis, and the more blue to the right of each graph, the better Lewis Lemon is doing. 2003 is the year of the Big Score. The blue clearly moves to the right from 2002 to 2003, and stays there for two years. Then it shifts back.In 2003, black students  at Lewis Lemon met or exceeded standards at 3 to 4 times the numbers that district or state black students did.

Third grade reading, white students.

So this is interesting in a couple ways. First, it’s clear that in the pivotal year of 2003, white performance actually declined slightly. Fewer students failed to meet standards, but fewer students exceeded them. Not a lot. But there’s no tremendous spike in white student performance in 2003.

And then, something that has gone completely unmentioned: the white student population collapsed to below testing levels immediately after 2003. This explains some of the falloff of the graph above–if whites comprised a decent chunk of the high scores, then their disappearance would impact the overall “meets or exceeds”.

It’s not clear to me whether the DI implementation was reading only, or reading and math. The news accounts all focus on reading, but more than one account mentions improved math scores. So I’ll include them. Here’s third grade, African American.

Here, Lewis Lemon was scoring better than blacks in the district and state before the DI implementation. The blue still shifts right, but it’s not as dramatic. In 2003, virtually every student met or exceeded expectations, but in every other year, both before and after the DI implementation, they were still doing very well.

Now white 3rd grade math:

So in 2002 through 2003,  the school saw a good bunch of whites move from “meets” to “exceeds”. Unlike reading, third grade white performance saw a decent boost, but it started a year earlier. Whites at Lewis Lemon performed better than the district.

And now, the fifth grade. Reading, black fifth graders:

Scores actually declined in 2003. 2005, the year that the third graders from 2003 were in fifth grade, sees a slight improvement. But not much of one–more than half failed to meet standards.

Fifth grade whites, reading:

Again, whites have their best year in 2002, and don’t show any particular spike. They also do better than black fifth graders (which is normal).

Fifth grade math:

Indifferent–except note the spike when the third graders of 2003 show up. It’s much more pronounced in math than it was in reading.

White fifth graders, to finish out.

So here’s one last way of looking at the data–compares blacks at Lewis Lemon to the whites at the school, district, and state level. Read down  on the left for 3rd and 5th reading, down on the right for 3rd and 5th math.

Once again a consistent pattern for  black third graders–big boost in 2003, slight fade in 2004, then tank. Black fifth graders see the boost exclusively in 2005, when the rock star 3rd grade class arrives,  holding on to their gains more so for math than for reading. Spike or not,  black third graders at Lewis Lemon do well in math compared to district whites from 2002-2005.

And here are the formal demographics reflecting the disappearance of white students from Lewis Lemon.

That’s a stark drop in a short time. 47% or so from 2001 to 2003, and 50% from 2003-2005.  It may have just been a drop in new students, but the third grade and fifth grade classes ran out of whites at the same time–I’d have expected the fifth grade cohort to have more white students for longer, in that case.

So restating the observations:

• Lewis Lemon black 3rd grade scores are stupendously high in 2003. The accusations of cheating were groundless, unless the cheaters carefully waited two years to boost the same kids’ scores when they hit fifth grade.
• However, black 3rd grade scores at Lewis Lemon were consistently higher than black scores district and state wide, before and after the program.
• White students saw no real benefit from the new program.
• Fifth graders, white or black, saw no real benefit from the program. The one strong category, fifth graders in 2005, is the boost of the third graders from 2003.

Questions:

• Is it possible that the 3rd grade class of 2003 was in some way extraordinary?
• Were the white and black 3rd grade students practically (that is, for some legal reason) separated? Did the black students have a different teacher than the white students?
• Did the white flight remove more high ability students? But even if it did, you’d expect the white kids to respond as well to DI as black kids,
• Could parents “opt out” of DI?
• Was the white flight out of Lewis Lemon exacerbated by the switch to DI? White students did not see the dramatic increases. Maybe they didn’t like the new regime?

Conclusions:

Well, it’d need a better analyst than me to evaluate the data. And if there’d been solid analysis and reporting at the time, we might have a better idea why the black third graders did so well. Clearly, the curriculum is a possibility for the higher reading and math scores.  But I can’t explain why subsequent third grade classes fell off in performance. And I really can’t figure why the white kids didn’t do well, unless it’s for the same reason that white kids don’t do KIPP.

Did the meta-analysis  include Lewis Lemon?

What I don’t see is a miracle. What I don’t see is  conclusive test evidence showing an obvious incremental increase in test scores at every level, in every demographic, or in every race. Given the actual, honest to god data, not a compiled version of it, broken down by both race and grade and score category, is easily available, I hope someone can go even deeper into the data and see if they find explanations or patterns that aren’t available in this surface level display of data.

Here’s what else I don’t see: any reason whatsoever for that domineering control freak superintendent to rip the program out of Lewis Lemon. I’m not a fan of D, but reading about this blatant obliteration of something that worked for the kids has been depressing. Shame on the media for not digging deep at the time and finding out why.

So there you go. Real data on Direct Instruction. Have at it. Draw your own conclusions.

Happy New Year.

Four Obvious Objections to Direct Instruction

Recently, I defended teachers from Robert Pondiscio’s accusatory fingerpointing. Why no, sir, twas not teachers at the heart of the foul deeds preventing DI’s takeover of the public schooling system.

I don’t have any great insights into why DI isn’t more popular. But any reasonable person should, without any research, have several immediate objections to accepting the Direct Instruction miracles at face value. Hear the tales about Project Followthrough and spend ten minutes reading about this fabulous curriculum, and a few minutes thought will give rise to the following obstacles.

The weird objection

I’ll have more to say later, hopefully, about the roots of Direct Instruction. But no research is necessary to see the B. F. Skinner echoes.  Direct Instruction looks much more like conditioning than education.  A curriculum sample (I can’t make it bigger, click to enlarge):

You’re thinking good heavens, those “signals” are just optional, right? Nope. This video , without prompting, tells the viewer that yes, “signals” are required.

Recently Michael Pershan observed that ” while schools are primarily in the business of teaching kids as much as we can, it’s not anyone’s only priority. There are other things that teachers, administrators, parents and kids value besides instructional efficiency.”

Yes. Many of us value public schools that don’t feel like a cult.

The age objection

From the meta-analysis that’s given rise to all the recent stories:

The strong pattern of results presented in this article, appearing across all subject matters, student populations, settings, and age levels, should, at the least, imply a need for serious examination and reconsideration of these recommendations.

It’s behind a paywall, but I can’t help but be skeptical. I’ve never heard of Direct Instruction implementations at high school.  High school is leagues harder than elementary school and middle school. How would DI work?

Teacher script: “Hamlet Act One Scene One Word One What Word?”
[tap]
Class: “Elsinore!”

Or math:

Teacher script: “Y=mx + b is the slope intercept form. Word m What Word?”
[tap]
Class: “Slope!”
Teacher: “Word b What Word?”
[tap]
Class: “Intercept!”

How many subjects have been broken down to that level? How many books have they scripted for instruction? Or is the high school curriculum like this US History sample, a few questions every paragraph?

I don’t know. I’d guess the researchers don’t know, either.

If DI’s curriculum isn’t entirely defined for high school students in all subjects, then how can the claim be made that DI works for all age levels?  How can we be sure that the gains made in elementary school aren’t subject to the dreaded fadeout? What if DI is simply a good method of teaching basic skills but won’t address the gaps that arise in high school?

Maybe answers–good answers, even–exist, maybe DI works for fifteen to eighteen year olds, maybe Romeo and Juliet can be broken down into tap-worthy chunks. Or maybe those writing paeans about Project Followthrough have no success stories about older kids to tell.

The money objection

There’s a new meta-analysis [that] documents a half-century of “strong positive results” for a curriculum regardless of school, setting, grade, student poverty status, race, and ethnicity, and across subjects and grades.–Robert Pondiscio(emphasis mine)

If it works for all income levels, why aren’t rich kids using it?

I mean, surely, this incredible curriculum is what they use at Grace Church School or Circle of Children to teach these exclusively and mostly white little preschoolers how to read. Distar is the gold standard at  exclusive Manhattan elementary schools. All the teachers are going word one, what word? (tap) and all the little hedge fund progeny obediently repeat the word, or Word.

Except, of course, that’s not the case at all. Check all the websites and you’ll see they brag about their inquiry learning and discovery-based curriculum.

 

Zig Engelmann has written that he focused his attention on the “neediest” children, but that his curriculum helps all students achieve at the highest level. In which case, Zig, go sell your curriculum to the most exclusive private schools. Public schools spend much time arguing that poor children deserve the same education rich children’s parents pay for.

The race objection

I almost left this section out, because it is necessarily more detailed and less flip than the others. At the same time, I don’t see how anyone can hear about DI the miracle and not ask about race, so here goes.

About thirty years ago, Lisa Delpit wrote a stupendous essay, The Silenced Dialogue that just obliterated the progressive approach to education, effectively arguing that underprivileged black children needed to be directly taught and instructed, unlike the children of their well-meaning progressive white teachers.  As I looked up her article to cite  her comments about the “language of power” I realized that Delpit actually discussed this using the context of Direct Instruction (Distar is the primary Engelmann brand):

Note that Delpit, who so accurately skewers progressives for withholding the kind of information that black children need, then rejects the notion of “separating” students by their needs.

She wants it both ways. She wants to acknowledge that some kids need this kind of explicit, structured curriculum while denying the inevitable conclusion that other kids don’t.

DI claims that all kids, regardless of race, see strong improvements.  But take a look at the videos, like this one from Thales Academy, and notice all the students reciting together. They all learn at exactly the same pace?

 

Really?

So I’m going to spoil alert this one. A quick google reveals that Direct Instruction doesn’t allow a student to progress until he or she has mastered the level, and yes, there is ability grouping.

History suggests that the students who move forward quickly will be disproportionately white and Asian, while the students who take much longer to reach mastery will be disproportionately black and Hispanic.

In fact, public schools are strongly discouraged from grouping by ability, and by discouraged I mean sued into oblivion. So how can Direct Instruction achieve its great results without grouping? And if DI helps all races equally, then won’t the existing achievement gap hold constant?

It’s quite possible that DI is an excellent curriculum for at risk kids, particularly those with weak skills or a preference for concrete tasks. It’s not credible that DI instituted in a diverse school won’t either lead to very bored students who don’t need that instruction or the same achievement and ability gaps we see in our current schools.

As I said, these are the relatively straightforward objections that, I think, make a hash out of Robert Pondiscio’s claim that teachers, those foul demons of public instruction, were the source of all DI discontent.  Next up, I’m going to look at some of the actual data behind the claims.

 

Why Not Direct Instruction?

Robert Pondiscio calls it the Rodney Dangerfield of curriculum as he berates the teaching community for disrespecting and neglecting  Zig Engelmann’s Direct Instruction program. Despite showing clear evidence of positive educational outcomes, Direct Instruction has been at best ignored, at worst actively rooted out for over forty years.

And whose fault is that?

..Direct Instruction, however effective, goes against the grain of generations of teachers trained and flattered into the certain belief that they alone know what’s best for their students.

Emphasis mine own, because oh, my goodness.

Trained and flattered.

Trained and flattered?

Trained?

Flattered?

Teachers?

I’ll leave you all to snorfle.

I do not dispute that many teachers think DI is creepy and horrible.  Here’s a fairly recent implementation [tap] that might [tap] help [tap] explain why [tap] teachers shudder. Word one, what word? Oorah!

But now, a question for serious people who want serious answers that don’t require the pretense that teachers are trained and flattered and capable of shutting down educational developments they dislike: why isn’t Direct Instruction more popular?

I’ve read Zig Engelmann’s book, Teaching Needy Kids in Our Backwards System,  and he doesn’t blame teachers. He thinks teachers are backwards and not terribly bright, but argues that most teachers introduced to his curriculum love it.

No, Engelmann puts the blame elsewhere.

 

For example, Direct Instruction unambiguously won Project Follow through. Originally, the program director had intended to identify winners and losers, to prevent schools from picking weak curriculum. But ultimately, the results were released without any such designation. Such a decision is well beyond any teacher’s paygrade.

According to Engelmann, the Ford Foundation was behind the effort to minimize his product’s clear victory. The foundation awarded a grant to a research project to evaluate the results.

The main purpose of the critique was to prevent the Follow Through evaluation results from influencing education policy. The panel’s report asserted that it was  inappropriate to ask, “Which model works best?” Rather, it should consider such other questions as “What makes the models work?” or “How can one make the models work better?”

Engelmann believes that Ford Foundation wanted to feel less foolish about funding all sorts of failed curriculum. I have no idea whether that’s true. But certainly Project Follow Through did not declare winners and losers, and thus from the beginning DI was not given credit for an unambiguously superior result.

Teachers didn’t turn Ford Foundation against DI.

But Engelmann and Becker were expecting decisionmakers to appreciate their success even if Project Follow Through didn’t designate them the victor. Becker wrote up their results for Harvard Educational Review, expecting tremendous response and got a few responses bitching about the study’s design.

I mean, cmon. Teachers don’t read research. That wasn’t us.

Engelmann and Becker fought for recognition all the way up the federal government food chain,  including politicians, and got no results. Shocking, I know.

Zig reserves his harshest criticism for district superintendents, describing a number of times when his program was just ripped out of schools despite sterling results. Parents, teachers, principals complained. One principal was fired for refusing to discontinue the program.

Throughout his memoir, Engelmann seems extremely perplexed, as well as angered, by his program’s failure, and to his credit is still determined to pound down the doors and win acceptance. His partner, Wesley Becker, was less copacetic. After years of rejection by his university and policymakers, Becker left education entirely and drank himself to death in less than a decade.   A few disapproving elementary school teachers aren’t going to induce that degree of existential despair.

Teachers didn’t kneecap Direct Instruction curriculum because it imposed an “intolerable burden” upon them, as Pondiscio dramatically proclaims. No. Decisionmakers killed DI programs. Time and again, management at the federal, state, and local level refuse to implement or worse, destroyed existing successful programs.

Blaming teachers and educators for what are manifestly management decisions is not only contradicted by all the available evidence, but failing to engage with a genuine mystery.

Why have so many districts refused to use Direct Instruction? Why has it been the target of so much enmity by power players in the educational field?

Those are questions that deserve investigation.

 

I did some more digging and have some data to talk about. I also want to discuss Engelmann’s book, since he often contradicts the claims made about his program.

But I’ll leave that for another day, because every so often I like to prove I can get under 1000 words.