Direct Instruction Miracle? The Lewis Lemon Case

Back in February, I observed that Direct Instruction hadn’t failed because of teacher disdain, as Robert Pondiscio charged, but because educational leadership from superintendents on up through US senators had not only refused to support the program, but in some cases actively and systemically ripped out implementations that seemed to be successful.  As I said at the time, I considered it a real mystery.

So I dug deep into Zig Engelmann’s book, Teaching Needy Kids in Our Backwards System looking for an example of a success story I could independently verify. He goes on at some length about both San Diego and Baltimore schools, but not in any way I could pin down DI vs. Non-DI. Quite often Engelmann would cite a newspaper article as evidence.

The Ventura County Star carried an article on March 15, 2003, titled “Effective Reading Program Must Go”, which indicated that the only school in Ventura County and one of 109 in the state to receive a citation for achieving exemplary progress was forced to drop their [DI] program.

No mention of the school. Or the superintendent. Or the degree to which test scores increased. One gets the distinct impression that Zig….well, read about it in the paper. He uses virtually identical wording in this Edweek article. California keeps detailed, easily searchable records; with the name of the school, the specifics of the decision could be confirmed.

The exception is Lewis Lemon.LewisLemoncite

But then, as happened consistently through the decades, a new superintendent came in and demanded Lewis Lemon abandon Direct Instruction and adopt “balanced literacy”. The principal, Tiffany Parker, flatly refused. She was accused of cheating, so the students were tested again. They passed the new test.  The district actually had to give back Reading First federal funds because they dropped DI. Apparently, the district blamed the principal for informing the feds that the district was no longer using the program, and Parker was demoted. She sued, and settled.  If you google this, you’ll find a New York Times article, a New York Sun article, this Heartland article, and references to many Rockford Register Star articles by Carrie Watters, who from what I can tell now works at an Arizona paper. The Rockford Register Star archive goes back before 2003, and has other articles by Carrie Watters, but contains none of the many stories Watters did on the Lewis Lemon controversy. Even Joanne Jacobs’ blog entry on Rockford, “They Messed with Success” has disappeared, which had me really freaked until I learned she was had temporarily moved her archives.

The story made national headlines when the superintendent decided, in 2005, to force Lewis Lemon off of DI into “balanced literacy”, while the strong student performance of 2003 only got covered locally. The New York Times piece went into a bit more depth, mentioning that the fifth graders didn’t do as well. Also, the reports weren’t always specific. Was it math or reading? Or both? So I went digging for data.

If contemporaneous reports are scarce, the contemporaneous data survives. Illinois test scores comparing Lewis Lemon performance to the district and state by race are all online. Here they are, and please check my results:

2001–the year they implemented DI
2002
2003
2004
2005–the year they were forced to discontinue DI
2006

So now I’m going to go all graph heavy, like Spotted Toad. First up: the top level comparison of Lewis Lemon scores to the district and state level for both 3rd and 5th grade (reading down).

LewisLemonOverall

Note the big ol’ spike in 2003. Lewis Lemon 3rd graders had far more students meet or exceed standards than the district or state averages in that year. But then, third graders at Lewis Lemon outscored the district average  every year both before and after DI was implemented in  Notice, too, that the fifth grade scores aren’t nearly as impressive, but that both math and English show a spike in 2005, which is the year the third graders of 2003 would have been in fifth grade.  Then again, fifth graders had much better reading scores in 2002 than in any time after DI implementation.

Lewis Lemon officials blame the lower fifth grade reading scores on a less than perfect implementation, and students who were further behind. On the surface, this seems unlikely, given the high scores in 2002. But there may be other causes.

So I disaggregated, first by race. Here are black and white 3rd grade reading score scores compared at the school, district, and state level, broken down by achievement category.

Up first, 3rd grade blacks:

LewisLemonBlack3rdReading

Now remember, this is Lemon Lewis black students compared to other black students only (not the entire population). Blue is Lemon Lewis, and the more blue to the right of each graph, the better Lewis Lemon is doing. 2003 is the year of the Big Score. The blue clearly moves to the right from 2002 to 2003, and stays there for two years. Then it shifts back.In 2003, black students  at Lewis Lemon met or exceeded standards at 3 to 4 times the numbers that district or state black students did.

Third grade reading, white students.

LewisLemonWhite3rdReading

So this is interesting in a couple ways. First, it’s clear that in the pivotal year of 2003, white performance actually declined slightly. Fewer students failed to meet standards, but fewer students exceeded them. Not a lot. But there’s no tremendous spike in white student performance in 2003.

And then, something that has gone completely unmentioned: the white student population collapsed to below testing levels immediately after 2003. This explains some of the falloff of the graph above–if whites comprised a decent chunk of the high scores, then their disappearance would impact the overall “meets or exceeds”.

It’s not clear to me whether the DI implementation was reading only, or reading and math. The news accounts all focus on reading, but more than one account mentions improved math scores. So I’ll include them. Here’s third grade, African American.

LewisLemonBlack3rdMath

Here, Lewis Lemon was scoring better than blacks in the district and state before the DI implementation. The blue still shifts right, but it’s not as dramatic. In 2003, virtually every student met or exceeded expectations, but in every other year, both before and after the DI implementation, they were still doing very well.

Now white 3rd grade math:

LewisLemonWhite3rdMath

So in 2002 through 2003,  the school saw a good bunch of whites move from “meets” to “exceeds”. Unlike reading, third grade white performance saw a decent boost, but it started a year earlier. Whites at Lewis Lemon performed better than the district.

And now, the fifth grade. Reading, black fifth graders:

LewisLemonBlack5thMath

Scores actually declined in 2003. 2005, the year that the third graders from 2003 were in fifth grade, sees a slight improvement. But not much of one–more than half failed to meet standards.

Fifth grade whites, reading:

LewisLemonWhite5thReadingAgain, whites have their best year in 2002, and don’t show any particular spike. They also do better than black fifth graders (which is normal).

Fifth grade math:

LewisLemonBlack5thMath

Indifferent–except note the spike when the third graders of 2003 show up. It’s much more pronounced in math than it was in reading.

White fifth graders, to finish out.

LewisLemonWhite5thMath

So here’s one last way of looking at the data–compares blacks at Lewis Lemon to the whites at the school, district, and state level. Read down  on the left for 3rd and 5th reading, down on the right for 3rd and 5th math.

LewisLemonblktowhite

Once again a consistent pattern for  black third graders–big boost in 2003, slight fade in 2004, then tank. Black fifth graders see the boost exclusively in 2005, when the rock star 3rd grade class arrives,  holding on to their gains more so for math than for reading. Spike or not,  black third graders at Lewis Lemon do well in math compared to district whites from 2002-2005.

And here are the formal demographics reflecting the disappearance of white students from Lewis Lemon.

LewisLemonDemographics

That’s a stark drop in a short time. 47% or so from 2001 to 2003, and 50% from 2003-2005.  It may have just been a drop in new students, but the third grade and fifth grade classes ran out of whites at the same time–I’d have expected the fifth grade cohort to have more white students for longer, in that case.

So restating the observations:

  • Lewis Lemon black 3rd grade scores are stupendously high in 2003. The accusations of cheating were groundless, unless the cheaters carefully waited two years to boost the same kids’ scores when they hit fifth grade.
  • However, black 3rd grade scores at Lewis Lemon were consistently higher than black scores district and state wide, before and after the program.
  • White students saw no real benefit from the new program.
  • Fifth graders, white or black, saw no real benefit from the program. The one strong category, fifth graders in 2005, is the boost of the third graders from 2003.

Questions:

  • Is it possible that the 3rd grade class of 2003 was in some way extraordinary?
  • Were the white and black 3rd grade students practically (that is, for some legal reason) separated? Did the black students have a different teacher than the white students?
  • Did the white flight remove more high ability students? But even if it did, you’d expect the white kids to respond as well to DI as black kids,
  • Could parents “opt out” of DI?
  • Was the white flight out of Lewis Lemon exacerbated by the switch to DI? White students did not see the dramatic increases. Maybe they didn’t like the new regime?

Conclusions:

Well, it’d need a better analyst than me to evaluate the data. And if there’d been solid analysis and reporting at the time, we might have a better idea why the black third graders did so well. Clearly, the curriculum is a possibility for the higher reading and math scores.  But I can’t explain why subsequent third grade classes fell off in performance. And I really can’t figure why the white kids didn’t do well, unless it’s for the same reason that white kids don’t do KIPP.

Did the meta-analysis  include Lewis Lemon?

What I don’t see is a miracle. What I don’t see is  conclusive test evidence showing an obvious incremental increase in test scores at every level, in every demographic, or in every race. Given the actual, honest to god data, not a compiled version of it, broken down by both race and grade and score category, is easily available, I hope someone can go even deeper into the data and see if they find explanations or patterns that aren’t available in this surface level display of data.

Here’s what else I don’t see: any reason whatsoever for that domineering control freak superintendent to rip the program out of Lewis Lemon. I’m not a fan of D, but reading about this blatant obliteration of something that worked for the kids has been depressing. Shame on the media for not digging deep at the time and finding out why.

So there you go. Real data on Direct Instruction. Have at it. Draw your own conclusions.

Happy New Year.

About educationrealist


16 responses to “Direct Instruction Miracle? The Lewis Lemon Case

  • sthomson1971

    Data for white students at the school is missing for 2004-2006.

  • Ken DeRosa

    I’ve learned the hard way that it’s difficult doing these kind of analyses based on very incomplete information and state data. As you point out more questions get raised than are answered.

    All I can say is that often when the top of the class is very small and already going to pass the state exam, they tend to get ignored by the school. They’ll get the weakest teacher, or lumped in with lower tracks so more help can be given to the kids at the margins, or who cares – they’re going to pass anyway.

    And that attrition rate (assuming it’s that and not something else, like a changing of boundaries or an influx of new minority students) is going to skew any results due to survivorship bias.

    • educationrealist

      You appear to be focused on the fact that white students didn’t improve and that DI can’t be blamed for that on the face of it. I agree. But where’s the reporting on it? What we need is reports or historical analysis that digs into the cause of this.

    • educationrealist

      Also, it’s really hard to believe that whites just loved the program when they were leaving in droves. I’d love to know if the unstated reason for the superintendent’s recalcitrance was due to fear of white flight throughout the district.

  • Michael Watts

    Can you be clearer about the timing of things? You make many references to student performance before, during, and after the DI implementation. But the only timing information I noticed in the piece was that the DI implementation began in 2001 and ended in 2005. All the data you present is from the years 2001-2006. It’s clear that 2006 is “after”. (Though with only a single “after” point, I wouldn’t want to be very confident in saying what performance was like “after” the program got ripped out.)

    It’s clear that 2002, 2003, and 2004 are “during” the program.

    It isn’t clear whether 2001 is “before” or “during” the program, and it also isn’t clear whether 2005 is “during” or “after”. And the same point applies as with 2006 — with between 0 and 1 “before” data points, it’s hard to be confident in saying what performance was like before the program was implemented. If 2003 can be anomalous, 2001 can be too.

    I would have assumed that the white flight was the reason for ripping out the program, but I have no particular reason for that. I was surprised to see you say

    Here’s what else I don’t see: any reason whatsoever for that domineering control freak superintendent to rip the program out of Lewis Lemon.

    after you gave thorough coverage to an obvious reason.

    I was even more surprised to see this comment:

    I’d love to know if the unstated reason for the superintendent’s recalcitrance was due to fear of white flight throughout the district.

    which kind of implies that you did see a reason for the superintendent’s behavior.

    • educationrealist

      Two things can be true: 1) the superintendent pulled the program because whites didn’t like it. 2) It’s a stupid reason to rip the program out when it had that sort of results for black kids–although I can’t tell if the 3rd graders were an exception or something that would have been the norm.

      Moreover, I’m a bit skeptical that the reporting would be *that* negligent. There should have been evidence of white parents bitching about the program at board meetings, and so on. So even though that seems possible, it’s also not impossible that the superintendent simply disliked the program.

  • Michael Pershan

    Interesting analysis! I agree that this doesn’t look like a miracle…but then again all this stuff is hard to look at, and ideally we’d want to compare schools to schools. Still, interesting.

    As far as why that superintendent messed with things: didn’t one of those pieces suggest it was so that he could provide district-wide PD in balanced literacy?

  • Mark Roulo

    Rockford, IL schools were under court supervised de-segregation oversight until 2002 when the district was “fully desegregated.” Local control was (in a limited form) returned in 2002 with the parents being given more (but not total) control over where their kids went to school.

    Googling suggests that Rockford, IL is pretty racially polarized. You can, as an example, search for “Rockford ranked 2nd worst city in America for racial equality” to find a 2015 article making that claim.

    I think there is a VERY good chance that Lewis Lemon is in a more black neighborhood and that once the white parents were allowed to stop busing their children there they did so. https://www.neighborhoods.com, for example, claims that “Neighborhoods and Homes nearLewis Lemon Elementary School” are 74% black and 10% non-hispanic white. This is fairly close to what your demographic curve tended towards.

    So, I think the white/black mix is a busing story rather than a Direct Instruction story. Prior to 2002 the district probably ran about 40% white because the white kids were bused in. After 2002, the parents were able to stop this. Maybe a bunch had their kids finish off their education at Lewis Lemon, but I can’t imagine that very many would actively sign up their 6 year olds to be bused there. Five years later Lewis Lemon had reverted to the neighborhood demographics.

  • Mark Roulo

    Something to keep in mind about the 3rd -> 5th grade scores is that the students in 5th grade are only sorta the same ones as in 3rd grade.

    http://www.illinoisreportcard.com reports that Lewis Lemon *today* has 32% “student mobility,” which is defined to be “the percentage of students who experienced at least one transfer in or out of the school between the first school day of October and the last school day of the year, not including graduates.”

    Two years of 32% student mobility and you’d expect about 0.68*0.68 ~= 50% of the 5th grade class to be the same students as the 3rd grade class from two years back.

    I do *NOT* know if the “student mobility” was similar back in the early 2000s, but that’s probably a better guess than that all the same students who scored well in 3rd grade in 2003 showed up at 5th graders in 2005.

  • Mark Roulo

    jrdougan: “In my past life doing consulting, I’ve seen has been many a manager who would rather fail the same way as everyone else rather than succeed using something non-standard.”

    Probably not quite.

    Managers tend to be reluctant to take RISKS and there is additional risk being non-standard. If the managers were *guaranteed* to be successful in a non-standard manner and *guaranteed* to fail conventionally, most would pick successful and on-standard. They are afraid of failing in a non-standard manner because that is worse than failing in the standard manner (“What were you thinking????” asks their former boss during the exit interview.)

    If someone else has already taken the risk of failure and we now have a success, the manager won’t intentionally change the success to a common failure. The manager might try to ‘standardize’ something that works, but only because he expects that the standardized approach will work, too. Again, if failure is guaranteed, then things are probably left alone.

    A different set of incentives come into play if folks can’t stand each other personally.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: