Jo Boaler’s Railside Study: The Schools, Identified. (Kind of.)

A brief, illustrative Jo Boaler anecdote by Dan Meyer, currently one of her doctoral students:

I was talking to Jo Boaler last night (name drop!) and she admitted she didn’t really get the whole blogging thing.

I laughed. Some background:

Jo Boaler, a Stanford professor, conducted a longitudinal study of three schools that’s widely known as the Railside paper. She presented the results to a standing room only crowd at the National Meeting of the National Council of Math Teachers in 2008, convincing almost everyone that “Railside” High School, a Title I, predominantly Hispanic high school outperformed two other majority white, more affluent schools in math thanks to the faculty’s dedication to problem-based integrated math, group work, and heterogeneous classes.

“Reform” math advocates, progressives whose commitment to heterogeneous classes has almost entirely derailed the rigor of advanced math classes at all but the most homogenous schools, counted this paper as victory and validation.

Three “traditionalists” were highly skeptical of Boaler’s findings and decided to go digging into the details: James Milgram, math professor at Stanford University, Wayne Bishop of CSU LA, and Paul Clopton, a statistician. They evaluated Boaler’s tests, the primary means by which Boaler demonstrated Railside’s apparently superior performance, and found them seriously wanting. They identified the schools and compared the various metrics (SAT scores, remediation rates) and demonstrated how Railside’s weak performance called Boaler’s conclusions into question. Their resulting paper, “A close examination of Jo Boaler’s Railside Report”, was accepted for publication in Education Next—and then Boaler moved to England. At that point, they decided not to publish the paper. All three men were heavily involved in math education and didn’t want to burn too many bridges with educators, who often lionize Boaler. One of the authors, James Milgram, a math professor at Stanford, posted the paper instead on his ftp site. Google took care of the rest.

The skeptics’ paper has stuck to Boaler like toilet paper on a stiletto heel; she’s written a long complaint about the three men’s “abusive” determination to get more information from her. From an Inside Higher Ed report on her complaint:

[S]he said she was prompted to speak out after thinking about the fallout from an experience this year when Irish educational authorities brought her in to consult on math education. When she wrote an op-ed in The Irish Times, a commenter suggested that her ideas be treated with “great skepticism” because they had been challenged by prominent professors, including one at her own university. Again, the evidence offered was a link to the Stanford URL of the Milgram/Bishop essay.

“This guy Milgram has this on a webpage. He has it on a Stanford site. They have a campaign that everywhere I publish, somebody puts up a link to that saying ‘she makes up data,’ ” Boaler said. “They are stopping me from being able to do my job.”

Boaler is upset because ordinary, every day, people aren’t merely taking her assertions at face value, but are instead challenging her authority with a link to a paper that, in her view, they shouldn’t even be able to read. So you can see why I laughed. This is a woman with absolutely no idea how the web works. “It’s not even peer-reviewed!!!” That people might find the ideas convincing and well-documented, with or without peer-review, isn’t an idea she’s really wrestled with yet.

Identifying the Schools
As I mentioned a while back, I had a strong reaction four years ago when reading an earlier work by Jo Boaler. A few months later, while still in ed school, I perused her Railside paper, which struck me as equally, er, not credible, a product of wishful deception, maybe? Or maybe just wishful thinking. I googled around to see if I was the only doubter and found the Milgram/Bishop/Clopton paper.

Railside High School

The article indicated that the three schools were identifiable. So I just googled algebra “bay area” boaler and in the first 2-3 pages I found this report on San Lorenzo High School:

San Lorenzo’s relationship with Stanford was based on their participation in a longitudinal study conducted by Professor Jo Boaler and her colleagues at the university. ….According to the CAPP liaison to the project, Weisberg, the researchers also found that SLHS math teachers rated high for their constructivist approach to teaching when compared to teachers at the other two high schools in their study.

Praised for their constructivist approach? In five minutes, I’d not only identified one of the schools. I’d identified the big Kahuna–Railside, the star of Boaler’s report, the school whose dedication to complex instruction, problem-based integrated math, and heterogeneous classes had propelled the Stanford professor to fame and glory. Bow to my greatness.

Happily, Boaler’s paper included CST scores for 2003, so I could match them up (as did MBC in their followup paper):

I could easily confirm that San Lorenzo High School CST scores for freshmen match exactly to Railside’s:

(you can confirm here, it’s in Alameda County. The Algebra column for freshmen only. See? 1% 15% 33% 36% 15%. 188 students. )

San Lorenzo is an California East Bay suburb, so I’m not sure why Boaler would describe Railside as “an urban school”. California has any number of high poverty, Title I suburban schools.

One down, two to go. But the original MBC paper didn’t specify how the men identified the schools, and google gave too many possibilities for the other two study participants. Besides, I had other things to do, like find a teaching job, so I put away childish things.

Greendale High School

Then four years later, Jo Boaler complains and, in his response, James Milgram explains how they identified the schools:

We took the data above from Table 5, and one of us…checked the entire publicly available 2003 California STAR data-base, looking for schools for which any column was identical to one of the columns in Table 5. In each case we found that there was one and only one school that had that data.

Hey. I could do that. I had Access (the database), even. Which you need, because the CST file is too big for Excel.

Using this method, I identified the other two schools.

I downloaded the 2003 data to a text file, imported it to Access. I know mySQL’s interface but have never used Access before. I feel sure there’s an easier way than the path I took, which was to treat poor Access like Excel: go to the TestResults table, highlight the “total students tested” row, and search for 125, looking to the right for 0,6,27,55,12. It sounded something like this:

ClicknoClicknoClicknoClicknoClicknoClicknoClicknoClicknoClickno ClicknoClicknoClicknoClickWAITcrap that was it!go back! What, there’s no Reverse?Christ?Where was it?crapcrapcrapscrollbackscrollbackClicknoClickn…yes! There it is!

What, you don’t see it? Click to enlarge:

I found Greendale!! Whoohoo!

All I had to do was tab to the left a bit and look up the school’s identifying number. Then I went to the form in Access to look up the school and tada!

According to Jo Boaler, “Greendale High School is situated in a coastal community, with very little ethnic or cultural diversity (almost all students are white).”.

Well, she’s half right. Greendale is definitely mostly white, but it’s in the mountains, not in the excessively wealthy mountains, in the much much much Greater Bay Area. Well, really, it’s juuuuust outside the much much much Greater Bay Area. Very pretty place. If you look at it on Google Maps, you would barely see blue, way off to the left.

It is not coastal.

Hilltop High School
Back to Access and clicknoclicknoclickgobackack!clicknoclickstop!tableft and there! I have Hilltop.

Here are the CST scores to match:

Boaler on Hilltop: “Hilltop High School is situated in a more rural setting, and approximately half of the students are Latino and half white.”

Demographics, right. Location, wrong. Boaler has just described Greendale’s location, not Hilltop’s. Find Hilltop’s town on a map and the blue is just to the left. One would describe Hilltop as “coastal”.

So Boaler flipped the school descriptions, but not the demographics? Was that on purpose, or an error?

I feel pretty confident, therefore, that in Boaler’s report:

• Railside High School is San Lorenzo High School, in San Lorenzo. Title I school, mostly Hispanic.
• Greendale High School is located in one of the mountain chains surrounding the Bay Area. Rural community, economically diverse, mostly white.
• Hilltop High School is in a coastal community just outside the Bay Area, half Hispanic, half white. Greendale and Hilltop are not neighbors, but much closer to each other than either is to the edges of the Bay Area, much less San Lorenzo.

I originally planned to reveal the names of all three schools. I used publicly available data and Boaler’s own study to identify them. The schools have nothing to be embarrassed about. They participated in a study to help further knowledge about effective math instruction. How is that a bad thing? Their scores are already available on government website. Boaler isn’t directly critical of any school. No downside is immediately apparent, at least to me.

But still. In San Lorenzo High School’s case, their participation is easily searchable, so I identified the school. But the other two schools take quite a bit of work to find in Google, and the principals might not want to wake up and find their schools in a blog, even if the news wasn’t bad. This way, they can have some warning—again, with the understanding that this is publicly available data. Using Access is the cleanest way to find them, but at the end of this post I will give some other info to help interested people identify them.

So What Does This Mean?

Well, let’s assume that I didn’t miss schools with identical CST scores (I checked every entry, but who knows, I might have clicked too fast) and that these are, in fact, the schools in the study.

With just a bit of effort, interested parties can now review the Milgram/Bishop/Clopton report and confirm its claims about the overall math performance of the three schools. I’ve spot checked a lot of it, and I haven’t found any errors yet.

I’m not terribly detail-oriented, yet I saw two huge issues.

First, the 2003 CST data I matched up? Boaler provides this data as an external validator, showing how well the Railside kids did compared to the other two groups, thanks to the superior instruction of reform math. As is evident from the screen prints of the actual CST data that Boaler is using freshman 2003 data. But in Table 6, reproduced here:

Boaler provides Year 3 data and clearly indicates that the students are juniors in 2003. The freshman algebra scores are not from her cohort. So why is she using this data as evidence of how great the program was? Shouldn’t she be using Algebra II data?

I went back two years to see what algebra scores were like, and discovered San Lorenzo High School (Railside) had fewer than ten freshmen taking algebra—in fact, the school has no math subject-specific scores at all. The other two schools did have freshmen algebra classes. So what, exactly, was Boaler comparing?

Milgram et al cover all of this in greater detail, and they also cover the other big red neon warning I see: if San Lorenzo High, which didn’t track, put all of its freshmen in algebra, while Greendale and Hilltop put their mid-to lower ability students in Algebra while the top freshmen took Geometry and Algebra II, then Boaler should not assert that San Lorenzo High is outperforming the other two schools based on freshman Algebra scores.

Of course, since she’s using the scores from the wrong cohort, she didn’t really demonstrate that the studied cohort from San Lorenzo HS outperformed the other two schools in the CST to begin with.

Why bother?

Like most mathematicians, MBC are vehemently opposed to reform math. Both Milgram and Bishop spend a lot of time working with parents or districts that are trying to get rid of reform curricula. In his rebuttal, Professor Milgram says,

Indeed, a high ranking official from the U.S. Department of Education asked me to evaluate the claims of [the Railside study] in early 2005 because she was concerned that if those claims were correct U.S. ED should begin to reconsider much if not all of what they were doing in mathematics education. This was the original reason we initiated the study, not some need to persecute Jo Boaler as she claims.

However, given both men’s determination to oppose reform math, and their willingness to work with parent groups organizing against reform math, Boaler believes, as Milgram says, that the paper was an attempt to discredit reform math, as opposed to an honest academic inquiry.

I have no opinion on that, but then I spend a lot of time on the Internet. MBC all seem pretty mild to me.

I’m not a traditionalist. I’ve written many times in this blog that for a pro-tracking, pro-testing discovery-averse teacher, I am stupendously squishy. Milgram, Bishop, Clopton, and Professor Wu would undoubtedly disapprove of my teaching methods. My kids sit in groups, I use a lot of manipulatives, I don’t lecture much or give notes, use lots of graphic organizers. To the extent I have a specialty, it lies in coddling low ability, low incentive kids through math classes whilst convincing them to learn something, and what they learn isn’t even close to the rigorous topics that real mathematicians want to see in math class. (Some lesson examples: real life coordinate geometry, modeling linear equations, triangle discovery, factoring trinomials, teaching trig and right triangles.) Nonetheless, I firmly believe that discovery, problem-based math, and complex instruction are ineffective with low to mid ability kids and think tracking or ability grouping is essential. So I’m not really tied to either camp in the math wars.

Besides, the math wars have largely been resolved. Lectures won’t work for low ability kids, but neither does discovery. High ability kids need fewer lectures, fewer algorithms, more open-ended problems, more challenges. Traditionalists have a lot of energy around reform math, but I think they could dial it back. For the most part, reform has lost in schools, particularly high schools.

Since Boaler will, if she acknowledges this post at all, complain about my motives, let me say that I am not a Boaler fan, but my disapproval is based purely on her opinions as revealed through her work: the Amber Hill/Phoenix Park paper, the Railside paper, and yeah, her recent bleat struck me as a big ol’ self-pity fest. But I’m not actively seeking to hurt her reputation, and while my tone is (cough) skeptical, I’m perfectly happy to learn that all of these questions I raise involve perfectly normal research decisions for academics.

However, I am constantly surprised at the unquestioning acceptance of educational research, particularly quantitative research.

Remember, this is a hugely significant paper in the math wars. Boaler is the hero who went out and “proved” that reform math gets better results. Suppose it’s academically acceptable for Boaler to assert that San Lorenzo High School algebra students outperformed the algebra students from two more affluent schools, based on the test results of students not in her study cohort. Would it nonetheless be important for education journalists to point out that the San Lorenzo students included the best students in the school, while the Greendale and Hilltop schools’ best students were in more advanced classes? And that a component of her success metric relied on scores of students who were two years behind her cohort?

To the extent I have an objective, there it is. Educational researchers may, in fact, engage in entirely acceptable behavior that nonetheless hides information highly relevant to the non-academic trying to use the research to figure out educational best practices.

Who’s responsible for bringing that information to light?

****************************************************************************

Identifying the schools

Ironically, when I was originally searching for the schools four years ago, I came across a link that identified Greendale. I just didn’t realize it for reasons that will be clearer once you find the link. Since MBC discuss the Greendale parents’ demand for a “traditional” program, and the school’s reluctant compliance, I tried to use that history to figure it out, googling (exactly): “interactive mathematics program” california high schools traditional. In the first couple pages, I found a link written by one of the MBC authors that references that parental demand as well. There are several schools mentioned in the paper, but only one of them is rural.

I’d also found a link with the Hilltop school in my initial search but had dismissed it, thinking the schools would all be in the Bay Area. But since MBC mentions that the school district forced Hilltop to cancel, I’d googled “interactive mathematics program” california district canceling. That will bring up, in the first two or three pages, a blog post from a once fairly well-known education specialty blogger (since gone inactive) on the school. This battle went on for some time, and the New York Times covered it earlier, but I won’t give the query for that.

A couple other clues: Many of Jo Boaler’s doctoral students posted in support of her complaint. An early supporter, who has a well-regarded math blog, taught at Greendale High School, although after the years of Jo Boaler’s study. That is probably not a coincidence. Jo Boaler thanks teachers in the paragraph in which she also mentions the schools that participated in her study. Maybe check out those teachers and see where they teach (or taught).