THE PROFESSORIATE FAILS US AGAIN: No one checked Reinhart and Rogoff’s work!

TUESDAY, APRIL 30, 2013

Part 2—The tale of the graduate student: We can think of two obvious questions about the latest giant bungle by the nation’s high-ranking professors.

Here’s the first question, and others have asked it: How did Carmen Reinhart and Kenneth Rogoff manage to bungle so badly?

People do make mistakes, of course, especially Harvard professors. But some of the errors by Reinhart and Rogoff do seem just a bit strange.

If you’re averaging data from twenty countries, how do you manage to leave out five countries? (Answer: You make an Excel coding error.) Once you’ve made your coding error, how do you fail to notice that your results don’t seem quite right?

We have learned to expect the worst from our highest-ranking authorities. Even so, some of the bungling in this case does seem somewhat strange.

That said, our second question is harder to answer—and we haven’t seen other folk ask it. This is our second question:

Why did three years go by before someone checked Reinhart and Rogoff’s work? And why was this important task left to three graduate students?

Here’s why that question is puzzling:

According to Paul Krugman and others (see below), many people were skeptical of Reinhart and Rogoff’s results from the first. And uh-oh! The study by the two professors turned out to be quite influential.

Powerful interests used their results to support austerity policies. According to Krugman, their study “instantly became famous; it was, and is, surely the most influential economic analysis of recent years.”

Under the circumstances, wouldn’t you think that somebody other than graduate students would have double-checked Reinhart and Rogoff’s results—results which didn’t quite seem to make sense? How did three long years go by before someone checked their work?

If you’re willing to wonder about that question, the BBC’s report on this matter is especially worthwhile. Writing for the BBC, Ruth Alexander interviewed Thomas Herndon, the University of Massachusetts graduate student who took the lead in double-checking Reinhart and Rogoff’s work.

This interview helps us see the way our society’s intellectual elites tend to function. Because alas! Herndon’s account of his adventure does seem sadly familiar.

Alas! Like a latter-day Joseph Campbell, Alexander seems to sketch the journey of the modern intellectual hero in her report on Herndon. As Herndon tells his story, Alexander records four stages on his journey:

His initial sense that something is wrong with some authority's findings: Herndon finds himself doubting Reinhart and Rogoff’s results.
His attempt to confirm that authority's findings: Herndon tries to replicate the professors’ findings but is unable to do it.
His assumption that his own lowly work must be wrong: Because Reinhart and Rogoff are famous professors, Herndon assumes that he, the graduate student, must be doing something wrong.
His eventual slaying of the dragon: Herndon asks the professors for their data, discovers that they have erred.

The most interesting part of Alexander’s report involves the third stage of this sadly familiar journey. Herndon describes a prevailing assumption—our highest-ranking intellectual elites simply can’t be wrong:

ALEXANDER (4/19/13): [W]hile US politicians were arguing over whether to inject more stimulus into the economy, the euro was creaking under the strain of forced austerity, and a new coalition government in the UK was promising to raise taxes and cut spending to get the economy under control, Thomas Herndon's homework assignment wasn't going well.

No matter how he tried, he just couldn't replicate Reinhart and Rogoff's results.

“My heart sank,” he says. "I thought I had likely made a gross error. Because I'm a student the odds were I'd made the mistake, not the well-known Harvard professors."

His professors were also sure he must be doing something wrong.

"I remember I had a meeting with my professor, Michael Ash, where he basically said, 'Come on, Tom, this isn't too hard—you just gotta go sort this out.’”

In this, the dark night of the grad student’s soul, everyone assumed that he, the lowly student, must be getting it wrong!

To his credit, Professor Ash finally urged young Herndon to ask Reinhart and Rogoff for their data. To their credit, Reinhart and Rogoff complied with this request.

When Herndon got the professors’ data, he spotted their mistakes. This brings us back to our puzzling question:

Why was it left to three graduate students to check the professors’ work? Since people had doubts about their study, why didn’t other professors ask to see their data?

Below, you see Krugman’s capsule account of these events. Why was the slaying of this dragon left to three graduate students?

KRUGMAN (4/19/13): [T]he truth is that Reinhart-Rogoff faced substantial criticism from the start, and the controversy grew over time. As soon as the paper was released, many economists pointed out that a negative correlation between debt and economic performance need not mean that high debt causes low growth. It could just as easily be the other way around, with poor economic performance leading to high debt. Indeed, that's obviously the case for Japan, which went deep into debt only after its growth collapsed in the early 1990s.

Over time, another problem emerged: Other researchers, using seemingly comparable data on debt and growth, couldn't replicate the Reinhart-Rogoff results. They typically found some correlation between high debt and slow growth—but nothing that looked like a tipping point at 90 percent or, indeed, any particular level of debt.

Finally, Ms. Reinhart and Mr. Rogoff allowed researchers at the University of Massachusetts to look at their original spreadsheet—and the mystery of the irreproducible results was solved. First, they omitted some data; second, they used unusual and highly questionable statistical procedures; and finally, yes, they made an Excel coding error. Correct these oddities and errors, and you get what other researchers have found: some correlation between high debt and slow growth, with no indication of which is causing which, but no sign at all of that 90 percent ''threshold.''

In Krugman’s account, the graduate students are described as “researchers at the University of Massachusetts.” This softens the blow as we ask once again:

Why didn’t other professors check Reinhart and Rogoff’s work?

There may be a good answer to that, but this pattern seems all too familiar. On the one hand, we think of the famous child who was able to spot his emperor’s lack of new clothes.

We also think of our own experience with a fraudulent test score-reporting practice affecting the entire state of Virginia. Tomorrow, in part 3 of this report, we will revisit that episode.

Three long and punishing years went by before anyone checked the professors’ work! In this badly fallen age, this is very much the way our intellectual elites tend to function.

Tomorrow: Nobody bothered to check. After we checked, no one tattled!