Evidence: Is It Really Overrated?

July 4th, 2014 at 11:44 am

By Jared Bernstein

A few weeks ago, during the evidentiary dustup between Piketty and the FT, I quasi-favorably quoted a Matt Yglesias line re empirical evidence being overrated. A number of readers were understandably unhappy with that assertion, arguing that they come here to OTE for fact-based analysis based on empirical evidence (with, admittedly, a fair bit a heated, if not overheated, commentary). If facts all of the sudden don’t matter anymore, why not just call it a day and join the Tea Party?

So let me add a bit more nuance. The statement is about the quality and durability of evidence, which is not only varied, but, at least in the economic policy world, increasingly problematic. A number of developments have significantly lowered the signal-to-noise ratio.

I’d divide the evidence problem into two separable categories. First, statistical issues about what’s “true” and what’s not, and second, ideological ways in which the noise factor is amplified at the expense of the signal. It’s this latter bit that’s arguably gotten worse.

–“Variable coefficients” Sounds wonky, but all’s I’m saying is that relationships change over time. As economies and societies evolve, as globalization increases, as technologies change, as cultural norms ebb and flow, we shouldn’t expect the relationships between inflation and unemployment, minimum wages and jobs, growth and inequality, education and pay, or pretty much anything else to stay the same. It’s rare for the “sign to flip” meaning I expect education and pay to remain positively correlated and inflation and unemployment to remain negatively correlated. But magnitudes change a lot. No elasticity is etched in stone! So be skeptical when someone tells you “this leads to that and therefore this is a good or bad idea!”

That’s “skeptical,” not cynical. Judicious use of data by unbiased practitioners can help inform our understanding of the relationship between this and that, at this juncture in time.

–Complexity As I stressed in the first link above, it is hard to know what to make of empirical evidence when it is based on simulated data, meaning data that the analyst has changed in ways that she believes are necessary (remember, we’re not talking ideological thumb on the scale yet). In my world, this happens a lot with income and wealth data (it was also the basis of the Piketty/FT argument).

Such data are often incomplete in ways that matter. They might, for example, leave out the value of government-provided benefits. So analysts are clearly justified in tacking on such things as the value of food stamps or medical benefits. But how to do so can be deceivingly tricky and there’s often no obvious right answer.

Health care benefits in the US provide the best example. If you just append the market value of coverage to the income of say, Medicaid recipients, as CBO does in some widely used tabulations, you’re inflating the incomes of the poor in ways that reflect not their buying power, but the widely documented excesses in the US health care system, including inflated salaries and over-priced medicines.

One study that employed this and other even more dubious methods ended up publishing results showing that between 1989 and 2007, when every other study showed income inequality growing, the income of the poorest 20% rose the most in real terms while that of the top 5% fell the most (see figure here).

With such analysis proliferating, you’ve really got to know what you’re looking at. I’m also increasingly drawn towards simple data that didn’t involve a lot of researcher manipulation, like these annual wage trends, for example.

–Noisy data I wrote about this yesterday re first quarter GDP—a huge negative outlier—but the fact that statistical evidence always has uncertainty bands around it gets quickly lost in the debate. I try to remind everyone, most notably myself, that the payroll jobs number upon which we all obsess every month has a 90% confidence interval of 90,000 jobs. Imagine how differently yesterday’s big jobs report—288K pop on payrolls—would have gone if the result was 200K, but in fact, that result is within the confidence interval (the CI means there’s a 90% chance that the true change in payrolls was 90K below or above the point estimate).

More so than in the past, we live in a world where money doesn’t just buy power and political access—that’s always been the case. It also buys the answers it wants, from faux-climate-scientists and “think tanks” that can generate whatever result you need. Recall, for example, the Heritage Foundation’s initial projections, later revised (or deep-sixed) as I recall, that “dynamic effects” of one of Paul Ryan’s budgets would drive the unemployment rate down to 2%.

You could just say, “well, I’m not going to believe evidence that comes from people or institutions with an ideological bias.” But that would be wrong. Paul Krugman is a “liberal” but in countless empirical columns and blogs, his analysis has been careful and accurate. The Center on Budget, EPI, CEPR too. Me too. That’s not to say they or I don’t make mistakes, of course. It’s that we’re almost always careful to know the data pretty well and not go beyond it. (I know I should balance the above with conservative examples—I think Chuck Lane at the Post often uses evidence effectively, also Jimmy P and others (Hassett, Strain) at AEI; also Doug Holtz-Eakin on a good day).

So what is a thinking person, who’s not a statistician, to do? Perhaps the answer comes from movie criticism. Before I had kids I used to really enjoy the movies (now I click on Netflix, immediately fall asleep, hopefully waking up in time to catch my laptop before it crashes to the floor). Figuring out which films to go see was vastly aided by knowing which critics to trust.

Same with evidence. I hesitate to start naming names or I’ll leave out some valued assets, but I already named some wonky types above. I’d add VOX, Upshot, Wonkbook, all of which I’d argue exist in no small part for this very reason—as go-to places for quality evidence (now that I’m writing for PostEverything, I’d shamelessly add them too). Dean Baker’s Beat the Press is must reading in this space.

In sum, let me rephrase the original point: careful evidence based on transparent data with respect for confidence intervals and variable coefficients is great. A lot of the rest is overrated.