Jumping To Conclusions

Jason Rylander is fat. So am I. Unfortunately, I can't claim that it's because of the Center for Disease Control and Prevention's new definition of "obesity," which ignores any distinction between muscle and flab. Mine really is all flab.

Still, it raises an important issue. The media is saturated with stories replete with numbers. Obesity is up X%. Teenage pregnancies are down Y%. Test scores are unchanged. Four out of five dentists recommend Trident sugarless gum for their patients who chew gum. Occasionally, we think about the implications of those statistics; more frequently, we let the pundits do it for us. But what we -- and the pundits -- virtually never do is ask what the numbers mean. What definitions were chosen? What methodology was used to gather the data?

Sometimes, the topic is trivial, as in this New York Times story which discusses the disputed methods of measuring movie box office data.

[W]eekend box-office figures released on Sunday and printed in many newspapers on Monday, including The New York Times, are based on actual box-office figures for Friday and Saturday plus each studio's guess about how its films will perform on Sunday. It is this wiggle room that has led many over the years to be overly optimistic about Sunday grosses in order to make their films No. 1 or to achieve some other goal.

[...]

Sunday guessing is not the only way that the system can be manipulated. Weekend box-office numbers also indicate the number of theaters in which a movie opens, but not the number of screens. A studio can release a movie on four or five screens at one multiplex and claim it as only one theater, raising the per-theater average for a film. Studios have often resisted, for this and other reasons, releasing the actual number of screens on which a movie opens.

But in other situations, the issue can be more serious. The supposedly rising obesity rate is leading to calls for public policy changes from all the usual suspects. (Coincidentally, all these policy changes will result in higher taxes and fewer freedoms for everyone.)

Economic policy, or at least punditry, is based on the Consumer Confidence Index. And yet, as the New Republic pointed out last year, the CCI is seriously flawed.

Although it's routinely described as a survey of 5,000 households, only about 3,500 generally return the form. The form essentially asks for a positive, negative, or neutral response to five questions about current and future business conditions.

So it's a poll.

Polls have their place, of course, but simply reporting that "x" percent of Americans surveyed feel "positive" about business conditions doesn't really seem like the kind of news that should be dominating business coverage and roiling the stock market. After all, polls from Harris and Gallup also address basic consumer confidence issues, and they never make the same splash that the confidence indexes do. Which goes to show that when you're trying to numberize a slippery idea like sentiment, an "index" trumps a "poll" every time.

How does the Conference Board convert its poll into an index? By combining the responses to its five questions and converting the resulting figure into a composite number "relative" to a benchmark score of 100.0 for 1985. (Why does 1985 equal 100 on this scale? Because it was "a basic, noneventful year," explains Lynn Franco, director of the Conference Board's Consumer Research Center, offering some insight into the formal science of consumer confidence.)

So you've got a survey. And yet, the number is treated as if it provides deep understanding about the state of the economy.

And how about the all-important Consumer Price Index, which measures the crucial inflation rate? Well, some of its flaws have been recognized and corrected in recent years, but there are still significant problems with both the construction of the statistic and the collection of the data.

In theory, the calculation of the index is simple. It is based on a marketbasket of 211 goods and services — medicine, education, entertainment and so on — bought by the average family. The prices are tracked over time.

But the task of calculation is daunting. As the bureau chooses among millions of products, it is constrained by budget limitations and saddled with old technology. Bureau agents roam stores, looking at price tags and writing prices on pieces of paper. They interview store executives, visit homeowners to determine housing prices and ask consumers to keep daily diaries of purchases.

All kinds of variables, including new products, mean that the bureau has a tough time keeping up.

These aren't esoteric concerns. They have real implications for all of us. The budget projections which drive taxing and spending in Washington rely upon statistics like these. Whether our taxes are cut (or hiked), whether interest rates will be reduced, whether social security will be reformed so that it can stay solvent longer -- these are all dependent on this sort of data. And that data is questionable.

And yet journalists generally treats this sort of data as holy writ. There's no acknowledgement that maybe everyone in the country didn't suddenly get fat. instead, the media jumps right to the question of "What should be done to solve this crisis?"