“Statistically” Speaking

Most people with an ounce of cynicism to their credit tend to discount statistics when the claims seem over-the-top or try to prove too much, and there is a good reason for that.  The conventional wisdom is that about 70% of statistics are made up on the spot.  See what I did there?

Seriously though, it turns out that statistics are so misunderstood or misapplied by the scientific community, that even the scientific method is regularly corrupted by the search for statistical significance.  The problem lies in the arbitrariness of the P-value (i.e., the level of confidence that the results of testing were not just a fluke), which is often 5%, even though there is no particular rhyme or reason for choosing this particular degree of certainty.  And when there’s no rationale serving as a backstop for when some hypothesis turns into a datum, those who make their living constructing and assembling such data might have reason to pad their findings.

In 2007, for instance, researchers combing the medical literature found numerous studies linking a total of 85 genetic variants in 70 different genes to acute coronary syndrome, a cluster of heart problems. When the researchers compared genetic tests of 811 patients that had the syndrome with a group of 650 (matched for sex and age) that didn’t, only one of the suspect gene variants turned up substantially more often in those with the syndrome — a number to be expected by chance.

When a researcher has been working on some hypothesis for months and years only to find out that there may be no statistical link between two phenomena, and that research grant is up for review (or worse, a corporate sponsor who is relying on the marketability of the data you’re working on), there seem to be all sorts of human reasons to find that one’s research is significant.  That’s simply part of the human condition; the problem is that statistics are pliable enough to fit those exogenously determined conclusions.

Any solution on the actual front lines of research would seem to require a greater degree of deference and independence for conducting scientific research than a “free market” would probably determine.  In a free market, the value of scientific research (and, as a result, the level of investment) would vary depending on the eventual practical application and usefulness of the resulting findings.  While that makes perfect sense, there is a tragedy of the commons inherent in the scientific method, which is that numerous scientists must be called upon to produce unuseful or unmarketable data (i.e., the rejected null hypotheses) as a necessary antecedent to the really useful findings.  Such science and research will be (and is) neglected if the market is the only force that determines where investment is directed.  The ivory tower doesn’t look so bad when barbarians are at the gate.

Furthermore, apart from the mere statistical failings of certain scientific surveys, there’s the additional issue of communicating the actual data to the population at large.  Here is a case where brevity is dangerously reductive and not so charming, except insofar as one enjoys being mislead.  For example, the relatively unsurprising claim that “regular moderate female drinkers were less likely to become obese after a 13 year study of more than 19,000 women” gets turned into the precocious claim that “Wine Doesn’t Make Women Fat, Report Claims.”  The actual hypothesis could be explained by dozens of behavioral phenomena apart from wine’s interaction with biology or chemistry, such as the fact that regular moderate women drinkers may be more likely to be socially interactive and therefore concerned with outward body image, but the headline version fairly implies that the chemical pr0cesses implicated by drinking wine do not result in additional weight gain.  To have an eye-catching, copy-selling story, there are strong incentives to oversell the significance of the data from a journalist’s perspective as well.

Such problems in media communications doesn’t even incorporate the possibility of sheer lack of sophistication or stupidity, which is probably the more important causal factor in real life–of course, how would I know which factor is more significant?

As I’ve said before, the practical upshot of the lack of scientific literacy in America is that individuals who do have that savvy shouldn’t keep it to themselves; medical students and science majors should continue to influence and lobby those who don’t take the time to acquire the knowledge themselves.  Go out there and instill a scientific sense of skepticism in your friends and family by questioning their assumptions.  That grain or two of salt is good for them–no matter what Michael Bloomberg says.