A television commercial shows a group of young people, each trailed by an animated set of vertical bars, indicating the strength of the person’s cell phone signal. The ad reminds me of when I first left medicine and entered the business world. I could have used a similar device—error bars hovering above anyone pitching an investment or deal, signaling the probability that the person is telling the truth.
As anyone who voiced skepticism of the amyloid hypothesis—the only “acceptable” area of research into Alzheimer’s disease for many neuroscientists a few years ago—knows, science, and biomedical science in particular, is less than a beacon of absolute truth. Even so, the believability of information is still a great dividing line between the scientific and business worlds.
Scientific and medical data do reflect the biases and inaccurate assumptions of the researchers involved; not every p-value greater than 0.05 (a frequent cut-off for statistical significance in the life sciences) reflects absolute truth. But the evidence basis of science—the need for reproducibility of experiments and the consistency of units of measurement—accounts for relatively high confidence in data quality.
Not so in the business world: We rely on anecdotes, retrospective data and data mining in ways that would be considered unacceptable at a medical conference or scientific meeting. The resulting conclusions can be highly inaccurate, and the inaccuracies are then magnified by the corporate investor roadshows and sales calls from investment bankers using these “proofs of concept” as evidence of value creation.
Those naive enough to accept these data as fact make costly mistakes.
To some extent this phenomenon is inevitable; reproducible experiments under controlled conditions are possible in the laboratory, but the isolation and study of individual variables in the economy, stock market or product market is impossible. And while statistical methods can correct for some of the variability from one sampling to another, isolating only the most meaningful factors—comparing, say, the economies of 1929 and 2007, or making accurate assumptions about the real effects of a drug being developed by a resource-constrained small company based on a small phase II trial—are at best compromises in truth.
More importantly, the business world is a sales culture, filled with zero-sum interactions where each participant recognizes that his or her counterpart will try to exploit any possible edge to gain an advantage. The unsuspecting scientist who ventures into this environment quickly learns not to overestimate the reliability of the data that he or she encounters, even scientific data. While some of the inaccuracies are unintentional byproducts of a less disciplined process of peer review, many are misleading or even fraudulent, including an impressive separation of Kaplan-Meier curves (which show a treatment effect over time versus an alternate treatment or a placebo) that are grossly exaggerated by misleading of the y-axis or a video showing a mouse “cured” of its spasticity after an instantly fatal injection of an experimental drug.
Most businessmen and women are honest and honorable (and not all scientists are). Even so, a good dose of scientific rigor and discipline is a better tool for diligence than hovering animated error bars.
