« Dear God, Make It Stop | Main | Cars of the Future »

September 16, 2010

Skepticism and the Scientific Method

Apropos of the post on skepticism (in which the avowed skeptic practiced skepticism towards others but was apparently untroubled by even the shadow of a doubt as to the self evident self evidentness of his own beliefs), this article explores the limits of science:

Good sense is the most fairly distributed commodity in the world, Descartes once quipped, because nobody thinks he needs any more of it than he already has. A neat illustration of the fact that gullibility seems to be a disease of other people was provided by Martin Gardner, a great American debunker of pseudoscience, who died this year. In the second edition of his “Fads and Fallacies in the Name of Science” (1957), Gardner reported that most of the irate letters he received in response to the first edition criticised only one of its 26 chapters and found the rest to be fine. Needless to say, readers disagreed about which chapter was the faulty one. Homeopaths objected to the treatment meted out to themselves, but thought that the exposé of chiropractors was spot on, and vice versa.

No group of believers has more reason to be sure of its own good sense than today’s professional scientists. There is, or should be, no mystery about why it is always more rational to believe in science than in anything else, because this is true merely by definition. What makes a method of enquiry count as scientific is not that it employs microscopes, rats, computers or people in stained white coats, but that it seeks to test itself at every turn. If a method is as rigorous and cautious as it can be, it counts as good science; if it isn’t, it doesn’t. Yet this fact sets a puzzle. If science is careful scepticism writ large, shouldn’t a scientific cast of mind require one to be sceptical of science itself?

There is no full-blown logical paradox here. If a claim is ambitious, people should indeed tread warily around it, even if it comes from scientists; it does not follow that they should be sceptical of the scientific method itself. But there is an awkward public-relations challenge for any champion of hard-nosed science. When scientists confront the deniers of evolution, or the devotees of homeopathic medicine, or people who believe that childhood vaccinations cause autism—all of whom are as demonstrably mistaken as anyone can be—they understandably fight shy of revealing just how riddled with error and misleading information the everyday business of science actually is. When you paint yourself as a defender of the truth, it helps to keep quiet about how often you are wrong.

Again (and for those who wondered what the point of my earlier post was, I was attempting to suggest that simply knowing what we ought to do doesn't mean we'll follow our own advice... especially when it conflicts with some deeply held belief or our own desire to think well of ourselves) I think the author is on the right track but he seems to be missing a critical piece of the puzzle: human nature.

As in, scientists are human beings first and scientists second.

The scientific method is designed to account for and correct the human tendency to bias but there's a huge flaw in the notion that scientists are any better than the rest of us at monitoring and correcting each other's mistakes: human nature.

Most laymen probably assume that the 350-year-old institution of “peer review”, which acts as a gatekeeper to publication in scientific journals, involves some attempt to check the articles that see the light of day. In fact they are rarely checked for accuracy, and, as a study for the Fraser Institute, a Canadian think-tank, reported last year, “the data and computational methods are so seldom disclosed that post-publication verification is equally rare.” Journals will usually consider only articles that present positive and striking results, and scientists need constantly to publish in order to keep their careers alive. So it is that, like the late comedian Danny Kaye, professional scientists sometimes get their exercise by jumping to conclusions. Historians of science call this bias the “file-drawer problem”: if a set of experiments produces a result contrary to what the team needs to find, it ends up filed away, and the world never finds out about it.

In a recent book, “Wrong: Why Experts Keep Failing Us—And How to Know When Not to Trust Them”, David Freedman, an American business and science journalist, does a sobering job of reviewing dozens of studies of ignorance, bias, error and outright fraud in recent academic science. He notes that discredited research is regularly cited in support of other research, even after it has been discredited. Trials of the safety and efficacy of drugs, which are often paid for by pharmaceutical companies, seem to be especially liable to errors of various sorts. That helps to explain why medicines that can do unexpected harm—such as Thalidomide, the sedative which was withdrawn in 1961 after causing deformities in babies, and Vioxx, a painkiller that had been used by 84m people before it was pulled in 2004—make it to the market.

This non-scientist would have a lot more confidence in the accuracy of science if she saw more scientists actually following the rules they claim place their work beyond dispute.

A little humility would go a long way, too.

Posted by Cassandra at September 16, 2010 08:36 AM

Trackback Pings

TrackBack URL for this entry:


I like this passage from William Langewiesche, in his book about the flight that landed successfully in the Hudson. Here, he is describing a conversation with the French engineer who led design of the Airbus flight-control system:

"Intelligence is not a prerequisite for safe flying, but an acceptance of human fallibility is, and the two are generally linked. Ziegler mentioned it on the banks of the Garonne. He has seen such variations over the years. He said that the mark of the great pilots he has known is that they admitted in advance to their capacity for error, and they addressed their mistakes vigorously after making them. He said, 'Vous savez, monsieur. L’Erreur est humaine.' Actually the Latin original, in full, goes 'Errare humanum est, sed perseverare diabolicum.' To err is human, but to persist is diabolical. Maybe it should be posted in polling stations. Certainly it should be posted in cockpits."

Maybe also in laboratories.

Posted by: david foster at September 16, 2010 10:37 AM

Historians of science call this bias the “file-drawer problem”: if a set of experiments produces a result contrary to what the team needs to find, it ends up filed away,....

This problem is easily obviated by following the (tongue-in-cheek) advice of a statistics professor of mine a while ago: "If your theory requires a straight line, collect two data points. If your theory requires a curve, collect three."

Unfortunately, several major "theories" today are being supported by taking that suggestion seriously.

Eric Hines

Posted by: E Hines at September 16, 2010 10:53 AM

...if a set of experiments produces a result contrary to what the team needs to find, it ends up filed away, and the world never finds out about it.

These days, it's easier just to torture the data until it confesses to whatever the team has decided the data should reveal.

Posted by: BillT at September 16, 2010 11:01 AM

I think, on the whole, scientists do a fairly good job of staying objective. But it's hard both to define objectivity and actually to be objective.

Mechanically, the peer review system is fatally flawed by the author choosing, for the most part, his reviewers, not publisher. I've heard one excuse for this being that the publisher isn't qualified to identify qualified reviewers. (!)

There's also the, often completely honestly (if ignorantly) arrived at, conclusion that I've answered the question (I know all the answers), so additional data, or more efficient descriptions of the world, need not be sought after. This was at the heart of the Papacy's dispute with Galileo, but it's ubiquitously extant in today's science, too. And in politics.

And the fundamental problem (not flaw) of science if Godel is right: the fundamental principles--the First Principles--can never be proved by the theory. Se we all just poke around the edges of the theory--doing engineering of the theory, not theoretical work with/about the theory.

And a lot of emotional investment in the theory gets built up--not to say funding continuation, position in the department, in the scientific community, etc. There's that human nature, again.

But we keep trying. We do get an occasional Galileo, Newton, Leibniz, Einstein, etc.

Eric Hines

Posted by: E Hines at September 16, 2010 11:08 AM

Global Warming. Now that was some seriously tortured data.

Posted by: Cricket at September 16, 2010 11:10 AM

This specifically is the problem with the AGW crowd. They set out with a goal in mind. Not a hypothesis, but a result they were going for (i.e. the earth is warming at a TERRIFYING rate and it's the fault of mankind). They collect and plot the data. There's nothing there to support their conclusion. The cut off the Medieval Warming Period and start their data with the Little Ice Age. Ok, now it's TRENDING hotter, but still not giving them anything to support their conclusion. So they "adjust the data" and suddenly you have Mann's "hockey stick graph". They publish. And when folks ask to see their data, they're told "no". At what point is this science? The answer is, at no point is it science. This is advocacy at its worst.

And yet, the Warmists still claim that it's "peer reviewed". But they won't say by whom, and they WILL NOT release the data. The scientific method is respected because it relies on the fact that your experiment is repeatable by ANYONE and yields the same (if not similar) results regardless of who performs the experiment.

Remember when that European team claimed they had achieved cold fusion with palladium? The world was in an uproar. Until they finally released the data on their experiment and... no one else could get anything even slightly like their reported "results". They were frauds. The difference between those frauds and the Warmists is that the cold fusion guys released their data and methods and could be proven to be liars in public. The AGW crowd will not release their data and methods. And the MSM is letting them hide it.

Posted by: MikeD at September 16, 2010 11:14 AM

Full disclosure: I have scientists and mathematicians in my family. They are openly scornful of religion because it's obvious to them that anything that cannot be scientifically or mathemagically proven is just a lot of superstitious bunk.

Of course I've pointed out several times that science has "proved" quite a few things that later turned out to be bunk and has no explanation for many other phenomena.

But their absolute and utter faith in science (and their own superiority) remains unshaken :p

Posted by: Cassandra at September 16, 2010 11:36 AM

And yet, the Warmists still claim that it's "peer reviewed". But they won't say by whom, and they WILL NOT release the data.

The data is essentially useless. The algorithm Mann used produces a "hockey stick" when *any* number -- even negative ones -- are plugged into the code.

And NASA/GISS has already "adjusted" historical temperature records and continues to "adjust" current data to prove that temperatures are still increasing -- even though CRU, their former ally, admits that hasn't been the case since 1998.

...it's obvious to them that anything that cannot be scientifically or mathemagically proven is just a lot of superstitious bunk.

My science prof in freshman year -- a physicist -- covered three blackboards with equations that mathematically proved that he didn't exist.

Posted by: BillT at September 16, 2010 12:58 PM

The scientific method is respected because it relies on the fact that your experiment is repeatable by ANYONE and yields the same (if not similar) results regardless of who performs the experiment.

There has never -- I repeat, never -- been any experiment which proves the CAGW (Church of Anthropogenic Global Warming) hypothesis that an increase in carbon dioxide in free convection will result in an increase in temperature. Increases in atmospheric carbon dioxide *follow* increases in temperature because, as the ocean warms, it releases some of the CO2 it holds in solution.

Posted by: BillT at September 16, 2010 01:10 PM

The contention that 'scientists' are unbiased acolytes of the Truth is little more than brand management and image marketing.

Posted by: ECM at September 16, 2010 01:30 PM

"Mathemagically," is that a slip, or is it something else? :)

I've been a working scientist for the past 25 years, and if I've found one thing to be true anamolous experimental findings often lead to profound new understanding.

As far as religion goes, I don't know many scientists who actually spend much time pondering it. In fact, many other scienetists who I know are religious and find no inherent contradiction.

Posted by: Allen at September 16, 2010 02:32 PM

The projections of CO2-based global warming cannot, of course, be proved experimentally, since we don't have a whole separate world to experiment with and a few hundred years to wait around for the results, but must depend on mathematical models. I wonder what the typical nonscientific AGW fanatic like Al Gore visualizes when he hears the term "mathematical model"....a hot chick with a PhD in topology?

Posted by: david foster at September 16, 2010 03:29 PM

They are openly scornful of religion because it's obvious to them that anything that cannot be scientifically or mathemagically proven is just a lot of superstitious bunk.

So they mathematically and scientifically proved how a battle would turn out?

That must have been an interesting "experiment".

Posted by: Ymarsakar at September 16, 2010 03:58 PM

E Hines-
I hate to say ANYTHING, since it's such an oddity to not see the same old same old rolled out about Galileo, but I got the impression the root of the dispute was that the G man was a lot like these scientists-- he decided he was right, even though he couldn't offer evidence, then he was rude about it. (Golly, someone who's really bright and yet lacks people skills-- NEVER seen that. ;^p)

I've been watching a similar trend in the "skeptical" community, but it's even more so-- seriously, you could start replacing words in some of the phrases and it would easily pass as something Huckabee or a born-again pastor would say. Kinda funny, really....

Posted by: Foxfier at September 16, 2010 04:48 PM

I believe the theory goes that if you can't, you just need more data.

Posted by: Foxfier at September 16, 2010 04:49 PM

No, no -- you need more *funding* so that you can collect additional data.


Data don't grow on trees, yanno -- unless you're a dendrochronologist...

Posted by: BillT at September 16, 2010 04:58 PM

People, people people! It's not Global Warming!! It's Global Climate Disruption. I know some of you are getting *old*..er, but you really must try and keep up....which, really shouldn't be all that difficult for ya since you're supposedly on that *downhill* run now.

*she says as she ducks incoming marmoset*

Posted by: DL Sly at September 17, 2010 08:06 AM

Skepticism and some humility are in order when it comes to mathematics and science.

"mathemagically" may well be an appropriate neologism describing the beliefs of some non-specialists, and regrettably some specialists as well.

What constitutes "proof" is not as cut and dried as some would have you think. Some accepted mathematical 'proofs' have later been found defectives, and much has been learned as a result. One of my math profs said, simply, that "a proof is an argument that convinces the audience at hand."

There are also some fundamental limitations to mathematical inquiry, embodied in results such as the "Halting Problem" and Godel's "incompleteness theorems". The first states that one cannot predict in general when an effective procedure for computing a given result will actually terminate. The latter essentially says that that in any mathematical construct as complicated as the counting numbers, there are true statements for which there are no proofs, where proof is a finite sequence of deductions from a set of hypotheses.

It is a healthy thing to acknowledge that we humans are finite beings in our earthly existence, and our investigations of reality consequently have limitations.

Posted by: Walking Horse at September 17, 2010 09:41 AM

...our investigations of reality consequently have limitations.

Those who believe their elegant computer models mirror actual reality should take those words to heart.

European airspace was closed for ten days after Eyjafjallajökull's April eruption based solely on a computer model of the density and spread of the ash plume, and the forecast was wildly different from the actual measurements.

Posted by: BillT at September 17, 2010 10:19 AM

That's because even given the vast capabilities our modern day computers have, they are still based, and depend completely, upon programs written by mere human beans -- warts and all.

Posted by: DL Sly at September 17, 2010 10:40 AM

Sometimes the warts show up in the programming.

The software guy trying to make sense of the HadCRUT code in order to correct it was f-bombing in his written notes by the third day...

Posted by: BillT at September 17, 2010 12:56 PM

Regarding MikeD's comment on cold fusion, several thousand positive replications of cold fusion have been reported in mainstream, peer-reviewed journals. I suggest you review this literature before commenting on this subject. See:


Posted by: Jed Rothwell at September 25, 2010 06:47 PM