Category: Signs of phony skepticism

Anomalies and Falsification in Science

Bluestone_the_Great_unmaskedIn science, an anomaly refers a new observation or experimental result that seems to fly in the face of conventional, well established scientific wisdom.

Anomalies are an important part of the scientific discovery process. Sometimes a particular anomaly or the collection of anomalies can precipitate a revolution in scientific thinking – dethroning an old way of thinking and bringing in a new one. A canonical example of this is the Michaelson Morley Experiment, which started the ball rolling towards the Theory of Relativity.

Anomalies are important, but they are also frequently misunderstood and frequently misused.

Peddlers of pseudo-science love anomalies. Anomalies are the raison d’etre of crank scientists the world around. It gives them a justification for their “new paradigm”. When asked what the scientific basis is for their claims they answer “because anomalies”. And there is an easy bait and switch going on:

“These anomalies falsify the scientific orthodoxy! So you might as well buy my theory instead, <wink> <wink>.”

The whole thing boils down to a confusion about falsifiability in science. Falsifiability is one of the key hallmarks of science. Philosopher Karl Popper defined science by the very idea that it makes claims that are testable and can therefore be proven wrong (falsified) by contradicting observations. Science, in the Popplerian sense, is always tentative. But, Richard Feynman says it better than I can:

(His whole segment on the scientific method can be found here. it’s great.)

Scientists take falsifiability very seriously. We take anomalies very seriously. But, the relationship between the two is where some people get confused. A good number of people are under the impression that theories are falsified with a dramatic moment of unmasking, as in “Scooby Doo, et al” (pictured above). But, science rarely happens this way. It is not so simple.

What we often refer to as a theory in science is not a singular thing. A theory is a collection of many ideas and explanations that systematically explain observations. Perhaps any one observation may be falsified, but it’s hard to falsify an entire theory in one fell swoop. In fact, if we were to throw away conventional scientific wisdom at the first sight of any anomalies, science would fundamentally lack stability.

A good scientist has the discipline to abandon an idea when it’s wrong. But, equally important is the discipline not to immediately give up on an otherwise good theory. A well-established or mature scientific framework is built on decades of observations, empirically established first-principles, and countless successful predictions. To dismiss mature science on the basis of a few anomalies would be premature, and in many cases wrong headed.

Scientists are constantly looking for new ways to push the envelope. We like anomalies because they’re way more interesting than confirming what we already know. We seek them out. We develop new instrumentation to give ourselves new sensitivity. We design new experiments to look in places we never looked before. And, the thing about venturing to new places is: you end up with a lot of false starts – mistakes, misunderstandings, experimental artifacts.

Anomalies are thus inevitable. So the right attitude is “trust but verify”, bearing in mind that verification takes time and work.

Sometimes anomalies completely dethrone the current paradigm. Far more often….they. just. don’t.

Here are some of the common fates of anomalies, from most likely to least:

1) It turns out to be an experimental artifact, a mistake. In my own field, the observation of neutrinos traveling faster than light turned out to be a consequence of a loose cable.

2) It is a real effect, but incomplete. A missing piece of the the observation, once found, restores consistency with the theory. Feynman gives a good example of this – superconductivity. At first, the discovery of superconductivity seemed to completely contradict the known understanding of atomic physics. Eventually it was realized that a very subtle quantum mechanical phenomena explained the effect. Once this is taken into account, atomic theory is again fully consistent.

3) It represents a real problem with the theory, but the essence of the theory survives with some modification (ranging anywhere from minor to major).

4) By itself or in combination with other anomalies, it dethrones the prevailing wisdom

If you’re a person who rejects broadly accepted science, you’re inclined to see any anomaly as your own Scooby Doo ending! It’s the moment you’ve been waiting for, when orthodoxy collapses and you are vindicated! It is also too easy to use these anomalies to draw inexperienced skeptics away from established science and into the rabbit hole of quack science. Which brings me to one last essential point:

Old theories are almost never dethroned by anomalies, UNTIL there is a superior alternative that explains both the old observations and the new anomalies. The really frustrating part of the pseudo-scientific bait-and-switch is the innuendo:

“Well if the mainstream science is wrong, then anyone’s guess is as good as anyone else’s. So, you might as well choose mine”

This is simply not true. In the face of deep rooted anomalies, one would be naive to blindly hold on to the established model. On the other hand, it is naive and unskeptical to settle for the first new theory to pass your way. The old paradigm survived a serious shaking and any new theory should do the same…and then some. Only hindsight is 20×20, and few people in history can claim to have successfully “picked winners” early in a scientific crisis.

So when people without any scientific credentials drive up in tinted vans, offering you the sweet indulgence of “anomalies”, just say NO and run to someone you trust :).

But seriously: I may sound like a defender of the orthodoxy. My colleagues and I take anomalies very seriously. They are the constant talk of the lunch table. The faster-than-light neutrino measurement, ridiculous as it seemed, prompted a serious and genuine discussion of theories that could violate relativity. A lot of work went in to understanding the measurement. The measurement was redone by two other experiments and neutrinos went back to being slower-than-light. And, in it’s own Scooby Doo style ending, the culprit turned out to be a loose wire.

An important, if not THE important, value of science is the willingness to revise or replace wrong ideas. But, as with anything, this is not a carte blanch principle. When anomalies appear, we should welcome them but we must also exercise care and patience.


Coherence between many lines of evidence: Part II

In the last post we discussed one of the key hallmarks of good scientific methodology: reproducible results across many different and complementary measurement techniques. In this post, we will discuss the ways in which phony scientific skeptics can spin these coherent results in order to make a particular measurement or conclusion look weak.

100 methods, 100 problems (Argument by Anecdote)

The more different methods corroborate a result, the more confidence we can place on that result: especially if the strengths of each different method can cover for each other’s weaknesses. Ironically, in the eye of a phony skeptic: the more techniques there are, the easier it is to make a given finding look weak. His equation is simple: more methods = more weaknesses.

No measurement technique is perfect.  For a finding supported by many different lines of evidence, a phony skeptic can produce a laundry list of the weaknesses of each method without ever putting them together to form a bigger, coherent picture. If a hundred different methods show the same conclusion, he will ignore or gloss-over the common conclusion and simply discuss all of the particular failings with each technique, taken by itself. This sort of laundry list is what I call “argument by anecdote”. And, for such a faux critic: numbers are all that matter. The more doubts he can throw your way, the more likely he hopes one of them will stick.

The thing that is inevitably missing is context. And, this is the distinguishing characteristic of true scientific skepticism. When a scientist tries to make a judgement on the strength of scientific evidence, context is her top priority. Those who wish to contradict science tend to value lists over context.

Asking Rhetorical Questions

Nothing is more frustrating to a scientist than an insincere rhetorical question. We work hard at what we do. We love tough questions, but we want them to be asked out of genuine curiosity. Unfortunately, rhetorical questions are an effective technique for casting doubt on scientific findings.

A common approach is the use of rhetorical questions to imply oversight on the part of scientists. People ask questions with a tone as if the scientific community did not even think to ask said questions. More often than not, these questions have been both asked and answered throughout the scientific literature. But, the kind of person who asks these questions typically has no real interest in finding out. Better to leave the question open-ended. Better to rely on innuendo.

My doctoral dissertation was based on research that I spent 6 years working on (along with 10 other colleagues). I will talk more about it in future posts. Suffice to say, I spent all of every day thinking about, quantifying, and testing all of the ways my measurement could go wrong. Way more time was spent on the error analysis than the measurement itself. This is typical in my field. It is certainly true that sometimes obvious questions are overlooked even by the experts. But, more often than not, if a question about a particular scientific finding seems pretty important, odds are high that the people who did the work already thought about it. I don’t expect or even want you to take that on faith. Next time someone asks an open-ended and accusatory question about a scientific finding, check for yourself. Email an expert. Search the literature. You will see that the people doing the research are thoughtful, thorough, and skeptical (it’s what we’re hired to do) and you will see that the people asking the accusatory questions often have a transparent agenda.

Purposely testing a technique under circumstances where you know it won’t work.

In the previous post I presented the example of a bathroom scale that works perfectly well in earth’s gravity. Given the assumption of earth gravity, the scale would give incorrect results if it were used on the moon. Now imagine someone purposely used the scale on the moon in order to discredit the reliability of the scale. Pretty dishonest, huh? Yet, this sort of approach is common place among those who have an axe to grind with certain scientific ideas.

Take the case of carbon dating: Carbon dating is based on the premise that a living organism incorporates C-14 from the atmosphere into is body (through plants breathing CO2 and animals eating the plants, for example). When that animal dies, the C14 starts to decay. How much of it has already decayed tells us how long ago it died. There are two significant preconditions for carbon dating to work: (1) It needs to have carbon and (2) it needs to have been primarily in direct contact with the atmosphere. Aquatic life, for example, is in contact with “old carbon” stored deep in sea. Thus, it is known to produce anomalous carbon dating results. Many people who spin doubt regarding radiometric dating will come up with countless lists of stories wherein samples-that-shouldn’t-be-carbon-dated for known reasons are dated and give nonsense results. This approach is a combination of my argument by anecdote (lists and doubts) and the asking of rhetorical questions (since the person listing these anecdotes does not mention that these samples were of a pedigree known to fail carbon dating).

In this article we focused on some aspects of false skepticism. As promised in my introduction to this blog, I hope to also provide a rubric for how to exercise legitimate scientific skepticism. Looking forward to your thoughts and comments!