The problem of Prior Probability, or why controlled trials of improbable propositions are a waste of time and money.

A common criticism aimed by supporters of alternative medicine modalities at those who are skeptical is that we are closed-minded and intolerant. "It can't hurt to study homeopathy, acupuncture, etc. We might find out that it actually works!" There are many studies that show that many types of CAM are at least mildly effective, but I would like to suggest that there are good reasons why a mild effect might show up in research, and why skeptics do not accept these types of results.

Some of the reasons a treatment may seem to work include the placebo effect as discussed previously, selection and observation bias, and unfortunately outright deception (this can be unintentional). This can result in studies being published that appear to show an effect. Often negative studies are not published at all (publication bias) as researchers and institutions often do not like to publish negative studies. Since the evidence-based medicine movement considers controlled trials the gold standard, reviews of CAM research can show potential mild effects or a need for further research that do not really exist. The prior probability of the modality may not be considered at all.

What is prior probability? It is simply defined as the scientific basis for something. CAM modalities such as Homeopathy, Reiki, Therapeutic Touch and other forms of "energy medicine" have extremely low prior probabilities because they either defy all known principles of physics, chemistry and biology (homeopathy) or are based on "energies" which no one can measure or demonstrate the existence of. Other types of CAM such as herbal medicine may have higher prior probabilities because they actually contain measurable amounts of chemicals that may or may not have an effect. This concept of prior probability is why skeptics say that extraordinary claims require extraoridnary evidence. An idea of how this applies can be gained from an understanding of Bayes theorem, which uses prior probability to evaluate the likelihood that a hypothesis may be true. Bayes theorem can be used in place of p-values (which can be misleading because they may not have a sound theoretical foundation). Kimball Atwood gave an excellent overview of these ideas at the Science-Based Medicine conferece jsut before TAM7 last week. His bibliography is here. This bibliography contains much more detailed discussions of these concepts than I have presented here.

A good understanding of basic statistics and of Bayes Theorem demonstrates why extraordinary evidence is required to prov extraordinary claims. When the plausibility and/or probability of a claim is exceedingly low, weak evidence is not sufficient to prove that it is true or even suficcient to justify spending further resources researching it.