Welcome back to "Doggerel," where I ramble on about words and phrases that are misused, abused, or just plain meaningless.
I'm often asked to try some form of woo, usually quackery, before I knock it, as if my personal experience will invalidate the absence of controlled studies or even the presence of several negative studies.
The big, inherent problem with such a thing is that I am biased. So are you. There's no getting around that... or is there?
The answer to that is controlled studies, preferably double-blinded, especially if subjective measurements are done. Granted, not everything can be double-blinded, but for most forms of such woo, it's usually not a problem.
For the topic of quackery: here's a quick un-nuanced explanation of how a double-blind clinical trial works:
First, get a large group of people. Randomly divide them into two groups: The control group (also known as the placebo group) and the experimental group. Both groups are treated the same, except in one aspect: Some get the treatment, and others get a fake treatment (sugar pill, whatever). No one involved in the process knows who's getting which until the results are measured. If there's a big enough difference (you can ask someone else how to determine statistical significance), then the treatment is probably having an effect.
Here's how it works out mathematically:
Control group's improvement = Coincidental recovery + placebo effect + 0
Experimental group's improvement = Coincidental recovery + placebo effect + Treatment effect.
If both groups get about the same amount of improvement, I think we can safely conclude the treatment's effect was zero.
That's fairly simple. Now onto things like psychics, astrology and so forth: Double-blinding can be a little trickier with some of these, but the end goal is the same: Information has to be tightly controlled.
An example: A medium claims to be able to perform a reading based on a photograph, and can tell whether or not the photographed person is alive or dead. Have a person without knowledge of the photos hand over a pile. Since he doesn't know, he can't accidentally give any hints. If the medium can correctly guess better than chance, that's positive evidence.
That sort of thing gets trickier (but often still manageable) if the medium requires contact with someone who knows the deceased. In human contact, there's always the risk of large information leakage. If double-blinding isn't possible, why should I believe that the information came from somewhere else if I may have provided it?
Additionally, such "try it yourself" pleas often call on me to rely on one data point. One. In such small sample sizes, the laws of probability can seem more dramatic. If the psychic or whatever makes one particularly lucky guess, it can seem really special since it's usually divorced from context: The psychic has probably made a large number of mistakes with other people. You have to look for the big picture. I'm not about to presume that everything is going to be typical with me.
That's why, even if I had a successful personal trial, I wouldn't be convinced: There's no way to be confident that I'd be a typical case. Large trials designed to eliminate bias are much less subject to human foibles and probabilistic snags.