Welcome back to "Doggerel," where I ramble on about words and phrases that are misused, abused, or just plain meaningless.I'm often asked to try some form of woo, usually quackery, before I knock it, as if my personal experience will invalidate the absence of controlled studies or even the presence of several negative studies.
The big, inherent problem with such a thing is that I am biased. So are you. There's no getting around that... or is there?
The answer to that is controlled studies, preferably double-blinded, especially if subjective measurements are done. Granted, not everything can be double-blinded, but for most forms of such woo, it's usually not a problem.
For the topic of quackery: here's a quick un-nuanced explanation of how a double-blind clinical trial works:
First, get a large group of people. Randomly divide them into two groups: The control group (also known as the placebo group) and the experimental group. Both groups are treated the same, except in one aspect: Some get the treatment, and others get a fake treatment (sugar pill, whatever). No one involved in the process knows who's getting which until the results are measured. If there's a big enough difference (you can ask someone else how to determine statistical significance), then the treatment is probably having an effect.
Here's how it works out mathematically:
Control group's improvement = Coincidental recovery + placebo effect + 0
Experimental group's improvement = Coincidental recovery + placebo effect + Treatment effect.
If both groups get about the same amount of improvement, I think we can safely conclude the treatment's effect was zero.
That's fairly simple. Now onto things like psychics, astrology and so forth: Double-blinding can be a little trickier with some of these, but the end goal is the same: Information has to be tightly controlled.
An example: A medium claims to be able to perform a reading based on a photograph, and can tell whether or not the photographed person is alive or dead. Have a person without knowledge of the photos hand over a pile. Since he doesn't know, he can't accidentally give any hints. If the medium can correctly guess better than chance, that's positive evidence.
That sort of thing gets trickier (but often still manageable) if the medium requires contact with someone who knows the deceased. In human contact, there's always the risk of large information leakage. If double-blinding isn't possible, why should I believe that the information came from somewhere else if I may have provided it?
Additionally, such "try it yourself" pleas often call on me to rely on one data point. One. In such small sample sizes, the laws of probability can seem more dramatic. If the psychic or whatever makes one particularly lucky guess, it can seem really special since it's usually divorced from context: The psychic has probably made a large number of mistakes with other people. You have to look for the big picture. I'm not about to presume that everything is going to be typical with me.
That's why, even if I had a successful personal trial, I wouldn't be convinced: There's no way to be confident that I'd be a typical case. Large trials designed to eliminate bias are much less subject to human foibles and probabilistic snags.
---
Doggerel Index
5 comments:
Interesting post. Especially in light of the fact that one of the things I've been using as "proof" of God in my own life is my personal experiences.
But considering that those experiences keep happening, I think there's still good reason for me to rely on those experiences.
Did that make sense at all?
I understand what you're saying, Amanda. An isolated event is next to meaningless, but a string of events is better. It's still not good, and won't pass as science, as there's one big other confounder: Confirmation Bias.
Humans naturally notice data that confirms their hypothesis a lot more than data that disconfirms it. Additionally, ambiguous data is often interpreted in a favorable manner.
Example: You pray for things every night. What you pray for varies in parts, while other parts are pretty constant. Most of the time, you probably don't see anything you pray for suddenly happen. But on occasions, you likely do. These occasions will then likely stand out in your mind as evidence that prayer works.
As time goes on, your list of examples of answered prayers grows. But what you're forgetting about is all of the unanswered prayers. Statistically speaking, it's not the number of answered prayers that will determine whether praying is valuable, but the frequency with which prayers are answered. To make a scientific conclusion, you'd have to keep track of both numbers, then compare this to chance expectations. If it's significantly higher than chance, you have evidence for the efficacy of prayer. If not, you have evidence against it.
Now, what about events that seem "miraculous"? That is, events that have quite a low probability of occuring yet do actually occur. How many "events" occur in your average day? Depending on how you stretch the definition, likely hundreds. With, let's say, 100 events every day, the chance that one of them will have a 1% or less chance of occuring is quite good. In a given year, you'll likely see an event with a 0.01% chance. In the course of a year, some human on earth is likely to experience an event with a probability of less than 1 in 10^14!
And that's where confirmation bias kicks in again. You see some ridiculously low-probability event, and it's up to you how you interpret it. A Christian might say it's because God did it. Me, I'd say it's just the inevitable result of having a ton of stuff happen that some of it will seem unlikely.
But the problem is when this is turned into evidence for the explanation. There's no way in which you can say God is a more plausible explanation for many events than Allah or Zeus or a Flying Spaghetti Monster is. In the end, all you really have evidence of is that rare events do occasionally happen.
So, what could you do to figure this stuff out for sure? As Bronze Dog said, a double-blinded, placebo-controlled experiment is always the best bet (and I'll point out that such studies on prayer have always shown no benefit to it). Aside from that, maybe you could pray for something that would completely beat any odds. For instance, take out a glass of water and pray for God to turn it into wine to prove his existence to you. If it does, that one instance incontrivertible proof that God exists and answers prayers, as the odds of it happening otherwise are so ridiculously low as to be nonexistant (hallucinations and quantum effects do give a possiblity, but it's negligible).
and I'll point out that such studies on prayer have always shown no benefit to it
And some recent ones have actually shown a detrimental effect (more than likely a coincidence or some self inflicted anticipatory mental and physical let down because they were expecting prayer to work for them).
Actually, when a study finds no statistically significant difference from control outcomes, we can't necessarily say that there is no difference. There might be some difference smaller than the "noise" of random variations, but we can't really tell. The wikipedia article http://en.wikipedia.org/wiki/Statistical_significance tells us that "Actually, statistics cannot be used to prove that there is exactly zero difference between two populations. Failing to find evidence that there is a difference does not constitute evidence that there is no difference."
That said, when analysis of experimental data reveals no statistically significant difference, any tiny difference observed may just as well have been due to random variation as any meaningful effect. Therefore, I'm not going to be impressed by any supposed treatment that can't provide convincing experimental results. Even some therapies that are supposed by evidence aren't always effective, so something with no clear evidence that it works at all would seem quite a poor candidate on which to spend time and money.
Regarding unlikely events, even before confirmation bias, people aren't always good at estimating how likely an event is anyway. They may fail to take account of reasonable causes for apparently unlikely events and, as noted, may fail to define the events they're considering properly. You've pointed out a couple, but there's also the observation that (assuming a 365-day year) a group with random, uniformly distributed birth dates is more likely than not to have two the same once it has at least 22 persons, even though naively we might suspect that we'd need to be much closer to 365 persons to get such a result.
Do note that statistics isn't my day job and I just did a quick computation for the birthdays, so either of my statements could be in error. If the probability computation is wrong, though, that just reinforces my claim that we have a hard time figuring out probabilities.
Well done, and many many many more people should know these things. My only niggle is that it should have more explanation on what "double blind" specifically means (what are the two things being blinded).
Post a Comment