Saturday, July 17, 2010

Doggerel #219: "I Did My Own Research!"

Welcome back to "Doggerel," where I ramble on about words and phrases that are misused, abused, or just plain meaningless.

The information age allows people incredible access to scientific data and exchange of information. Unfortunately, it also allows incredible access to urban legends and crazed rants that would normally be relegated to badly copied manifestos handed out on the street.

What separates real research from from what the typical woo does is discrimination: You have to know what forms of evidence are more reliable than others. A double-blind control study is superior to a case study, which is superior to an anecdote. A video recording is superior to human memory. Quality also matters more than quantity, most of the time. A thousand eyewitnesses are less objective than a handful of cameras.

Woo "research" typically involves collecting anecdotes, a form of cherry-picking. In alternative medicine, for example, positive outcomes for an alleged remedy are popularized, but, because of the culture, negative outcomes are generally ignored, usually leading to the patient to quietly change remedies. The result is a large collection of isolated positive anecdotes divorced from the larger context of the world.

Real research has to look at large numbers. With small groups, and worse, individual test subjects, there is little ability to rule out coincidences. Scientific studies require large numbers of subjects or trials because large numbers make coincidence less likely of an explanation. Most people simply have trouble thinking about the large numbers involved in the world, and how millions of people experiencing the world over many years can produce all sorts of "unlikely" events. We use the scientific method because of such shortcomings in our thinking.

Another form of false research comes in the form of blindly trusting people with fancy degrees or even Nobel Prizes. Being a scientist, earning a degree, or being awarded a prestigious prize is not a certification of papal infallibility. Scientists are people. People can make mistakes. Working in a manner that minimizes the chance of mistakes is what makes your conclusions more likely to be accurate. It doesn't matter who you are. High quality work is high quality work. Knowing how to distinguish real science from anecdote and hearsay is what separates a researcher from just another student of Google University.

7 comments:

Rhoadan said...

Quality also matters more than quality, most of the time.

I'm guessing that one of those "quality"s, probably the second one, should be a "quantity."

Bronze Dog said...

Oops, editinating.

Valhar2000 said...

You silly geeks who live in their mothers' basements and have not travelled the world! IF you had, you would know that all these scientific mumbo-jumbo is unnecessary! Real men just know shit!

Dark Jaguar said...

Been a while since I'd seen an update here. Nice, "did my own research" is an annoying one I've been hearing more often lately.

Yakaru said...

Well doing my own research works for me! (/sarcasm)

Don said...

A buddy of mine is dealing with some drama right now with his ex taking their 18 month old daughter to a chiropractor. He tried politely explaining the scientific evidence to her, referencing experts and peer-reviewed research, and she countered with her own "research": she's talked to other mommies who take their kids to have their bones cracked, and she talked to the chiropractor himself.

Yup, that's some solid research right there.

djfav said...

Perhaps she can be plied with the humor of Eddie Izzard.

Toothache? CRACK YOUR BONES!

Headache? CRACK YOUR BONES!

Stomachache? CRACK YOUR BONES!

Common cold? CRACK YOUR FUCKING BONES!