I put this under the FAQ section to be a guide for folks to help them critically assess what they read about fitness. There are some contributors to this site who I sincerely hope will add to this thread because I know they have a great deal of expertise in doing this very thing. The excerpt below is from Christian Finn's newsletter:
One of the best places to get your hands on a goldmine of
fat-burning secrets and natural muscle-building techniques that
can save you years of wasted time and effort is in research
journals.
Most studies are available in the public domain - you can read
them for yourself if you dig hard enough. Though sifting through
the research may seem a daunting task, it doesn't have to be.
Here are several important questions to ask.
Was the study published in a peer-reviewed journal?
Not all studies are published in peer-reviewed journals. Those
that are carry more credibility because they undergo scrutiny
from a panel of experts.
The review process can last many months. During this time, the
author may have to revise their article based on feedback from
the reviewer. Some studies are rejected altogether.
In the area of exercise science, popular journals include
Medicine and Science in Sports and Exercise, the Journal of
Applied Physiology, the European Journal of Applied Physiology
and Acta Physiologica Scandinavica. The Journal of Strength
and Conditioning Research also publishes some useful research,
though the quality is not always as high as the other
journals mentioned.
Was the research funded by a company with a financial interest
in the outcome? Every study has to be paid for by somebody.
Just because a trial has been funded by a company with a
vested interest in the outcome doesn't mean you should ignore
it. However, be very cautious if the research is sponsored,
especially if drugs or supplements are involved.
According to an article in The Observer (December 7, 2003),
pharmaceutical giants hire ghostwriters to produce articles -
then put doctors' names on them! Many articles written by
so-called independent academics may have been penned by writers
working for agencies that receive huge sums from drug
companies to plug their products.
What was the group of the test subjects? Were they old or
young, trained or untrained?
Some strength-training studies involving sedentary or elderly
groups show large gains in strength - several hundred percent
in some cases. This may sound impressive. But it may have been
experienced by individuals whose strength in the leg press
improved from 10 pounds to 30 pounds. At 30 pounds, they're
still very weak when compared with younger people.
In a recent class-action lawsuit involving Cytodyne
(manufacturers of the ephedra-free diet product
Xenadrine-EFX), judge Ronald Styn of San Diego Superior Court
concluded that the advertising for Xenadrine RFA-1 - the
now discontinued ephedra-based diet supplement sold by
Cytodyne - excluded, misstated and overstated scientific
findings.
A typical claim, that Xenadrine users had a more than 3,000
percent greater fat loss, was based on a small company-financed
study showing that users lost 1.93 percent of their body fat,
while those who did not use it lost 0.05 percent.
According to Styn, such claims are a "distortion" intended to
mislead consumers. Cytodyne argue that its claims were
supported by the research, and that its advertising hyperbole
was simply the "puffery" commonly indulged in by other
manufacturers.
Was the study double-blinded with a placebo and control group?
Studies involving food supplements, sports drinks, energy bars
and other performance aids usually follow this format. A
placebo is a "fake" supplement used to reduce the influence of
faith and belief in a treatment on the results of a study.
Double-blinded means that neither the researcher nor the test
subject knows which supplement or drug they are getting. If
either person knows, it can have a big influence on the
results.
What are the confounding factors? In many studies, there are
confounding factors or variables which aren't always measured.
Put simply, these are hidden factors that vary between groups
and which scientists may attribute to other variables being
measured.
The link between diet and heart disease, for example, is
always a controversial one. Large studies may show an
association or link between various nutrients in the diet and
an increase or decrease in the risk of heart disease.
But association does not mean causation.
There is research, for example, to show that people who eat
breakfast are less likely to suffer from obesity or diabetes.
But is it eating breakfast that's solely responsible for this?
People who eat breakfast may be more likely to engage in
other healthy behaviors, such as eating more fruit and
vegetables. Maybe people who don't eat breakfast are less
likely to exercise, or more likely to consume a diet high in
sugar or trans-fatty acids.
Some studies show that people watching TV for more than four
hours each day are more likely to be obese than those watching
TV for less than one hour.
Does this mean that the TV is making you fat? Are broadcasters
secretly implanting subliminal messages in their programs
compelling you to eat more? Does your TV emit a "secret" form
of radiation developed in a top-secret government laboratory
designed to create more compliant and docile citizens by
turning them into couch potatoes?
If I were a conspiracy theorist, the answer would probably be
yes.
But it's far more likely that watching TV replaces physical
activity. And it's this drop in physical activity, rather than
the TV itself, which is responsible for the weight gain.
There are statistical adjustments you can apply to a study if a
confounder is measured. However, not all confounding variables
are measured, or even identified.
Tip # 3: Look for evidence rather than proof.
Finding answers to these questions does not guarantee an end to
the confusion. The "right" and "wrong" way to train with
weights, for example, is still a subject that continues to
confound consumers.
In January 2000, a study conducted at the University of
Florida at Gainesville raised eyebrows when it suggested that
one set of an exercise results in as much improvement as doing
three sets. In October of the same year, a Ball State
University study found that multiple sets work better!
This kind of conflicting evidence prompts many frustrated
people to throw up their arms in frustration at the fact that
"even the experts can't agree."
Of course, important variations in the way the trials were
conducted explain the different results. However, by the time
reports of the study reach your favorite newspaper or magazine,
these subtle differences are often left out or ignored.
One hot topic is whether overweight people can lead a healthy
life. Several prominent health experts suggest that it's
possible to be fit and fat.
Possible? Yes. Likely? Researchers can't agree.
People who are obese tend to be at greater risk of diabetes
and heart disease. Proponents of the fit-and-fat theory believe
that exercise reduces the risk, regardless of whether you lose
weight. After all, people who don't exercise, whether they're
overweight, thin or someone in between, are often vulnerable to
the same diseases.
Keep in mind that knowledge changes and evolves. Science is a
process, and there are always holes in what we know. A study
should be taken as evidence, not proof. No single set of
guidelines should be followed blindly.