By Jose Antonio PhD FNSCA FISSN.
A pithy summary
- The typical ‘Hierarchy of Evidence’ is wrong.
- The gold standard in science will always be the randomized controlled trial or RCT (i.e. the original investigation).
- If you truly want to change a field, you have to conduct the RCT.
- Reviews and meta-analyses exist only because scientists have conducted prior RCTs.
- Reviews and meta-analyses do not provide new (i.e. original data) information.
- If you’re lazy, read reviews and meta-analyses.
- If you’re ambitious, make sure you read RCTs.
- If you profess to be a scientist, then there better be RCTs in your CV.
- If you’re a Monday morning quarterback, I can’t help you there.
- If your mommy didn’t hug you enough as a young lad, so sad.
I’ve seen this pic about as often as I’ve seen sand on a beach. Funny how social media works. One person puts up this pic and before you know, you’ve got your loyal band of merry followers parroting the same old silliness. The speed in which these pics are posted is inversely proportional to amount of thought given to the veracity of the pic. This evidence hierarchy pyramid is at best misleading and at its worst, just downright wrong. Have patience grasshopper; I’ll explain why. First some background. Variations of this pyramid exist with Cochrane Reviews sometimes at the top (i.e., it’s still a Review no matter what adjective you place in front of it). Either way, what is typically grouped at the top of the evidence pyramid are Reviews and Meta-Analyses. Ostensibly that represents the crème de la crème of science. Below that we find Randomized Controlled Trials or RCTs, Cohort/Case Control studies, Case Studies/Reports, Animal Studies, In Vitro data etc.
In the category of Sports Science, this pyramid makes no sense. Why? Because the way to make changes in a field isn’t by writing umpteen review articles and/or meta-analyses. How do you make changes in a field?
If you answered, “you have to generate original data” then go to the head of the class. Original data. That’s right. You have to do the RCT. Or do a cohort/case control study. Though if you’re studying human performance, a cohort study is about as useful as chopsticks in a hot dog eating contest. Even though case studies get the shaft much of time, these too are important. Particularly if case studies involve what you’ve done with hundreds of athletes. Strength coaches who have worked at the collegiate or professional level with perhaps thousands of athletes must be doing something right. Nonetheless, the gold standard is the RCT.
Review papers and meta-analyses are nice and all; I’ve even written a bunch. Heck, I recommend to my students that to get an initial bird’s eye view of a topic, reading review papers and meta-analyses is a great way to start. But the key word is ‘start.’ If all you relied on were reviews, that would be like watching the finish line of the Boston Marathon. You’d have no clue as to what transpired during the first mile, mile 13, or Heartbreak Hill (mile 21). You’d have no clue as to how these runners trained for the marathon. All you’d know is how they finished. That’s a typical review. You get the finish line.
Meta-analyses are even more problematic. You’re basically ‘combining’ studies that may be as disparate as hot dogs and cotton candy. The hope is that you can narrow down a bunch of RCTs into a single number and conclusion. That would be like describing the United States by doing a meta-analysis of the 50 states. It would go something like this:
The USA is 78% white, 13% black and 17% Hispanic with a small minority of Asians, Pacific Islanders etc. The median income is $53,000, the percentage of college grads about 29% and number of those living below poverty about 15%. Get my drift? I won’t list every possible way to 
describe the USA but let’s just say you can’t ‘meta-analyze it’ in a fashion that makes any sense. Clearly, there are MANY ways to ‘explain’ the USA. There are statistics galore you can use to describe the USA to someone who has never been there. But in the end, it’s a bit like grabbing water. Does anyone actually think living in Oklahoma is anything like Northern California? Is Texas anything like Massachusetts? Is the Sunshine State anything like Iowa? Not only is the racial make-up entirely different between these states, but geography, political leanings, and average income differ. Heck, you could probably divide the USA into 6 different countries. In the state of Florida alone, the Miami area is virtually a different country compared to north Florida. It makes you wonder if you’re in the same state. But I do love South Florida. 
So next time you think a meta-analysis is the best arbiter of science, try describing the United States using a meta-analysis. Good luck. (Note: I am a co-author on a caffeine meta-analysis as well as several review papers; so I’m not ‘bashing’ these types of publications for the sake it. I’m trying to get you to think beyond the obvious).
Getting back to RCTs. If you want to truly change a field, you need to do the RCT. RCTs are the bread and butter of science. Without RCTs, there are no reviews or meta-analyses. The field of sports science in particular needs more RCTs. Reviews and meta-analyses exist at the behest of RCTs. So why on Earth would anyone think RCTs are less important than reviews/meta-analyses?
Androgens – Roid Rage Nonsense
Back in the 1970s and 80s, the American College of Sports Medicine (ACSM) published their Position Stands on Anabolic-Androgenic Steroids. [1] Their original Position in 1977 basically stated that anabolic steroids don’t work and are dangerous. Their revised position in 1987 
stated they might work but are still dangerous. The funny part about the 1987 revision was they changed their conclusions based on no new data. Yes you read that correctly. Nothing changed in terms of original data. That left my head scratching. Moving on. When I examined the literature way back when (and I read EVERY original paper on this subject), I had a sneaking suspicion that the ACSM was flat-out wrong on both counts. In 1996, I co-authored a review paper stating that “the use of moderate doses of androgens results in side effects that are largely benign and reversible. It is our contention that the incidence of serious health problems associated with the use of androgens by athletes has been overstated.”[2] My review was nice and all, but it really won’t change anyone’s mind. It won’t change a field. So what did change the prevailing view of androgens? If you answered an RCT, you get an “A.”
Dr. Bhasin published the classic paper on androgens in the New England Journal of Medicine.[3] His research team found that “supraphysiologic doses of testosterone, especially when combined with strength training, increase fat-free mass and muscle size and strength in normal men.” So anabolic steroids work and the side effects were…oh wait…they didn’t find any. And they found no changes in mood or behavior. Yep. The myth of ‘roid rage busted. This RCT was the tipping point in the study of androgens.
Creatine – The Tipping Point
Dr. Roger Harris published the seminal paper on creatine.[4] That original investigation and the subsequent economic boom that resulted from the sales of creatine 
monohydrate has done more to change the sports nutrition industry (business and science) than any single supplement. Creatine gave respect to the supplement industry that otherwise was known more for selling protein powder that tasted like dirty socks soaked in sour milk. The study of creatine has generated 100s of RCTs and has probably been the topic of study for many a Masters and PhD student.
Caffeine – thank Dr. David Costill for this one
The 1978 study published in MSSE gave Dr. Costill more notoriety 
than his entire body of work (which was a LOT!). He showed that caffeinated coffee improved cycling performance compared to the placebo. Coffee and caffeine lovers rejoiced. Again, it was an RCT that changed the field of caffeine/exercise. There are literally hundreds of studies on caffeine and exercise. You can thank Dr. Costill for that.
The Big Picture
Having written many reviews, they do have their value. Often times reviews, and in particular the ISSN Position Stands, can give you a snapshot of the current science as we know it. Most of us don’t have time to sift through the umpteen studies on a particular subject. Hence, reviews (and meta-analyses) have an important role. Like I said before; it’s a good place to start. In fact, a great review or meta-analysis might generate the questions that will spur scientist(s) into conducting the appropriate RCT. Perhaps the RCT/Cohort/Case Study and the Review/Meta-Analysis categories are like two wheels of a bicycle. Both wheels are important. However, you can’t have the latter without the former.
Thus, don’t delude yourself into thinking that reviews/meta-analyses represent the most important pieces of evidence. They are not. They exist only because hundreds of scientists have done the hard work of performing RCTs. Heck, anyone can write a review. Just sit in your underwear, PubMed search whatever topic you like, and write. Believe me. I done my fair share of underwear-sitting and writing. I kinda like it actually. 
Performing an original investigation is the bread and butter of science. Original data gives a field credibility. Imagine if you were to magically delete EVERY study ever done in the world of sports nutrition and strength and conditioning. What would you have left? A bunch of angry trolls fighting online. “One set is better than three sets.” “No 10 sets is better than 3 sets.” “Squeezing my butt is best for the glutes.” “No way Jose. Squats are the best butt exercise.” “Low carb is best.” “No high carb is best.” The endless drivel would be enough to shut down Facebook. And the list goes on and on. How should these disputes be resolved. If you answered ‘science’ than go to the head of the class. Science, particularly original investigations (i.e., RCTs), are the only way to objectively resolve disputes. So unless you’ve actually conducted a study, you’ll honestly have no idea what it’s like. Doing research is a brutal teacher.
I love it when an original investigation is posted on social media. The keyboard warriors, Monday-morning-quarterbacks and University of Google “PhDs” come out in full force with their critiques of the study. “Why is the sample size so small?” “Why’d you use male subjects only and not female?” “This study is invalid because it is sponsored by a private company.” Folks who critique scientific studies (i.e., those whose experience with science is reading Discover 
magazine and playing “Operation”) are often criticized for their lack of scientific expertise. Hence, they often resort to the intellectually lazy argument of “that’s an ad hominem attack.” Judge the message, not the messenger. Sure in the world of frickin’ Utopia, we have endless hours in the day to vet everything said by everyone. But alas, I don’t live in that world. In the interest of being pragmatic (and so as to not infringe on my beach time), judgements have to be made on the messenger. And guess what. I’ll take the advice of my internal medicine doctor over some dipshit who memorizes the WebMD website. Oops. Is that an ad hominem attack? Ask me if I care. Yes it does matter WHO gives advice. Just because you have fingers, a laptop and can type search words in Google, Wikipedia or any other third party site does not make you an expert. (For an interesting piece on ‘expertise,’ read this: http://thefederalist.com/2014/01/17/the-death-of-expertise/ ).
Trying to explain the intricacies of conducting a study would be like dry-land swimming. Sure. I can explain the stroke. I can show you videos of swimming. I’ll even let you look at the water. But unless you jump in the water, you really will have zero idea as to the process of swimming (i.e., research). Unfortunately the ratio of scientist to Monday-morning-quarterback ain’t good. It’s something like for every one genuine scientist, there are probably enough Monday-morning-QBs to fill AT & T Stadium in Dallas Texas.
Summary
I’ve published in the categories of RCTs, Reviews, Meta-Analyses, Case Studies and Animal Research. They are all important though in different ways. Keyboard warriors and Monday morning quarterbacks notwithstanding, if you want to really change a field and I mean really change it, you have to do original research. You can blog until your blue in the face and it won’t change a damn thing. It would be like putting lipstick on pig and entering it in beauty contest. Cheers.
References
1. ACSM Position Stand on Anabolic Steroids: http://journals.lww.com/acsm-msse/Citation/1987/10000/Position_Stand_on_The_Use_of_Anabolic_Androgenic.23.aspx
2. Street C, Antonio J, Cudlipp D: Androgen use by athletes: a reevaluation of the health risks. Can J Appl Physiol 1996, 21:421-440.
3. Bhasin S, Storer TW, Berman N, Callegari C, Clevenger B, Phillips J, Bunnell TJ, Tricker R, Shirazi A, Casaburi R: The effects of supraphysiologic doses of testosterone on muscle size and strength in normal men. N Engl J Med 1996, 335:1-7.
4. Harris RC, Soderlund K, Hultman E: Elevation of creatine in resting and exercised muscle of normal subjects by creatine supplementation. Clin Sci (Lond) 1992, 83:367-374.
