How Do We Reach Our Conclusions?
What Does Our Research Process Look Like?
It’s extremely important to us here at VAGA that you know exactly how we go about doing our research.
We believe that the internet is completely infested with health and fitness sites making completely unfounded, unsubstantiated, and unwarranted claims. They speak with an air of authority without ever presenting evidence.
The worst sites make an appeal to authority – they use a doctor as their ‘front man’ and expect you to just believe everything they publish because this supposed ‘expert’ says it is so.
The vast majority of websites we see talking about biohacking, supplementation, and human enhancement do make efforts to provide evidence to back up their claims. But providing studies is only one step to being truly impartial. Readers also need to know that you are following the scientific method when doing research, or when conducting product reviews.
Basically, they need to know that you’re being consistent in your approach.
This is where most websites fail.
The individuals behind them never disclose their review methodology. They don’t discuss their approach to research, and they never explicitly state what their criteria is for a good, valid study.
In that environment, it is easy to cherry-pick studies to suit your desired conclusions. If you don’t disclose your review process then it’s easy to spoof the results.
If you aren’t consistent across substances, you can subconsciously back up your pre-existing biases by not taking one substance as seriously as another.
Obviously, this isn’t a very scientific way to go about doing research.
We think we can do better.
Our Research Approach – What makes a good study?
It’s easy to cherry-pick evidence to back up the conclusions that you have already decided on.
There are a lot of published, peer-reviewed papers out there. They all have different approaches, different methodologies, and different authors. Some are extremely robust and valid, some not so much. Many will be heavily influenced by commercial interests; others will be influenced by the researchers’ innate biases.
The point we’re getting at is, there are a lot of studies out there. Many of them contradict one another.
As such, the simple act of citing a research paper does not mean that you are right.
It just means that at least one researcher interpreted their results in a way that suits your beliefs.
So does that mean that citing studies is useless?
NO! Of course it doesn’t!
It just means that we need to prioritize certain studies over others, and that the amount of evidence pointing in a certain direction matters a great deal.
We ascribe value to a study roughly according to where it sits on the hierarchy of scientific evidence. This hierarchy is nicely encapsulated in this – admittedly oversimplified – image:
As you can see, the least reliable evidence is personal, anecdotal reports and expert opinion. This is why we loathe so many supplement sites out there right now . They either appeal to the fact that their editor has a certain degree (which they take to mean they don’t need to provide proof), or they just tell you how they felt when using a certain substance.
Of course anecdotal data can be useful – it can tell you what kind of effects you might expect from a given substance. It might also protect you from seriously dangerous substances.
But it cannot prove whether or not a substance works. It certainly can’t tell us how well a substance works.
The same goes for expert opinion. Never assume that a “doctor” on a website is a medical doctor. Always do background checks – if they’re genuinely a medical professional, this should be easy to do.
Sadly, there are plenty of popular health and fitness “experts” who pose as medical doctors when they actually are nothing of the sort. Take a look at some of the most prominent “doctors” of online health and fitness websites – some of them are chiropractors (not recognized as medicine), and others are “natural health physicians” (also not recognized as medicine).
Instead of making these appeals to authority or to anecdotal claims, we try to stick to evidence from the top of the evidence hierarchy as far as that is possible.
When we must use inferior forms of evidence – such as case control studies or animal studies – we will always make you aware of the fact that such evidence has its limitations.
We will try to make you aware of how much of this evidence there is, how strong it seems, and what we make of it.
What we will never do is draw strong conclusions from such evidence.
There are some other criteria that we check studies for before we allow them to sway out opinions. We always look to see if a study:
- Has commercial bias
- Uses a small sample group
- Uses an unusual form of a substance
- Conflates multiple substances
- Uses a biased sample
- Is dated/has been contradicted by a lot of subsequent research
If a paper passes all of these checks and it looks like rock solid science, we’ll use it to form our opinions.
Obviously, our opinions about certain substances are colored by our personal experiences. This is inevitable, and not entirely harmful. But we also ensure that our personal anecdotes take a backseat to what the clinical literature has to say.
How We Review Supplements
Our review process for supplements is always the same.
First, we identify a candidate for review. We look for stacks that people are interested in, which look vaguely interesting, or which we think might become popular in the future.
We then go about combing through the formula, evaluating the ingredients, and checking the doses against both the scientific literature and the industry recommendations. We also compare them to the competition, but this isn’t as important as comparing it to effective doses used in clinical trials.
We’ll also usually do some digging on the manufacturer. If we find a trove of customer complaints or shady behavior, we’ll let you know!
Then, once it’s all put together, we publish!
Got Questions Or Suggestions?
Got some questions for us about our research?
Have a suggestion for an article or a product review?
Spotted a mistake in our work?
Get in touch today and let us know! Visit our contact page to see the different ways you can talk to us.