Background
I’m teaching a seminar course to our PhD students and last week someone made mention of the small print in scientific publications and this reminded me of an old study with some interesting small print that I never bothered to write up, as it had no chance of ever being published at the time. Self help books tell us Don’t sweat the small stuff and it’s all small stuff. In the scientific field that advice is problematic for several reasons
- Often we don’t know what is truly small stuff
- The devil may well be in the details
- Small stuff can have a huge impact on data interpretation, inferences, and policy decisions
The study in question
In 2001, one of the leading cardiovascular specialty journals published a study entitled “Cardiovascular thrombotic events in controlled, clinical trials of rofecoxib”(Konstam et al. 2001). This was a meta-analysis of 23 RCTs examining the drug rofecoxib (Vioxx) and the abstract (“The BIG PRINT”) is shown here
The results (more BIG PRINT)
Small print
There is some revealing information in the small footnote that accompanied the publication and is reproduced below
All authors either were employees or consultants of the drug manufacturer. Moreover the stdy was fast tracked (before journals were even formally doing fast tracking) with 1 day between submission and acceptance! The study appeared online 12 days after submission. The backdrop for this rapidity was the publication in JAMA at the end of August 2001 looking at the 3 published rofecoxib RCTs that concluded there was an increased CV risk. This footnote certainly raises disconcerting flags mandating a closer look at the results and conclusions, which, as shown next, were hardly found to be reassuring.
Interpretative problems
- 20 of the 23 trials were unpublished
- The interpetation of the forest plot is incorrect. Consider the rofecoxib versus placebo comparison the OR = 0.84, 95%CI 0.51 - 1.38. The correct interpretation is not that there is no evidence for an excess of CV events but rather the study is underpowered as the data is compatible with sampling errors of 49% reduction up to a 38% increase in events. As the magnitude of these possibilities are certainly clinically meaningful the result should be interpreted as inconclusive. Similar comments apply to the rofecoxib versus non-naproxen NSAIDs. Remember “absence of evidence is not evidence of absence”(Altman and Bland 1995)
- The biggest interpretative error is in the 3rd comparison where these apologists claim the statistically significant difference between rofecoxib and naproxen were not due to a danger with rofecoxib but rather to the beneficiasl antiplatelt effects of naproxen.
- What is the evidence of these naproxen benefits? Well there are no references in the paper that support this statement and the reason for that lack is shown in this PubMed search done at the time of this publication
The follow-up
Rofecoxib stayed on the market until Sept 2004 when the manufacturer voluntarily withdrew it. A report from the FDA suggested that “An estimated 88 000-140 000 excess cases of serious coronary heart disease probably occurred in the United States over the market life of rofecoxib. The US national estimate of the case-fatality rate (fatal acute myocardial infarction plus sudden cardiac death) was 44%, which suggests that many of the excess cases attributable to rofecoxib use were fatal.”
Returning to the original topic of “small print”, the footnote on page 1 of the paper with the 1 day timeline of submission and acceptance and the extensive conflicts of interest (COI) was a red flag as to possible quality issues. While the COI among the authors is nothing new, the actions of the journal were more surprising. An examination of the full paper certainly confirmed these initial suspicions of highly suspect quality.
References
Citation
@online{brophy2025,
author = {Brophy, Jay},
title = {Small Print},
date = {2025-01-27},
url = {https://brophyj.com/posts/2024-12-30-my-blog-post/},
langid = {en}
}