One of the news feeds I subscribe to (ACC CV news) reported the headline “Ketogenic Diet Could Reduce The Risk For All-Cause Mortality By Up To 24%” with a reference to an article published in Nature Scientific Reports(Qu, Huang, and Rong 2024). Moreover it was reported that the results “suggest the keto diet did not increase the risk for cardiovascular disease-related death despite its content possibly contributing to a higher risk for cardiovascular diseases.” Sounds too good to be true so I had a look.
The journal
I had previously heard of the journal, principally through its Nature branding but was quite unaware of much else, including its overall quality. Their website claims “Scientific Reports is the 5th most-cited journal in the world, with more than 734,000 citations in 2023”. Impressive! I also found their impact factor (IF) was 3.8. Now we are all aware of the quality limitations of IF but these numbers do raise some questions. \[Impact\,Factor = \frac{
Number\,of\,Citations\,in\,2023\,to\,articles\,from\,2021\,and\,2022}{
Number\,of\,Articles\,published\,in\,2021\,and\,2022}\] Thus while the number of reported citations does seem impressive, the IF suggests either many of these citations refer to reports prior to 2021, these citations wouldn’t count in the IF, or that the journal publishes an extraordinary high number of articles each year that dilutes the IF.
The authors
Authors and Affiliations
Xiaolong Qu Department of Cardiovascular Medicine, Renji Hospital, Shanghai Jiaotong University School of Medicine, 160 Pujian Road, Pudong New Area, Shanghai, 200120, China
Lei Huang & Jiacheng Rong
Department of Cardiology, Renji Hospital Ningbo Branch, Shanghai Jiao Tong University School of Medicine, 1155 Binhai 2nd Road, Hangzhou Bay New District, Ningbo, 315336, China
According to Google Scholar they have previously published 8 articles together.
The data
This study used 43,776 adults from the National Health and Nutrition Examination Survey (NHANES) conducted between 2001 and 2018 to investigate the potential association between dietary ketogenic ratio (DKR) and both all-cause mortality and cardiovascular disease (CVD) mortality.
Their results
Using an adjusted Cox proportional hazards regression analysis the authors report a significant inverse association was observed between the dietary ketogenic ratio (DKR) and all-cause mortality (HR = 0.76, 95% CI = 0.63–0.9, P = 0.003). However, no significant association with cardiovascular mortality was found (HR = 1.13; CI = 0.79–1.6; P = 0.504).
Their conclusions
“In the adult population of the United States, adherence to a KD exhibits potential in reducing all-cause mortality risk while not posing an increased threat of CVD-related fatalities.”
My issues (in no particular order)
Measurement error: The exposure variable DKR is never measured. The authors report DKR is calculated based on an equation proposed in a 1980 publication entitled “The ketogenic diet: mechanism of anticonvulsant action”(Withrow 1980). The exact equation and the population is was derived from and relationship to the study population is unknown. This is in addition to the usual measurement errors associated with dietary 24 hour recall studies.
Selection bias: The study population of eligible adults was 50,201 and 43,776 were analysed. It is unknown if this missing 13% resulted in selection bias but the possibility certainly exists.
Inferential errors: i) The authors report “no significant association with cardiovascular mortality was found (HR = 1.13; CI = 0.79–1.6; P = 0.504).” Classic error of “absence of evidence is not evidence of absence”[Altman and Bland (1995)](Gelman and Stern 2006). The data is compatible upon repeated sampling with a potential 60% increase in cardiovascular mortality. ii) Questionable causal inferences “Additional investigations are warranted to comprehensively comprehend the mechanisms underlying these observed associations and ascertain the feasibility and long-term sustainability of implementing a ketogenic diet for population-wide health benefits.”
Poor reporting: i) The actual DKR values are not provided. Instead we are only given quartile distributions among the deceased and survivors.ii) no discussion of the impact of researcher degrees of freedom in the choice of the 2 adjusted models iii) no discussion of model mis-specification iv) no verification of the Cox model assumptions.
Questionable data analysis: Here is a summary of the key elements of the baseline data according to survival status.
This is worrisome and provides stimulus to look at the outcomes. Obviously without the raw data, we are limited to looking at the unadjusted models. The authors report the following for the unadjusted model
Unadjusted model
Q1
Q2
Q3
Q4
1
1.03(0.96, 1.10)
1.05(0.97, 1.12)
1.05(0.98, 1.13)
Now this can be verified using the baseline data provided Here’s the calculation for each quartile:
Q1: Risk = 1563 / (1563 + 9374)
Q2: Risk = 1540 / (1540 + 9414)
Q3: Risk = 1498 / (1498 + 9443)
Q4: Risk = 1453 / (1453 + 9491)
\[Risk\,ratio_{Q2,Q1} = \frac{Q2\,risk}{Q1\,risk} = \frac{1540 / (1540 + 9414)}{1563/(1563 + 9374) } = 0.984\] Results for the other RR and 95%CI are easily calculated
Code
library(epiR)# Define the counts of deceased and survival for each quartiledeceased<-c(1563, 1540, 1498, 1453)survival<-c(9374, 9414, 9443, 9491)# Calculating the risk ratio and 95% CI using Q1 as the exposed controlresults<-lapply(2:4, function(i){# Create the 2x2 table for each comparison with correct exposure designation# Exposed (Q1) first, then Unexposed (Q2, Q3, Q4)table<-matrix(c(deceased[i], survival[i], deceased[1], survival[1]), nrow =2, byrow =TRUE)colnames(table)<-c("Deceased", "Survival")rownames(table)<-c(sprintf("Q%d (Unexposed)", i), "Q1 (Exposed)")# Calculate the risk ratios using epi.2by2epiR::epi.2by2(table, method ="cohort.count", conf.level =0.95)})results
[[1]]
Outcome + Outcome - Total Inc risk *
Exposed + 1540 9414 10954 14.06 (13.41 to 14.72)
Exposed - 1563 9374 10937 14.29 (13.64 to 14.96)
Total 3103 18788 21891 14.17 (13.72 to 14.64)
Point estimates and 95% CIs:
-------------------------------------------------------------------
Inc risk ratio 0.98 (0.92, 1.05)
Inc odds ratio 0.98 (0.91, 1.06)
Attrib risk in the exposed * -0.23 (-1.16, 0.69)
Attrib fraction in the exposed (%) -1.65 (-8.50, 4.76)
Attrib risk in the population * -0.12 (-0.92, 0.69)
Attrib fraction in the population (%) -0.82 (-4.13, 2.39)
-------------------------------------------------------------------
Uncorrected chi2 test that OR = 1: chi2(1) = 0.242 Pr>chi2 = 0.622
Fisher exact test that OR = 1: Pr>chi2 = 0.628
Wald confidence limits
CI: confidence interval
* Outcomes per 100 population units
[[2]]
Outcome + Outcome - Total Inc risk *
Exposed + 1498 9443 10941 13.69 (13.05 to 14.35)
Exposed - 1563 9374 10937 14.29 (13.64 to 14.96)
Total 3061 18817 21878 13.99 (13.53 to 14.46)
Point estimates and 95% CIs:
-------------------------------------------------------------------
Inc risk ratio 0.96 (0.90, 1.02)
Inc odds ratio 0.95 (0.88, 1.03)
Attrib risk in the exposed * -0.60 (-1.52, 0.32)
Attrib fraction in the exposed (%) -4.38 (-11.47, 2.26)
Attrib risk in the population * -0.30 (-1.10, 0.50)
Attrib fraction in the population (%) -2.14 (-5.48, 1.09)
-------------------------------------------------------------------
Uncorrected chi2 test that OR = 1: chi2(1) = 1.633 Pr>chi2 = 0.201
Fisher exact test that OR = 1: Pr>chi2 = 0.205
Wald confidence limits
CI: confidence interval
* Outcomes per 100 population units
[[3]]
Outcome + Outcome - Total Inc risk *
Exposed + 1453 9491 10944 13.28 (12.65 to 13.93)
Exposed - 1563 9374 10937 14.29 (13.64 to 14.96)
Total 3016 18865 21881 13.78 (13.33 to 14.25)
Point estimates and 95% CIs:
-------------------------------------------------------------------
Inc risk ratio 0.93 (0.87, 0.99)
Inc odds ratio 0.92 (0.85, 0.99)
Attrib risk in the exposed * -1.01 (-1.93, -0.10)
Attrib fraction in the exposed (%) -7.64 (-15.02, -0.73)
Attrib risk in the population * -0.51 (-1.31, 0.29)
Attrib fraction in the population (%) -3.68 (-7.05, -0.42)
-------------------------------------------------------------------
Uncorrected chi2 test that OR = 1: chi2(1) = 4.735 Pr>chi2 = 0.030
Fisher exact test that OR = 1: Pr>chi2 = 0.031
Wald confidence limits
CI: confidence interval
* Outcomes per 100 population units
So if the unadjusted results are totally different from what has been reported, how could anyone have the slightest confidence in a magically derived adjusted model, especially when considered in the context of the reservations noted above?
Should the authors maintain their confidence in their findings they should share the dataset and the statistical code that generated these findings.
The problems herein are aggravated by the imprimatur provided by publication in a Nature journal and of using a highly recognizable dataset which has historically been analyzed with much rigour and care.
According to Altmetric, this article is in the 98th percentile (ranked 3,956th) of the 209,240 tracked articles of a similar age in all journals!
Now that I have vented, it is time to get back to other activities.
References
Altman, D. G., and J. M. Bland. 1995. “Absence of Evidence Is Not Evidence of Absence.” Journal Article. BMJ 311 (7003): 485. https://doi.org/10.1136/bmj.311.7003.485.
Gelman, A., and HS. Stern. 2006. “The Difference Between ‘Significant’ and ‘Not Significant’ Is Not Itself Statistically Significant.” Journal Article. Am Statist 60: 328–31.
Qu, Xiaolong, Lei Huang, and Jiacheng Rong. 2024. “The Ketogenic Diet Has the Potential to Decrease All-Cause Mortality Without a Concomitant Increase in Cardiovascular-Related Mortality.” Journal Article. Scientific Reports 14 (1): 22805. https://doi.org/10.1038/s41598-024-73384-x.
Withrow, C. D. 1980. “The Ketogenic Diet: Mechanism of Anticonvulsant Action.” Journal Article. Adv Neurol 27: 635–42.
Citation
BibTeX citation:
@online{brophy2024,
author = {Brophy, Jay},
title = {Reflections on Peer Review - Part 2},
date = {2024-10-31},
url = {https://brophyj.com/posts/2024-10-31-my-blog-post/},
langid = {en}
}
