Departments of Medicine, Epidemiology and Biostatistics, McGill University
2024-11-28
Conflicts of interest
No conflicts of interest
To the best of my knowledge I’m equally disliked, or at best deemed irrelevant, by all drug and device companies
AI generated image
Objectives
Review of some probability concepts
Review of statistical inference concepts
Realization of their practical applications
Are statistics (math) important for MDs?
2012 Harvard study suggests the answer is YES
What are the benefits of “good” stats knowledge?
Informs proper research methods for the
collection and analysis of the data
acknowledgement of uncertainty
presentation and interpretation of the results
Facilitates
the creation and understanding of knowledge
formation of reasonable conclusions
good decision making
Metascience - study of science itself
Hypothetico-deductive model of the scientific method
Metascience
Where this talk concentrates
Metascience
Plenty of other places to go wrong besides “stats”
Statistical inference
Statistical inference is using data analysis to infer (learn) from a sample about the underlying population (without it we’re left simply with our data)
Often involves “noisy” sample data, so variability and uncertainty must be accounted for
Inferences require referral to common statistical distributions (or else simulations)
The area under the curve (AUC) of the probability density function from the assumed statistical model corresponds to the probability of occurence for that random variable
Statistical models alone are insufficient for causality, remember \[association \ne causation\]
Probability
Probability branch of mathematics of how likely events occur -> statistical inferences
Contrasting views of probability
Null hypothesis significance testing (frequentist) inference: views probability as a long run frequency. Parameters considered as fixed but unknown quantities, so can’t make probability statements about them. Answers questions like “What should I decide given my data, controlling the long run proportion of mistakes I make at a tolerable level.”
Bayesian inference: views probability as a calculus of beliefs, parameters are considered random variables with probability distributions that follow the rules of probability. Answers questions like “Given my prior beliefs and the objective information from the data, what should I now believe?”
Basic Probability Distributions
Understanding probability requires understanding some posible models (& their assumptions)
IOW, what underlying probability distributions could have generated the observed data.
Here are common probability distributions according to discrete or continuous data.
Normal and t distributions
Normal distributions are common due to CLT, many random variables that are the sum of independent processes, such as measurement errors, are often close to normal.
Mathematically the normal distribution is expressed as \[f(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{-\frac{(x-\mu)^2}{2 \sigma^2}}\]
where \({\mu}\) is the mean or expectation of the distribution (and also its median and mode)
\({\sigma^{2}}\) is the variance and \({\sigma }\) is the standard deviation
Graphically this may be plotted as shown
Student’s t is a continuous distribution with “fatter” tails (depending on the degrees of freedom)
Uniform distribution
The uniform distribution assumes that all continuous outcomes between boundaries have an equal probability of occurring.
Of interest in Bayesian statistics as a prior non-informative distribution which allows the observed data to completely dominate the final (posterior) distribution.
Mathematically this is expressed as \[PDF(x) = \frac{1}{b-a} \,\, for \, a \le x \le b\] PDF = probability density funcion
For discrete data, continuous models inappropriate (cf negative counts are impossible). A binomial distribution describes the number of successes in a fixed number of independent Bernoulli trials, where each trial has the same probability of success.
Mathematically, \[P(X = k) = {n \choose k} * p^k * (1 - p)^{n-k}\] • P(X=k) is the probability of having k successes in n trials
• \({n \choose k}\) is the binomial coefficient, the number of ways to choose k from n trials
• p is the probability of success in each trial
• 1-p is the probability of failure in each trial
Binomial distributions can model coin flips, where there are two possible outcomes (heads or tails) and each trial is independent of the others.
But many, many other more important applications for binary outcomes in various fields, including medicine
Poisson Distribution
Another discrete probability distribution - # events occurring in a fixed time interval with a constant rate.
Mathematically, \[P(X = k) = \frac{\lambda^k * e^{-\lambda}}{k!}\]
• P(X=k) is the probability of observing k events in a given period of time
• \(\lambda\) is the expectation of the # events over the same time period
• variance = mean (E(Y))
\[binomial \:P(\textit{k} \:successes \:in \:\textit{n} \:trials) \approx poisson \:P(k \:with \:
\lambda = \textit{np})\] If n -> \(\infty\) & p (any trial success) -> 0
Better appreciated by plotting the equation
How well do we understand probability?
A case study
A 2024 publication “One-Year Outcomes of Transseptal Mitral Valve in Valve in Intermediate Surgical Risk Patients” in Circulation: Cardiovascular Interventions based on 50 MVIV patients, concluded in symptomatic patients with a failing mitral bioprosthesis that
“Mitral valve-in-valve with a balloon-expandable valve via transseptal approach in intermediate-risk patients was associated with improved symptoms and quality of life, adequate transcatheter valve performance, & no mortality or stroke at 1-year follow-up.”
I wondered what is the probability of seeing 0 deaths if truly no difference in mortality between MVIV and a redo with an estimated 1 year STS mortality = 4%?
I also wondered what would be my colleagues’ probability estimates.
Q3. Given only this study, what is P(MVIV is as safe or safer than a redo)?
< 25 %
25 - 50%
51 - 75%
76 - 95%
> 95%
Quiz 1 Results
15 (42%, n= 36) MUHC cardiology staff and 4 (18%, n = 22) fellows replied Q1. P(0 deaths | true death rate 4%)?
Q2. P(0 deaths | true death rate 5.6%)? (i.e. 40% increase)?
Lot’s of variability, n’est-ce pas?
Quiz 1 Q1 Discussion
Use either a Poisson distribution (counts) or a binomial distribution (independent Bernouilli trials) where each of the 50 subjects is considered as alive or dead with a probability of 4%. These calculations can be done with the above equations or more easily with any software that includes Poisson or Binomial distributions.
Q1. P(0 deaths | true death rate 4%)?
With one line of code in R db <- 100*dbinom(c(0:6),50, .04)
Assuming a binomial distribution with an event (death) probability of 4%, the probability for 0, 1, 2, 3, 4, 5, 6 events is 13%, 27.1%, 27.6%, 18.4%, 9%, 3.5%, respectively.
Assuming a Poisson distribution with an event (death) rate of 2 (# deaths in 50 individual in 1 year with 4% expected mortality), the probability for 0, 1, 2, 3, 4, 5, 6 events is 13.5%, 27.1%, 27.1%, 18%, 9%, 3.6%, respectively.
Again from a single line of R code dp <- 100*dpois(c(0:6),2) Therefore the answer to Q1 is 10 - 14.9%
Quiz 1 Q1 Discussion
This data can also be visualized emphasizing the similarity between the Poisson and Binomial distributions.
Again with a single line of R code dp1 <- 100*dpois(c(0:6),2.8)
Assuming a P(2.8) (rate = # deaths in 50 individual in 1 year with 5.6% expected mortality = 50*.056), the probability for 0, 1, 2, 3, 4, 5, 6 events is 6.1%, 17%, 23.8%, 22.2%, 15.6%, 8.7%, respectively.
Assuming a B(n, 5.6%), the probability for 0, 1, 2, 3, 4, 5, 6 events is 5.6%, 16.6%, 24.2%, 22.9%, 16%, 8.7%, respectively.
Visually
Clearly, if the expected death rate is higher the curves shift left meaning the probability of observing 0 deaths will fall with increasing mortality rates, so
The answer to Q2 is 5 - 9.9%
Quiz 1 Q3 Results
Q3. P(MVIV as safer or safer than redo)?
IOW P(MVIV mortality <4%)
Again lot’s of variability, n’est-ce pas?
Quiz 1 Q3 Answer & Discussion
Q3. P(MVIV as safer or safer than redo)
Ideal outcome from both interventions in the same patient but impossible. So do RCTs & hope that subjects in each arm are exchangeable. Here, no RCT but assuming the STS model is accurate then the MVIV patients would have had a 4% mortality if, instead of MVIV, they had had a surgical redo. Simulate a dataset with the 50 observed MVIV results and 50 counterfactual subjects receiving a redo operation with a 4% mortality rate.
Use binomial distributions (dead or alive) but include the explanatory variable treatment (logistic regression).
Family: binomial
Links: mu = identity
Formula: success | trials(total) ~ Tx
Data: dat1 (Number of observations: 2)
Draws: 4 chains, each with iter = 10000; warmup = 5000; thin = 1;
total post-warmup draws = 20000
Regression Coefficients:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
Intercept 0.98 0.02 0.93 1.00 1.00 1229 2480
Txredo -0.04 0.04 -0.13 0.03 1.01 703 521
Draws were sampled using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).
As expected, mean mortality difference (MVIV - redo) is - 4% but this Bayesian model also now gives us a measure of the associated uncertainty with 95% CrI -13% to 3%
Quiz 1 Q3 Discussion
The probability density for this mortality difference is plotted here
This shows a 12% probability (blue AUC) that MVIV patients in this study would have a worse outcome with a surgical redo, provided their counterfactual mortality has been well predicted with the STS model. IOW, P(MVIV is as safe or safer than a redo) = 88.3% (grey AUC).
Therefore the answer to Q3 is 76-95%
Quiz 1 How did we do?
For Q1, Only 16% (3/19) correctly estimated was a 10-14.9% probability of observing 0 deaths, if true rate was 4%. 7 of 19 (37%) estimated this probability as < 5%.
For Q2, 68% (13/19) respondents correctly reasoned that if the true underlying mortality increased, less likely to observe 0 deaths but a third (6/19) didn’t.
For Q3 Only 2 (10.5%) responses correctly estimated P(MVIV as safe or safer than a redo) = 76-95%. 12 of 19 (63%) of respondents estimated this probability at being under 50%.
For Q1-3, No one gave all 3 correct answers
Limitations
i) small sample size of respondents
ii) unknown if respondents have better, worse or same quantitative skills as the non-respondents.
Notwithstanding limitations, suggests additional quantitative probability training may be helpful.
(In)famous P value
Definition 𝑝-value is the probability under the null hypothesis and an assumed data-generating model, that an appropriate test statistic would be as or more extreme than what was observed. \(P(data | H_o, \ model\))
Quiz 2 Q1
Consider an experiment which is quite encouraging with a true effect size that is 2 standard errors from zero.
Imagine a simulation where the same experiment can be repeated 10 times.
What do you think is the range of the 10 p values?
1. 0.02 to 0.05
2. 0.002 to 0.05
3. 0.0002 to 0.05
4. 0.0002 to 0.5
Again one line of R code set.seed(123) # for reproducibility p_value <- 2*(1 - pnorm(abs(rnorm(10, 2, 1)))) # take 10 random draws from N(2,1)
Print the p values associated with the 10 random draws from a N(2,1) distribution
0.1500, 0.0767, 0.0003, 0.0384, 0.033,
0.0002, 0.0138, 0.4623, 0.1891, 0.1201
Quiz 2 Q1 Answer
Consider an experiment which is quite encouraging with a true effect size that is 2 standard errors from zero. What do you think is the range of the 10 p values?
Quiz 2 Q1 What says ChatGPT4?
“If we assume minor variations around the 2 standard errors effect size, p-values might typically range from just below 0.0456 to slightly above it. For example, if some experiments measure the effect size at slightly less than 2 or slightly more than 2 standard errors, the p-values could range approximately between 0.04 and 0.05.”
THIS IS WRONG.
Quiz 2 Q1 Answer
How can this be?
Even though we KNOW here that the effect = 2, according to the p value about 1/2 the time p > 0.05 meaning we should accept the null hypothesis of no difference!
Quiz 2 Q1 ChatGPT4 after shown simulation?
ChatGPT4: “Your simulation highlighted the variability that can occur in experimental results due to random sampling, which indeed can lead to a much broader range of p-values. It demonstrates that even if the true effect size is relatively large, the exact p-values obtained in any given experiment can vary widely.”
Me: “If you ask the same question tomorrow or in the future, will you update your response accordingly?”
ChatGPT4: “My response will certainly take into account this discussion and the insights from your simulation.”
IT ABSOLUTELY DIDN’T AS WHEN ASKED THE SAME QUESTION 1 WEEK LATER IT GAVE THE SAME INCORRECT RESPONSE.
Quiz 2 Q1 Answers
The larger points are:
The p-value is a statement relative to the null hypothesis of no effect. It doesn’t have much of an interpretation relative to a real, nonzero effect.
The p-value is super-noisy. It’s a weird nonlinear transformation of the z-score with nonintuitive behavior.
Even ChatGPT4 doesn’t realize how nosiy p values are!
Bayes’ Theorem -> probability statements about hypotheses, model parameters or anything else that has associated uncertainty
Advantages
treats unknown parameters as random variables -> direct and meaningful answers (estimates)
allows integration of all available information -> mirrors sequential human learning with constant updating
allows consideration of complex questions / models where all sources of uncertainty can be simultaneously and coherently considered
Disadvantages
subjectivity (?) problem of induction (Hume / Popper - difficulty generalizing about future)
Frequentist vs Bayesian (summary)
Frequentist
Bayesian
Probability is “long-run frequency”
Probability is “degree of certainty”
\(Pr(X \mid \theta)\) is a sampling distribution (function of \(X\) with \(\theta\) fixed)
\(Pr(X \mid \theta)\) is a likelihood (function of \(\theta\) with \(X\) fixed)
No prior
Prior
P-values (NHST)
Full posterior probability model available for summary/decisions
Confidence intervals
Credible intervals
Violates the “likelihood principle”1: Sampling intention matters Corrections for multiple testing Adjustment for planned/post hoc testing
Respects the “likelihood principle”: Sampling intention is irrelevant No corrections for multiple testing No adjustment for planned/post hoc testing
Objective?
Subjective?
Probabilities with 2 arm studies
A 2023 paper in the NEJM concluded “In patients with refractory out-of-hospital cardiac arrest, extracorporeal CPR and conventional CPR had similar effects on survival with a favorable neurologic outcome”
Are the results truly similar?
What do MUHC cardiologists think?
Quiz 3 Q1
I surveyed MUHC cardiologists (51% response rate, 18 of 35) and asked their probabilities that eCPR was superior to cCPR
Only 2 of the cardiologist gave >50% probability of eCPR being superior
Quiz 3 Q2
The respondents were next provided with 2 previous trials of e-CRP with these results
name
n_ecrp
survival_ecrp
n_ctl
survival_ctl
ARREST
15
6
15
1
PRAGUE
124
38
132
24
Both trials showed improved 30 day survival with eCPR.
The respondents were then asked to update their previous probabilities that eCRP is better than cCRP
Quiz 3 Q2
Slight overall shift in >50% from 2 to 5 respondents but still collectively 13/18 < 50%
Quiz 3 Q2
How did MUHC cardiologists update their beliefs with additional evidence?
Observations: i) Original authors claimed “similarity” but actually greater 50% probability of eCPR superiority based on their data and 76-95% with prior data ii) 16/18 MUHC cardiologists increased their probability estimate with the new data but iii) their original low estimate has a persistent grounding effect on revised estimates
Conclusions
Gut1 (heuristic) probability estimates are widely variable & often very far from probabilistic quantified effects
Statistical literacy depends on understanding probability distributions.
Statistical literacy improves diagnostic and therapeutic decision-making.
Undergraduate and graduate medical education should consider improving their quantitative training.
Acknowledgements
My former PhD supervisor, Professor (emeritus) Lawrence Joseph, arguably Canada’s first Bayesian biostatistician
Fonds de Recherche du Québec (Santé) whose salary support (1999 - 2023) allowed me to pursue these statistical musings.
