Probability and Bayes

Why a positive test can still mean you probably don’t have it, and what that means for AI classifiers.

This one’s part of How Models Learn Pro — every lesson, every interactive, and new ones as they land. About 11 minutes.

See Pro plans Sign in

Already Pro? Sign in and you’ll go straight in.

Inside “Probability and Bayes”

What you’ll learn

Probability as counting
Joint and conditional probability
P(A | B) is not P(B | A)
Bayes’ theorem
Back to the studies
Why this matters for AI

Key terms

Prevalence (base rate): The share of a group that has a condition before any test is done: P(condition).
Joint probability: The probability that two things are both true, such as having a condition and testing positive.
Conditional probability: P(A | B): the probability of A among only the cases where B is true. In general it is not the same as P(B | A).
Sensitivity: P(positive | has the condition): the share of real cases a test catches. Called recall in machine learning.
Specificity: P(negative | doesn’t have the condition): the share of people without it whom the test correctly clears.
Bayes’ theorem: P(A | B) = P(B | A) · P(A) / P(B): a way to turn P(B | A) into P(A | B) using the base rate.
Natural frequencies: Expressing probabilities as counts out of a whole group, such as “51 out of 10,000”.
Precision: Of the cases a classifier flags as positive, the share that really are: TP / (TP + FP).

Sources

Lehman et al. (2017), National Performance Benchmarks for Modern Screening Digital Mammography, Radiology 283(1) — BCSC, US, 1,682,504 screens (2007–2013): sensitivity 86.9%, specificity 88.9%, 5.1 cancers detected and 0.8 missed per 1,000 screens, PPV1 4.4%; cancer = diagnosed within 12 months.
Sprague et al. (2017), National Performance Benchmarks for Modern Diagnostic Digital Mammography: Update from the Breast Cancer Surveillance Consortium, Radiology 283(1) — BCSC, US, 401,548 diagnostic exams in 265,360 women (2007–2013): sensitivity 87.8%, specificity 90.5%, cancer detection rate 34.7 and false-negative rate 4.8 per 1,000, abnormal interpretation rate 12.6%, PPV2 27.5%; positive = BI-RADS 4 or 5; cancer = diagnosed within 1 year.
Gigerenzer & Hoffrage (1995), How to Improve Bayesian Reasoning Without Instruction: Frequency Formats — Bayesian calculations are simpler with frequency formats; untrained participants gave Bayesian answers in 46–50% of cases with frequencies vs 16–28% with probabilities (Psychological Review 102(4)).
Goodfellow, Bengio & Courville, Deep Learning, ch. 3: Probability and Information Theory — Frequentist and Bayesian probability follow the same rules; conditional probability; Bayes’ rule.
Google Machine Learning Crash Course: Accuracy, precision and recall — Precision = TP / (TP + FP); recall = TP / (TP + FN); imbalanced datasets.
Hopkins, Reeber, Forman & Suermondt (1999), Spambase, UCI Machine Learning Repository — 4,601 real emails (1,813 spam) with 57 features, CC BY 4.0 (doi:10.24432/C53G6X). The spam-filter step scores held-out emails from it.
scikit-learn: LogisticRegression — The model behind the spam scores (L2, C = 1, lbfgs), trained offline by data-pipeline/latent_data/spam_scores.py: held-out AUC 0.980; precision 92.5% at 40% spam, 51.5% at 5%.