Review of "Timeless Algorithms: The Foundational Ideas" by Gary Sutton

A.C. Jokela — Sat, 17 Oct 2026 02:00:00 GMT

Disclosure: I have been a technical reviewer for Manning on several projects. My copy of Timeless Algorithms was my compensation for reviewing pre-publication material of an unrelated book.

Introduction

Timeless Algorithms: The Foundational Ideas is Gary Sutton's latest book for Manning, following Statistics Slam Dunk and Statistics Every Programmer Needs. I have wanted someone to write this book for years. Take the dozen papers and books that modern machine learning rests on, reconstruct the problem each one solved, and show with equations and worked examples why the idea still holds up. The lineup:

Thomas Bayes's essay on inverse probability (1763)
Fisher on maximum likelihood (1922)
Neyman and Pearson on hypothesis testing (1933)
Shannon's theory of communication (1948)
Bellman's Dynamic Programming (1957)
Raiffa and Schlaifer's Applied Statistical Decision Theory (1961)
Vapnik's The Nature of Statistical Learning Theory and the soft-margin support vector machine (1995)
Breiman's random forests (2001)
MacKay's Information Theory, Inference, and Learning Algorithms (2003)
The deep learning review by LeCun, Bengio, and Hinton (2015)
"Attention Is All You Need" (2017)
The scaling laws of Kaplan and colleagues (2020)

In 2017, my last required course for graduate school was pattern recognition. It wasn't even called machine learning. It was mostly formal proofs, and only in the final week did we touch on a newfangled idea called "neural networks," along with something called "deep learning." LeCun, Bengio, and Hinton had already reviewed deep learning in Nature two years earlier, and "Attention Is All You Need" came out that same year. Sutton's lineup has the same shape as that semester, with deep learning arriving in chapter 11 of 13, except the book gives it three chapters instead of a week.

The book contains no code by design. Sutton's welcome letter names the target reader: someone who has used regression, built a model, or interpreted a dashboard, and now wants to understand why the methods work.

I read the fifth Manning Early Access Program (MEAP) release. It runs 13 chapters across roughly 380 pages and is complete through the final chapter, but it has no references, index, or appendices yet. Since the book uses worked examples in place of code, I recomputed all of them. That turned out to be the most revealing part of the review. The arithmetic is almost always right. The setups often aren't.

The Hidden Stack

Chapter 1 opens with a pilot flying through fog whose autopilot suddenly disengages, and argues that practitioners who run models they don't understand are in the same position. Sutton's remedy is a framework he calls the "hidden stack of modern intelligence." It has five layers, from bottom to top:

Raw data and feature engineering
Modeling and decision frameworks
Algorithmic assumptions
Mathematical foundations
Epistemology and ethics

Figure 1.1 is a good graphic: a pyramid beside a grid, with filled circles showing which layers each of the twelve works touches. A lending example then walks up the stack, from credit data to the question of whether accurate default predictions should be used to deny loans.

Page 4 of the MEAP v5 PDF, with Figure 1.1. I removed the licensing line at the foot of the page, which carries my email address.

It is a promising device, and the book never uses it again. No chapter after the first mentions the stack or names any of its layers; I searched the full text. The chapter 1 tags don't match the chapters they describe:

MacKay is tagged "Mathematical Foundations," yet chapter 10, his chapter, contains no equations.
Kaplan's scaling laws are tagged "Epistemology & Ethics," yet the ethics content in chapter 13 comes to one paragraph on how few organizations can afford frontier training.
Breiman is tagged "Epistemology & Ethics" too. Chapter 9 uses a loan-approval tree and discusses proxy features in variable importance, but never raises proxy discrimination.

A framework that appears only in the introduction is just a table of contents. Sutton should either end every chapter by placing its idea in the stack, which would give the book the spine it lacks, or cut the device.

What Works

The best pages put two ideas side by side and let the numbers argue. Chapter 3 reruns chapter 2's rare-disease test (prevalence 1 in 1,000, 99% sensitivity, 1% false positives) through Fisher's eyes. Bayes gives a 9% probability of disease. Fisher's likelihood ratio says a positive result is 99 times more likely under disease than under health. Same data, same model, two different questions, and the reader sees exactly where the prior enters. The same chapter derives ordinary least squares as maximum likelihood under Gaussian errors, then fits a ten-point study-hours data set by hand. Every number checks out: a cross-product sum of 77.15, a sum of squares of 82.50, and a slope of 0.935.

Chapter 6's knapsack walkthrough is the best-executed example in the book. It builds five tables for a capacity of 7, with every cell correct, then a backtracking table that recovers the optimal items. Chapter 5's mutual information example (100 patients, blood pressure against disease) correctly computes H(Y) = 0.97, H(Y|X) = 0.85, and I = 0.12 bits. It then points out that decision-tree information gain is the same calculation put to a different use. That is the one place in the Shannon chapter where the bridge to machine learning is built from numbers rather than asserted.

Chapter 11 takes a seven-parameter network through a forward pass, a backward pass, and an update. It then runs a second forward pass to show the loss falling from 0.39 to 0.37, a step most introductions skip. The chapter derives binary cross-entropy exactly from Fisher's Bernoulli likelihood. It also makes a point many textbooks blur: a predicted probability of 0.68 says nothing about how confident the network should be in its own weights.

Chapter 12 computes a single attention head by hand, from token IDs through the softmax over the vocabulary. It keeps the attention softmax and the output softmax carefully separate, and its explanation of causal masking is clear.

A few other ideas stayed with me:

Chapter 7 separates Shannon's information from decision value. A small change in a posterior that crosses a treatment threshold is worth more than a large change that leaves the decision alone.
Chapter 8 explains the canonical SVM hyperplane well. Since f(x) and 2f(x) have the same sign, you can fix the scale so the closest points sit at ±1, and the margin then becomes 2/‖w‖. That makes minimizing ‖w‖ feel inevitable rather than arbitrary.
Chapter 9 is unusually honest about what random forests cannot do. More trees do not cure leakage, out-of-bag error becomes optimistic once you tune against it, and two copies of the same temperature reading, one in Celsius and one in Fahrenheit, will split the permutation importance between them.
Chapter 13 warns that log-normal processes can impersonate power laws over a limited range.

Where the Papers Should Be

The chapter openers each name a specific work. Most chapters then teach the textbook version of a method with the founder's name attached, and leave out what the work actually contained.

Chapter 4 is the clearest case. Neyman and Pearson's 1933 paper is famous for one result: when testing a simple hypothesis against a simple alternative, the likelihood ratio test is the most powerful test at a given significance level. The book puts that lemma in a sidebar without a single equation. It spends the chapter on a seven-step testing procedure that Sutton acknowledges Neyman and Pearson never wrote down as such.

Neyman and Pearson's real departure from Fisher was taking the alternative hypothesis seriously, which means Type II errors and power. None of the chapter's three worked examples computes power. The clinical-trial example needed it most. Fourteen of 20 patients recover against a 50% baseline, the p-value is 0.058, and the book correctly declines to reject the null. But the rejection region of 15 or more recoveries has an actual size of 2.1%, not 5%. And if the drug really raised recovery to 70%, the test would detect it only 42% of the time. That number is the Neyman-Pearson lesson, and the chapter leaves it out.

The pattern repeats elsewhere:

Chapter 2 never works Bayes's own problem: a ball on a table with a uniform prior and binomial evidence. It is the one example that would show a full posterior distribution being updated.
Chapter 5 gives Shannon's actual theorems (source coding, channel capacity, noisy-channel coding) about five pages with no equations. Four entropy calculations take longer, because each one appears twice: first in words, then in numbers.
Chapter 6 names Bellman's principle of optimality once and never states it. About 20 of its 33 pages go to Fibonacci, knapsack, and coin change, while the Bellman equation, the part reinforcement learning depends on, gets three pages and no numbers. Its section on shortest routes leaves out Bellman's own shortest-path algorithm.
Chapter 8 is named for Vapnik's The Nature of Statistical Learning Theory, yet gives statistical learning theory about three pages and no equations: no VC bound, and no link between margin and capacity. It never reaches the dual problem, so the coefficients on the support vectors simply appear. Kernels show up only at prediction time, which hides why the kernel trick works. The hinge loss is printed but never named, though it is the clearest line from Vapnik to how models are trained today.
Chapter 11 is built around the 2015 deep learning review but covers almost none of it beyond backpropagation. Convolutional networks get a sidebar. Recurrent networks and LSTMs are absent, though the next chapter is titled "Beyond recurrence."
Chapter 12 reports none of the transformer paper's results: no 65-million-parameter base model and no BLEU scores. It also lacks the matrix form of the attention equation and the positional-encoding formula, and never explains the encoder-decoder design that its own Figure 12.1 reproduces. Its examples are chatbot prompts, not translation.

Chapter 13 covers a quantitative paper and reports none of its quantities. Kaplan's exponents (roughly 0.076 for parameters, 0.095 for data, and 0.050 for compute) never appear. Neither does the finding that the optimal model size grows with about the 0.73 power of compute. That omission hides the sharpest illustration of the theme the chapter keeps returning to, diminishing returns: at an exponent of 0.076, halving the loss through parameters alone takes about 9,000 times as many parameters.

Chinchilla, the 2022 result that overturned Kaplan's allocation advice, gets a single anonymous sentence: "Subsequent research refined that allocation." Meanwhile, both chapter 1 and chapter 13 credit Kaplan with the idea that parameters, data, and compute must grow in balance. That was Chinchilla's message; Kaplan's allocation leaned heavily toward model size. Figure 13.2 adds to the confusion. Its "compute-efficient frontier" slopes downward along a single fixed-compute contour, while the paragraph below it says the optimum moves upward and to the right.

Chapter 10 is the extreme case. Its 15 pages on MacKay contain no equations, no figures, and no example from MacKay's book. More than half the chapter recaps chapters 2 through 9 or previews chapters 11 through 13. On page 276, Sutton writes that the chapter "deliberately extends" MacKay's insight; the summary then credits MacKay with that extension.

Test the Examples Like Code

In a book with no code, the worked examples do the work code would do, so they need the same testing. The arithmetic passes almost everywhere. Several setups fail, and when they do, they undercut the lesson they were built to teach.

Chapter 7 asks whether a physician should order a second, "more accurate" test before treating. The second test is 95% sensitive and 90% specific. Chapter 2's original test was 99% on both measures, so the "more accurate" test is worse on both. The chapter concludes that the expected value of sample information is zero: no result can change the decision, so the test isn't worth its cost of 5 utility units. That holds only because the test is weak. Rerun the analysis with a test that really is more accurate, such as a repeat of the 99/99 test, and the information is worth 7.9 units. That is more than the cost, and the lesson reverses. The chapter never shows a case where information is worth buying, which is the case decision analysis exists for.

Chapter 3 is the maximum likelihood chapter, yet it introduces its logistic regression example with "Suppose the optimization yields" coefficients of −6 and 1.2. The actual maximum likelihood estimates for the book's ten data points are −5.82 and 1.30. With the real fit, a student who studied five hours has a 66% chance of passing, not the 50% the book reports. The reader never sees the chapter's central estimator actually being run.

Chapter 5's neural-network example claims to show cross-entropy loss falling during training. What it computes is the entropy of the network's prediction, which drops from 3.32 bits to 0.44. Against a one-hot target, the cross-entropy is −log₂(0.95), or 0.074 bits. The difference matters, because a network that is confidently wrong also has low entropy. The cross-entropy formula never appears in the chapter, and KL divergence appears nowhere in the book. The same chapter has two more slips:

It says entropy falls as outcomes become more evenly distributed. Entropy rises.
Its loan-approval tree splits a root of 10 approvals and 10 denials into branches that add up to 11 and 9.

Chapter 6's coin-change recurrence, as printed, counts ordered sequences. Evaluated as written, it finds 18 ways to make 12 cents from pennies, nickels, and dimes, not the 4 its own tables show.

Chapter 11's pedestrian-detection network takes as input a single hand-engineered feature, which is exactly what the chapter says deep learning replaced. With the given weights, its output stays between 0.378 and 0.802 for every possible input. At the book's braking threshold of 0.3, it brakes for everything.

All of these are fixable, and catching them is what MEAP is for. But they suggest a process that checks the arithmetic without checking whether each example demonstrates what the surrounding text claims.

History and Credit

A book organized around foundational works is partly a history book, and the history needs work. Three sidebars get basic facts wrong:

Andrew Viterbi was on the UCLA faculty, not USC's, when he published his algorithm in 1967. That removes the premise of the sidebar celebrating him and Bellman as colleagues.
Bellman's brain tumor came in 1973, when he was 52 or 53, not "in his mid-forties."
The Bayes sidebar calls Pascal a contemporary of Bayes. Pascal died in 1662; Bayes was born around 1701.

The conceptual history matters more. The Fisher passage I praised above has Fisher deliberately ignoring the disease's prevalence. Fisher objected to priors with no frequency behind them; when the prior was a known frequency, as prevalence is, he accepted Bayes's theorem. Chapters 4 and 10 also fold Fisher's p-values into the Neyman-Pearson framework, the very distinction the two camps fought over for decades.

Credit also flows too easily to the headline name:

Shannon. John Tukey coined the word "bit," and Hartley defined logarithmic information twenty years before Shannon. Neither appears.
Decision trees. The information-gain splitting rule the chapter teaches is the criterion from Quinlan's ID3, and Quinlan is never mentioned.
Breiman. Random feature selection predates his 2001 paper. Tin Kam Ho's random decision forests (1995) gave each tree a random subset of features, and Amit and Geman (1997) drew a fresh subset at each split. Breiman's own paper credits both, yet the chapter presents the idea as his addition.
Vapnik. Corinna Cortes is credited once for the soft-margin SVM. Ten pages later, "Vapnik addressed this limitation" alone, and chapter 1 credits the soft margin to Vapnik alone too.
The transformer. The chapter is titled after Google Brain, although half of the eight authors were at Google Research. It also says Vaswani and his Brain colleagues "made the decisive move" of dropping recurrence, which the paper's contribution footnote credits to Jakob Uszkoreit of Google Research.
Scaling laws. Baidu's 2017 scaling-law work preceded Kaplan, and Kaplan's own paper calls it "perhaps the closest" work to its own. The book never mentions it.

Chapters 12 and 13 are also the only chapters titled after companies rather than people.

The Prose

Sutton writes in a formal, cautious register that rarely lands a memorable sentence, and he says most things several times:

Some form of "foundation" appears 146 times.
"Transparent" appears 14 times in the 26 pages of chapter 4, where each of the seven steps opens by restating every step before it.
Chapter 6 tells the reader five times in five pages that the dynamic programming structure "remains intact" or "remains unchanged."
Each subsection of chapter 5's communication-theory section closes with some variant of "continue to shape modern communication systems."
Chapter 9 draws the Vapnik-versus-Breiman contrast six times.

Chapter 10's unifying template (what remains unresolved, what information is available, how it is transformed, what limits the result) fits every method in the book, so it explains none of them. I estimate a quarter of the text could go without losing anything.

There are also drafting marks a copy edit will catch. Chapter 3 has run-in headings labeled "EXAMPLE," "INTERPRETATION," and "TRANSITION" that look like an outline template, and two consecutive sections both titled "Maximum likelihood estimation." A beverage fill-volume example refers to "the patient's test result." A heading spells Schlaifer as "Schalifer." Equations lose parentheses and subscripts often, which hurts more in a book with no code to fall back on.

Who Should Read It

In its current state, Timeless Algorithms suits a reader with little formal statistics who wants to know where the vocabulary of machine learning comes from, and who will tolerate repetition to get there. Much of the early material is first-course statistics: the addition rule illustrated with a single die, the complement rule with two dice, a test-selection table of t-tests and chi-square tests. Readers who have taken that course, or an undergraduate machine learning course, will get more from the sources. Shannon's 1948 paper is still readable, the transformer paper is short, and MacKay's book is free online and better than any summary of it.

The fixes are clear, and most are editorial:

Put each work's results on the page. That means Kaplan's exponents, the Neyman-Pearson lemma with a power calculation, the transformer's model size and scores, and Bellman's principle in his own words.
Test every worked example like code. Check the arithmetic, and check that the example demonstrates its claim.
Use the hidden stack in every chapter or drop it.
Cut a quarter of the text.
Build the bridge the book gestures toward but never crosses. Large language models are trained by minimizing cross-entropy, a quantity built on Shannon's entropy that is also Fisher's negative log-likelihood. Their quality is reported as perplexity. Shannon himself generated text from n-gram models of English in 1948. That one thread runs through chapters 3, 5, 11, 12, and 13, and it appears nowhere in the book.

The premise deserves a book that delivers on it. MEAP v5 isn't that book yet, but the knapsack walkthrough, the Fisher-versus-Bayes contrast, and the hand-computed attention head show that Sutton can write it. There is still time before final publication.

TinyComputers.io (Posts about statistics)