Lab 7 — Instructor answer key
Companion file: solution.R (in this folder), the lab chunk code with the three TODOs (1a, 2a, 3a) completed, in chunk order. Run it from the lab root (Rscript solutions/solution.R) and it prints every checkpoint in the handout plus the extras quoted below (the straight tails of the natural spline, a second contrast on the flat stretch of age, the omitted-variable δ of the notes’ section 2, the per-fold super learner weights, what happens without cross-fitting, the common slips in each TODO, and the median-over-fold-draws recipe). About ten seconds in all.
Contents: TODO code answers · expected checkpoints · model answers Q1–Q6 · understanding notes
TODO code answers
| TODO | Answer |
|---|---|
| 1a | estimate = sum(cvec * coef(fit)) and se = sqrt(as.numeric(t(cvec) %*% V %*% cvec)): the contrast vector against the coefficients, and the contrast vector around the HC3 matrix. Equivalent: as.numeric(cvec %*% coef(fit)) and drop(crossprod(cvec, V %*% cvec)). A student who writes vcov(fit) in place of V gets 0.40 / 0.64 / 0.70 / 0.64 instead of 0.40 / 0.63 / 0.71 / 0.63 (the model-based and HC3 matrices nearly agree here; the handout asks for HC3 for consistency with Weeks 7 to 9, not because it matters in this fit). A student who puts X in place of cvec gets a non-conformable-arguments error: the contrast is a difference of two rows, not the two rows. |
| 2a | family = binomial() and validRows = folds. family = "binomial" (a string) also works. With gaussian() by mistake the super learner fits a linear-probability model to the exposure: its risk is 0.1826 instead of 0.1827, the scores run from 0.011 to 0.636, and the Step 3 slope is 3.381 instead of 3.380; the slip is nearly harmless here because no propensity score sits near 0 or 1, which is not a reason to make it. With validRows left out (or a fresh split(sample(...))), CV.SuperLearner() draws its own folds, so the two residuals come from different fold assignments; the estimator still runs and gives a nearby number, but the cross-fitting argument assumes one partition serves both nuisance models, and the handout says so (“the SAME fold assignment”). |
| 3a | y_til <- Y - cv_y$SL.predict, a_til <- A - ps (or A - cv_a$SL.predict), and lm(y_til ~ -1 + a_til) (equivalently ~ 0 + a_til or ~ a_til - 1). With an intercept left in, lm(y_til ~ a_til) gives an intercept of 0.007 and a slope of 3.3797: harmless, because both residuals are already centred, which is Robinson’s point. With the residuals reversed, lm(a_til ~ -1 + y_til) gives 0.011, a different quantity (the regression of the exposure residual on the outcome residual). Using cv_y$library.predict (a matrix, one column per candidate) in place of SL.predict fails on dimensions. |
Expected checkpoints
Verified 2026-11-17 on R 4.6.2 (SuperLearner 2.0.42, gam 1.22.7, earth 5.3.5, sandwich 3.1.2). The fold draw and both super learners are seeded (2026; seeds 1 and 2 for the two extra fold draws), and the learners themselves are deterministic, so these reproduce exactly under R’s default RNG.
setup: 1,566 rows x 8 columns
Step 1: age bands (n, mean change): 276 3.96 / 214 4.30 / 177 3.66 / 197 4.36 / 234 2.23 / 186 1.79 /
132 1.06 / 85 -1.10 / 53 -3.48 / 12 -8.75
knots: df 3 -> 36, 50; df 5 -> 32, 39, 47, 55; boundary knots 25, 74
basis df 3 = 3 columns (first row 0.0992 0.5732 -0.3448); coef(fit_ns3) 3.77 / -1.08 / -5.95 / -10.43
AIC: linear 10831.50 / df 3 10805.74 / df 5 10806.65 / knots 45, 60 10804.94
key-only: coef(fit_kn) 3.98 / -3.24 / -8.45 / -11.49 (same df, different columns)
key-only: second differences of the df-3 curve: 2.7e-15 beyond age 74 (a line), 0.018 between the knots
key-only: max |curve - curve| over ages 25-74: df3 vs knots 1.04 kg, df3 vs df5 1.63, linear vs df3 4.42
key-only: HC3 half-width (1.96 se) at ages 25 / 50 / 74: linear 0.69 / 0.43 / 1.03; df 3 1.06 / 0.64 / 2.24;
df 5 1.27 / 0.78 / 2.89; knots 0.94 / 0.56 / 2.90; fitted at 74: -1.89 / -6.31 / -7.94 / -7.35
contrast 65 vs 40 (HC3): linear -3.73 (0.40); df 3 -6.39 (0.63); df 5 -6.45 (0.71); knots -6.28 (0.63)
check: diff of two predictions, df 3 = -6.39364
key-only: the same with vcov(fit): se 0.40 / 0.64 / 0.70 / 0.64
key-only: 40 vs 25 (the flat stretch): linear -2.24 (0.24); df 3 +0.23 (0.74); df 5 +0.58 (0.76); knots -0.14 (0.64)
key-only: 74 vs 60 (the steep end): linear -2.09 (0.22); df 3 -5.89 (1.04); df 5 -8.04 (1.62); knots -7.17 (1.58)
qsmk, HC3: linear confounders 3.338 (0.476); df-3 spline confounders 3.433 (0.473)
key-only: adjusted AIC 10733.8 -> 10704.4; F-test of the 8 spline columns in the outcome model F = 5.7, p = 3.6e-07;
in a linear model of qsmk on W: F = 2.95, p = 0.0028, R2 0.0485 -> 0.0628
key-only: delta = -0.0944 = (3.338 - 3.433) exactly
key-only: mean age 42.8 (kept smoking) vs 46.2 (quit); share who quit by age band 25-35 .208, 35-45 .246,
45-55 .252, 55-65 .373, 65-75 .338
Step 2: n 1566, quit 403, mean change 2.64; key-only: var(Y) 62.09 (sd 7.88), var(A) 0.1912
library: SL.mean SL.glm SL.gam_1 SL.gam_2 SL.gam_3 SL.earth; fold sizes 314 313 313 313 313
outcome SL: risk 56.423 (se 3.44); Discrete SL 56.460; mean 62.149; glm 57.406; gam_1 56.353; gam_2 56.385;
gam_3 56.513; earth 56.701
weights (averaged): mean .030 / glm .020 / gam_1 .592 / gam_2 .095 / gam_3 .201 / earth .062
key-only: by fold, gam_1 carries 0.000 / 0.631 / 0.976 / 0.914 / 0.438 and gam_3 0.859 / 0.065 / 0 / 0.051 / 0.033;
the discrete SL picked gam_3 in fold 1 and gam_1 in folds 2-5; out-of-fold MSE 56.43 = 9.1% of var(Y) explained
PS SL: risk 0.18274 (se 0.0053); Discrete SL 0.18303; mean 0.19122; glm 0.18373; gam_1 0.18259; gam_2 0.18257;
gam_3 0.18283; earth 0.18741
weights (averaged): mean .133 / glm .076 / gam_1 .622 / gam_2 .072 / gam_3 .079 / earth .019
PS quantiles (0 / 5 / 50 / 95 / 100%): kept smoking .089 .130 .232 .402 .639; quit .087 .162 .269 .477 .686
key-only: 17 scores below 0.10, 6 above 0.60, none above 0.90; mean PS .245 / .288 by group, .256 overall (.257 quit)
key-only: discrete SL picked gam_1 in four folds and glm in fold 3
key-only: family = gaussian() by mistake: risk 0.1826, scores 0.011 to 0.636, slope 3.381
Step 3: residual means 0.0102 / 0.0009; slope 3.37971, HC3 SE 0.47015, t 7.19, p 1e-12; by hand 3.379712
comparison: linear 3.338 (0.476) / spline 3.433 (0.473) / super learner R-on-R 3.380 (0.470)
key-only: model-based SE 0.4359; 95% Wald interval 2.46 to 4.30; sd(y_til) 7.514, sd(a_til) 0.4276, cor 0.192
key-only: with an intercept 0.0071 / 3.3797; residuals reversed 0.0109
key-only: WITHOUT cross-fitting (SuperLearner() once on all the data): sd 7.381 / 0.4225; slope 3.363 (HC3 0.467);
in-sample MSE 54.44 vs 56.43 out of fold; full-data weights outcome .012 0 .534 0 .102 .352, exposure .124 0 .409 0 .468 0
seeds: seeds 1, 2 -> 3.358 (HC3 0.470), 3.430 (0.469); with seed 2026: median 3.380, SD of the three 0.037 = 8% of the SE
key-only: adjusted SE sqrt(median(se^2 + (est - median)^2)) = 0.4707 vs 0.4701
Model answers
Any phrasing that names the right mechanism is fine. Students should quote the numbers.
Q1. df counts the columns the spline adds to the design matrix. For ns() without its own intercept column that is the number of interior knots plus one (the notes’ table: “df (= interior knots + 1)”), so df = 3 buys two interior knots and df = 5 buys four, and ns() puts them at quantiles of age (the tertiles 36 and 50; the quintiles 32, 39, 47, 55) unless we hand it knots =, in which case df is implied by the count: two knots at 45 and 60 is df = 3 again. The two df = 3 fits span different three-dimensional families of curves, bent at different ages, so their coefficients are weights on different columns and cannot be compared (3.77 / −1.08 / −5.95 / −10.43 against 3.98 / −3.24 / −8.45 / −11.49); their predictions can be, and they nearly coincide: the two curves are never more than 1.04 kg apart over ages 25 to 74, the contrasts are −6.39 and −6.28, the AICs 0.8 apart. That is the reason to report a contrast of predictions: the quantity of interest, the expected difference between two ages, is a linear combination of all the coefficients, \(c^{\intercal}\hat\beta\), with variance \(c^{\intercal}Vc\), and no single coefficient of a spline answers a question (the column ns(age, 3)2 is a bent function of age with no units anyone wants). Contrasts also make fits with different bases comparable, which coefficients never are. What keeps the curve from curling at the ends is the constraint that gives the natural spline its name: beyond the boundary knots (25 and 74, the range of age) the second and third derivatives are forced to zero, so the two end pieces are straight lines. The key checks it: the second differences of the df = 3 predictions over ages 74 to 90 are \(2.7 \times 10^{-15}\), zero to machine precision, against 0.018 between the knots. The data are thin there (12 people over 70): the HC3 half-width at 74 is 2.2 to 2.9 kg against 0.6 to 0.8 at 50, so the uncertainty shows the thinness, but a cubic left free in the tails (the B-spline of the notes’ figure) would have chased those twelve points and the natural spline cannot.
Q2. The age contrast illustrates the second perspective: in the crude model age sits in the exposure’s chair, the shape of its relation with weight change is the thing reported, and the straight line reports a curve that is not there. It averages the flat stretch and the fall into one slope, so it understates the 65-versus-40 drop by 2.7 kg and, on the flat stretch, invents a decline: from 25 to 40 the line says −2.24 kg (0.24) where the three splines say +0.23, +0.58, and −0.14 (SEs 0.6 to 0.8), nothing. At the steep end it does the reverse (74 versus 60: −2.09 against −5.89 to −8.04). For a mis-modelled confounder to bias the exposure coefficient, the leftover \(g(C) - \gamma C\) must be correlated with the exposure given the other terms; in the notes’ words, \(X\) must depend on \(C\), and more exactly on the nonlinear part of \(C\), since the linear part is already in the model. The bias \(\delta\) is the coefficient on \(X\) in a regression of the leftover on \(X\) and \(C\); the key computes it (regress the spline fit’s nuisance part on qsmk and the linear confounders) and gets −0.0944, which is exactly 3.338 − 3.433. Both ingredients are present here, each modestly: the eight spline columns improve the outcome fit (F = 5.7, p < 10⁻⁶; AIC 10733.8 to 10704.4) and they are associated with quitting (F = 2.95, p = 0.003; the share who quit rises from 0.21 under 35 to 0.37 at 55 to 65, roughly where the weight-change curve bends), but the bias is a product of the two and the product is a tenth of a kilogram, a fifth of the standard error. So the small move tells us only the second thing: in these data the bends in age and the other three confounders were nearly orthogonal to quitting, so the leftover had almost nowhere to go. It does not certify the linear model in general. The notes’ simulation shows the same mechanism producing a bias that no sample size removes when the exposure depends strongly on the curved part of the confounder; whether it does is a property of the population, and we did not know it was small here until we fit the spline, which is the argument for fitting it.
Q3. The weights are the coefficients of a regression of the outcome on the level-one data, the matrix of held-out predictions with one column per candidate, fitted by non-negative least squares with no intercept and normalized to sum to one (method.NNLS; the notes’ steps 4 and 5). They answer “which convex combination of the candidates’ out-of-sample predictions tracks the outcome best.” Because the handout cross-fits, the weights are computed inside each of the five outer training sets, by the inner five-fold cross-validation, and the printed vector is their average; the per-fold weights (key-only) swing from 0.98 on the df-2 GAM in fold 3 to 0.86 on the df-5 GAM in fold 1, which says the ranking among the GAMs is a near tie decided by which observations share a fold. The discrete super learner reports the single candidate with the smallest cross-validated risk in each training set: here the df-2 GAM in four folds and the df-5 GAM in one, and its own cross-validated risk is the Discrete SL row of the summary, 56.46, a little above the super learner’s 56.42 and above the df-2 GAM’s own 56.35 (the fold-1 pick was wrong for fold 1’s validation set). The super learner is a convex combination, so it can only beat its best candidate when the candidates err in different places; these six all estimate the same smooth function of six covariates and disagree only in details, so their held-out predictions are nearly collinear and no mixture of them can do much better than the best one (the theory says it does asymptotically as well as the best candidate, up to a term of order log(number of candidates)/n, and better only when the mixture adds something). The spread the candidates are competing over is also tiny next to the noise: the flexible candidates sit within 0.35 kg² of one another on a risk of 56, of which almost all is irreducible (the super learner explains 9.1% of the variance of weight change, 62.1 kg²), and each fold’s risk estimate carries a standard error of 3.4. SL.mean earns its place three ways. Its risk is the variance of \(Y\) (62.15), the floor that lets every other number be read as a share of variance explained. It is the safe fallback when the flexible candidates overfit, which is why a library should always contain the simple candidates (Phillips et al. 2023); here they were not needed, and the weight says so. And a small weight on the mean shrinks the combined prediction toward the grand mean, a mild regularization. In the propensity model it took 0.13, a sign that the exposure is harder to predict (the logistic model removes only 4% of the variance of qsmk) and the flexible candidates noisier there.
Q4. A flexible fit predicts the observations it was trained on too well: it has absorbed some of each row’s own noise into its prediction, so the in-sample residual is too small for exactly the rows whose noise it absorbed, and the prediction for row \(i\) is no longer independent of row \(i\)’s error. Robinson’s argument for the final regression needs the nuisance predictions to behave like fixed functions of \(W\), uncorrelated with the residual noise; out-of-fold predictions are, because the fit that produced them never saw the row. Smaller residuals are therefore not better residuals; they are the fit’s optimism, and feeding them to the least squares step carries the optimism into \(\hat\psi\) (the own-observation or overfitting bias that Chernozhukov et al. 2018 and Zivich and Breskin 2021 document; cross-fitting is the remedy). The key runs the comparison: with each super learner fit once to all the data, the in-sample mean squared error is 54.4 against 56.4 out of fold, the residual SDs shrink (7.38 against 7.51; 0.4225 against 0.4276), and the slope is 3.363 (0.467) against 3.380 (0.470). The difference is small here because the library is tame (GAMs with at most five degrees of freedom on six covariates cannot memorize 1,566 people); with trees or boosting in the library it can be large, and the recipe does not depend on knowing in advance which case we are in. The range of propensity scores speaks to positivity: every confounder pattern that produced quitters also produced non-quitters, with probabilities bounded away from 0 and 1 (17 of 1,566 scores below 0.10, none above 0.90, the same 0.09-to-0.69 range in both groups). A score near 0 or 1 would mark a stratum of \(W\) in which one exposure level was (nearly) impossible, where no estimator can learn the contrast from the data and where inverse probability weights explode. It says nothing about exchangeability: a propensity score is built from the measured confounders and cannot see an unmeasured one. Why the exposure model must be right: the slope converges to \(\mathrm{Cov}(\tilde Y, \tilde A)/\mathrm{Var}(\tilde A)\) with \(\tilde A = A - \hat e(W)\). If \(\hat e\) is right, the error of the outcome model, \(\hat m - m\), is a function of \(W\) alone and is uncorrelated with \(A - e(W)\) (its conditional mean given \(W\) is zero), so it drops out of the numerator and the estimator is consistent even with a poor outcome model, only less precise. If \(\hat e\) is wrong, the exposure residual carries a leftover \(e(W) - \hat e(W)\) that is a function of \(W\): the denominator gains its variance, which is noise as far as the signal is concerned, and the numerator gains only the product of the two nuisance errors, so the estimate attenuates toward the null by the share of the residual exposure variance that is exposure-model error (the notes’ section 5.3, the same arithmetic as measurement error in a regressor). AIPW combines the two nuisance functions in an estimating equation whose bias is the product of the two errors in both places, so either model being right suffices; the partially linear model’s double robustness is only in rates (two moderately accurate fits leave a negligible product), not in specification. In exchange it never inverts a probability, so scores near 0 or 1 do not produce extreme weights.
Q5. Not a different number but a different warrant for it. The three agree because, in these data, the linear adjustment happened to be adequate; before Step 2 we had assumed that, and now we have checked it with a procedure that could have said otherwise (had the super learner put its weight on MARS or the df-5 GAM and moved the estimate, we would have learned that the straight lines were hiding confounding, as the notes’ simulation shows they can). We also gained a standard error that remains valid when the data choose the functional form, at no cost in precision (0.470 against 0.476), and insurance for the next dataset, where the linear model may not be adequate and where we would have had no way to know. The two conditions: identification, exchangeability given \(W\) (with positivity, checked by eye in Step 2, and consistency), the Week 8 assumptions, which the super learner does not supply (it adjusts for what it is given; here six confounders, where Hernán and Robins and the notes use nine, so the lab’s \(\psi\) is identified under a stronger assumption than the book’s, a teaching simplification worth saying aloud); and the model, one constant additive effect, \(E(Y \mid A, W) = \psi A + g(W)\) with no exposure-confounder interaction. If the effect varies across people along a covariate that also drives exposure, residual-on-residual regression converges to a weighted average of individual effects with weights \(\{A - \pi(W)\}^2\), largest near a propensity score of one half, and not to the average treatment effect (section 5.3). The HC3 standard error of ror is usable because its target is a single parametric number and Robinson’s theorem covers it: when both nuisance fits converge faster than \(n^{-1/4}\) and the residuals are cross-fitted, the nuisance errors enter \(\hat\psi\) only through their product, which is negligible on the \(\sqrt{n}\) scale, so the final step behaves like an ordinary regression of one fixed residual on another, with a limiting normal distribution centred on \(\psi\) and a sandwich variance that coincides with the influence-function variance of AIPW and TMLE (section 5.2). The unusable standard error of section 4.4 belonged to a different object: the fitted curve itself, a nonparametric function whose bias shrinks more slowly than its standard deviation, or a plug-in built on it; and, for the chosen-df spline, to an lm() that computes the standard error as if the degrees of freedom had been fixed in advance. Here the nuisance function has been projected out and the inferential object is the slope of the final regression. The model-based standard error that Robinson proposed is 0.436 here against HC3’s 0.470: the residual variance is not constant, and Lab 6’s lesson applies.
Q6. The spread comes from the random partition into folds. A different fold assignment changes which observations train the nuisance models for which, so every out-of-fold prediction changes a little, the two residuals change, and \(\hat\psi\) with them. It is randomness the estimator introduced, Monte Carlo rather than sampling variability, and the HC3 standard error does not contain it because the sandwich is computed for a fixed set of residuals: it estimates how \(\hat\psi\) would vary across repeated samples from the population, with the fold assignment held as given. Both kinds of variability are real. Here the second is small, an SD of 0.037 across the three draws, 8% of the standard error (a sixth in the notes, with nine confounders and a different library); with smaller samples, larger libraries, or tree learners it can approach the standard error (Naimi et al. 2024, who found average treatment effect estimates that changed materially with the seed). A paper should state the number of folds, the library and the weighting method, the software and package versions, the seed, and whether the reported number is one fold draw or an aggregate over several; and it should report the spread across draws, as the handout’s optional chunk does. The reasonable repair is to repeat the whole cross-fitting over \(S\) fold assignments and combine: Chernozhukov et al. (2018) take the median of the \(S\) estimates and report a variance that adds the between-draw spread, \(\tilde\sigma^2 = \mathrm{median}_s\{\hat\sigma_s^2 + (\hat\psi_s - \tilde\psi)^2\}\), so that the fold-draw randomness that the single-draw sandwich omits is put back. On the three draws here the median is 3.380 and the adjusted standard error 0.4707 against 0.4701, nothing, which is the honest finding for these data and would not have been known without doing it. (Increasing \(K\), so that each nuisance model is fit on a larger share of the data, also reduces the draw-to-draw spread; it does not remove it.)
Understanding notes
- Seeds and versions.
set.seed(2026)runs before the fold draw and again before eachCV.SuperLearner()call, because the inner cross-validation draws its own folds;SL.gam,SL.earth,SL.glm, andSL.meanare deterministic given the folds, so the fold draws are the only randomness. Checkpoints verified on R 4.6.2 with SuperLearner 2.0.42, gam 1.22.7, earth 5.3.5. If a student’s numbers differ in the third decimal, suspect a package version; if in the first, a skippedset.seed(), a re-runsample(), or a droppedinnerCvControl(the default innerVis 10, which changes every number and doubles the time). inner <- rep(list(list(V = 5)), K). Passing the one-elementlist(list(V = 5))works too but makes the package warn “Only a single innerCvControl is given, will be replicated across all cross-validation split calls” on every run; pre-replicating it is identical in effect and silent.method.NNLSis the package default for both families; the handout writes it out so the two calls read alike.method.NNloglikweights a binomial super learner by held-out log-likelihood instead; not needed here.SL.gamandmgcv.SL.gamfitsgam::gam()with a smoothing splines(x, deg.gam)on every column with more than four distinct values and the rest linear, sosexandraceenter linearly anddeg.gamis the spline’s degrees of freedom, not a polynomial degree. Ifmgcvis attached (bylibrary(mgcv), or byggplot2forgeom_smooth(method = "gam")),SL.gamwarns about the maskedgam()on every fit; restart R.- The
Super LearnerandDiscrete SLrows ofsummary()are the cross-validated risks of the whole procedures (weights or pick estimated inside each training set, scored on the held-out fold); the candidate rows are each candidate’s own risk on the same folds. The super learner need not beat its best candidate in a finite sample, and here it does not (56.42 against 56.35). - Six confounders, not nine. The handout’s \(W\) is age, sex, race, smoking intensity, years smoked, and baseline weight. Hernán and Robins (and the notes’ section 5.2) also adjust for education, exercise, and physical activity. That is why the notes’ estimate is 3.34 (0.47) and the lab’s 3.38 (0.470), and why the lab’s \(\psi\) is identified under a stronger exchangeability assumption; Q5’s model answer says so. A student who adds the three factors (as indicator columns, since
SL.gamneeds numeric columns) will land near the notes’ number. - A different library from the notes. The notes tune a hand-written natural-spline learner (
SL.nsadd); the lab uses the package’s built-inSL.gamat three degrees of freedom instead, at the instructor’s request (built-in learners only, so that students can read the code).SL.polymarswas tried and dropped: its binomial branch prints C-level messages on every fold and takes forty seconds. Students who compare the two libraries will see different weights for the same reason. - The δ identity (key-only, Step 1) is exact because the spline model’s column space contains the linear model’s:
ns()plus an intercept spans every linear function of its variable, so the linear fit is a projection of the spline fit and the difference between the twoqsmkcoefficients is exactly the coefficient ofqsmkin a regression of the spline fit’s nuisance part onqsmkand the linear confounders, which is the notes’ definition of \(\delta\). It is a good board demonstration of section 2 if the lab runs long. - The figure clips the y-axis at ±20 kg (
coord_cartesian); 51 of the 1,566 changes lie beyond it (the range is −41 to +49). Clipping withcoord_cartesian()only changes the view;ylim()would drop the points from the fit of any smoother, which is the usual trap. - Runtime. About ten seconds for the whole script on a laptop; the two super learners take about two seconds each and the optional seeds chunk about four. Nothing is slow enough to trim.
- What this lab does not do. It does not fit AIPW or TMLE (the notes mention them as the estimators that drop the constant-effect restriction), does not put effect modification back (\(\tilde A \times V\) terms), does not treat a continuous exposure, and does not show
bs()orrcs()(the notes’ table of spline families). It never bootstraps: the house rule of section 4.4 is not needed because residual-on-residual regression has a valid sandwich, which is the point of Step 3.