pacman::p_load(tidyverse, splines, SuperLearner, gam, earth, sandwich, lmtest)
nhefs <- read_csv("data/nhefs.csv", show_col_types = FALSE) |>
filter(!is.na(wt82_71)) |>
select(wt82_71, qsmk, age, sex, race, smokeintensity, smokeyrs, wt71)
dim(nhefs)Lab 7 — Fit a Partially Linear Model
EPID 785R · Regression and Study Design
Introduction
The lecture notes on flexible regression followed one object, the nuisance function, from the wrong scale to the right estimator. A continuous confounder entered as a straight line leaves confounding in the exposure coefficient that no sample size removes; a continuous exposure entered as a straight line reports a curve that is not there. Splines repair both by adding columns to the design matrix. Cross-validation chooses how many, the super learner weights the candidates, and what the data choose the usual standard error no longer covers. The partially linear model gets the standard error back: the super learner builds the nuisance functions, and a regression of one residual on another turns them into an effect estimate whose standard error comes off the sandwich of Week 7.
In this lab we walk that path once, on the NHEFS data of Weeks 5 to 9: the effect of quitting smoking on weight change between 1971 and 1982, adjusting for a few confounders. We will:
- fit the same natural cubic spline three ways with
ns(), at two settings of the degrees of freedom and at knots we place ourselves, read the fits through predictions and one contrast rather than through coefficients, and then move the spline to its usual job, a nuisance term beside the one coefficient we care about; - let a super learner choose the nuisance models, one for the outcome and one for the propensity score, each cross-fitted;
- combine the two in residual-on-residual regression, the partially linear model’s estimator, and compare its answer and its standard error with the parametric fits of Step 1.
Deliverables (yours to keep, nothing is submitted): this document rendered to HTML with your answers in the answer blocks. Three chunks contain TODOs with ??? placeholders (they error until fixed). When they all run, flip eval: false to eval: true in the YAML header and re-render. The whole document runs in well under a minute.
Background for this lab: the lecture notes Flexible Regression: Splines and the Super Learner. Step 1 follows sections 3.4 and 3.5 (reading a spline fit; the spline functions in R), Step 2 follows section 4 (the super learner), and Step 3 follows section 5 (the partially linear model). The question, and the data, are the running example of Hernán and Robins (2020).
1 Before you start (~3 min)
- Open
lab7.Rprojin RStudio. - The data ship with the lab:
data/nhefs.csvis the NHEFS file distributed with Hernán and Robins’ Causal Inference: What If (also at https://miguelhernan.org/s/nhefs.csv). - Packages: the tidyverse,
splines(ships with R),sandwichandlmtest(Labs 2 to 6), and three new ones thatpacmaninstalls if needed:SuperLearner,gam, andearth.
The chunk below loads the packages and the data and keeps the 1,566 people with a weight recorded at both visits, the spine of Weeks 5 to 9: weight change (wt82_71, in kg), quitting smoking between the visits (qsmk), and six confounders.
Checkpoint: 1,566 rows and 8 columns.
2 Step 1 — Three natural splines, and one contrast (~15 min)
Weight change against age first, with nothing else in the model. The table (provided) gives the mean change in five-year age bands.
nhefs |>
mutate(age_band = cut(age, breaks = seq(25, 75, by = 5), include.lowest = TRUE)) |>
group_by(age_band) |>
summarise(n = n(), mean_change = round(mean(wt82_71), 2))Checkpoints: the mean change sits near 4 kg from 25 into the mid-40s (3.96, 4.30, 3.66, 4.36), falls through the 50s (2.23, 1.79, 1.06), and drops past 60 (−1.10, −3.48, −8.75, the last on 12 people). A straight line cannot stay flat for twenty years and then bend.
2.1 The same spline, set three ways
ns(age, df = k) spends k columns on age. With no knots given, it puts k − 1 interior knots at quantiles of age and the boundary knots at the range of age; beyond the boundary knots the curve is a straight line (section 3.5). The fourth fit below places the knots where the table says the curve bends: two interior knots, so df = 3 again, the same number of columns as the second fit, in different places.
fit_lin <- lm(wt82_71 ~ age, data = nhefs) # the reference
fit_ns3 <- lm(wt82_71 ~ ns(age, df = 3), data = nhefs) # 2 interior knots, at the tertiles
fit_ns5 <- lm(wt82_71 ~ ns(age, df = 5), data = nhefs) # 4 interior knots, at the quintiles
fit_kn <- lm(wt82_71 ~ ns(age, knots = c(45, 60)), data = nhefs) # knots we chose
attr(ns(nhefs$age, df = 3), "knots") # where ns() put them
attr(ns(nhefs$age, df = 5), "knots")
head(ns(nhefs$age, df = 3), 3) # the basis: three columns of age
coef(fit_ns3) # weights on those columns
AIC(fit_lin, fit_ns3, fit_ns5, fit_kn)Checkpoints: with df = 3 the knots land at ages 36 and 50; with df = 5 at 32, 39, 47, and 55. The basis is three columns, each a smooth function of age, and the coefficients are weights on them (3.77, −1.08, −5.95, −10.43) that say nothing on their own. AIC: 10831.5 for the line, 10805.7 (df 3), 10806.7 (df 5), and 10804.9 (our knots). Every spline beats the line by about 25 units; the three splines sit within two units of one another.
2.2 Reading the fits through predictions
A spline fit returns predictions. The function X_at() (provided) evaluates a fit’s design matrix on new data; the knots travel with the fit, so ns() is rebuilt at the same places, which is what predict() does under the hood. From the design matrix on a grid of ages we get each curve and its HC3 band.
X_at <- function(fit, newdata) model.matrix(delete.response(terms(fit)), data = newdata)
grid <- data.frame(age = 25:74)
curve_of <- function(fit, label) {
X <- X_at(fit, grid)
V <- vcovHC(fit, type = "HC3") # Week 7's sandwich
yhat <- as.vector(X %*% coef(fit))
se <- sqrt(diag(X %*% V %*% t(X)))
tibble(fit = label, age = grid$age, yhat, lower = yhat - 1.96 * se, upper = yhat + 1.96 * se)
}
curves <- bind_rows(curve_of(fit_lin, "linear"),
curve_of(fit_ns3, "ns, df = 3"),
curve_of(fit_ns5, "ns, df = 5"),
curve_of(fit_kn, "ns, knots at 45 and 60")) |>
mutate(fit = factor(fit, levels = unique(fit)))
knots <- tibble(fit = factor(c(rep("ns, df = 3", 2), rep("ns, df = 5", 4),
rep("ns, knots at 45 and 60", 2)), levels = levels(curves$fit)),
knot = c(attr(ns(nhefs$age, df = 3), "knots"),
attr(ns(nhefs$age, df = 5), "knots"), 45, 60))
ggplot() +
geom_point(data = nhefs, aes(age, wt82_71), colour = "grey60", alpha = .2, size = .6) +
geom_vline(data = knots, aes(xintercept = knot), linetype = "dotted") +
geom_ribbon(data = curves, aes(age, ymin = lower, ymax = upper), fill = "#D55E00", alpha = .25) +
geom_line(data = curves, aes(age, yhat), colour = "#D55E00", linewidth = 1) +
facet_wrap(~ fit) +
coord_cartesian(ylim = c(-20, 20)) + # 51 of the 1,566 changes lie beyond 20 kg
labs(x = "Age in 1971 (years)", y = "Weight change, 1971 to 1982 (kg)") +
theme_classic()Then one contrast: the expected weight change at 65 against 40. With \(c = x(65) - x(40)\), the difference between two rows of the design matrix, the contrast is \(c^{\intercal}\hat\beta\) and its variance is \(c^{\intercal} V c\), with the HC3 sandwich as \(V\) (section 3.4; the contrast vector of Week 8’s bonus material). Complete TODO (1a).
contrast_65_40 <- function(fit) {
X <- X_at(fit, data.frame(age = c(65, 40)))
cvec <- X[1, ] - X[2, ]
V <- vcovHC(fit, type = "HC3")
# TODO (1a): the contrast c'beta, and its standard error sqrt(c'Vc)
c(estimate = sum(??? * coef(fit)),
se = sqrt(as.numeric(t(cvec) %*% ??? %*% cvec)))
}
round(rbind(linear = contrast_65_40(fit_lin),
ns_df3 = contrast_65_40(fit_ns3),
ns_df5 = contrast_65_40(fit_ns5),
knots = contrast_65_40(fit_kn)), 2)
diff(predict(fit_ns3, newdata = data.frame(age = c(40, 65)))) # check: a difference of two predictionsCheckpoints: the line says a 65-year-old gains 3.73 kg less than a 40-year-old (SE 0.40); the three splines say 6.39 (0.63), 6.45 (0.71), and 6.28 (0.63) kg less. The check reproduces the df = 3 estimate, −6.39.
2.3 The spline as a nuisance term
For the causal question the spline does not model the exposure; it models the confounders, so that the one coefficient we care about is not biased by a confounder forced into a straight line. The target stays a single number; only the nuisance part of the model bends (provided).
fit_adj_lin <- lm(wt82_71 ~ qsmk + age + sex + race + smokeintensity + smokeyrs + wt71,
data = nhefs)
fit_adj_ns <- lm(wt82_71 ~ qsmk + ns(age, df = 3) + sex + race + ns(smokeintensity, df = 3) +
ns(smokeyrs, df = 3) + ns(wt71, df = 3), data = nhefs)
hc3 <- function(fit, term) coeftest(fit, vcov = vcovHC(fit, type = "HC3"))[term, 1:2]
round(rbind(linear_confounders = hc3(fit_adj_lin, "qsmk"),
spline_confounders = hc3(fit_adj_ns, "qsmk")), 3)Checkpoints: quitting goes with 3.34 kg more weight gain (HC3 SE 0.476) when every confounder enters linearly, and 3.43 kg (0.473) when the four continuous confounders enter as df = 3 splines.
Q1. The df = 3 spline and the spline with knots at 45 and 60 spend the same number of columns on age, in different places. Their coefficients differ, yet their contrasts (−6.39 and −6.28) and their AICs (10805.7 and 10804.9) nearly agree. What does df count in ns(), and how many interior knots does df = 5 buy? Why do we report a contrast of predictions rather than a coefficient? And what feature of a natural spline keeps the curve from curling at ages 25 and 74, where the data are thin?
YOUR ANSWER:
Q2. The line says 65-year-olds gain 3.7 kg less than 40-year-olds; every spline says about 6.4 kg less. Section 2 of the notes gave two perspectives on a mis-specified curvilinear relation: a confounder entered linearly leaves residual confounding in the exposure coefficient, and an exposure entered linearly reports a curve that is not there. Which perspective does the contrast illustrate? Then, in the adjusted models, the qsmk coefficient moved only from 3.34 to 3.43 when age and the other continuous confounders went from lines to splines. What has to be true of a confounder’s relation with the exposure for its mis-modelled relation with the outcome to bias the exposure coefficient? Does the small move tell us the linear model was adequate, or only that this bend did not matter much here?
YOUR ANSWER:
3 Step 2 — The super learner chooses the nuisance models (~12 min)
Step 1 chose df = 3 for every continuous confounder; nothing in the data was asked. Section 4 of the notes replaced that choice with cross-validation: fit each candidate on part of the data, score it on the rest, and weight the candidates by non-negative least squares on their held-out predictions. Here the candidates are the two adjustment models of Step 1 with the exposure term left out, the spline one at three settings, and two more.
The outcome, the exposure, and the confounders, as the package wants them: a vector, a vector, and a data frame (provided).
Y <- nhefs$wt82_71
A <- nhefs$qsmk
W <- nhefs |> select(age, sex, race, smokeintensity, smokeyrs, wt71) |> as.data.frame()
n <- nrow(W)
c(n = n, quit = sum(A), mean_change = round(mean(Y), 2))3.1 The library
To the SuperLearner package a learner is a function that fits and then predicts, and it ships with dozens of them (listWrappers() prints the menu). We use four. SL.mean and SL.glm are the simple candidates, the sample mean and every column linear (Step 1’s linear adjustment, without the exposure term); they stay in the library so the super learner can fall back on them when the flexible candidates overfit. SL.gam is Step 1’s spline adjustment, again without the exposure: a smoothing spline on every continuous column of W with the binary columns linear, and its df (deg.gam) is the tuning knob; create.Learner() makes one copy per df so that the data can choose it. SL.earth (multivariate adaptive regression splines) chooses its own knots as it fits.
gam_df <- create.Learner("SL.gam", tune = list(deg.gam = c(2, 3, 5)))
lib <- c("SL.mean", "SL.glm", gam_df$names, "SL.earth")
lib # SL.gam_1, _2, _3 are df 2, 3, 5(SL.gam warns on every fit if the mgcv package is also loaded, which ggplot2 does for geom_smooth(method = "gam"); restart R if that happens.)
3.2 The outcome model, cross-fitted
Two layers of cross-validation (section 5.2). The outer loop of CV.SuperLearner(), with \(K = 5\) folds, is the cross-fitting: each fold’s predictions come from a super learner that never saw it. Inside each training set, the super learner’s own five-fold cross-validation picks the weights. One fold assignment, drawn once, will serve both nuisance models, so that their residuals line up on the same folds.
K <- 5
set.seed(2026)
folds <- split(sample(1:n), rep(1:K, length.out = n))
inner <- rep(list(list(V = 5)), K) # the inner 5-fold CV, one entry per outer fold
set.seed(2026)
cv_y <- CV.SuperLearner(Y = Y, X = W, family = gaussian(),
SL.library = lib, method = "method.NNLS",
cvControl = list(V = K, validRows = folds),
innerCvControl = inner)
summary(cv_y) # CV risk (MSE) of the super learner and of each candidate
round(colMeans(coef(cv_y)), 3) # the weights, averaged over the K outer fitsCheckpoints: the super learner’s cross-validated risk is 56.42 (mean squared error, in kg²); the candidates run from 56.35 (SL.gam_1, df 2) to 62.15 (the mean). The weights, averaged over the five outer fits, put 0.59 on the df-2 GAM, 0.20 on the df-5 GAM, 0.10 on the df-3 GAM, 0.06 on MARS, and 0.03 and 0.02 on the mean and the linear model.
3.3 The propensity score, cross-fitted on the same folds
The same call with the exposure as the outcome and the binomial family (section 5.2: \(E(A \mid W)\) is the propensity score). Complete TODO (2a): the family, and the folds.
set.seed(2026)
# TODO (2a): the binomial family, and the SAME fold assignment as the outcome model
cv_a <- CV.SuperLearner(Y = A, X = W, family = ???,
SL.library = lib, method = "method.NNLS",
cvControl = list(V = K, validRows = ???),
innerCvControl = inner)
summary(cv_a)
round(colMeans(coef(cv_a)), 3)
ps <- cv_a$SL.predict # out-of-fold P(A = 1 | W), in row order
round(do.call(rbind, tapply(ps, A, quantile, c(0, .05, .5, .95, 1))), 3)Checkpoints: risk 0.1827 for the super learner, against 0.1912 for the mean and 0.1837 for the logistic model; the weights put 0.62 on the df-2 GAM and 0.13 on the mean. The out-of-fold propensity scores run from 0.09 to 0.64 among those who kept smoking and from 0.09 to 0.69 among those who quit, with medians 0.23 and 0.27: the two groups cover the same range, and no one sits near 0 or 1.
Q3. The outcome super learner put 0.59 of its weight on the df-2 GAM, 0.20 on the df-5 GAM, and almost nothing on the linear model, and its cross-validated risk (56.42) was not below that of its best candidate (56.35). Say in plain words what the weights are and how they were obtained (section 4.3). What would the discrete super learner have reported, and why can the super learner not beat its best candidate by much when the candidates’ risks sit within a few units of one another? Why keep SL.mean in a library in which it earns 3% of the weight?
YOUR ANSWER:
Q4. Each person’s propensity score in ps came from a super learner fit to the other four folds. Why does the recipe insist on that, when a fit to all five folds would produce smaller residuals (section 5.2)? The scores run from 0.09 to 0.69 in both exposure groups: which identification assumption does that range speak to? And why does residual-on-residual regression need the exposure model to be right, when AIPW does not (section 5.3)?
YOUR ANSWER:
4 Step 3 — Residual-on-residual regression (~10 min)
Robinson’s trick (section 5.1): subtract \(E(Y \mid W)\) from \(Y\) and \(E(A \mid W)\) from \(A\), and the nuisance function \(g(W)\) drops out of the partially linear model \(Y = \psi A + g(W) + \varepsilon\). What is left is a regression through the origin of the one residual on the other, with the same \(\psi\). Both nuisance predictions are the out-of-fold ones from Step 2. Complete TODO (3a): the two residuals and the no-intercept regression.
# TODO (3a): each variable minus its cross-fitted prediction, then the slope through the origin
y_til <- ??? - cv_y$SL.predict # the outcome with E[Y | W] removed
a_til <- A - ??? # the exposure with E[A | W] removed
round(c(mean_y_til = mean(y_til), mean_a_til = mean(a_til)), 4) # both near zero
ror <- lm(y_til ~ ??? + a_til)
coeftest(ror, vcov = vcovHC(ror, type = "HC3"))
sum(a_til * y_til) / sum(a_til^2) # by hand: a no-intercept slope is sum(xy) / sum(x^2)
round(rbind(linear_confounders = hc3(fit_adj_lin, "qsmk"),
spline_confounders = hc3(fit_adj_ns, "qsmk"),
super_learner_ror = hc3(ror, "a_til")), 3)Checkpoints: both residuals have means within 0.01 of zero. The slope is 3.380 kg with an HC3 standard error of 0.470, and the by-hand ratio reproduces it. Beside the parametric fits of Step 1: 3.338 (0.476) with linear confounders, 3.433 (0.473) with splines, 3.380 (0.470) with the super learner.
4.1 The fold draw is random too (optional)
Cross-fitting adds a random element of its own: another fold assignment gives a slightly different estimate. Two more seeds (provided; about ten seconds).
ror_with_seed <- function(seed) {
set.seed(seed)
f <- split(sample(1:n), rep(1:K, length.out = n))
cy <- CV.SuperLearner(Y = Y, X = W, family = gaussian(), SL.library = lib,
cvControl = list(V = K, validRows = f), innerCvControl = inner)
ca <- CV.SuperLearner(Y = A, X = W, family = binomial(), SL.library = lib,
cvControl = list(V = K, validRows = f), innerCvControl = inner)
coef(lm(I(Y - cy$SL.predict) ~ -1 + I(A - ca$SL.predict)))[[1]]
}
round(sapply(c(1, 2), ror_with_seed), 3)Checkpoints: seeds 1 and 2 give 3.358 and 3.430, against 3.380 at seed 2026: a spread of a few hundredths of a kilogram, about a tenth of the standard error.
Q5. Three estimates of one quantity: 3.34 (0.476) with linear confounders, 3.43 (0.473) with splines, 3.38 (0.470) with the super learner. The number did not move. What did we gain? Then: under what two conditions, one about identification and one about the model, is \(\psi\) the average treatment effect (section 5.1)? The notes called the standard error that lm() prints for a cross-validated spline unusable (section 4.4); why is the HC3 standard error of ror usable?
YOUR ANSWER:
Q6. Three fold assignments gave 3.380, 3.358, and 3.430. Where does that spread come from, given that the data never changed, and why does it not appear in the HC3 standard error? What should a paper that reports this estimate state, and what would be a reasonable way to make the number less dependent on one draw (Naimi et al. 2024; Chernozhukov et al. 2018)?
YOUR ANSWER:
References
- Hernán MA, Robins JM. Causal Inference: What If. Chapman & Hall/CRC, 2020. (The data and the question.)
- Durrleman S, Simon R. Flexible Regression Models with Cubic Splines. Stat Med. 1989;8(5):551–561. (Natural cubic splines, Step 1.)
- Perperoglou A, Sauerbrei W, Abrahamowicz M, Schmid M. A Review of Spline Function Procedures in R. BMC Med Res Methodol. 2019;19:46. (
ns()and its relatives, Step 1.) - Harrell FE. Regression Modeling Strategies. Springer, 2001. (Knots at quantiles, Q1.)
- van der Laan MJ, Polley EC, Hubbard AE. Super Learner. Stat Appl Genet Mol Biol. 2007;6(1):Article 25. (Step 2.)
- Polley E, LeDell E, Kennedy C, van der Laan M. SuperLearner: Super Learner Prediction. R package. (The
SuperLearnerpackage, Step 2.) - Phillips RV, van der Laan MJ, Lee H, Gruber S. Practical Considerations for Specifying a Super Learner. Int J Epidemiol. 2023;52(4):1276–1285. (The library, Q3.)
- Robinson PM. Root-N-Consistent Semiparametric Regression. Econometrica. 1988;56(4):931–954. (Step 3.)
- Chernozhukov V, Chetverikov D, Demirer M, Duflo E, Hansen C, Newey W, Robins J. Double/Debiased Machine Learning for Treatment and Structural Parameters. Econometrics J. 2018;21(1):C1–C68. (Cross-fitting, Step 3 and Q6.)
- Zivich PN, Breskin A. Machine Learning for Causal Inference: On the Use of Cross-fit Estimators. Epidemiology. 2021;32(3):393–401. (Cross-fitting, Q4.)
- Naimi AI, Mishler AE, Kennedy EH. Challenges in Obtaining Valid Causal Effect Estimates with Machine Learning Algorithms. Am J Epidemiol. 2023;192(9):1536–1544. (Why a plug-in super learner loses its standard error, Q5.)
- Naimi AI, Yu Y-H, Bodnar LM. Pseudo-random Number Generator Influences on Average Treatment Effect Estimates Obtained with Machine Learning. Epidemiology. 2024;35(6):779–786. (Q6.)
- Naimi AI, Jin Q, Yu Y-H, Parisi SM, Bodnar LM. Residual-on-Residual Regression as a Tool for Effect Estimation in Observational Data. arXiv:2606.30976, 2026. (The estimator of Step 3, as presented in the notes.)