Module 2 · Homework 3

Worked answers with reasoning. References point to the lecture notes PDF and the notes pages. Questions 12–15 use hw3_heart.csv (297 patients; age, sex, cp, trestbps, chol, fbs, restecg, thalach, exang, oldpeak, slope, ca, thal, and class 0–4; no missing values).

← Back to notes index

Q1 — Why cross-validation is harder with survival data

Answer: The presence of censored outcomes requires special handling when estimating prediction error.

The CV machinery — single split, LOOCV, $k$-fold, generalized CV — is identical to the regression/classification case. What breaks is scoring the held-out fold: for a test subject with $X_i \le t$ and $\delta_i = 0$ you do not know whether $T_i \le t$ or $T_i > t$, so they can be counted as neither case nor control. The notes’ fix (§1.8.5) is the IPCW estimator — weight the subjects whose status is known by $1/\hat{G}(\cdot)$, where $\hat{G}$ is a Kaplan–Meier estimate of the censoring survival function, and let censored-before-$t$ subjects contribute only through $\hat{G}$. This is why the Video 24 LOOCV switched to the Brier score and the Video 26 CV loop used timeROC inside each fold.

Why the others are wrong:

• Proportional hazards may not hold — a model-diagnostic issue (§1.7), unrelated to how folds are scored.
• KM curves cannot be estimated within folds — false; KM is estimated within the test fold, to build the IPCW weights.
• Survival times are always normally distributed — false; they are positive and right-skewed (§1.6), and this would make CV easier, not harder.
• Hazard ratios cannot be compared across folds — false, and CV compares prediction accuracy (AUC, Brier), not coefficients.

Q2 — What the AUC represents

Answer: The probability that a randomly chosen positive case is ranked higher than a randomly chosen negative case.

Notes §1.8.2: $\text{AUC} = P(X > Y)$, with $X$ the score of a random $D^{+}$ subject and $Y$ the score of a random $D^{-}$ subject — the Mann–Whitney interpretation, estimated nonparametrically as $\frac{1}{mn}\sum_i\sum_j I(X_i > Y_j)$ over all case–control pairs. It is a single-number summary of accuracy across all thresholds $\theta$.

Why the others are wrong:

• “A perfect test has AUC $= 0.5$” — backwards: a perfect test has AUC $= 1$; $0.5$ is the diagonal, a coin flip.
• “Proportion of true positives among all positive predictions” — that is the PPV, $a/(a+b)$, and it depends on one threshold and on prevalence.
• “The error rate of the classifier” — misclassification at one threshold; AUC integrates over thresholds.
• “Difference between sensitivity and specificity” — not a defined accuracy measure; both are threshold-specific.

Q3 — Choosing a threshold on the ROC curve

Answer: F.

The axes are sensitivity ($y$) and $1 - $ specificity $=$ FPR ($x$). Apply the priorities in order:

1. Maximize sensitivity — only F and G sit at the top of the plot, sensitivity $= 1$. A–E are all lower and are eliminated.
2. Minimize FPR among the survivors — F is at $x \approx 0.67$, G at $x = 1$ (G calls everyone positive: sensitivity 1, specificity 0, the top-right corner of every ROC curve). F wins.

The trap is D: it is the point closest to the upper-left corner, which is the usual heuristic for an “optimal” $\theta$ (notes §1.8.2). But the question sets a lexicographic rule — sensitivity first, no trade-off allowed — and D’s sensitivity is below 1. As the notes say, “the optimal value depends heavily on the costs of each type of error”; here the stated costs say a false negative is unacceptable, so the upper-left heuristic does not apply.

Q4 — Which measures are “lower is better”?

Answer: Brier score and Mean squared error (MSE). Not AUC, PPV, or C-index.

The two checked items are the course’s two loss functions (§1.8.1): they measure error, so zero is the ideal. The other three measure accuracy or agreement, so one is the ideal.

Q5 — Which predictor to add, by incremental value

Answer: Age.

$IV(Z_b) = \text{AUC}{p(Z_b)} - \text{AUC}{p(-Z_b)}$. Age and breast biopsy tie at the largest value, $0.10$, so AUC alone does not decide. The tie-break is the reason incremental value exists in the first place (notes §1.8.3): the question is whether a predictor is worth the cost and burden of obtaining it. The notes’ own lists:

Same gain, essentially zero cost → age.

Why the others are wrong:

• Breast biopsy — identical $0.10$, but invasive and expensive; only justified if it added something age does not.
• Stool DNA test — smaller gain ($0.06$) and high cost.
• Cholesterol — negligible gain ($0.02$); a routine lab, but it adds almost nothing.
• All of these — the question asks for the single best addition, and “add everything” defeats the purpose of weighing cost against incremental value.

Q6 — How generalized cross-validation estimates test error

Answer: By repeatedly splitting the data into training and test sets at random, and averaging the test errors.

Notes §1.8.4, generalized CV: pick a training proportion $h$ (e.g. $0.75$), draw a random training set of size $h \cdot N$, fit on it, compute the test error $TE_m$ on the remainder; repeat $M$ times ($M = 100$ or $500$) with a fresh random draw each time — the test sets are not mutually exclusive — and average:

It is the single-split procedure repeated many times, which removes the single split’s high variance while letting you choose how much data goes to training.

Why the others are wrong:

• Single observation held out — that is LOOCV ($k = N$).
• $k$ equal folds, averaged — that is $k$-fold CV; its folds are mutually exclusive and each observation is tested exactly once, unlike GCV’s overlapping random draws.
• Training error on the full dataset — not cross-validation at all; it “can dramatically underestimate the test error rate.”

Q7 — Which CV procedure is deterministic?

Answer: Leave-one-out cross-validation.

LOOCV holds out observation 1, then 2, …, then $N$ — every observation is the test set exactly once, so there is nothing to randomize. Notes §1.8.4: “performing LOOCV multiple times will always yield the same results: there is no randomness in the training/test set splits.” The Video 24 script says the same in code comments — LOOCV: “we should all get the same answer here”; 5-fold: “we will all get different answers.”

(The price of determinism is $N$ model fits, which is why $k = 5$ or $10$ is the usual practical choice.)

Q8 — 5-fold vs. 8-fold CV estimates

Answer: 0.02

The $k$-fold CV estimate is the average of the $k$ per-fold test errors (notes §1.8.4), $\frac{1}{k}\sum_{j=1}^{k} TE_j$:

The sample size ($N = 500$) does not enter: each fold’s error is already a rate, and the folds are of roughly equal size so a plain average is the estimate. The 8-fold estimate is a little lower, consistent with larger training sets (7/8 of the data vs. 4/5) producing slightly better models.

Script: hw3_q8.R.

Q9 — Purpose of IPCW

Answer: IPCW corrects for censored subjects by giving extra weight to subjects who are not censored.

Notes §1.8.5, nearly verbatim: “IPCW estimators correct for censored subjects by giving extra weight to subjects who are not censored. Each subject $i$ is weighted by the inverse of an estimate of the conditional probability of having remained uncensored until time $t$.” With $G(t) = P(C > t)$ estimated by Kaplan–Meier on the censoring times,

— subjects observed past $t$, or with an observed event before $t$, get “extra credit” for having made it that far uncensored, and stand in for those whose status is unknown.

Why the others are wrong:

• Confounding variables / unmeasured covariates — that is inverse probability of treatment weighting (causal inference), not censoring weighting. IPCW addresses missing outcome status, not confounding.
• “Sets weights to zero for censored subjects, effectively ignoring them” — the half-truth. Subjects censored before $t$ do get $\widehat{W}_i = 0$ in the sums, but the notes are explicit that they are not ignored: “we use everyone to estimate $\hat{G}(\cdot)$,” and the whole point of the method is to avoid the bias that dropping them would cause.
• Time-varying covariates — a different topic (§1.9, tmerge).

Q10 — Which are time-varying covariates?

Answer: Blood pressure, Weight, Medication adherence. Not race/ethnicity, blood type, or place of birth.

A time-varying (time-dependent) covariate is one whose value can change during follow-up, so $Z(t)$ rather than $Z$ (notes §1.9). The notes’ own examples are pollution exposure, blood glucose, medication adherence, alcohol consumption, and transplant status.

The distinction matters because a time-varying covariate must be entered through person-time splitting (tmerge, Surv(tstart, tstop, event)) and licenses statements about the hazard but not naive survival prediction — the Stanford transplant lesson.

Q11 — Cumulative incidence under competing risks

Answer: 0.14

Notes §1.10: with competing causes $k = 1, \dots, K$, the cumulative incidence function (sub-distribution function) for cause $k$ is

— the probability of having had a cause-$k$ event by time $t$. With no censoring this is simply the proportion of the cohort with a cause-$k$ event by $t$:

The remaining $310/550 = 0.564$ are event-free at 5 years, and the three probabilities sum to 1 — the competing-risks analogue of $S(t) + F(t) = 1$. Note what you must not do: treat death as censoring and compute a Kaplan–Meier $1 - \hat{S}(5)$ for renal failure. That would give $158 / (550 - 82)$-type quantities that overstate the incidence, because a patient who died can no longer have renal failure. The CIF accounts for this.

Script: hw3_q11.R — the arithmetic plus an Aalen–Johansen cross-check via survfit with a multi-state outcome, which returns the same 0.2873 / 0.1491.

Q12 — AUC of resting blood pressure for heart disease

Answer: 0.87

heart = read.csv("hw3_heart.csv")
heart$hd = 1*(heart$class > 0)          # 216 with disease, 81 without

Resting blood pressure is the “test”; higher values should indicate disease (cases average 140 mmHg vs. 118 for controls), so in pROC terms direction = "<" — the opposite of the notes’ cognitive-score example, where lower scores flagged disease. Always set it explicitly.

The AUC is the Mann–Whitney probability (notes p. 41)

over all $216 \times 81 = 17{,}496$ case–control pairs, with $X$ the blood pressure of a case and $Y$ that of a control. Computed directly, and via pROC:

auc.hand = mean(outer(bp[hd == 1], bp[hd == 0], function(a, b) (a > b) + 0.5*(a == b)))
pROC::roc(heart$hd, heart$trestbps, direction = "<")$auc

Both give $0.8686 \to \mathbf{0.87}$: a randomly chosen patient with heart disease has a higher resting blood pressure than a randomly chosen patient without it about 87% of the time. No model is fit and no cross-validation is used — this is the raw-data roc(response, predictor) form from the Video 20 script, and since there is no model there is no training/test distinction to worry about.

Script: hw3_q12_15.R — Q12–Q15 share the dataset, so they live in one script. The hand computation needs only base R; the pROC line is a cross-check and runs only if the package is installed.

Q13 — Incremental value of resting blood pressure

Answer: 0.20

$IV(\text{trestbps}) = \text{AUC}{p(\text{trestbps})} - \text{AUC}{p(-\text{trestbps})}$ (notes §1.8.3). Both models are fit and scored on the full dataset — the training AUC, as the question instructs — using the Video 20 recipe: glm(..., family = "binomial"), predict(type = "response"), then roc(hd, preds, direction = "<").

m1 = glm(hd ~ age + thalach + chol,            family = "binomial", data = heart)
m2 = glm(hd ~ age + trestbps + chol + thalach, family = "binomial", data = heart)

A large gain — consistent with Q12, where blood pressure alone already reached 0.87, and with the Model 2 coefficient table, where trestbps has by far the largest $z$ ($7.5$). Both are the “with vs. without” comparison from the Video 20 aSAH example, where dropping s100b cost 0.10; here the biomarker matters even more.

Script: hw3_q12_15.R.

Q14 — AIC of the two logistic models

Answer: 105.60

$\mathrm{AIC} = -2\,l_a + 2a$ (notes p. 33), where $a$ counts every estimated coefficient including the intercept — $4$ for Model 1 and $5$ for Model 2. From the two glm fits of Q13:

R’s AIC() gives the same numbers as the hand formula. The difference is positive and large: adding trestbps raises the log-likelihood by 53.8 (worth 107.6 on the $-2l$ scale) for a penalty of only 2, so Model 2 is strongly preferred — the AIC view of the same conclusion the incremental value in Q13 reached. (Since Model 1 is nested in Model 2, a likelihood ratio test would give $\chi^2_1 = 107.6$, $p \approx 0$.)

Script: hw3_q12_15.R.

Q15 — LOOCV Brier score, all 13 predictors

Answer: 0.07

The 13 predictors are every column except the row index and class: age, sex, cp, trestbps, chol, fbs, restecg, thalach, exang, oldpeak, slope, ca, thal. LOOCV is the Video 24 loop — fit on $N - 1$, predict the held-out patient, score it — and because a single held-out observation cannot produce an AUC, the per-patient error is the Brier score term $(p_i - A_i)^2$ (notes §1.8.2, §1.8.4):

for (i in 1:n) {
  m = glm(hd ~ age + sex + cp + ... + thal, family = "binomial", data = heart[-i, ])
  p = predict(m, newdata = heart[i, ], type = "response")
  brier[i] = (p - heart$hd[i])^2
}
mean(brier)

No seed is needed — LOOCV is deterministic (Q7), so everyone gets this value. The training Brier score on the same model is 0.055, so the usual optimism is visible but modest with 297 patients and 13 predictors.

Coding note. The variables are used as they come from the CSV — sex, fbs, exang logical, everything else numeric — matching how the course scripts treat such data. Treating the integer-coded categoricals (cp, restecg, slope, thal) as factors instead gives a LOOCV Brier of 0.079, which would round to 0.08; the question gives no instruction to recode, so 0.07 is the answer to the question as asked.

Script: hw3_q12_15.R — includes the factor-coded sensitivity check.