Spring 2010

Course Materials:

• Syllabus
• Project Assignment

Reading:

You are encouraged to read all of each chapter. There is an immense wealth of material in each chapter. But to better help you match your reading focus with what will be covered in lecture, since we cannot possibly cover all of the book in lecture, the following is a week-by-week reading guide.

• Week 1: Chapter 2
• Week 2: Chapter 3.1-3.4, 3.8
• Week 3: Chapter 4.1-4.5
• Week 4: Chapter 4.6
• Week 5: Chapter 5.1-2, 5.4-5, and 5.6

This chapter covers various methods relying on basis expansions, including issues of regularization when using such expansions. We will focus on the case of bases of splines (i.e., a certain class of piecewise polynomials). Later in the semester, when we get to support vector machines (SVMs), in Chapter 12, we will return to Section 8 of Chapter 5 and discuss regularization in the context of reproducing kernel Hilbert spaces.

• Week 6: Chapter 7.1-8, 7.10

We will skip Chapter 6 and go directly to Chapter 7. Having encountered the need for model selection in a number of the earlier chapters, we will now examine it directly as an issue in and of itself. The first three sections are background on the general problem, particularly the issue of bias-variance trade-off. Section 4 introduces the important concept of training error 'optimism', as an estimate of expected error. Sections 5-7 then present different perspectives on obtaining analytic model selection criteria. You can focus primarily on the AIC criterion; BIC and MDL are similar in spirit. Sections 10 and 11 discuss computationally intensive methods of model selection. We will concentrate on the method of cross-validation (Section 10), and skip the bootstrap method (Section 11).

• Week 7: Chapter 8.1-8.3, 8.7, 8.8

  We will concentrate in this chapter on model averaging, as a general principle, and focus on three ways to go about it: bagging, Bayesian model averaging, and frequentist model averaging. These are covered in 8.7, 8.3, and 8.8, respectively. You should also read 8.2, which provides basic background material on classical likelihood-based parameter inference versus inference using bootstrap principles. But we will not cover this in lecture.

• Week 8: Chapter 9.1-9.2

From this chapter we will take only two topics: generalized additive models (i.e., Ch9.1) and tree-based methods (i.e., Ch 9.2).

• Week 9: Chapter 10.1-10.5, 10.7-10.8; 15.1-15.3

  We will look at two of the most popular off-the-shelf methods for statistical learning i.e., boosting and random forests. Readings focus primarily on the most standard versions of these methods.

• Week 10: Chapter 11
• Week 11: Chapter 12.1-12.3 (Optional: Chapter 5.8)

  We will study support vector machines, and in conjunction with that we'll also examine the framework of reproducing kernel Hilbert spaces (RKHS).

Articles of Interest:

• Friedman, J.H. (1997). On Bias, Variance, 0/1-Loss, and the Curse-of-Dimensionality. Data Mining and Knowledge Discovery, 1, 55-77.
• Meinshausen, N., Meier, L., and Buhlmann, P. (2009). p-values for High-Dimensional Regression. Journal of the American Statistical Association, 104,1671-1681.

Homeworks:

• Homework 1: Problems 3.4, 3.6, 3.12, and 3.27. (To be presented Monday, Jan 25) [Solutions]
• Homework 2: Problem 4.2. (To be presented Monday, Feb 1) [Solutions]

In addition, do the following data assignment, to be turned in as a short report on Monday, Feb 1. Implement (either writing your own computer program or using functions from, say, R or Matlab) a quadratic discriminant analysis by fitting a separate Gaussian model per class. Apply this to the vowel data in your book, available here. Compute the misclassification rate and use appropriate visualizations to demonstrate the classifications you make. In addition,implement a linear discriminant analysis method on the expanded input space that incorporates the predictors in linear, quadratic, and interaction terms (i.e., in the sense of the last paragraph on page 102 in your text). Compare your results to those that you obtain using QDA.

• Homework 3: Problems 4.5 and 4.6. (To be presented Monday, Feb 8) [Solutions]
• Homework 4: Problems 5.4, 5.7, and 5.9. (To be presented TUESDAY, Feb 16) [Solutions]

In addition, do the following data assignment, to be turned in as a short report on Friday, Feb 19. Analyze the 'ozone' data of your authors, predicting ozone as a function simulataneously of all three other variables, using regression splines. Produce an analysis in the spirit of the example in Section 5.2.2. Compare the regression spline fits to what you obtain, on a variable by variable basis, with cubic smoothing splines.

• Homework 5: Problem 7.4.

We will have only one problem due on Monday i.e., the derivation of the fundamental relationship between Err_in, err, and op. This should not take all of Monday's lecture; the rest will be spent in formal lecture on Ch8 material.

• Homework 6: Problem 8.6(a),(b),(c) and extra. (Due Friday, March 5, in class.)

The first three parts of this problem ask you to construct uncertainty bands (i.e., confidence bands and posterior bands) using standard frequentist asymptotics (part a), Bayesian methods (part b), and bootstrap methods (part c). In addition, I would like you to add a part "d" to this problem, wherein you look at the accuracy of bagging for prediction. In particular, I would like you to use cross-validation to estimate the error associated with (i) the standard smoothing spline estimator in part (a), and (ii) a bagging version of this estimator.

• Homework 7: Problem 9.5
• Homework 8: Problems 10.1, 10.2, and 15.4
• Homework 9: Problems 12.1 and 12.2