Resampling Methods for Prediction Error Assessment and Model Selection

2 RESAMPLING METHODS FOR PREDICTION ERROR ASSESSMENT AND MODEL SELECTION  

The outcome y can be predicted from the collection of features z with the use of various statistical models. Subsequently a given prediction rule can be written as rj z), where x reflects the dependence of the built rule on the observed random sample (see Fig. 10.1). 

To assess how well a given rule predicts, loss functions may be employed. For each observation, a loss function L(y rj[zd) simply measures the error of the mle by determining the difference between what is predicted by the mle and the tmth, that is, the difference between rj zd and y,. The choice of loss function is dependent on the scale of y. If y is continuous, frequent choices of L(y rj z,)) are as follows  

If y is binary, the most common choice is the indicator loss function, also referred to as the misclassification of the prediction mle  

Here the indicator function, /(y, rj(z,)), equals unity if the mle s prediction is incorrect and zero if it is correct. A more thorough discussion of loss function options is left for the end of the chapter, including the scenario when y is a time to event which is not always observed, that is, censored. 

---

Classical Resampling Methods for Prediction Error Assessment and Model Selection... 

---