Root mean square error in prediction

Root mean square error in prediction (RMSEP) (or root mean square deviation in prediction, RMSDP). Also known as standard error in prediction (SEP) or standard deviation error in prediction SDEP), is a function of the prediction residual sum of squares PRESS, defined as... 

For each reduced data set, the model is calculated and responses for the deleted objects are predicted from the model. The squared differences between the true response and the predicted response for each object left out are added to PRESS ( prediction error sum of squares). From the final PRESS, the (or R ) and RMSEP ( root mean square error in prediction) values are usually calculated [Cruciani, Baroni et al, 1992]. 

A different metric used to compare models with different ranges/standard deviations of the y value, is the root mean square error in prediction (RMSE/RMSEP). 

Once the calibration model is obtained, the validation is performed by applying the calibration model (the B matrix) to median spectra obtained from a set of validation images, for which the reference concentration value is known. As in any other calibration procedure, the predictions obtained can be used to evaluate the accuracy and the precision of the bulk concentration estimates by examining the bias and the root mean square error in prediction (RMSEP). 

With cross-validation [28], the same objects are used both for model estimation and testing. A few objects are left out from the calibration data set and the model is calibrated on the remaining objects. Then the values for the left-out objects are predicted and the prediction residuals are computed. The process is repeated with another subset of the calibration set, and so on until every object has been left out once then all prediction residuals are combined to compute the validation residual variance and root mean square error in prediction (RMSEP). It is of utmost importance that the user is aware of which level of cross-validation one wants to validate. For example, if one physical sample is measured three times, and the objective is to establish a model across samples, the three replicates must be held out in the same cross-validation segment. If the objective is to validate the repeated measurement, keep out one replicate for all samples and generate three cross-validation segments. The calibration variance is always the same it is the validation... 

The number of significant PLS LVs was chosen by considering two approaches first, the commonly used leave one out cross vahdation (LOO) procedure [68] and the second called leave one producer out (LOP). In this latter approach, a producer at time was left out from the model and considered as test set. The procedure was done six times, obtaining six different root mean square errors in prediction (RMSEP-LOP). 

Chemometrics in Process Analytical Technology (PAT) 409 The main figure of merit in test set validation is the root mean square error of prediction (RMSEP) ... 

NIR models are validated in order to ensure quality in the analytical results obtained in applying the method developed to samples independent of those used in the calibration process. Although constructing the model involves the use of validation techniques that allow some basic characteristics of the model to be established, a set of samples not employed in the calibration process is required for prediction in order to conhrm the goodness of the model. Such samples can be selected from the initial set, and should possess the same properties as those in the calibration set. The quality of the results is assessed in terms of parameters such as the relative standard error of prediction (RSEP) or the root mean square error of prediction (RMSEP). 

Root Mean Square Error of Prediction (RMSEP) Plot (Model Diagnostic) The validation set is employed to determine the optimum number of variables to use in the model based on prediction (RMSEP) rather than fit (RMSEO- RM-SEP as a function of the number of variables is plotted in Figure 5.7S for the prediction of the caustic concentration in the validation set, Tlie cuive levels off after three variables and the RMSEP for this model is 0.053 Tliis value is within the requirements of the application (lcr= 0.1) and is not less than the error in the reported concentrations. 

Root Mean Square Error of Prediction (RMSEP) Plot (Model Diagnostic) The new RMSEP plot in Figure 5-100 is more well behaved than the plot shown in Figure 5-93 (with the incorrect spectrum 3). A minimum is found at 3 factors with a corresponding RMSEP that is almost two orders of magnitude smaller than the minimum in Figure 5-93- The new RMSEP plot shows fairly ideal behavior with a sharp decrease in RMSEP as factors are added and then a slight increase when more than three factors are included. 

Root Mean Square Error of Prediction (RMSEP) Plot (Model Diagnostic) Prediction error is a useful metric for selecting the optimum number of factors to include in the model. This is because the models are most often used to predict the concentrations in future unknown samples. There are two approaches for generating a validation set for estimating the prediction error internal validation (i.e., cross-validation with the calibration data), or external validation (i.e., perform prediction on a separate validation set). Samples are usually at a premium, and so we most often use a cross- validation approach. 

Root Mean Square Error of Prediction (RMSEP) Plot (Model Diagnostic) The RMSEP versus number of factors plot in Figure 5.113 shows a break at three factors and a leveling off after six factors. Tlie RMSEP value with six factors (0,04) is comparable to the estimated error in the reported concentrations (0.033), indicating the model is predicting well At this point we tentatively choose a rank six model. The rank three model shows an RMSEP of 0.07 and may well have been considered to be an adequate model, depending on how well the reference values are known. 

Root Mean Square Error of Prediction (RMSEP) Plot (Model Diagnostic) The RMSEP plot for the MCB model is shown in Figure 5.127. Although the shape of this RMSEP plot is not ideal, it does not exhibit erratic behavior. Tlie first minimum in this plot is at four factors with a lower minimum at six factors. In Section 5.2.1.2, nonlinear behavior was suspected as the root cause of the failure of the DCLS method. Tlicreforc, it is reasonable that a PLS model re-... 

The first PC quantified the moisture content as evident from the strong contribution around 1930 nm region, which represents the combination band of the -OH stretching and -OH bending vibrations. The second PC quantified the baseline shift due to changing sample density as a result of changing compression pressures. The third PC quantified the changes in MCC structure as evident from its resemblance with NIR spectrum collected on the 100% MCC powder sample. The root mean-square errors of prediction for different sample properties are summarized in Table 14. 

Figure 3. Reconstructions of (A) diatom-based and (B) chrysophyte-based monomeric Al for Big Moose Lake, and diatom-based monomeric Al for (C) Deep Lake, (D) Upper Wallface Pond, and (E) Windfall Pond in the Adirondack Mountains, New York. Reconstructions are bounded by bootstrapping estimates of the root mean-squared error of prediction for each sample. Bars to the right of each reconstruction indicate historical (H) and Chaoborus-based (C) reconstructions of fishery resources. The historical fish records are not continuous, unlike the paleolimnological records. Intervals older than 1884 are dated by extrapolation. (Reproduced with permission from reference 10.

At this point, it is worth noting that the same validation methods that are used to avoid overfitting of quantitative models (Section 8.3.7) can also be used to avoid overfitting in qualitative models. The only difference is that the figure of merit in this case is not the Root Mean Squared Error of Prediction (RMSEP), but rather the percent correct classification or %CC ... 

The model was also validated by full cross-validation. Four PCs were necessary to explain the most variation in the spectra (99.9%), which best described the molecular weight. The root mean square error of prediction... 

The b vector chosen by the validation procedure can be employed prospectively to predict concentrations of the analyte of interest in independent data. Similar to the calculation of RMSECV, the root mean square error of prediction (RMSEP) for an independent data set is defined as the square root of the sum of the squares of the differences between predicted and reference concentrations. 

Figures 11 and 12 illustrate the performance of the pR2 compared with several of the currently popular criteria on a specific data set resulting from one of the drug hunting projects at Eli Lilly. This data set has IC50 values for 1289 molecules. There were 2317 descriptors (or covariates) and a multiple linear regression model was used with forward variable selection the linear model was trained on half the data (selected at random) and evaluated on the other (hold-out) half. The root mean squared error of prediction (RMSE) for the test hold-out set is minimized when the model has 21 parameters. Figure 11 shows the model size chosen by several criteria applied to the training set in a forward selection for example, the pR2 chose 22 descriptors, the Bayesian Information Criterion chose 49, Leave One Out cross-validation chose 308, the adjusted R2 chose 435, and the Akaike Information Criterion chose 512 descriptors in the model. Although the pR2 criterion selected considerably fewer descriptors than the other methods, it had the best prediction performance. Also, only pR2 and BIC had better prediction on the test data set than the null model.

The two regression models have lower prediction accuracy than the random-function model when assessed using cross-validated root mean squared error of prediction . This quantity is simply the root mean of the squared cross-validated residuals y( (,)) - for i = 1,..., n in the numerator of (20). The cross-... 

---