RMSECV validation

For cross-validation the formula is as for test-set validation but should formally be reported as RMSECV. Validation residual, explained variances... 

Root mean square (RMS) granularity, 19 264 Root-mean-squared error of cross-validation (RMSECV), 6 50-51 Root-mean-squared error of calibration (RMSEC), 6 50-51... 

Figure 4.37. RMSECV PCA from leave-one-out cross-validation for PCA Example 2. 

Root an Square Error of Cross Validation for PCA Plot (Model Diagnostic) Figuir 4.83 shows that the RMSECV PCA decreases significantly after the first and ssond PCs arc added, but the deercase is much smaller when additional PCs are aiifed. This implies that a two-component PCA. model is appropriate. 

A common approach to cross-validation is called leave-one-out" cross-validation. Here one sample is left out, a PC model with given number of factors is calculated using the remaining samples, and then the residua of the sample left out is computed. This is repeated for each sample and for models with 1 to n PCs. The result is a set of cross-validation residuals for a given number of PCs. The residuals as a function of the number of PCs can be examined graphically as discussed above to determine the inherent dimensionality. In practice, the cross-validation residuals are summarized into a single number termed the Root Mean Squared Error of Cross Validation for PCA (RMSECV PCA), calculated as follows ... 

Root Mean Square Ei ror of Cross-Validation for PCA Plot (Model Diagnostic) Figure 4.63 displays the RMSECV PCA vs. number of principal components for the class B data from a leave-one-out cross-validation calculation. The RMSECy PCA quickly drops and levels off at two principal components, consistent with the choice of a rank tv- o model. 

The b vector chosen by the validation procedure can be employed prospectively to predict concentrations of the analyte of interest in independent data. Similar to the calculation of RMSECV, the root mean square error of prediction (RMSEP) for an independent data set is defined as the square root of the sum of the squares of the differences between predicted and reference concentrations. 

In the following sections, we review the application of Raman spectroscopy to glucose sensing in vitro. In vitro studies have been performed using human aqueous humor (HAH), filtered and unfiltered human blood serum, and human whole blood, with promising results. Results in measurement accuracy are reported in root mean squared error values, with RMSECV for cross-validated and RMSEP for predicted values. The reader is referred to Chapter 12 for discussion on these statistics. 

Raman spectra in the range 1545-355 cm 1 were selected for data analysis. An average of 27 (461/17) spectra were obtained for each individual with a 3 min integration time per spectrum. Each spectrum was obtained with excitation power 300 mW and integration time equivalent to 3 min. Spectra from each volunteer were analyzed using PLS with leave-one-out cross-validation, with eight factors retained for development of the regression vector. For one subject, a mean absolute error (MAE) of 7.8% (RMSECV 0.7 mM) and an R2 of 0.83 were obtained. 

The cross-validation approach can also be used to estimate the predictive ability of a calibration model. One method of cross-validation is leave-one-out cross-validation (LOOCV). Leave-one-out cross-validation is performed by estimating n calibration models, where each of the n calibration samples is left out one at a time in turn. The resulting calibration models are then used to estimate the sample left out, which acts as an independent validation sample and provides an independent prediction of each y, value, y(i), where the notation i indicates that the /th sample was left out during model estimation. This process of leaving a sample out is repeated until all of the calibration samples have been left out. The predictions y(i) can be used in Equation 5.10 to estimate the RMSECV. However, LOOCV has been shown to determine models that are overfitting (too many parameters are included) [7, 8], The same is true for v-fold cross-validation, where the calibrations set is split into... 

An important issue in PCR is the selection of the optimal number of principal components kopt, for which several methods have been proposed. A popular approach consists of minimizing the root mean squared error of cross-validation criterion RMSECV,. For one response variable (q = 1), it equals... 

The R-RMSECV values are rather time consuming because, for every choice of k, they require the whole RPCR procedure to be performed n times. Faster algorithms for cross validation are described [80], They avoid the complete recomputation of resampling methods, such as the MCD, when one observation is removed from the data set. Alternatively, one could also compute a robust R2-value [61], For q= 1 it equals ... 

Table 4.1. Leave-one-out cross-validation results from Tuckerl- and A-PLS on sensor) data for one to four components ( LV) for prediction of salt content. The percentage o variation explained (sum-squared residuals versus sum-squared centered data) is shown foi fitted modes (Fit) and for cross-validated models (Xval) for both X (sensory data) and Y (salt). The root mean squared error of cross-validation (RMSECV) of salt (weight %) is also provided.

Figure 10.36. Cross-validation results of X(ixjK). Legend RMSECV stands for root-mean-squared error of cross-validation and represents the prediction error in the same units as the original measurements.

Error types can be e.g. root mean square error of cross validation (RMSECV), root mean square error of prediction (RMSEP) or predictive residual sum of squares (PRESS). 

Table 1. Comparison of three PLS models in the Slurry-Fed Ceramic Melter data set. The variance in both blocks of data and the Root Mean Square Error of Calibration (RMSEC), Cross-validation (RMSECV) and Prediction (RMSEP) are compared.

Several applications can be found in literature regarding the use of NIR for the paediction of the main physical and rheological parameters of pasta and bread. De Temmerman et al. in 2007 proposed near-infrared (NIR) reflectance spectroscopy for in-line determination of moisture concentrations in semolina pasta immediately after the extrusion process. Several pasta samples with different moisture concentrations were extruded while the reflectance spectra between 308 and 1704 ran were measured. An adequate prediction model was developed based on the Partial Least Squares (PLS) method using leave-one-out cross-validation. Good results were obtained with R2 = 0,956 and very low level of RMSECV. This creates opportunities for measuring the moisture content with a low-cost sensor. 

Another example of applying chemometrics to separations data is depicted in Figures 8 and 9. Here, interval PLS (iPLS) was applied to blends of oils in order to quantify the relative concentration of olive oil in the samples (de la Mata-Espinosa et al., 2011b). iPLS divides the data into a number of intervals and then calculates a PLS model for each interval. In this example, the two peak segments which presented the lower root mean square error of cross validation (RMSECV) were used for building the final PLS model. 

A few of the predictions for imseen data obtained from leave-one-out cross-validation are shown in figures 11 and 12. The square root of MSECV (RMSECV) is calculated for each individual outpnit variable in order to obtain a measure of the standard deviation of the error of prediction. The same predictions made from the overall model built on all the available data are shown in figures 13 and 14. 

PLSR was used to develop a prediction model in the entire wave range from 4000 cm" to 10000 cm-i. Cross validation was applied to the calibration set. Each time, one sample was taken out from the calibration set. A calibration model was established for the remaining samples and the model was then used to predict the sample left out. Thereafter, the sample was placed back into the calibration set and a second sample was taken out. The procedure was repeated until all samples have been left out once. The root mean square error of cross validation (RMSEcv) was calculated for each of all wavelength combinations. The best principal component (PC) number with the highest Rev (correlation coefficient of cross validation) and lowest RMSEcv value was selected. 

The total sample sets were separated into calibration set and validation set. Cross validation was first used in calibration sample set to find the optimal principle component number. From figure 4 we can see the best principle component nmnber to be 10 with corresponding highest Rev of 0.91 and lowest RMSEcv of 0.41. Model accuracy was then evaluated on the validation set using the root mean square error of prediction (RMSEP), correlation... 

RMSEcv root mean square error of cross validation... 

Figure 6 Measured versus predicted plots from the PLSR analyses. M/G ratio measured by NMR without (a) and with (b) water suppression (zgpr) versus the M/G ratio predicted from the Raman spectra. RMSECV = root mean square error of cross validation. Both models are based on two PLS components.

In Figure 4 the root mean square error (RMS) and the root mean square error of cross validation (RMSECV) of different data processing methods and parameters are shown. As expected, the RMSECV is larger than the RMS for each method. The larger errors of the IHM are due to the non-perfect description of the pure spectra. Interestingly, CPR shows for a set of ranks (number of components used for description of the spectra) and power coefficients the lowest errors. In this example of the mixture of water and oil, this is attributed to the fact, that CPR not only considers the correlation, but also the variance with a power coefficient. 

The best estimate of future performance of a calibration model is the RMSEP. Concentration estimates, c, in the RMSEP are determined by applying the calibration model to a subset of data that was not employed in determining the model parameters. The RMSEP may be calculated for a validation set in order to determine the optimal number of factors in a model or to a test set in order to test the performance of the optimal model on future data. If an external subset of data is not available to optimize the calibration model, the RMSEP can be estimated by the RMSECV. The concentration estimates of Equation (10.18) are determined in CV by iteratively removing (and replacing) each sample from the data set. When a sample is removed from the data set, a calibration model is constructed from the remaining samples. The property of the removed sample is then estimated by the calibration model. RMSEC is a measure of how well the calibration model fits the calibration set. This is potentially the least informative of the three statistics. RMSEC is an extremely optimistic estimation of the model performance. As more factors are included in the calibration model, the RMSEC always decreases. 

---