RMSEP prediction

The Standard Error of Prediction (SEP) is supposed to refer uniquely to those situations when a calibration is generated with one data set and evaluated for its predictive performance with an independent data set. Unfortunately, there are times when the term SEP is wrongly applied to the errors in predicting y variables of the same data set which was used to generate the calibration. Thus, when we encounter the term SEP, it is important to examine the context in order to verify that the term is being used correctly. SEP is simply the square root of the Variance of Prediction, s2. The RMSEP (see below) is sometimes wrongly called the SEP. Fortunately, the difference between the two is usually negligible. 

The Root Mean Standard Error of Prediction (RMSEP) is simply the square root of the MSEP. The RMSEP is sometimes wrongly called the SEP. Fortunately, the difference between the two is usually negligible. 

Root-mean-squared error of prediction (RMSEP), 6 50-51... 

Unfortunately, definitions, nomenclature, and abbreviations used for performance criteria are sometimes confusing (Frank and Todeschini 1994 Kramer 1998). For instance in the abbreviations MSEC, PRESS, RMSEP, SEC, SEE, SEP, E means error or estimate, R means residual or root, and S means squared or standard or sum. To make it not too complicated, at least in these examples, C is always calibration, M is mean, and P is prediction. 

The root mean squared error (RMSE) is the square root of MSE, and can again be given for calibration (RMSEC or RMSECAL), CV (RMSECV or RMSECv) or for prediction/test (RMSEP, or RMSEtest). In the case of a negligible bias, RMSEP and SEP are almost identical, as well as MSEniST and SEP2. 

Figure 3.10 Hallmark signature of significant sampling bias as revealed in chemometric multivariate calibrations (shown here as a prediction validation). Crab sampling results in an unacceptably high, irreducible RMSEP. While traditionally ascribed to measurement errors, it is overwhelmingly due to ISE.

The pilot study showed good prospects for predicting crystallization point temperatures directly from the acoustic signatures of the liquid feed into the granulator with an indicated prediction error (root mean square error of prediction) RMSEP = 1.1 °C. 

Figure 9.23 Prediction resuits for ammonia, validated with two-segment cross validation (test set switch). Slope = 0.96. RMSEP = 0.48% ammonia.

The full-scale industrial experiment demonstrated the feasibility of a convenient, nonintrusive aconstic chemometric facility for reliable ammonia concentration prediction. The training experimental design spanned the industrial concentration range of interest (0-8%). Two-segment cross-validation (test set switch) showed good accnracy (slope 0.96) combined with a satisfactory RMSEP. It is fully possible to further develop this pilot study calibration basis nntil a fnll industrial model has been achieved. There wonld appear to be several types of analogous chemical analytes in other process technological contexts, which may be similarly approached by acoustic chemometrics. 

Chemometrics in Process Analytical Technology (PAT) 409 The main figure of merit in test set validation is the root mean square error of prediction (RMSEP) ... 

NIR models are validated in order to ensure quality in the analytical results obtained in applying the method developed to samples independent of those used in the calibration process. Although constructing the model involves the use of validation techniques that allow some basic characteristics of the model to be established, a set of samples not employed in the calibration process is required for prediction in order to conhrm the goodness of the model. Such samples can be selected from the initial set, and should possess the same properties as those in the calibration set. The quality of the results is assessed in terms of parameters such as the relative standard error of prediction (RSEP) or the root mean square error of prediction (RMSEP). 

Root Mem Square Error of Prediction (RMSEP) (Model Diagnostic) The RMSEP is anciaer diagnostic for examining the errors in the predicted concentrations. Whie the statistical prediction error discussed earlier quantifies preci-... 

The RMSEP characterizes both the accuracy and precision errors expected for future predictions. 

Root Mea Square Error of Prediction (RMSEP) (Model Diagnostic) The RMSEP values for all four components are numerically summarized in Table 5.6. They are large owing to the bias in the predictions. Several reasons for this bias can be proposed, including an inaccurate reference method, transcription errcKS, poor experimental procedures, changes in densiw and/or pathlength, l t scatter in the instrument or sample, chemical interactions,... 

The RMSEP plotted versus the number of variables included in the model for componem A indicates that the optimum model contains one variable (see Figure 5-67). The prediction error with one variable (0.016) is greater than the known coneersration error of 0.010, which 1 an indication that tiic niodd is not overfittii. Assuming the model is correct, the fact that the RMSEP is greater than e errors in the known concentrations is due to errors in R. 

Root Mean Square Error of Prediction (RMSEP) Plot (Model Diagnostic) The validation set is employed to determine the optimum number of variables to use in the model based on prediction (RMSEP) rather than fit (RMSEO- RM-SEP as a function of the number of variables is plotted in Figure 5.7S for the prediction of the caustic concentration in the validation set, Tlie cuive levels off after three variables and the RMSEP for this model is 0.053 Tliis value is within the requirements of the application (lcr= 0.1) and is not less than the error in the reported concentrations. 

To select m optimum rank, the RMSEP is evaluated with models constructed using different numbers of factors. The RMSEP decreases when the predicted valu (c.) is close to the known value (cp and, therefore, small RMSEP values are desired (see Equation 5-36). To choose the optimum number of factors,.2plot of RMSEP versus the number of factors is examined. This plot typically Resents results from models constructed using more factors than are expecasd to be significant. 

Root Mean Square Error of Prediction (RMSEP) Plot (Model Diagnostic) The new RMSEP plot in Figure 5-100 is more well behaved than the plot shown in Figure 5-93 (with the incorrect spectrum 3). A minimum is found at 3 factors with a corresponding RMSEP that is almost two orders of magnitude smaller than the minimum in Figure 5-93- The new RMSEP plot shows fairly ideal behavior with a sharp decrease in RMSEP as factors are added and then a slight increase when more than three factors are included. 

The prediction ability as measured by RMSEP is similar for MCB and ODCB and is roughly a factor of 3 smaller than for the other two components. One explanation for this is the high degree of correlation between the spectra of EB and CUM (see Figure 5.26 in Section 5.2.1.2) compared with the MCB and ODCB. 

RMSEP. see Root mean square error, of prediction... 

Sion, the RMSEP shown in Equation 5.15 summarizes both the precision and accuracy of future predictions ... 

Summat of Prediction Diagnostic Tools for ICLS, Example 2 Based on the prediction diagnostics, the conclusion is that the predicted values for 98 of 99 prediction samples are reasonable. Based on the range of validation concentration residuals (see Figure 5.56), the errors in the predicted caustic concentrations of the unknovsrns are expected to be within 0.17 wt.% corresponding to an RMSEP of 0.06 wt.%. 

Root Mean Square EtTor of Prediction (RMSEP) Plot (Model Diagnostic) The number of variables to include is finalized using a validation procedure that accounts for predictive ability. There are two approaches for calculating the prediction error internal cross-validation (e.g., Icave-one-out cross-validation with the calibration data) or external validation (i.e.. perform prediction... 

The RMSEC in Table 5.10 (0.018) is larger than the RMSEP (0.016). This seems incorrect, because prediction errors are npically larger than fit errors, but these numbers are not statistically different. 

The RMSEP is a more realistic estimate of predictive abilit than R.MSEC. 

FIGURE 5,78. RMSEP for the prediction of the caustic concentration as a function of the number of variables using a separate validation set. 

Root Mean Square Error of Prediction (RMSEP) Plot (Model Diagnostic) The RMSEP versus number of factors plot in Figure 5.113 shows a break at three factors and a leveling off after six factors. Tlie RMSEP value with six factors (0,04) is comparable to the estimated error in the reported concentrations (0.033), indicating the model is predicting well At this point we tentatively choose a rank six model. The rank three model shows an RMSEP of 0.07 and may well have been considered to be an adequate model, depending on how well the reference values are known. 

Root Mean Square Error of Prediction (RMSEP) Plot (Model Diagnostic) The RMSEP plot for the MCB model is shown in Figure 5.127. Although the shape of this RMSEP plot is not ideal, it does not exhibit erratic behavior. Tlie first minimum in this plot is at four factors with a lower minimum at six factors. In Section 5.2.1.2, nonlinear behavior was suspected as the root cause of the failure of the DCLS method. Tlicreforc, it is reasonable that a PLS model re-... 

---