Number of samples in the calibration set

In this chapter as a continuation of Chapters 58 and 59 [1, 2], the confidence limits for the correlation coefficient are calculated for a user-selected confidence level. The user selects the test correlation coefficient, the number of samples in the calibration set, and the confidence level. A MathCad Worksheet ( MathSoft Engineering Education, Inc., 101 Main Street, Cambridge, MA 02142-1521) is used to calculate the z-statistic for the lower and upper limits and computes the appropriate correlation for the z-statistic. The upper and lower confidence limits are displayed. The Worksheet also contains the tabular calculations for any set of correlation coefficients (given as p). A graphic showing the general case entered for the table is also displayed. 

Because the number of samples in the calibration set is small, all of the TEA samples (c bration and validation) are used to construct a TEA SIMCA model that is tested against the validation samples from the rest of the training set... 

Model Validation Validation of the calibration model is crucial before prospective application. Two types of validation schemes can be adopted internal and external. Internal validation, or cross-validation, is used when the number of calibration samples is limited. In cross-validation, a small subset of calibration data is withheld from the model building step. After the model is tested on these validation spectra, a different subset of calibration data is withheld and the b vector is recalculated. Various strategies can be employed for grouping spectra for calibration and validation. For example, a single sample is withheld in a leave-one-out scheme, and the calibration and validation process is repeated as many times as the number of samples in the calibration data set. In general, leave- -out cross-validation can be implemented with n random samples chosen from a pool of calibration data. 

In order to use the F statistic tables properly, it is also necessary to know the degrees of freedom in both the numerator (vj and denominator (vj) of the F ratio value. For F ratios based on PRESS values, the number of samples used to calibrate the model has been suggested as the proper value for both. Therefore, in the case of a cross-validation, the degrees of freedom would be the total number of sample in the training set minus the number left out in each group. For a validation set prediction, they would be the total number of samples in the training set. 

For the purposes of this section, error is simply the difference between the value of the y variable predicted by a regression and the true value (sometimes called the expected value). Naturally, it is impossible to know the true value, so we are forced to settle for using the best available referee value for the y variable. (Note it is possible that the "best available referee values" can have larger errors than the predicted values produced by the calibration.) We will follow the common convention and name the expected value of the variable y and the predicted value of the variable y, pronounced "Y-hat." Then the error is given by p -y. We will also denote the number of samples in a data set by the letter n. 

With this background infonnation on the inverse methods, it is instructive to examine the calculations for the inverse model in more detail. In Equation 5-23, the key to the model-building step is the inversion of the matrix CR ). This is a squire matrix with number of rows and columns equal to the number of measurement variables (nvars). From theory, a number of independent samples in the calibration set greater than or equal to nvars is needed in order to invert this matrix. For most analytical measurement systems, nvars (e.g., number of wavelengths) is greater than the number of independent samples and therefore RTr cannot be directly inverted. However, with a transformation, calculating she pseudo-inverse of R (R is possible. How this transformation is accomplished distinguishes the different inverse methods. 

Usually, PRESS should be calculated separately for each predicted component, and the calibration optimized individually for each component. For preliminary work, it can be convenient to calculate PRESS collectively for all components together, although it isn t always possible to do so if the units for each component are drastically different or scaled in drastically different ways. Calculating PRESS collectively will be sufficient for our purposes. This will give us a single PRESS value for each set of results Kl, through K25,, . Since not all of the data sets have the same number of samples, we will divide each of these PRESS values by the number of samples in the respective data sets so that they can be more directly compared. We will also divide each value by the number of components predicted (in this case 3). The resulting press values are compiled in Table 2. 

Often the number of samples for calibration is limited and it is not possible to split the data into a calibration set and a validation set containing representative samples that are representative enough for calibration and for validation. As we want a satisfactory model that predicts future samples well, we should include as many different samples in the calibration set as possible. This leads us to the severe problem that we do not have samples for the validation set. Such a problem could be solved if we were able to perform calibration with the whole set of samples and validation as well (without predicting the same samples that we have used to calculate the model). There are different options but, roughly speaking, most of them can be classified under the generic term "cross-validation . More advanced discussions can be found elsewhere [31-33]. 

Corti et al. analyzed ranitidine and water in sample tablets [79], Production samples, when made under control, contain a narrow range of values for active concentration. It is then difficult to cull any number of samples to generate a desired range of sample values in the calibration set to cover a 20% (90 to 110% of label claim) range as done for typical HPLC methods. The result is a diminished correlation coefficient due to the small range of values in the calibration set. 

Because there are a finite number of samples in the set used for prediction, in many cases the number of factors that gives a minimum PRESS value can still be overfit for predicting unknown samples. In other words, there is a statistical possibility that some of the noise vectors from the spectral decomposition may be present in more than one sample. These vectors can appear to improve the calibration by a small amount when, by random correlation, they are added to the model. However, if these exact same noise vectors are not present in future unknown samples (and most likely they will not be), the predicted concentrations will have significantly larger prediction errors than if those additional vectors were left out of the model. 

There are three rules of thumb to guide us in selecting the number of calibration samples we should include in a training set. They are all based on the number of components in the system with which we are working. Remember that components should be understood in the widest sense as "independent sources of significant variation in the data." For example, a... 

The most common and intuitive method for the determination of this number of eigenvalues is called Cross Validation. The idea is to remove one (or several samples) from the calibration set, use what is left for the computation of a new calibration, and use it to predict the quality of the removed sample(s). Each prediction is compared with the actual quality that is known as the removed sample really is part of the total calibration set. In a loop all samples are removed either one by one or in groups and after recalibration with the reduced calibration set their qualities are predicted and compared with the true values. In order to determine the best number of eigenvectors this procedure is repeated in a big loop systematically trying all numbers of eigenvectors. This complete procedure is called Cross Validation. 

The factors that influence the optimal cross-validation method, as well as the parameters for that method, are the number of calibration samples (AO, the arrangement order of the samples in the calibration data set, whether the samples arise from a design of experiments (DOE, Section 12.2.6), the presence or absence of replicate samples, and the specific objective of the cross-validation experiment. In addition, there are two traps that one needs to be aware of when setting up a cross-validation experiment. 

Infrared data in the 1575-400 cm region (1218 points/spec-trum) from LTAs from 50 coals (large data set) were used as input data to both PLS and PCR routines. This is the same spe- tral region used in the classical least-squares analysis of the small data set. Calibrations were developed for the eight ASTM ash fusion temperatures and the four major ash elements as oxides (determined by ICP-AES). The program uses PLSl models, in which only one variable at a time is modeled. Cross-validation was used to select the optimum number of factors in the model. In this technique, a subset of the data (in this case five spectra) is omitted from the calibration, but predictions are made for it. The sum-of-squares residuals are computed from those samples left out. A new subset is then omitted, the first set is included in the new calibration, and additional residual errors are tallied. This process is repeated until predictions have been made and the errors summed for all 50 samples (in this case, 10 calibrations are made). This entire set of... 

Once ftE calibration and test sets are defined, SIMC. models are estimated using ini rank estimates Offom PCA) and default class volumes (from the software fKrkage). The samples in the test set arc predicted and the results evaimtedaa detennine if modification of the tanks and/or class volumes is needed (see Habit 4 for details of this evaluation). If the number of misclassifi-cations is anacccptable, the rank and class volume parameters are adjusted and the pswress repeated. 

---