Testing set

To understand what training and test sets are, and how to make use of them... 

In Eq. (14) "est stands for the calculated (estimated) response, exp" for the experimental one, and n and m are the numbers of objects in the training set and the test set respectively. CoSE stands for Compound Standard Error. As an option, one can employ several test sets, if needed. 

The easiest way to extract a set of objects from the basic dataset, in order to compile a test set, is to do so randomly. This means that one selects a certain number of compounds from the initial (primary) dataset without considering the nature of these compounds. As mentioned above, this approach can lead to errors. 

Another method of detection of overfitting/overtraining is cross-validation. Here, test sets are compiled at run-time, i.e., some predefined number, n, of the compounds is removed, the rest are used to build a model, and the objects that have been removed serve as a test set. Usually, the procedure is repeated several times. The number of iterations, m, is also predefined. The most popular values set for n and m are, respectively, 1 and N, where N is the number of the objects in the primary dataset. This is called one-leave-out cross-validation. 

Oui recommendation is that one should use n-leave-out cross-validation, rather than one-leave-out. Nevertheless, there is a possibility that test sets derived thus would be incompatible with the training sets with respect to information content, i.e., the test sets could well be outside the modeling space [8]. 

Hence, the main danger in the process of compiling test sets remains. Fortunately, there are some other approaches which, although they may look more sophisticated, do diminish the possibiHty of such incompatibilities. 

First, one can check whether a randomly compiled test set is within the modeling space, before employing it for PCA/PLS applications. Suppose one has calculated the scores matrix T and the loading matrix P with the help of a training set. Let z be the characteristic vector (that is, the set of independent variables) of an object in a test set. Then, we first must calculate the scores vector of the object (Eq. (14)). 

The methods discussed above enable one to examine carefully die content of test sets, thus improving the quality of modeling. 

Therefore the 28 analytes and their enantiomers were encoded by the conformation-dependent chirality code (CDCC) and submitted to a Kohoiien neural network (Figure 8-1 3). They were divided into a test set of six compounds that were chosen to cover a variety of skeletons and were not used for the training. That left a training set containing the remaining 50 compounds. 

A test set is loaded whose object properties are to be predicted. The different propei ty classes are identified by different colors. 

Model building consists of three steps training, evaluation, and testing. In the ideal case the whole training data set is divided into three portions, the training, the evaluation set, and the test set. A wide variety of statistical or neural network... 

Finally, a model has to be tested using an independent data set with compounds yet completely unknown to the model the test set. The complete process of building a prediction model is depicted in Figure 10.1-1 as a flow chart. 

Recently, several QSPR solubility prediction models based on a fairly large and diverse data set were generated. Huuskonen developed the models using MLRA and back-propagation neural networks (BPG) on a data set of 1297 diverse compoimds [22]. The compounds were described by 24 atom-type E-state indices and six other topological indices. For the 413 compoimds in the test set, MLRA gave = 0.88 and s = 0.71 and neural network provided... 

A data set can be split into a training set and a test set randomly or according to a specific rule. The 1293 compounds were divided into a training set of 741 compounds and a test set ot 552 compounds, based on their distribution in a K.NN map. From each occupied neuron, one compound was selected and taken into the training set, and the other compounds were put into the test set. This selection ensured that both the training set and the test set contained as much information as possible, and covered the chemical space as widely as possible. 

Multiple linear regression analysis is a widely used method, in this case assuming that a linear relationship exists between solubility and the 18 input variables. The multilinear regression analy.si.s was performed by the SPSS program [30]. The training set was used to build a model, and the test set was used for the prediction of solubility. The MLRA model provided, for the training set, a correlation coefficient r = 0.92 and a standard deviation of, s = 0,78, and for the test set, r = 0.94 and s = 0.68. 

Figure 10.1-3. Predicted versus experimental solubility values of 552 compounds in the test set by a back-propagation neural network with 18 topological descriptors.

When structure-property relationships are mentioned in the current literature, it usually implies a quantitative mathematical relationship. Such relationships are most often derived by using curve-fitting software to find the linear combination of molecular properties that best predicts the property for a set of known compounds. This prediction equation can be used for either the interpolation or extrapolation of test set results. Interpolation is usually more accurate than extrapolation. 

The validation of the prediction equation is its performance in predicting properties of molecules that were not included in the parameterization set. Equations that do well on the parameterization set may perform poorly for other molecules for several different reasons. One mistake is using a limited selection of molecules in the parameterization set. For example, an equation parameterized with organic molecules may perform very poorly when predicting the properties of inorganic molecules. Another mistake is having nearly as many fitted parameters as molecules in the test set, thus fitting to anomalies in the data rather than physical trends. 

In order to parameterize a QSAR equation, a quantihed activity for a set of compounds must be known. These are called lead compounds, at least in the pharmaceutical industry. Typically, test results are available for only a small number of compounds. Because of this, it can be difficult to choose a number of descriptors that will give useful results without htting to anomalies in the test set. Three to hve lead compounds per descriptor in the QSAR equation are normally considered an adequate number. If two descriptors are nearly col-linear with one another, then one should be omitted even though it may have a large correlation coefficient. 

Fig. 4. Schematic of an ultrasonic nondestmctive testing set-up of material of length, T.

Many of the tools are aimed at classification and prediction problems, such as the handwriting example, where a training set of data vectors for which the property is known is used to develop a classification rule. Then the rule can be appHed to a test set of data vectors for which the property is... 

Pure, low temperature organic Hquid viscosities can be estimated by a group contribution method (7) and a method combining aspects of group contribution and coimectivity indexes theories (222). Caution is recommended in the use of these methods because the calculated absolute errors are as high as 100% for individual species in a 150-compound, 10-family test set (223). A new method based on a second-order fit of Benson-type groups with numerous steric correctors is suggested as an alternative. Lower errors are claimed for the same test set. 

The assumption in step 1 would first he tested hy obtaining a random sample. Under the assumption that p <. 02, the distrihiition for a sample proportion would he defined hy the z distrihiition. This distrihiition would define an upper hound corresponding to the upper critical value for the sample proportion. It would he unlikely that the sample proportion would rise above that value if, in fact, p <. 02. If the observed sample proportion exceeds that limit, corresponding to what would he a very unlikely chance outcome, this would lead one to question the assumption that p <. 02. That is, one would conclude that the null hypothesis is false. To test, set... 

Owing to the greater test uneertainties assoeiated with field testing as eompared to planned shop testing, the warranty and guarantee should be modified to take into aeeount the nonideal test set-up. 

---