Training and Test Datasets

The final stage of compiling a maximally refined dataset is to split it into a training and test dataset. The definition of a test dataset is an absolute must during learning, as, in fact, it is the best way to validate the results of that learning. 

Although the problem of compilation of training and test datasets is crucial, unfortunately no de-faao standard technique has been introduced. Nevertheless, we discuss here a method that was designed within our group, and that is used quite successfully in our studies. The method is mainly addressed to the task of finding and removing redundancy. 

The first step to test this hypothesis is the division of the dataset into a training and a test dataset. Two-thirds (3685 compounds) of the entire dataset were assigned to the training dataset and the remaining third (1828 compounds) to the test dataset by a random selection with the restriction that the ratio of hits to non-hits in the entire dataset is preserved in both the training and test dataset. 

Figure 3. Frequency ratio of factors relate to respective training and testing datasets. (A) Elevation (B) Slope angle (C) Slope aspect (D) Slope curvature (E) Topographic position (F) Distance from main surface ruptures (G) PGA (H) Distance from roads (T) NDVI (J) Distance from drainages (K) Lithology (L) Distance from all faults.

Biometric system performs matching in pattern recognition problems between the training and test dataset for unknown features which would later determine the class (identity) of these unknown features. As a result of this learning technique, individuals can be identified for security and privacy purposes. 

Since equal numbers of disordered and ordered residues were used for training and testing, prediction success would be about 50% if disordered and ordered sequences were the same. In contrast to this 50% value, prediction success rates for the short, medium, long, and merged datasets were 69% 3%, 74% 2%, 73% 2%, and 60% 3%, respectively (Romero et al., 1997b), where the standard errors were determined over about 2200, 2600, 2000, and 6800 individual predictions, respectively. 

This method for preventing overfitting requires that there are enough samples so that both training and test sets are representative of the dataset. In fact, it is desirable to have a third set known as a validation set, which acts as a secondary test of the quality of the network. The reason is that, although the test set is not used to train the network, it is nevertheless used to determine at what point training is stopped, so to this extent the form of the trained network is not completely independent of the test set. 

Golbraikh, a. Tropsha, a. Predictive QSAR modeling based on diversity sampling of experimental datasets for the training and test set selection. 

Figure 2.2 The generation of a drug-likeness model includes the following steps. Assemble a set of molecules for which the property to be learned is already known. Calculate descriptors for structures. Divide the dataset into training and test sets. Put the test set aside. Present the training set to the machine learning algorithm to build a model. Sometimes at this stage a...

The optimum size and representativeness of training and test sets for calibration modelling are a big subject. Some chemometricians recommend using hundreds or thousands of samples, but this can be expensive and time consuming. In some cases a completely orthogonal dataset is unlikely ever to occur and field samples with diese features cannot be found. Hence there is no perfect way of validating calibration... 

The structural variance of the dataset was analyzed with principal component analysis (PCA) [9] performed on the complete set of ALMOND descriptors calculated for the compounds which comprised the training and test sets. The first two components explained 35% of the stmctural variance of the dataset. Figure 9.1 shows that no structural outliers are present in the dataset and that the training and test sets share similar chemical space. 

The test set compounds were randomly selected from each group to represent the activity span uniformly. The PCA of the datasets was performed to assess structural variance of both training and test sets. The PCA scores plot (Fig. 9.6) showed the absence of any structural outlier in the dataset and that the training set and test set shared the same chemical space. 

Molecules in the dataset were divided into a training set (331 compounds) and a test set (39 molecules). The test set was chosen to cover the activity data span of the two subsets of data uniformly. The dataset was divided into three groups according to the activity value. (2.02-1.45, 1.43-0.95, 0.90.3) Then 13 molecules from each group were randomly chosen. The principal component analysis of the complete dataset was performed to analyze the structural variance of both the training and test sets. The PCA scores plot (Fig. 9.10) showed that there was no structural outlier present in the dataset and that the training set and test set shared the same chemical space. 

Most, if not all, QSAR methods require selection of relevant or informative descriptors before modeling is actually performed. This is necessary because the method could otherwise be more susceptible to the effects of noise. The a priori selection of descriptors, however, carries with it the additional risk of selection bias [73], when the descriptors are selected before the dataset is divided into the training and test sets (Figure 6.6A). Because of selection bias, both external validation and cross validation could significantly overstate pre-... 

A diverse set of 4173 compounds was used by Karthikeyan et al. [133] to derive their models with a large number of 2D and 3D descriptors (ah calculated by MOE following 3D-structure generation of Concord [134]) and an artificial neural network. The authors found that 2D descriptors provided better prediction accuracy (RMSE = 48-50°C) compared to models developed using 3D indexes (RMSE = 55-56°C) for both training and test sets. The use of a combined 2D and 3D dataset did not improve the results. 

---