Training and Testing Sets

The guidelines above should be followed if possible, but you should be aware that in real-world applications it may be difficult or impossible to meet them. For example, certain categories just do not occur very often. In such cases you will need to use some of the performance metrics discussed later in this chapter for assessing reliability of the ANN. Familiarity with your data is also important here. Do the values of your variables really range over their expected values for the cases in your data set If not, you may want to consider gathering more data. 

The test set should contain only cases that are not in the training set. If a test set case is also in the training set, you will merely be testing the ability of the 

---

To understand what training and test sets are, and how to make use of them... 

As a result, VEGA creates a PDF file that contains all the information about the prediction, including the final assessment of the prediction, the list of the six most similar compounds found in the training and test set of the model, the list of all Applicability Domain indices and a reasoning on SAs with a brief explanation of their meaning. 

This method for preventing overfitting requires that there are enough samples so that both training and test sets are representative of the dataset. In fact, it is desirable to have a third set known as a validation set, which acts as a secondary test of the quality of the network. The reason is that, although the test set is not used to train the network, it is nevertheless used to determine at what point training is stopped, so to this extent the form of the trained network is not completely independent of the test set. 

The solubility is expressed as logarithm of molar fractions log(S). A recommended partition of the data into training and test sets is also taken from the mentioned paper. Six outliers described in Huanxiang et al. (2005) were removed from the considerations. SMILES notations of organic solvents for this study have been obtained with ACD/ChemSketch software (http //www.acdlabs.com/) according to CAS numbers from US National Library of Medicine (http //toxnet.nlm.nih.gov/). 

Numerical data on the correlation weights and numbers of SMILES attributes in the training and test sets are presented in Table 14.2. One can see from Table 14.2 that two SMILES attributes are absent in the training set. Their correlation weights are defined as being equal to zero. In other words, these attributes are removed from the modeling process. 

Experimental values of the fullerene C60 solubility, values of the DCW for organic solvents under consideration, and the details of partition of the data into the training and test sets are shown in Table 14.4. The considered model is displayed in Figs. 14.1 and 14.2, for training and test sets, respectively. The model of the fullerene C60 solubility obtained in the first ran of the Monte Carlo method optimization is as follows ... 

A recent new trend called Active Learning substitutes the often assumed static setting of training and test set in which a learning machine is applied by the probably more realistic scenario of a continuous flow of data. The outcome of experiments influences the choice and generation of subsequent data points [155]. Active Learning provides tools that help select the most promising next subset of data to be subjected to experimentation [156]. 

Golbraikh, a. Tropsha, a. Predictive QSAR modeling based on diversity sampling of experimental datasets for the training and test set selection. 

Composition of the Training and Test Sets for the Case Study3... 

Set method of designing Training and Test Sets. The sixth Training Set (named Random) was constructed by randomly selecting molecules for the Training Set with the remainder becoming the Test Set. Statistical analysis of the predicted bioactivities was performed in Microsoft Excel 97 unless so noted. 

From the above methods of constructing QSAR models several important parameters can be realized. The composition of the Training and Test Sets play an important role in the ability of the model to predict the bioactivities of the known and novel compounds. The methods used to set up the molecules, specifically the partial charges, can have a large impact on the information extracted from the model. The type of QSAR model to construct (traditional, 3D, D) will dictate the type of information gathered from the model. 

The construction of the Training and Test Sets can have a significant impact on the ability of the model. In the traditional QSAR portion, Bioheavy models were able to adequately predict the original bioactivities for the Training and Test Set for the Hansch ( R2 = 0.86, Q2 = 0.78, R2 = 0.58) and MOE ( R2 = 0.79, Q2 — 0.69, R2 — 0.66) descriptors. This was not the case when the Biolite models were confronted with the same task. The Biolite models were unable to predict the original bioactivities for the Test Sets even though the models were able to predict the bioactivities for the Training Set Hansch descriptors ( R2 = 0.91, Q2 = 0.86, R2 = 0.00) and MOE Descriptors ( R2 = 0.84, Q2 = 0.77, R2 = 0.09). 

As has been discussed, there are several key steps to keep in mind when performing a QSAR study. They may seem minor details or trivial, yet all are important for obtaining a usable final model. From the molecules chosen for the Training and Test Sets to the number of descriptors used to create the model, all aspects of how the model was created are valuable in assessing the worthiness of the model or determining where errors may have occurred in construction of the model. The following are questions to ask when performing a QSAR study ... 

---