Robustness testing level selection

In robustness tests, the selected range between the levels should represent the variability that can occur when transferring the method (4, 5,16,18,25). However, specifications to estimate such variability are not given in regulatory documents, such as the ICH guidelines. Often the extreme levels are chosen based on personal experience, knowledge, or intuition. Sometimes they are defined as nominal level + x%. However, this approach based on relative variation is not appropriate because the absolute variation then depends on the value of the nominal level (18). Another systematic approach defines the levels based on the precision or the uncertainty with which they can be set (5, 18). The uncertainty can be estimated for the nominal factor level (18, 26). If the uncertainty or absolute error on a measured pH value... 

The Scheffe procedure is powerful because of it robustness, yet it is very conservative. Type I error (the false positive rate) is held constant at the selected test level for each comparison. 

AIMS/OBJECTIVES AND STEPS IN A ROBUSTNESS TEST SELECTION OF FACTORS AND LEVELS... 

Qualitative factors are also frequently considered in a robustness test. " For CE methods, factors such as the batch or manufacturer of the capillary, reagent or solvent can be selected. When evaluating the influence of such qualitative factor, the analyst should be aware that the estimated effect is only valid or representative for the examined discrete levels and not for any other level of that factor, and certainly not for the whole population. For example, when examining two capillaries X and Y, the estimated effect only allows drawing conclusions about these two capillaries and not about other capillaries available on the market. Such approach allows evaluating whether capillary Y is an alternative for capillary X, used, for instance, to develop the method. 

In a robustness test the following steps can be identified (a) identification of the variables to be tested, (b) definition of the different levels for the variables, (c) selection of the experimental design, (d) definition of the experimental protocol, (e) definition of the responses to be determined, (f) execution of the experiments and determination of the responses of the method, (g) calculation of effects, (h) statistical and/or graphical analysis of the effects, and (i) drawing chemically relevant conclusions from the statistical analysis and, if necessary, taking measures to improve the performance of the method. A general overview of robustness testing can be found in [35). 

The levels selected in a robustness test are different from those at which factors are evaluated in method optimization. For optimization purposes the variables are examined in a broad interval. In robustness testing the levels are much less distant. They represent the (somewhat exaggerated) variations in the values of the variables that could occur when a method is transferred. For instance, in optimization the levels for pH would be several units apart, while in robustness testing the difference could be 0.2 pH units. The levels can for instance be defined based on the uncertainty with which a factor level can be set and re.set 36 and usually they are situated around the method (nominal) conditions if the method specifies pH 4.0, the levels would be 3.9 and 4.1. The experimental designs used are in both situations the same and comprise fractional factorial and Plackett-Burman designs. 

The main goal of a robustness test is to examine potential sources (factors) causing variability in one or more responses of the method. To identify those sources, a number of factors, usually specified with a nominal level in the operating procedure of the method, are selected. These factors are then varied in an interval, representative for the fluctuations in the nominal factor levels,... 

In method optimization, the range between the levels is much larger than in robustness tests. Often, the range selected for a factor in optimization represents the broadest interval in which the factor can be varied with the technique considered. In practice, the examined range is chosen based on earlier gathered knowledge and/or information from the literature. 

In robustness testing, the extreme levels are most frequently chosen symmetrically around the nominal for mixture-related and quantitative factors. However, for some factors, an asymmetric interval might better represent the reality or better reflect the change in response occurring. A first example is the capillary temperature. Suppose a capillary temperature of 15 °C is prescribed. Symmetric levels, selected based on uncertainty are, for instance, 10 °C and 20 °C. However, many cooling systems do not allow temperatures of more than 10 °C below room temperature therefore, 10 °C may not be attained accurately by the instrument. The lowest extreme level could then be taken equal to the nominal (15 °C). 

The four factors in Table 2.4 were selected from a robustness test on a CE method to determine rufloxacin hydrochloride in coated tablets (29). AU factors were quantitative (A-D) and their extreme levels are situated symmetrically around the nominal. 

The diligent analyst would develop a robust method with rigorous matrix effect tests on multiple lots, including hemolyzed and lipidemic samples. An initial test would be a spike-recovery evaluation on at least six individual lots. Samples should be spiked at or near the LLOQ, and at a high level near the ULOQ. If matrix interference were indicated by unacceptable relative error (RE) percentage in certain lots, the spiked sample of the unacceptable lots should be diluted with the standard calibrator matrix to estimate the minimum dilution requirement (MDR) at and above which the spike-recovery is acceptable. The spike-recovery test should then be repeated with the test samples diluted at the MDR. Note that this approach will increase the LLOQ for a less sensitive assay. If sensitivity is an issue, then other venues will be required to address the matrix effect problem. For example, the method can be modified to include sample clean-up, antibodies and/or assay conditions may be changed, or the study purpose may be tolerable to acknowledge that the method may not be selective for a few patients (whose data may require special interpretation). 

Reverse-phase protein arrays offer a robust new method of quantitatively assessing expression levels and the activation status of a panel of proteins. For this purpose, the lysate of protein(s) of interest is arrayed without selection via a capture molecule. This array can then be queried with an antibody or ligand probe, or an unknown biological component. Since an individual test sample is immobilized in each array spot, this array can be composed of a variety of different patient samples. Each array is incubated with one detection protein or antibody, and a single end point is measured across the arrayed cohort and can be directly compared across multiple samples. Replicates can be reproducibly printed at a given sitting, increasing quality control over a series of queried arrays (reviewed in [33]). 

When the EP comprises linear computations (linear in the observations) such as simple differences, y - B, or linear least squares or linear multivariate computations, initial normality (of the observations y) is preserved for the estimated quantities. Non-linear computations, such as arise commonly in iterative model selection and peak search routines, produce estimated parameters having non-normal distributions (59). Caution is in order, in those cases, in applying "normal" values of test statistics to calculate 1 and Cl s. (Other factors to consider are the extent of non-linearity, the level of confidence or significance [1-a], and the robustness of the statistic in question.)... 

The WEKA software suite [23] has been used in carrying out the experiments. The results were evaluated using Accuracy (Acc). For the training and validation steps, we used k-fold cross-validation with k = 10. Cross-validation is a robust validation method for variable selection [24]. Repeated cross-validation (as calculated by the WEKA environment) allows robust statistical tests. We also use the measurement provided automatically by WEKA Coverage of cases (0.95 level). 

In methods like TLC, validation must be performed in more levels. In the basic level, a scanner must be checked and calibrated according to the manufacturer s specifications. In the second level, it must be validated with a special test plate. This must be done at least once in a year, in order to obtain data about mechanical robustness, repeatability of measurements, monochromator accuracy, baseline noise, and the signal-to-noise ratio. This test plate is prepared by a vendor, and its purpose is detection of any possible malfunction. Too often, use of this informative but time-consuming and complicated test operation is questionable, especially because people in the laboratory are usually not able to repair a scanner, so a service-call is required. In order to know the actual quality of a scanner, a user must prepare a simpler procedure. This procedure can use a part of a vendor test plate, but it is better for each user to prepare an additional in-house test plate. The purpose of such a test is to check the quality of a scanner under working conditions on a daily basis. For accurate results, a third level may be introduced. The best way is to make a system-suitability test on each plate, selecting one spot on a plate, and measuring it with the working paramaters. 

---