Statistical algorithms

A statistical algorithm, also known as linear regression analysis, for systems where Y (the random variable) is linearly dependent on another quantity X (the ordinary or controlled variable). The procedure allows one to fit a straight line through points xi, y0, X2,yi), x, ys),..., ( n,yn) where the values jCi are defined before the experiment and y values are obtained experimentally and are subject to random error. The best fit line through such a series of points is called a least squares fit , and the protocol provides measures of the reliability of the data and quality of the fit. 

One approach is based on the assumption that the spectrum does not change its intensity from frame to frame other than due to noise and cosmic rays. Therefore, by comparing multiple neighboring frames, a statistical algorithm can be used to identify cosmic rays. Another solution compares adjacent pixels in the same spectrum and detects abrupt jumps in intensity from pixel to pixel. Once a cosmic ray contaminated pixel is identified, its value can be replaced by the average of neighboring pixels. 

Exploratory data analysis (EDA). This analysis, also called pretreatment of data , is essential to avoid wrong or obvious conclusions. The EDA objective is to obtain the maximum useful information from each piece of chemico-physical data because the perception and experience of a researcher cannot be sufficient to single out all the significant information. This step comprises descriptive univariate statistical algorithms (e.g. mean, normality assumption, skewness, kurtosis, variance, coefficient of variation), detection of outliers, cleansing of data matrix, measures of the analytical method quality (e.g. precision, sensibility, robustness, uncertainty, traceability) (Eurachem, 1998) and the use of basic algorithms such as box-and-whisker, stem-and-leaf, etc. 

Disadvantages arise mainly from the complexity of the statistical algorithms and the fact that fitting models to data is time consuming. The first-order (EO) method used in NONMEM also results in biased estimates of parameters, especially when the distribution of inter individual variability is specified incorrectly. The first-order conditional estimation (EOCE) procedure is more accurate but is even more time consuming. The objective function and adequacy of the model are based in part on the residuals, which for NONMEM are determined based on the predicted concentrations for the mean pharmacokinetic parameters rather than on the predicted concentrations for each individual. Therefore, the residuals are confounded by intraindividual, inter individual, and linearization errors. 

Similar conditions related to the sample size apply to the investigations with neural networks, which are, in fact, nothing more than a more complex statistical algorithm. Since a network minimizes an overall error, the proportion of types of data in the set is critical. A network trained on a data set with 900 good cases and 100 bad ones will bias its decision toward good cases, as this allows the algorithm... 

The use of RDF descriptors suggests a way to calculate a single value that describes the diversity of a data set by means of descriptive statistics. A series of statistical algorithms is available for evaluating the similarity of larger data sets. By statistical analysis, the diversity — or similarity — of two data sets can be characterized. Two methods are straightforward. 

The choice of reference spectra is carried out, on one hand, manually from real (deterministic) spectra, with the consideration of determined spectra (nitrate, nitrite, surfactants). It is completed automatically, on the other hand, with a mathematical procedure [15] allowing the selection of the more relevant spectra for the model, able to explain by a linear combination, the shape of UV spectra of water or wastewater. This last procedure can be replaced by any advanced statistical algorithms (PCA or PLS, for example) or by commercial software (such as UVPro from Secomam). 

---