Normally-distributed noise

Now we are ready to consider what happens if the data are noisy. We will take the data we just used and add some noise to it. We will add normally distributed noise to each wavelength of each spectrum at a level of approximately 5%. It is important to understand that, within a given spectrum, the particular amount of noise added to each wavelength is independent of the noise added to the other wavelengths. And, of course, the noise we add to each spectrum is independent of the noise added to the other spectra. In other words, there is no correlation to the noise. Figure 39 contains a plot of the data before and after the addition of the noise. Figure 40 show two other views of the data after the additon of the noise. 

To better understand this, let s create a set of data that only contains random noise. Let s create 100 spectra of 10 wavelengths each. The absorbance value at each wavelength will be a random number selected from a gaussian distribution with a mean of 0 and a standard deviation of 1. In other words, our spectra will consist of pure, normally distributed noise. Figure SO contains plots of some of these spectra, It is difficult to draw a plot that shows each spectrum as a point in a 100-dimensional space, but we can plot the spectra in a 3-dimensional space using the absorbances at the first 3 wavelengths. That plot is shown in Figure 51. 

Figure 50. Some spectra consisting of pure, normally distributed noise.

Figure 52. Eigenvalues (...) and reduced eigenvalues ( ) for the spectra consisting of pure, normally distributed noise.

Purpose Generate a data set that superimposes normally distributed noise on a linear calibration model to study the effects of the adjustable parameters. A whole calibration—measurement—evaluation sequence can be optimized for quality of the results and total costs. 

Here again, our task is simplified by the two facts we have mentioned above first, that we can reuse many of the results we obtained previously for the case of Normally distributed noise, and second, that the nature of uniformly distributed noise characteristics simplify the mathematical analysis. Our first step in this analysis starts with equation 44-71, that we derived previously in Chapter 44 referenced as [5] as a general description of noise behavior ... 

The figure corresponding to Figure 45-9 (which appeared in Chapter 45 [6]) that was calculated for Normally distributed noise is Figure 46-15, which presents the results of calculating the variance of the two terms of equation 44-77 for uniformly distributed noise instead. We note that while these terms follows the same trends as the Normally distributed errors, these errors do not become appreciable until Er has fallen below 0.6, which corresponds to the point where values occur close to or less than zero. For values of Er below 0.6 the values of both terms of equation 44-77 become very large and erratic. 

Equation 49-130 is now exactly in the form of 2fWP = Y [F(X) F(X)] (times a scaling factor) as we started with in equation 49-121, and is now in a form that can be more easily worked with. More importantly, it is also in a form that is useful and convenient it is in the form of T times a multiplying factor. It now remains to find out the nature and behavior of the multiplying factor. We will therefore now investigate the behavior of equation 49-130, similarly to the way we investigated equation 49-126, and for that matter, the corresponding equation 43-62 for the case of Normally distributed noise [4],... 

Measurment Residual Plot There are residual plots for each unknown sample for every SIMCA model. Tlie residual spectra for samples that belong to a class are expected to resemble in magnitude and shape normally distributed noise as fotsrd in the training set Depending on the structure of the residuals, it may be possible to identify failures in the instrument (e.g., excessive noise) or chemical differences between tlie calibration and unknown samples (e.g., peaks in the residuals). The residual plot may help identify why a sample is not classified iiso any given class. 

As mentioned previously, the task of model-based data fitting for a given matrix Y is to determine the best rate constants defining the matrix C, as well as the best molar absorptivities collected in the matrix A. The quality of the fit is represented by the matrix of residuals, R = Y - C x A. Assuming white noise, i.e., normally distributed noise of constant standard deviation, the sum of the squares, ssq, of all elements is statistically the best measure to be minimized. This is generally called a least-squares fit. 

SlMl.dat Section 1.4 Five data sets of 200 points each generated by SIM-GAUSS the deterministic time series sine wave, saw tooth, base line, GC-peak, and step function have stochastic (normally distributed) noise superimposed use with SMOOTH to test different filter functions (filer type, window). A comparison between the (residual) standard deviations obtained using SMOOTH respectively HISTO (or MSD) demonstrates that the straight application of the Mean/SD concept to a fundamentally unstable signal gives the wrong impression. 

Nine error distributions were generated according to a contaminated normal distribution noise = (l-e) N(0, a ) + e N(5, (11)... 

In spite of its prevalence in the fluorescence decay literature, we were not universally successful with this fitting method. Most reports of hi- or multiexponential decay analysis that use a time-domain technique (as opposed to a frequency-domain technique) use time-correlated photon counting, not the impulse-response method described in Section 2.1. In time-correlated photon-counting, noise in the data is assumed to have a normal distribution. Noise in data collected with our instrument is probably dominated by the pulse-to-pulse variation of the laser used for excitation this variation can be as large as 10-20%. Perhaps the distribution or the level of noise or the combination of the two accounts for our inconsistent results with Marquardt fitting. 

FIGURE 6.3.1 Data on particle size distributions for the determination of nucieation and particle growth rates generated by forward simulations of the population balance equation with the addition of 3% normally distributed noise. (From Mahoney 2000.)... 

---