Solution to the Functional Estimation Problem

B. Neural Network Solution to the Functional Estimation Problem.449... 

For an introduction to NNs and their functionality, the reader is referred to the rich literature on the subject (e.g., Rumelhart et al, 1986 Barron and Barron, 1988). For our purposes it suffices to say that NNs represent nonlinear mappings formulated inductively from the data. In doing so, they offer potential solutions to the functional estimation problem and will be studied as such. 

The magnitude of /emp(g) will be referred as the empirical error. All regression algorithms, by minimizing the empirical risk /emp(g), produce an estimate, g(x), which is the solution to the functional estimation problem. 

Recall that Equation 13.18 is exactly the same as the linear prediction Equation 12.16, where = fli, 02,..., Op are the predictor coefficients and x[n] is the error signal e n. This shows that the result of linear prediction gives us the same type of transfer function as the serial formant synthesiser, and hence LP can produce exactly the same range of frequency responses as the serial formant S5mthesiser. The significance is of course that we can derive the linear prediction coefficients automatically fi om speech and don t have to make manual or perform potentially errorful automatic formant analysis. This is not however a solution to the formant estimation problem itself reversing the set of Equations 13.14 to 13.18 is not trivial, meaning that while we can accurately estimate the all-pole transfer function for arbitrary speech, we can t necessarily decompose this into individual formants. 

In this study the problem of estimating an unknown function from its examples is revisited. Its mathematical description is attempted to map as closely as possible the practical problem that the potential NN user has to face. The objective of the chapter is twofold (1) to draw the framework in which NN solutions to the problem can be developed and studied, and (2) to show how careful considerations on the fundamental issues naturally lead to the Wave-Net solution. The analysis will not only attempt to justify the development of the Wave-Net, but will also refine its operational characteristics. The motivation for studying the functional estimation problem is the derivation of a modeling framework suitable for process control. The applicability of the derived solution, however, is not limited to control implementations. 

Before attempting to formalize and solve the problem in mathematical terms, it is instructive to recognize the characteristics that are inherent to the functional estimation task and not attributable to the particular tool used to solve it. This will also set the ground on which different solutions to the problem can be analyzed and compared. 

The remainder of this chapter is structured as follows. In Section II the problem of deriving an estimate of an unknown function from empirical data is posed and studied in a theoretical level. Then, following Vapnik s original work (Vapnik, 1982), the problem is formulated in mathematical terms and the sources of the error related to any proposed solution to the estimation problem are identified. Considerations on how to reduce these errors show the inadequacy of the NN solutions and lead in Section III to the formulation of the basic algorithm whose new element is the pointwise presentation of the data and the dynamic evolution of the solution itself. The algorithm is subsequently refined by incorporating the novel idea of structural adaptation guided by the use of the L" error measure. The need... 

The formulation of the parameter estimation problem is equally important to the actual solution of the problem (i.e., the determination of the unknown parameters). In the formulation of the parameter estimation problem we must answer two questions (a) what type of mathematical model do we have and (b) what type of objective function should we minimize In this chapter we address both these questions. Although the primary focus of this book is the treatment of mathematical models that are nonlinear with respect to the parameters nonlinear regression) consideration to linear models linear regression) will also be given. 

Dynamic sets of process-model mismatches data is generated for a wide range of the optimisation variables (z). These data are then used to train the neural network. The trained network predicts the process-model mismatches for any set of values of z at discrete-time intervals. During the solution of the dynamic optimisation problem, the model has to be integrated many times, each time using a different set of z. The estimated process-model mismatch profiles at discrete-time intervals are then added to the simple dynamic model during the optimisation process. To achieve this, the discrete process-model mismatches are converted to continuous function of time using linear interpolation technique so that they can easily be added to the model (to make the hybrid model) within the optimisation routine. One of the important features of the framework is that it allows the use of discrete process data in a continuous model to predict discrete and/or continuous mismatch profiles. 

---