The Newton method

The main idea of the Newton jnethod is to try to solve tlu problem of ininimizat.ion in one stej)  

To determine this specific direction, Am, let us calculate the misfit functional for this first iterat ion 

Using an adjoint operator for the Frcxdict derivative, we find 

Note that, according to Theorem 81 of Appendix D, the first variation of the rni.sfit functional at the minimum must he equal to zero  

It is (liffieull, to find the exact solution of the opeiator e([uation (5.37). How-ev( r, one can siinplilv tliis problem liy linearization of tin- o])erator A(m(, + Am), using a, I reche( deii -a(i c opi rator  

---

Another option is a q,p) = p and b q,p) = VU q). This guarantees that we are discretizing a pure index-2 DAE for which A is well-defined. But for this choice we observed severe difficulties with Newton s method, where a step-size smaller even than what is required by explicit methods is needed to obtain convergence. In fact, it can be shown that when the linear harmonic oscillator is cast into such a projected DAE, the linearized problem can easily become unstable for k > . Another way is to check the conditions of the Newton-Kantorovich Theorem, which guarantees convergence of the Newton method. These conditions are also found to be satisfied only for a very small step size k, if is small. 

A simple way of finding the roots of an equation, other than by divine inspiration, symmetry or guesswork is afforded by the Newton method. We start at some point denoted x l along the x-axis, and calculate the tangent to the curve at... 

The most frequently used methods fall between the Newton method and the steepest descents method. These methods avoid direct calculation of the Hessian (the matrix of second derivatives) instead they start with an approximate Hessian and update it at every iteration. 

Roots of implicit equations and extrema of functions the Newton method... 

Table 3.2. Calculation of N/2 as a solution to the equation x2 — 2 = 0 by the Newton method. Compare with the true value of 1.41421.

The Newton method can also be used to find the extremum (maximum or minimum) of a function/(x), i.e., the value for which the first derivative/ (x) is zero. The iterative search for the extremum is implemented by a formula derived from equation (3.1.29)... 

Table 3.3. Iterative calculation of the age of an isochron with a slope of0.256 in the 207Pbl204Pb vs 206Pb/204Pb diagram by the Newton method.

The Newton method can be extended to several variables in order to find the zeroes of n functions/ in n variables x which we lump as the vector x = [x,..., x ]T, i.e., to solve the system of equations... 

Successive iterations converge extremely fast. After the fifth step, the results hardly change (Table 5.24) which, using the Newton method outlined in Section 3.1, indicates an age of T = 2.9065 Ga. 

The Newton method will set the next estimate x k+ to the minimum point... 

The general features of the Newton method are very well known. Nevertheless, it is perhaps worthwhile to offer a very brief review for the scalar case, which is finding a solution to F(y) = 0. The algorithm is stated as... 

In Fig. 15.8 notice that during the time integration, the steady-state residuals increased for a period as the transient solution trajectory climbed over a hill and into the valley where the solution lies. This behavior is quite common in chemically reacting flow problems, especially when the initial starting estimates are poor. In fact it is not uncommon to see the transient solution path climb over many hills and valleys before coming to a point where the Newton method will begin to converge to the desired steady-state solution. 

The Newton method fares no better here see the code below and its output on the next page. 

The equilibrium distribution of particles over sites of various types was found from system (A.4)-(A.12) by any iterative method (as rule, the Newton method is preferable) for selected 6, or. P sets. 

We are now ready to implement the Newton method. The D row is an approximation to C and we wish to correct D. For details of the Newton method used on a set of nonlinear equations, see a text like Press et al. [452]. More briefly here, Taylor expansion of the system (8.66) around the current D to the corrected D + d where d, is the correction term row, produces the set of equations linear in d,... 

Essentially everything has now been given. The six-equation set must be solved numerically, and the Newton method works very well, requiring normally only 2 3 iterations at most, since the changes over a given time interval are relatively small. For this purpose the unknowns are gathered into the unknowns vector X = [C 0 CB 0 G A G B G c p ]T Further treatment is now confined to a concrete example. 

The Newton/sparse matrix methods now used by electrical engineers have become the solution method of choice. Hutchison and his students at Cambridge were among the first chemical engineers to publish this approach, in the early 1970s. They used a quasi-linear model rather than a Newton one, but the ideas were really very similar. (It appears that the COPE flowsheeting system of Exxon was Newton based it existed in the mid-1960s but slowly evolved into a sequential modular system. One must assume the Newton method failed to compete.)... 

One continuation method reconstructs exactly the Newton method when t moves in the positive direction. Think of the surface that corresponds to summing the squares of the functions one wishes to drive to zero. If the Newton method flounders in a local hole in this surface where the bottom of the hole does not reach down to zero, and thus where the equations do not have a solution, it would be very useful to climb out of the hole by going in the reverse of the Newton direction (i.e., by simply reversing the sign on f), hopefully over the top of a nearby ridge and down the other side into a hole where a solution does exist. A continuation method does just this. 

In describing the steps of a CG method to solve Ax = — b, the residual vector Ax + b is useful. We define r = —(Ax + b) and use the vectors dA below to denote the CG search vectors (for reasons that will become clear in the Newton Methods section). The solution x can then be obtained by the following procedure, once a starting point Xq is specified.78 79... 

On the contrary, the Newton method gives robusmess and quadratic rate of convergence. The main drawback of this algorithm is the computation of a Jacobian matrix and the resolution of a large linear system at each step. This method has been largely described and used successfully. The reader is refered to [9] for more details. 

As described in 3.2, the solution of the discretized non linear system of equations is difficult to compute. This is now well known and various adaptations of the Newton method... 

Figure 5-4 The plot of the misfit functional value as a function of model parameters m. In the framework of the Newton method one tries to solve the problem of minimization in one step. The direction of this step is shown by the arrows in the space M of model parameters and at the misfit surface.

Of course, it is usually not enough to use only one iteration for the solution of a nonlinear inverse problem in the framework of the Newton method (because we used the linearized approximation (5.38)). However, we can construct an iterative process based on the relationship (5.43) ... 

---