Empirical risk minimization

Support vector machine (SVM) is originally a binary supervised classification algorithm, introduced by Vapnik and his co-workers [13, 32], based on statistical learning theory. Instead of traditional empirical risk minimization (ERM), as performed by artificial neural network, SVM algorithm is based on the structural risk minimization (SRM) principle. In its simplest form, linear SVM for a two class problem finds an optimal hyperplane that maximizes the separation between the two classes. The optimal separating hyperplane can be obtained by solving the following quadratic optimization problem ... 

Support vector machine (SVM) is a widely used machine learning algorithm for binary data classification based on the principle of structural risk minimization (SRM) [21, 22] unlike the traditional empirical risk minimization (ERM) of artificial neural network. For a two class problem SVM finds a separating hyperplane that maximizes the width of separation of between the convex hulls of the two classes. To find the expression of the hyperplane SVM minimizes a quadratic optimization problem as follows ... 

According to the principle of empirical risk minimization (ERM) it is necessary to depress the training error. But this is not enough, since the risk of prediction still contains another term for risk due to overfitting ... 

A common belief is that because SVM is based on structural risk minimization, its predictions are better than those of other algorithms that are based on empirical risk minimization. Many published examples show, however, that for real applications, such beliefs do not carry much weight and that sometimes other multivariate algorithms can deliver better predictions. 

The minimization of the expected risk given by Eq. (1) cannot be explicitly performed, because P(, y) is unknown and data are not available in the entire input space. In practice, an estimate of 7(g) based on the empirical observations is used instead with the hope that the function that minimizes the empirical risk 7g p(g) (or objective function, as it is most commonly referred) will be close to the one that minimizes the real risk 7(g). 

In light of the previous discussion and contrary to the established practice, we propose the use of the maximum absolute error (6) as the empirical risk to be minimized because it offers the following advantages ... 

We want to prove that, if this is the case, then only solutions with Kg) < e will be produced by the minimization of the empirical risk, and convergence in this weak sense will be guaranteed. Let g be a function such that Kg ) > s. Then from Eq. (25)... 

The magnitude of /emp(g) will be referred as the empirical error. All regression algorithms, by minimizing the empirical risk /emp(g), produce an estimate, g(x), which is the solution to the functional estimation problem. 

Learning Problem. At every instant l let Zt = (x, y() e Rkxli = 1,2,...,/ be a set of data drawn with some unknown probability distribution, P(, y) and Gt, a space of functions. Find a function g/(x) X - Y belonging to G( that minimizes the empirical risk Iemp(g). 

Given a set of functions, the relationship between the empirical risk and the actual risk for this set of functions is one of most important research directions, which is known as the bounds on generalization ability of learning machines. As for binary classification problems, the following basic bounds describing the generalization ability of a threshold real-valued function (also known as indicator function) that minimize the empirical risk functional [131] ... 

According to Theorem 2.2, given some selection of learning machines whose empirical risk is zero, one wants to choose that learning machine whose associated set of functions has minimal VC dimension. At present, for the y-margin separating hyperplane, we quote an important theorem without proof as follows. For more details, see [132]. 

The VC confidence term in Eq. [8] depends on the chosen class of functions, whereas the empirical risk and the actual risk depend on the particular function obtained from the training algorithm. It is important to find a subset of the selected set of functions such that the risk bound for that subset is minimized. A structure is introduced by classifying the whole class of functions into nested subsets (Figure 20), with the property dvc,i < dwc,i < dye,3- For each subset of functions, it is either possible to compute dye or to get a bound on the VC dimension. Structural risk minimization consists of finding the subset of functions that minimizes the bound on the actual risk. This is done by training for each subset a machine model. For each model the goal is to minimize the empirical risk. Subsequently, one selects the machine model whose sum of empirical risk and VC confidence is minimal. 

An approach that is sometimes helpful, particularly for recent pesticide risk assessments, is to use the parameter values that result in best fit (in the sense of LS), comparing the fitted cdf to the cdf of the empirical distribution. In some cases, such as when fitting a log-normal distribution, formulae from linear regression can be used after transformations are applied to linearize the cdf. In other cases, the residual SS is minimized using numerical optimization, i.e., one uses nonlinear regression. This approach seems reasonable for point estimation. However, the statistical assumptions that would often be invoked to justify LS regression will not be met in this application. Therefore the use of any additional regression results (beyond the point estimates) is questionable. If there is a need to provide standard errors or confidence intervals for the estimates, bootstrap procedures are recommended. 

There is a lack of empirical surveys discussing how to increase safety and resilience in the oil and gas industry in a proactive manner. By safety we mean freedom from unacceptable risks resilience is defined as the ability of a system or an organization to react to and recover from disturbances at an early stage, with minimal effect on the dynamic stability , both from Hollnagel (2006). By proactive we mean acting in anticipation of future problems . 

---