Home » Numerical Techniques Using C++ » METHOD OF LEAST SQUARES || CURVE-FITTING


What is empirical law?

In various branches of Applied Mathematics, it is required to express a given data obtained from observations, in the form of a law connecting the two variables involved. Such a law inferred by some scheme, is known as the empirical law.

Example: If we need to obtain a law connecting the length and the temperature of a metal bar. The length of metal bar is measured at various temperatures. Then by different methods (Curve fitting, Scatter diagram, etc), a law is obtained that represents the relationship existing between temperature and length of metal bar for the observed values. Then this relationship can be used to predict the length at an arbitrary temperature.

What is curve fitting?

To find a relationship between the set of paired observations (say) x and y, we plot their corresponding values on the graph, taking one of the values along x-axis and other along the y-axis. To find the equation of the curve of ‘best fit’ which may be the most suitable for predicting the unknown values. The process of finding such an equation of ‘best fit’ is known as curve-fitting.

Principle of Least Squares.

The Principle of Least Squares was suggested by a French Mathematician Adrien Marie Legendre in 1806. It states that ‘ The curve of best fit is that for which e’s (errors) are as small as possible i.e., the sum of the squares of the errors is a minimum.

The principle of least squares, provides an elegant procedure of fitting a unique curve to a given data.
Let the curve y=a + bx+ cx2 + …….+kxm …………..(1)
be fitted to the set of data points (x1, y1), (x2, y2), ………, (xn, yn).
Now we have to determine the constants a, b, c, …., k such that it represents the curve of best fit. In case n=m, on substituting the values (xi , yi) in (1), we get ‘n’ equations from which a unique set of ‘n’ constants can be found. But when n>m, we obtain n equations which are more than the m constants and hence cannot be solved for these constants. So we try to determine the values of a, b, c, ….., k which satisfy all the equations as nearly as possible and thus may give the best fit. In such cases, we apply the principle of least squares.

At x=xi, the observed(experimental) value of the ordinate is yi and the corresponding value on the fitting curve (1) is a + bxi+ cxi2 + …….+kxim (=Ƞi, say) which is the expected (or calculated) value (see figure). The difference fo the observed and the expected values i.e., yi – Ƞi (= ei ) is called the error at x=xi. Clearly some of the errors e1, e2, ….., en will be positive and others negative. Thus to make the sign of each error equal, we square each of them and form their sum i.e. E= e12 + e22 + … + en2.
So when E is minimum the curve is the curve of ‘best fit’.

method of least squares

Method of Least squares.

Now we learn how to use Least squares method, suppose it is required to fit the curve
y= a + bx + cx2
to a given set of observations
(x1,y1), (x2,y2), …. , (x5,y5).
For any xi, the observed value is yi and the expected value is
Ƞi = a + bxi + cxi
so that the errors ei = yi – Ƞi.
Therefore, The sum of the squares of these errors is
E = e12 + e22 + … + e52
= [y1 -(a + bx1 + cx12)]2 + [y2 -(a + bx2 + cx22)]2 + ……… + [y5 -(a + bx5 + cx52)]2
For E to be minimum, we have

method of least squares

Equation (1) simplifies to
y1 + y2 + … + y5 = 5a + b(x1 + x2 + … + x5) + c( x12 + x22 + … + x52)
Σyi = 5a + b Σxi + c Σxi2 ………………..(4)
similarly (2) and (3) becomes
Σxi yi = aΣxi+ bΣxi2 + cΣxi3 ………………..(5)
Σxi2yi = aΣxi2 + bΣxi3+ cΣxi4 ………………..(6)

The equations (4), (5) and (6) are known as Normal equations and can be solved as simultaneous equations in a, b, c. The values of these constants when substituted in (1) give the desired curve of best fit.