4. Least Squares Fitting

Question 1

What is the general linear regression function for a line?

Answer 1

y=a+bx

Question 2

In the function y=a+bx, what do ‘a’ and ‘b’ represent?

Answer 2

a= intercept
b=gradient

Question 3

What is the formula for total error, ‘E’, in the method of least squares?

What symbols should be drawn on top of and below the ‘sum of’ symbol, ‘∑’

Answer 3

E= ∑(yi-(a+bxi))²

n on top and i=1 on the bottom

Question 4

What part of the error formula do you need to change if you are using a non linear fitting function?

E= ∑(yi-(a+bxi))²

For example if the function is y=ax+b/x

Answer 4

You change: (a+bxi)

It would become:

E= ∑(yi-(axi+b/xi))²

Question 5

What are the general steps to Least Squares Regression, i.e. proving the normal equations at a particular linear or non-linear function

For example using: ∑(yi-a-bxi)²

Answer 5

• Write out the total error, ‘E’ formula: E= ∑(yi-(a+bxi))²

-If non-linear, replace ‘a+bxi’ with the relevant residuals for ‘a’ and ‘b’

• Set the partial derivatives of ‘a’ and ‘b’ with respect to ‘E’ equal to 0
• Derive the function to get: ∑2(yi-a-bxi) x (-1) = 0
• Rearrange to get: -2∑(yi-a-bxi) x 1 = 0
• Remove ‘-2’ and multiply to get: ∑(yi-a-bxi) = 0
• Expand the brackets to get: ∑yi-∑a-∑bxi = 0
• Change ‘∑’ on any residuals without xi to ‘n’, and factor out the residual from any sum to get: ∑yi-na-b∑xi = 0
• Rearrange to solve for ∑yi (in this case) and multiply by -1 to get a positive function: na+b∑xi=∑yi

Question 6

What is the following function referred to as:

E= ∑(yi-(a+bxi))²

Answer 6

The objective function

Question 7

Why do we take the first derivation of ‘a’ and ‘b’ with respect to E?

Answer 7

The total error can only reach minimum at stationary points, hence the partial derivatives are set simultaneously to 0

Question 8

What is the general algorithm for the Least Squares Fitting process?

Answer 8

1.) Construct the objective function (the total error):
E= ∑(yi-(a+bxi))²

2.) Evaluate partial derivatives of the objective function with respect to the unknown coefficients

3.)Equate the partial derivatives to zero and solve the resulting system of linear equations

4.) The found fitting coefficients produce the best fit

Question 9

How would you derive:
E= ∑(yi-a-bxi)²

With respect to:

1.) ‘a’

2.) ‘b’

Answer 9

1.) wrt ‘a’: ∑2(yi-a-bxi)-(1)

2.) wrt ‘b’: ∑2(yi-a-bxi)-(xi)

Question 10

Why is -2 factored out of the expression during least squares fitting

Answer 10

As the equation is set to zero, multiplying it by -2 still equals zero, so it can be removed

Question 11

Why is ‘∑’ replaced with ‘n’ when considering a constant without ‘xi’ in the process?

Answer 11

∑a = na

The sum of all a values is also equal to the number of values multiplied by a

Question 12

How should the equations be rearranged before multiplying everything by (-1)

Answer 12

Put all unknown variables on the left and all known variables on the right, i.e. ‘a’ and ‘b’ coefficients on the left and ‘y’ coefficients on the right

Question 13

Solve the following problems:

i.) b/xi x xi

ii.) b/xi x 1/xi

iii.) axi² x xi²

iv.) b/xi x xi²

v.) axi⁵⁄₂ x xi⁵⁄₂

vi.) bxi-¹⁄₂ x xi⁵⁄₂

Answer 13

i.) b

ii.) b/xi²

iii.) axi⁴

iv.) bxi

v.) axi¹⁰⁄₂ or axi⁵

vi.) bxi⁴⁄₂ or bxi²

Question 14

How can b/√xᵢ be re-written in a simpler way for computation

Answer 14

bxi⁻¹⁄₂

Question 15

How do you know what values to tabulate and sum in the part b questions?

Answer 15

You need to sum all of the values in the normal equations given, for example ∑x or ∑y

Question 16

How do you solve the normal equations when you have substituted your values in?

Answer 16

Multiply each equation by the factor of ‘a’ in each other equation, this will allow you to substitute one equation from another in order to solve for ‘b’.

For example

(1) 5a + 7.5b = 100 (x3) = 15a + 22.5b = 300

(2) 3a + 9b = 105 (x5) = 15a + 45b = 525

(2) - (1) = 22.5b = 225 ; therefore b = 225/22.5 = 10

Question 17

Once you have solved for ‘b’, how do you solve for a?

For example: b=10, and, 5a +7.5b=100

Answer 17

a= (100-7.5(10))/5=5

Substitute ‘b’ back into one of the original equations and solve for ‘a’.

Question 18

When you have solved for both ‘a’ and ‘b’, how do you present your answer?

Use y=ax²+b/x as an example where a=5 and b=10

Answer 18

The non-linear regression equation is: y=5x²+10/x

Question 19

The correlation coefficient ‘r’ is used to measure the accuracy of fitting a straight line through data points. What values does it range between?

Answer 19

-1 to +1

Question 20

Describe the correlation of the following ‘r’ values:

r=1
r=0.8
r=0
r=-0.8
r=-1

Answer 20

r=1 is perfect positive correlation
r=0.8 or greater is high positive correlation
r=0 is no correlation
r=-0.8 or less is high negative correlation
r=-1 is perfect negative correlation