Many a times, you may not have the privilege or knowledge of the physics of the problem to dictate the type of regression model. You may want to fit the data to a polynomial. But then how do you choose what order of polynomial to use.
Do you choose based on the polynomial order for which the sum of the squares of the residuals, Sr is a minimum? If that were the case, we can always get Sr=0 if the polynomial order chosen is one less than the number of data points. In fact, it would be an exact match.
So what do we do? We choose the degree of polynomial for which the variance as computed by
Sr(m)/(n-m-1)
is a minimum or when there is no significant decrease in its value as the degree of polynomial is increased. In the above formula,
Sr(m) = sum of the square of the residuals for the mth order polynomial
n= number of data points
m=order of polynomial (so m+1 is the number of constants of the model)
Let’s look at an example where the coefficient of thermal expansion is given for a typical steel as a function of temperature. We want to relate the two using polynomial regression.
Temperature |
Instantaneous Thermal Expansion |
^{o}F |
1E-06 in/(in ^{o}F) |
80 |
6.47 |
40 |
6.24 |
0 |
6.00 |
-40 |
5.72 |
-80 |
5.43 |
-120 |
5.09 |
-160 |
4.72 |
-200 |
4.30 |
-240 |
3.83 |
-280 |
3.33 |
-320 |
2.76 |
If a first order polynomial is chosen, we get
, with Sr=0.3138.
If a second order polynomial is chosen, we get
with Sr=0.003047.
Below is the table for the order of polynomial, the Sr value and the variance value, Sr(m)/(n-m-1)
Order of polynomial, m |
Sr(m) |
Sr(m)/(n-m-1) |
1 |
0.3138 |
0.03486 |
2 |
0.003047 |
0.0003808 |
3 |
0.0001916 |
0.000027371 |
4 |
0.0001566 |
0.0000261 |
5 |
0.0001541 |
0.00003082 |
6 |
0.0001300 |
0.000325 |
So what order of polynomial would you choose?
From the above table, and the figure below, it looks like the second or third order polynomial would be a good choice as very little change is taking place in the value of the variance after m=2.
This post is brought to you by Holistic Numerical Methods: Numerical Methods for the STEM undergraduate at http://numericalmethods.eng.usf.edu
Subscribe to the feed to stay updated and let the information follow you.
Hi, how exactly do you calculate Sr? I read your definition of it several times but still clues. Please let me know. Thanks.
LikeLike
Hello Ken:
To find what Sr is about http://numericalmethods.eng.usf.edu/mws/gen/06reg/mws_gen_reg_txt_straightline.pdf
To see how Sr is calculated for this example, go to page 7 of http://numericalmethods.eng.usf.edu/mws/gen/06reg/mws_gen_reg_txt_nonlinear.pdf
LikeLike
Very nice article but I cannot seam to know how to interpret the following sentence:
We choose the degree of polynomial for which the variance as computed by
Sr(m)/(n-m-1)
is a minimum or when there is no significant decrease in its value as the degree of polynomial is increased
So, what does “significant decrease” mean? Is it statistical significance test? Or you just set a threshold, say 0.001, and choose the order when the change in the Sr(m)/(n-m-1) drops below this threshold?
Could you help me out on this issue?
Thanks!
LikeLike
There is no rule of thumb that I know of. If there is a minimum at a low order of polynomial, that is an indication of the optimum polynomial as well. If there is any threshold, it should be a relative number.
LikeLike
Thanks for your answer!
I am working to find an algorithmic/automatic way of predicting the order of the polynomial.
But with not much luck!
However I found ways to predict the order of the polynomial and accept that it can overestimate the true order… it can do that with 83% accuracy.
I’m still crunching numbers!
LikeLike
This is very nice technique, but I cannot understand the meaning of the denominator ‘n-m-1’.
Is it a weighting?
Thanks!
LikeLike
Hi, this is good article.
However I cannot understand the meaning of (n-m-1).
Is this a weighting?
Also, is there any reference of this article?
Please tell me it.
Thank you.
LikeLike
Yes, it is weighting. If n=m+1, then Sr=0. So it weights reduction is Sr and increase in polynomial order.
LikeLike
Hi, thank you for this informative article. However, I don’t understand why n-m-1 is used as the denominator instead of n-1.
LikeLike
Rewrite n-m-1 as n-(m+1). The numerator is Sr for a polynomial of order m. So when n=m+1, then Sr=0. So as m increases, Sr, and n-(m+1) both decrease. And since we want something that is optimum, we give them equal weight. Plot n-m-1 vs m and Sr vs m separately to see what happens.
LikeLike
Why would I want to provide equal weightage to n-m-1? Isn’t the Sr satisfactory enough?
LikeLike
If Sr is given all the weight, then the optimum order of polynomial m is n-1. This is the case where the polynomial goes through all the data points and Sr=0. But regression is all about finding a simplified curve to represent the data.
LikeLike