Finding the optimum polynomial order to use for regression

Many a times, you may not have the privilege or knowledge of the physics of the problem to dictate the type of regression model. You may want to fit the data to a polynomial. But then how do you choose what order of polynomial to use.

Do you choose based on the polynomial order for which the sum of the squares of the residuals, Sr is a minimum? If that were the case, we can always get Sr=0 if the polynomial order chosen is one less than the number of data points. In fact, it would be an exact match.

So what do we do? We choose the degree of polynomial for which the variance as computed by

Sr(m)/(n-m-1)

is a minimum or when there is no significant decrease in its value as the degree of polynomial is increased. In the above formula,

Sr(m) = sum of the square of the residuals for the mth order polynomial

n= number of data points

m=order of polynomial (so m+1 is the number of constants of the model)

Let’s look at an example where the coefficient of thermal expansion is given for a typical steel as a function of temperature. We want to relate the two using polynomial regression.

 Temperature Instantaneous Thermal Expansion oF 1E-06 in/(in oF) 80 6.47 40 6.24 0 6.00 -40 5.72 -80 5.43 -120 5.09 -160 4.72 -200 4.30 -240 3.83 -280 3.33 -320 2.76

If a first order polynomial is chosen, we get

$alpha=0.009147T+5.999$, with Sr=0.3138.

If a second order polynomial is chosen, we get

$alpha=-0.00001189T^2+0.006292T+6.015$ with Sr=0.003047.

Below is the table for the order of polynomial, the Sr value and the variance value, Sr(m)/(n-m-1)

 Order of polynomial, m Sr(m) Sr(m)/(n-m-1) 1 0.3138 0.03486 2 0.003047 0.0003808 3 0.0001916 0.000027371 4 0.0001566 0.0000261 5 0.0001541 0.00003082 6 0.0001300 0.000325

So what order of polynomial would you choose?

From the above table, and the figure below, it looks like the second or third order polynomial would be a good choice as very little change is taking place in the value of the variance after m=2.

This post is brought to you by Holistic Numerical Methods: Numerical Methods for the STEM undergraduate at http://numericalmethods.eng.usf.edu

Author: Autar Kaw

Autar Kaw (http://autarkaw.com) is a Professor of Mechanical Engineering at the University of South Florida. He has been at USF since 1987, the same year in which he received his Ph. D. in Engineering Mechanics from Clemson University. He is a recipient of the 2012 U.S. Professor of the Year Award. With major funding from NSF, he is the principal and managing contributor in developing the multiple award-winning online open courseware for an undergraduate course in Numerical Methods. The OpenCourseWare (nm.MathForCollege.com) annually receives 1,000,000+ page views, 1,000,000+ views of the YouTube audiovisual lectures, and 150,000+ page views at the NumericalMethodsGuy blog. His current research interests include engineering education research methods, adaptive learning, open courseware, massive open online courses, flipped classrooms, and learning strategies. He has written four textbooks and 80 refereed technical papers, and his opinion editorials have appeared in the St. Petersburg Times and Tampa Tribune.

12 thoughts on “Finding the optimum polynomial order to use for regression”

1. Ken says:

Hi, how exactly do you calculate Sr? I read your definition of it several times but still clues. Please let me know. Thanks.

Like

2. Odo Bogdan says:

Very nice article but I cannot seam to know how to interpret the following sentence:

We choose the degree of polynomial for which the variance as computed by
Sr(m)/(n-m-1)
is a minimum or when there is no significant decrease in its value as the degree of polynomial is increased

So, what does “significant decrease” mean? Is it statistical significance test? Or you just set a threshold, say 0.001, and choose the order when the change in the Sr(m)/(n-m-1) drops below this threshold?
Could you help me out on this issue?
Thanks!

Like

1. There is no rule of thumb that I know of. If there is a minimum at a low order of polynomial, that is an indication of the optimum polynomial as well. If there is any threshold, it should be a relative number.

Like

1. Odo Bogdan says:

I am working to find an algorithmic/automatic way of predicting the order of the polynomial.
But with not much luck!
However I found ways to predict the order of the polynomial and accept that it can overestimate the true order… it can do that with 83% accuracy.

I’m still crunching numbers!

Like

3. John says:

This is very nice technique, but I cannot understand the meaning of the denominator ‘n-m-1’.
Is it a weighting?
Thanks!

Like

4. comcom says:

Hi, this is good article.

However I cannot understand the meaning of (n-m-1).
Is this a weighting?

Thank you.

Like

1. Yes, it is weighting. If n=m+1, then Sr=0. So it weights reduction is Sr and increase in polynomial order.

Like

5. yc says:

Hi, thank you for this informative article. However, I don’t understand why n-m-1 is used as the denominator instead of n-1.

Like

1. Rewrite n-m-1 as n-(m+1). The numerator is Sr for a polynomial of order m. So when n=m+1, then Sr=0. So as m increases, Sr, and n-(m+1) both decrease. And since we want something that is optimum, we give them equal weight. Plot n-m-1 vs m and Sr vs m separately to see what happens.

Like

6. ivanjemic says:

Why would I want to provide equal weightage to n-m-1? Isn’t the Sr satisfactory enough?

Like

1. If Sr is given all the weight, then the optimum order of polynomial m is n-1. This is the case where the polynomial goes through all the data points and Sr=0. But regression is all about finding a simplified curve to represent the data.

Like