Finding the optimum polynomial order to use for regression


Many a times, you may not have the privilege or knowledge of the physics of the problem to dictate the type of regression model. You may want to fit the data to a polynomial. But then how do you choose what order of polynomial to use.

Do you choose based on the polynomial order for which the sum of the squares of the residuals, Sr is a minimum? If that were the case, we can always get Sr=0 if the polynomial order chosen is one less than the number of data points. In fact, it would be an exact match.

So what do we do? We choose the degree of polynomial for which the variance as computed by

Sr(m)/(n-m-1)

is a minimum or when there is no significant decrease in its value as the degree of polynomial is increased. In the above formula,

Sr(m) = sum of the square of the residuals for the mth order polynomial

n= number of data points

m=order of polynomial (so m+1 is the number of constants of the model)

Let’s look at an example where the coefficient of thermal expansion is given for a typical steel as a function of temperature. We want to relate the two using polynomial regression.

Temperature

Instantaneous Thermal Expansion

oF

1E-06 in/(in oF)

80

6.47

40

6.24

0

6.00

-40

5.72

-80

5.43

-120

5.09

-160

4.72

-200

4.30

-240

3.83

-280

3.33

-320

2.76

If a first order polynomial is chosen, we get

alpha=0.009147T+5.999, with Sr=0.3138.

If a second order polynomial is chosen, we get

alpha=-0.00001189T^2+0.006292T+6.015 with Sr=0.003047.

Below is the table for the order of polynomial, the Sr value and the variance value, Sr(m)/(n-m-1)

Order of

polynomial, m

Sr(m)

Sr(m)/(n-m-1)

1

0.3138

0.03486

2

0.003047

0.0003808

3

0.0001916

0.000027371

4

0.0001566

0.0000261

5

0.0001541

0.00003082

6

0.0001300

0.000325

So what order of polynomial would you choose?

From the above table, and the figure below, it looks like the second or third order polynomial would be a good choice as very little change is taking place in the value of the variance after m=2.

Optimum order of polynomial for regression

This post is brought to you by Holistic Numerical Methods: Numerical Methods for the STEM undergraduate at http://numericalmethods.eng.usf.edu

Subscribe to the feed to stay updated and let the information follow you.

Advertisements

Published by

Autar Kaw

Autar Kaw (http://autarkaw.com) is a Professor of Mechanical Engineering at the University of South Florida. He has been at USF since 1987, the same year in which he received his Ph. D. in Engineering Mechanics from Clemson University. He is a recipient of the 2012 U.S. Professor of the Year Award. With major funding from NSF, he is the principal and managing contributor in developing the multiple award-winning online open courseware for an undergraduate course in Numerical Methods. The OpenCourseWare (nm.MathForCollege.com) annually receives 1,000,000+ page views, 1,000,000+ views of the YouTube audiovisual lectures, and 150,000+ page views at the NumericalMethodsGuy blog. His current research interests include engineering education research methods, adaptive learning, open courseware, massive open online courses, flipped classrooms, and learning strategies. He has written four textbooks and 80 refereed technical papers, and his opinion editorials have appeared in the St. Petersburg Times and Tampa Tribune.

10 thoughts on “Finding the optimum polynomial order to use for regression”

  1. Very nice article but I cannot seam to know how to interpret the following sentence:


    We choose the degree of polynomial for which the variance as computed by
    Sr(m)/(n-m-1)
    is a minimum or when there is no significant decrease in its value as the degree of polynomial is increased

    So, what does “significant decrease” mean? Is it statistical significance test? Or you just set a threshold, say 0.001, and choose the order when the change in the Sr(m)/(n-m-1) drops below this threshold?
    Could you help me out on this issue?
    Thanks!

    Like

    1. There is no rule of thumb that I know of. If there is a minimum at a low order of polynomial, that is an indication of the optimum polynomial as well. If there is any threshold, it should be a relative number.

      Like

      1. Thanks for your answer!

        I am working to find an algorithmic/automatic way of predicting the order of the polynomial.
        But with not much luck!
        However I found ways to predict the order of the polynomial and accept that it can overestimate the true order… it can do that with 83% accuracy.

        I’m still crunching numbers!

        Like

  2. Hi, this is good article.

    However I cannot understand the meaning of (n-m-1).
    Is this a weighting?

    Also, is there any reference of this article?
    Please tell me it.

    Thank you.

    Like

    1. Rewrite n-m-1 as n-(m+1). The numerator is Sr for a polynomial of order m. So when n=m+1, then Sr=0. So as m increases, Sr, and n-(m+1) both decrease. And since we want something that is optimum, we give them equal weight. Plot n-m-1 vs m and Sr vs m separately to see what happens.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s