We have observations (t_i, y_i) and want a curve that describes their pattern. Fitting means choosing the coefficients of that curve so that its predictions are close to the observations.

16.1 choose a curve

A polynomial of degree p has the form

\widehat{y}(t) = a_0 + a_1 t + a_2 t^2 + \cdots + a_p t^p.

The hat on \widehat{y} means a predicted value. The coefficients a_0, a_1, \ldots, a_p are the numbers we fit.

Degree (also called order) Shape Coefficients to fit
0 A horizontal line 1
1 A straight line: intercept and slope 2
2 A parabola: also able to bend 3

For a straight line, a_0 is the predicted value at t=0, and a_1 is the change in the prediction per unit of time. Higher degrees allow more complicated shapes.

16.2 what do we minimize?

For each observation, measure the vertical difference between the observed value and the curve at the same time. This difference is the residual:

e_i = y_i - \widehat{y}(t_i).

A point above the curve has a positive residual; a point below has a negative one. In least squares, we choose the coefficients that minimize

S = \sum_{i=1}^{n} e_i^2 = \sum_{i=1}^{n} \left[y_i - \widehat{y}(t_i)\right]^2.

We square the residuals so that positive and negative differences do not cancel. Squaring also gives large residuals more influence. Each observation has equal weight in this lesson.

16.3 how is the fit performed?

  1. Choose the degree. This decides which shapes the model can take.
  2. Use all the observations together. Each choice of coefficients gives predicted values, residuals, and a value of S.
  3. Find the coefficients with the smallest S. For a polynomial, a least-squares solver calculates these coefficients directly. You do not need to search by hand.

The degree stays fixed during that calculation. You choose the degree; the fitting algorithm chooses the coefficients.

For degree 0, the best constant is the mean of the observations. For degree 1, the algorithm finds the best intercept and slope. A fitted curve generally does not pass through every point.

16.4 fit lab

The lab starts with observations only. Turn on Show fit when you are ready to reveal the curve.

Change one setting at a time. Try to predict the effect before moving a slider.

Open the fit lab in a full browser tab

Try these three experiments:

  1. Find a line yourself. Choose “Roughly straight” and turn on “Try a straight line by hand”. Change the intercept and slope to reduce S, then ask for the least-squares line.
  2. Give the curve more freedom. Keep the points fixed and increase the degree. With 12 points, try degree 11; with 30 points, you can go up to degree 29. Watch S, then reveal the curve that generated the data. Does the smaller error always mean a better description of that curve?
  3. Move one point. Drag a point upward. Which parts of the fitted curve change? Watch the curve and its total squared error.

16.4.1 a smaller error is not the whole story

On the same observations, increasing the polynomial degree cannot increase the minimum S: the simpler polynomial is still one of the available choices. But extra flexibility can also fit the noise. A curve that closely follows the observed points may behave poorly between them or outside their time range.

Choose the degree with your scientific question and the expected shape in mind. The lowest error on the fitted observations is only part of that decision.

16.5 a small Python example

Here we choose degree 1. The call to Polynomial.fit finds its coefficients; the remaining lines evaluate the predictions and calculate S. Change degree to 0 or 2 and compare.

a small Python example 
import numpy as np
import matplotlib.pyplot as plt
from numpy.polynomial import Polynomial

t = np.array([0., 1., 2., 3., 4., 5.])
y = np.array([1., 1.9, 2.2, 3.1, 3.8, 4.])

degree = 1
model = Polynomial.fit(t, y, deg=degree)
predicted = model(t)
residuals = y - predicted
S = np.sum(residuals**2)
print(f"Sum of squared residuals: {S:.3f}")

fine_t = np.linspace(t.min(), t.max(), 200)
fig, ax = plt.subplots()
ax.scatter(t, y, label="Observations")
ax.plot(fine_t, model(fine_t), label="Least-squares fit")
ax.vlines(t, predicted, y, color="tab:red", alpha=0.5, label="Residuals")
ax.set(xlabel="Time t", ylabel="Value y")
ax.legend()
plt.show()

The lab’s expandable Python section uses your current points and settings. See the NumPy documentation for the fitting function.

In the next lesson, we will use this same polynomial-fitting idea inside a small moving neighborhood and keep the fitted value at its center.