22  substituting outliers

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib.dates import DateFormatter
import matplotlib.dates as mdates
import seaborn as sns
sns.set(style="ticks", font_scale=1.5)  # white graphs, with large and legible letters
from scipy.signal import savgol_filter

Ok. We found the outliers. Now what?!

As usual, it depends.

22.1 Do nothing

Assuming the outlier indeed happened in real life, and is not the result of faulty data transmission or bad data recording, then excluding an outlier might be the last thing you want to do. Sometimes extreme events do happen, such as a one-in-a-hundred-year storm, and they have a disproportionate weight on the system you are studying. The outliers might actually be the most interesting points in your data for all you know!

In case the outliers are not of interest to you, if you are using robust methods to analyze your data, you don’t necessarily need to do anything either. For instance, let’s say that you want to smooth your time series. If instead of taking the mean inside a sliding window you choose to calculate the median, then outliers shouldn’t be a problem. Test it and see if it’s true. Go on.

# a smooth signal with a single outlier
rng = np.random.default_rng(1)
signal = pd.Series(np.sin(2 * np.pi * np.arange(200) / 50) + 0.1 * rng.normal(size=200))
signal.iloc[100] = 8

# a sliding mean and a sliding median, with the same window
fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(signal, color='lightgray', label='data')
ax.plot(signal.rolling(11, center=True).mean(), color='tab:blue', label='sliding mean')
ax.plot(signal.rolling(11, center=True).median(), color='tab:orange', label='sliding median')
ax.legend(frameon=False);

For many things you need to do (not only smoothing), you might be able to find robust methods. What do you do if you have to use a non-robust method? Well, then you can substitute the outlier for two things: NaN or imputed values.

22.2 NaN

Substitute outliers for NaN.

NaN means “Not a Number”, and is what you get when you try to perform a mathematical operation like 0/0. It is common to see NaN in dataset rows when data was not collected for some reason.

This might seem like a neutral solution, but it actually can generate problems down the line. See this example:

# example using numpy
series = np.array([2, 4, 5, np.nan, 8, 15])
mean = np.mean(series)
print(f"the series average is {mean}")

the series average is nan

A single NaN in your time series ruins the whole calculation! There is a workaround though:

mean = np.nanmean(series)
print(f"the series average is {mean}")

the series average is 6.8

You have to make sure what is the behavior of each function you use with respect to NaNs, and if possible, use a suitable substitute.

The same example in pandas would not fail:

date_range = pd.date_range(start='2024-01-01', periods=len(series), freq='1D')
df = pd.DataFrame({'series': series}, index=date_range)
mean = df['series'].mean()
print(f"the series average is {mean}")

the series average is 6.8

22.3 impute values

To “impute values” means to fill in the missing value with a guess, an estimation of what this data point “should have been” if it were measured in the first place. Why should we bother to do so? Because many tools that we know and love don’t do well with missing values.

We learned about the Savitzky-Golay filter for smoothing data. See what happens when there is a single NaN in the series:

steps = np.random.default_rng(0).integers(low=-2, high=2, size=100)
data = steps.cumsum()
date_range = pd.date_range(start='2023-01-01', periods=len(data), freq='1D')
df = pd.DataFrame({'series': data}, index=date_range)
df.loc['2023-02-05', 'series'] = np.nan

df['sg'] = savgol_filter(df['series'], window_length=15, polyorder=2)

def concise(ax):
    locator = mdates.AutoDateLocator(minticks=3, maxticks=7)
    formatter = mdates.ConciseDateFormatter(locator)
    ax.xaxis.set_major_locator(locator)
    ax.xaxis.set_major_formatter(formatter)

fig, ax = plt.subplots(figsize=(8,4))
ax.plot(df['series'], color="tab:blue", label="series with 1 NaN")
ax.plot(df['sg'], color="tab:orange", label="SavGol filter has many more NaNs")
concise(ax)
ax.legend(frameon=False);

We will deal with this topic in the chapter “filling missing values”. There, we will learn a few methods to fill in missing data.

22.4 whatever you choose, flag it

Never overwrite the raw data. Keep the original column, and add new columns: the cleaned data, and a flag that says which points were changed. Months from now, you (or someone else) will want to know what was measured and what was a guess.

df['temperature_clean'] = df['temperature'].where(~is_outlier)   # NaN where we found an outlier
df['flagged'] = is_outlier

There is another good reason to keep the flags: they tell us how our sensors are doing. If the number of flagged points grows with time, or if data goes missing every time there is a storm, the sensor needs attention. Outliers are not only a nuisance, they are also information.