How to Clean Time Series Data in Python

<p>Real-world time series data is rarely clean. Sensors drop out, systems clock-drift, pipelines duplicate records, and manual data entry introduces mistakes. By the time a dataset reaches your notebook, it has passed through collection, transmission, and storage, each step a potential source of corruption.</p> <p>Cleaning time series data is harder than cleaning tabular data because time is a structural constraint. You can't shuffle rows or impute a missing value with a column mean without pulling future data into a past observation. Every cleaning decision has to respect temporal ordering, or it breaks the integrity of everything built on top of it.</p> <p>This guide walks through the full cleaning pipeline in Python: from raw data arrival to a dataset ready for feature engineering or modelling. We'll cover missing value detection and imputation, outlier identification and treatment, duplicate handling, frequency alignment, noise smoothing, and schema validation, applied to sample sensor data throughout.</p> <p><a href="https://github.com/balapriyac/data-science-tutorials/blob/main/time-series-data-cleaning/time_series_data_cleaning.ipynb">You can get the Colab notebook from GitHub and follow along</a>.</p> <h2 id="heading-prerequisites">Prerequisites</h2> <p>To follow along to this guide, you'll need to be:</p> <ul> <li><p>Comfortable working with Python and pandas DataFrames</p> </li> <li><p>Familiar with time-indexed data</p> </li> <li><p>Aware of what feature engineering and machine learning modelling involve at a high level</p> </li> </ul> <p>We'll use <code>pandas</code> and <code>numpy</code> for data manipulation, <code>scipy</code> for signal smoothing and statistical tests, <code>scikit-learn</code> for anomaly detection, and <code>statsmodels</code> for seasonal decomposition. Install them before running any code in this guide:</p> <pre><code class="language-bash">pip install pandas numpy scipy scikit-learn statsmodels </code></pre> <h2 id="heading-table-of-contents">Table of Contents</h2> <ul> <li><p><a href="#heading-how-to-audit-your-time-series-before-cleaning-it">How to Audit Your Time Series Before Cleaning It</a></p> </li> <li><p><a href="#heading-how-to-reindex-to-a-canonical-frequency">How to Reindex to a Canonical Frequency</a></p> </li> <li><p><a href="#heading-how-to-handle-missing-values">How to Handle Missing Values</a></p> <ul> <li><p><a href="#heading-forward-fill-for-step-function-signals">Forward Fill — For Step-Function Signals</a></p> </li> <li><p><a href="#heading-time-weighted-interpolation-for-continuous-signals">Time-Weighted Interpolation — For Continuous Signals</a></p> </li> <li><p><a href="#heading-seasonal-decomposition-imputation-for-long-gaps">Seasonal Decomposition Imputation — For Long Gaps</a></p> </li> </ul> </li> <li><p><a href="#heading-how-to-detect-and-handle-outliers">How to Detect and Handle Outliers</a></p> <ul> <li><p><a href="#heading-z-score-with-rolling-window">Z-Score with Rolling Window</a></p> </li> <li><p><a href="#heading-iqr-based-outlier-detection">IQR-Based Outlier Detection</a></p> </li> <li><p><a href="#heading-isolation-forest-for-multivariate-outlier-detection">Isolation Forest — For Multivariate Outlier Detection</a></p> </li> <li><p><a href="#heading-outlier-treatment">Outlier Treatment</a></p> </li> </ul> </li> <li><p><a href="#heading-how-to-remove-duplicates">How to Remove Duplicates</a></p> </li> <li><p><a href="#heading-frequency-alignment-and-resampling">Frequency Alignment and Resampling</a></p> </li> <li><p><a href="#heading-smoothing-noise">Smoothing Noise</a></p> <ul> <li><p><a href="#heading-exponential-weighted-moving-average">Exponential Weighted Moving Average</a></p> </li> <li><p><a href="#heading-savitzky-golay-filter">Savitzky-Golay Filter</a></p> </li> </ul> </li> <li><p><a href="#heading-schema-and-sanity-validation">Schema and Sanity Validation</a></p> </li> <li><p><a href="#heading-the-complete-cleaning-checklist">The Complete Cleaning Checklist</a></p> </li> </ul> <h2 id="heading-how-to-audit-your-time-series-before-cleaning-it">How to Audit Your Time Series Before Cleaning It</h2> <p>The first rule of data cleaning is: look before you cut. Before imputing, smoothing, or dropping anything, you need a complete picture of what's wrong and where.</p> <p>A good audit covers the following:</p> <ul> <li><p>The time index: Is it regular? Are there gaps?</p> </li> <li><p>Missing value distribution: Are missing values random or clustered?</p> </li> <li><p>Value range: Are there obvious gaps or sensor failures?</p> </li> <li><p>Duplicate timestamps</p> </li> </ul> <p>Let's spin up a sample dataset (with some of the above problems):</p> <pre><code class="language-python"># Simulate one week of smart grid voltage readings (hourly) # with realistic problems injected periods = 168 index = pd.date_range("2024-06-01", periods=periods, freq="H") voltage = ( 230.0 + 3.5 * np.sin(2 * np.pi * np.arange(periods) / 24) + np.random.normal(0, 1.2, periods) ) # Inject problems voltage[14:17] = np.nan # sensor dropout: 3 consecutive missing voltage[42] = np.nan # isolated missing voltage[78] = 312.4 # spike outlier voltage[101:104] = np.nan # another dropout voltage[130] = 187.2 # dip outlier series = pd.Series(voltage, index=index, name="voltage_v") # --- Audit --- print("=== TIME SERIES AUDIT ===") print(f"Period: {series.index.min()} → {series.index.max()}") print(f"Observations: {len(series)}") print(f"Expected freq: {pd.infer_freq(series.index)}") print(f"\nMissing values: {series.isna().sum()} ({series.isna().mean()*100:.1f}%)") print(f"Value range: [{series.min():.2f}, {series.max():.2f}]") print(f"Mean ± Std: {series.mean():.2f} ± {series.std():.2f}") # Identify consecutive missing runs missing_mask = series.isna() missing_runs = [] run_start = None for i, (ts, is_missing) in enumerate(missing_mask.items()): if is_missing and run_start is None: run_start = ts elif not is_missing and run_start is not None: missing_runs.append((run_start, missing_mask.index[i - 1])) run_start = None print(f"\nMissing runs ({len(missing_runs)} total):") for start, end in missing_runs: print(f" {start} → {end}") </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext">=== TIME SERIES AUDIT === Period: 2024-06-01 00:00:00 → 2024-06-07 23:00:00 Observations: 168 Expected freq: h Missing values: 7 (4.2%) Value range: [187.20, 312.40] Mean ± Std: 230.22 ± 7.81 Missing runs (3 total): 2024-06-01 14:00:00 → 2024-06-01 16:00:00 2024-06-02 18:00:00 → 2024-06-02 18:00:00 2024-06-05 05:00:00 → 2024-06-05 07:00:00 </code></pre> <p>This audit gives you a map of the damage before you start cleaning. The key task is distinguishing between <strong>isolated missing values</strong>, which are imputable with local context, and <strong>missing long runs</strong>, which may need a different strategy or flagging for downstream consumers.</p> <h2 id="heading-how-to-reindex-to-a-canonical-frequency">How to Reindex to a Canonical Frequency</h2> <p>Before imputing missing values, you need to confirm your time index is actually <em>regular</em>. A common problem in ingested time series is that missing timestamps are simply absent rather than represented as <code>NaN</code> rows — which means a <code>.fillna()</code> call will never find them.</p> <pre><code class="language-python"># Simulate a sensor feed with missing timestamps (not just missing values) irregular_index = index.delete([14, 15, 16, 42, 101, 102, 103]) irregular_series = series.dropna().reindex(irregular_index) print(f"Original length: {len(series)}") print(f"Irregular length: {len(irregular_series)}") print(f"Inferred freq: {pd.infer_freq(irregular_series.index)}") # None = irregular # Reindex to the full canonical hourly grid canonical_index = pd.date_range( start=irregular_series.index.min(), end=irregular_series.index.max(), freq="H" ) reindexed = irregular_series.reindex(canonical_index) print(f"\nAfter reindex:") print(f"Length: {len(reindexed)}") print(f"Missing values: {reindexed.isna().sum()}") print(f"Inferred freq: {pd.infer_freq(reindexed.index)}") </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext">Original length: 168 Irregular length: 161 Inferred freq: None After reindex: Length: 168 Missing values: 7 Inferred freq: h </code></pre> <p><code>pd.infer_freq</code> returning <code>None</code> is your signal that the index has gaps. After reindexing to the canonical grid, missing timestamps become explicit <code>NaN</code> rows, and now your imputation logic can find them.</p> <h2 id="heading-how-to-handle-missing-values">How to Handle Missing Values</h2> <p>Not all missing values should be handled the same way. A single isolated missing reading in a smooth signal is best filled with interpolation. A 3-hour sensor dropout in a volatile signal, however, might be better flagged than fabricated. Strategy should match both gap length and signal behavior.</p> <h3 id="heading-forward-fill-for-step-function-signals">Forward Fill — For Step-Function Signals</h3> <p>Forward fill is appropriate when the variable holds its last known value until something changes it — a machine state, a setpoint, a categorical flag.</p> <pre><code class="language-python"># Equipment operating mode — a step signal mode_data = pd.Series( ["running", "running", np.nan, np.nan, "idle", "idle", np.nan, "running"], index=pd.date_range("2024-06-01", periods=8, freq="H"), name="operating_mode" ) filled_mode = mode_data.ffill() print(pd.DataFrame({"original": mode_data, "ffill": filled_mode})) </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext"> original ffill 2024-06-01 00:00:00 running running 2024-06-01 01:00:00 running running 2024-06-01 02:00:00 NaN running 2024-06-01 03:00:00 NaN running 2024-06-01 04:00:00 idle idle 2024-06-01 05:00:00 idle idle 2024-06-01 06:00:00 NaN idle 2024-06-01 07:00:00 running running </code></pre> <h3 id="heading-time-weighted-interpolation-for-continuous-signals">Time-Weighted Interpolation — For Continuous Signals</h3> <p>For continuous sensor readings, linear interpolation weighted by time handles irregular gaps correctly because it doesn't assume equal spacing.</p> <pre><code class="language-python"># Fill the voltage series using time-based interpolation voltage_clean = reindexed.interpolate(method="time") # Compare original vs filled around the first gap gap_window = voltage_clean["2024-06-01 12:00":"2024-06-01 18:00"] original_window = reindexed["2024-06-01 12:00":"2024-06-01 18:00"] comparison = pd.DataFrame({ "original": original_window, "interpolated": gap_window.round(3), "was_missing": original_window.isna(), }) print(comparison) </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext"> original interpolated was_missing 2024-06-01 12:00:00 230.290355 230.290 False 2024-06-01 13:00:00 226.798197 226.798 False 2024-06-01 14:00:00 NaN 226.848 True 2024-06-01 15:00:00 NaN 226.897 True 2024-06-01 16:00:00 NaN 226.947 True 2024-06-01 17:00:00 226.996356 226.996 False 2024-06-01 18:00:00 225.410371 225.410 False </code></pre> <h3 id="heading-seasonal-decomposition-imputation-for-long-gaps">Seasonal Decomposition Imputation — For Long Gaps</h3> <p>For gaps longer than a few steps in a seasonal signal, interpolating across the gap ignores the seasonal pattern. A better approach is to decompose the series, impute each component separately, then reconstruct.</p> <pre><code class="language-python">from statsmodels.tsa.seasonal import seasonal_decompose # Use a longer series for decomposition (needs enough periods) long_voltage = pd.Series( 230.0 + 3.5 * np.sin(2 * np.pi * np.arange(336) / 24) + np.random.normal(0, 1.0, 336), index=pd.date_range("2024-06-01", periods=336, freq="H") ) # Inject a 6-hour gap long_voltage.iloc[100:106] = np.nan # Interpolate first to give decompose a complete series to work with temp_filled = long_voltage.interpolate(method="time") decomp = seasonal_decompose(temp_filled, model="additive", period=24) # Reconstruct: trend + seasonal + zero residual for missing positions reconstructed = long_voltage.copy() missing_idx = long_voltage[long_voltage.isna()].index reconstructed[missing_idx] = ( decomp.trend[missing_idx].fillna(method="ffill") + decomp.seasonal[missing_idx] ) print(f"Missing before: {long_voltage.isna().sum()}") print(f"Missing after: {reconstructed.isna().sum()}") print("\nFilled values at gap:") print(reconstructed[missing_idx].round(3)) </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext"> original interpolated was_missing 2024-06-01 12:00:00 230.290355 230.290 False 2024-06-01 13:00:00 226.798197 226.798 False 2024-06-01 14:00:00 NaN 226.848 True 2024-06-01 15:00:00 NaN 226.897 True 2024-06-01 16:00:00 NaN 226.947 True 2024-06-01 17:00:00 226.996356 226.996 False 2024-06-01 18:00:00 225.410371 225.410 False </code></pre> <p>The seasonal decomposition imputation respects the time-of-day pattern. As you can see, the filled values aren't a flat line across the gap but follow the expected daily curve.</p> <h2 id="heading-how-to-detect-and-handle-outliers">How to Detect and Handle Outliers</h2> <p>Outliers in time series are trickier than in tabular data because context matters. For example, an unusually high or low voltage might be a sensor spike or a genuine grid event. You need methods that use <em>temporal context</em>, not just global statistics.</p> <h3 id="heading-z-score-with-rolling-window">Z-Score with Rolling Window</h3> <p>A global Z-score misses local anomalies in non-stationary series. A rolling Z-score flags values that are unusual <em>relative to their local neighbourhood</em>.</p> <p><strong>Note</strong>: A <strong>non-stationary series</strong> is a time series whose statistical properties—such as mean, variance, or trend—change over time instead of remaining constant.</p> <pre><code class="language-python">window = 24 # 24-hour rolling window roll_mean = voltage_clean.rolling(window, center=True, min_periods=1).mean() roll_std = voltage_clean.rolling(window, center=True, min_periods=1).std() rolling_z = (voltage_clean - roll_mean) / roll_std threshold = 3.0 outliers_z = rolling_z[rolling_z.abs() &gt; threshold] print(f"Rolling Z-score outliers detected: {len(outliers_z)}") print(outliers_z.round(3)) </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext">Rolling Z-score outliers detected: 2 2024-06-04 06:00:00 4.646 2024-06-06 10:00:00 -4.484 Name: voltage_v, dtype: float64 </code></pre> <p>Z-score outlier detection works best for approximately Gaussian (normal) distributions because it assumes the data is centered around a mean with symmetric spread measured by standard deviation.</p> <h3 id="heading-iqr-based-outlier-detection">IQR-Based Outlier Detection</h3> <p>The interquartile range (IQR) method is more robust for detecting outliers in non-Gaussian distributions. The interquartile range (IQR) is the difference between the third quartile (Q3) and the first quartile (Q1), representing the spread of the middle 50% of the data.</p> <pre><code class="language-python">Q1 = voltage_clean.quantile(0.25) Q3 = voltage_clean.quantile(0.75) IQR = Q3 - Q1 lower_bound = Q1 - 1.5 * IQR upper_bound = Q3 + 1.5 * IQR outliers_iqr = voltage_clean[ (voltage_clean &lt; lower_bound) | (voltage_clean &gt; upper_bound) ] print(f"IQR bounds: [{lower_bound:.2f}, {upper_bound:.2f}]") print(f"Outliers detected: {len(outliers_iqr)}") print(outliers_iqr.round(2)) </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext">IQR bounds: [220.16, 239.46] Outliers detected: 2 2024-06-04 06:00:00 312.4 2024-06-06 10:00:00 187.2 Name: voltage_v, dtype: float64 </code></pre> <h3 id="heading-isolation-forest-for-multivariate-outlier-detection">Isolation Forest — For Multivariate Outlier Detection</h3> <p>When you have multiple sensors, an isolated reading on one channel might look normal, but its combination with readings from other channels reveals the anomaly. Isolation Forest handles this naturally.</p> <pre><code class="language-python"># Build a multi-sensor DataFrame np.random.seed(42) n = 200 sensor_df = pd.DataFrame({ "voltage_v": 230 + 3 * np.sin(2 * np.pi * np.arange(n) / 24) + np.random.normal(0, 1, n), "current_a": 15 + 0.8 * np.sin(2 * np.pi * np.arange(n) / 24) + np.random.normal(0, 0.3, n), "frequency_hz": 50 + np.random.normal(0, 0.05, n), }, index=pd.date_range("2024-06-01", periods=n, freq="H")) # Inject a multivariate anomaly — voltage drops, current spikes together sensor_df.iloc[88, 0] = 194.2 # voltage dip sensor_df.iloc[88, 1] = 28.7 # current surge (consistent with fault) clf = IsolationForest(contamination=0.02, random_state=42) sensor_df["anomaly_score"] = clf.fit_predict(sensor_df[["voltage_v", "current_a", "frequency_hz"]]) anomalies = sensor_df[sensor_df["anomaly_score"] == -1] print(f"Anomalies detected: {len(anomalies)}") print(anomalies[["voltage_v", "current_a", "frequency_hz"]].round(2)) </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext">Anomalies detected: 4 voltage_v current_a frequency_hz 2024-06-02 07:00:00 234.75 15.84 49.90 2024-06-04 06:00:00 233.09 15.82 50.15 2024-06-04 16:00:00 194.20 28.70 50.08 2024-06-06 05:00:00 235.09 15.41 49.91 </code></pre> <p>In practice you'd follow up anomaly scores with domain-specific threshold rules.</p> <h3 id="heading-outlier-treatment">Outlier Treatment</h3> <p>Once outliers are identified, you can handle them in several ways:</p> <ul> <li><p>Cap them using Winsorization by limiting extreme values to a threshold.</p> </li> <li><p>Replace them with interpolated or estimated values.</p> </li> <li><p>Flag them so the model can handle them appropriately.</p> </li> </ul> <pre><code class="language-python"># Winsorize: cap at the IQR bounds voltage_winsorized = voltage_clean.clip(lower=lower_bound, upper=upper_bound) # Replace outliers with time-interpolated values voltage_outlier_fixed = voltage_clean.copy() voltage_outlier_fixed[outliers_iqr.index] = np.nan voltage_outlier_fixed = voltage_outlier_fixed.interpolate(method="time") print("Outlier treatment comparison:") for ts in outliers_iqr.index: print(f"\n {ts}") print(f" Original: {voltage_clean[ts]:.2f}") print(f" Winsorized: {voltage_winsorized[ts]:.2f}") print(f" Interpolated: {voltage_outlier_fixed[ts]:.2f}") </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext">Outlier treatment comparison: 2024-06-04 06:00:00 Original: 312.40 Winsorized: 239.46 Interpolated: 232.01 2024-06-06 10:00:00 Original: 187.20 Winsorized: 220.16 Interpolated: 231.43 </code></pre> <p>Winsorization preserves the point but clips it to a plausible range — useful when you want to retain the information that something anomalous happened. Interpolation treats the outlier as if it were missing — better when you believe the reading is simply wrong.</p> <h2 id="heading-how-to-remove-duplicates">How to Remove Duplicates</h2> <p>Duplicate timestamps are common when data pipelines retry on failure. Unlike tabular duplicates, time series duplicates aren't always identical, a retry might deliver a slightly different reading for the same timestamp.</p> <pre><code class="language-python"># Inject duplicate timestamps with slightly different values (retry scenario) dup_index = index.tolist() dup_index.insert(20, index[20]) # exact duplicate timestamp dup_index.insert(55, index[55]) # retry duplicate dup_values = voltage_clean.tolist() dup_values.insert(20, voltage_clean.iloc[20]) dup_values.insert(55, voltage_clean.iloc[55] + 0.7) # slightly different value dup_series = pd.Series(dup_values, index=pd.DatetimeIndex(dup_index), name="voltage_v") print(f"Length with duplicates: {len(dup_series)}") print(f"Duplicate timestamps: {dup_series.index.duplicated().sum()}") # Strategy 1: keep first (original reading) dedup_first = dup_series[~dup_series.index.duplicated(keep="first")] # Strategy 2: keep mean (average across retries) dedup_mean = dup_series.groupby(level=0).mean() print(f"\nAfter dedup (keep first): {len(dedup_first)}") print(f"After dedup (mean): {len(dedup_mean)}") # Show the retry duplicate ts_retry = index[55] print(f"\nRetry duplicate at {ts_retry}:") print(f" Values: {dup_series[ts_retry].values.round(3)}") print(f" Keep first: {dedup_first[ts_retry]:.3f}") print(f" Mean: {dedup_mean[ts_retry]:.3f}") </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext">Length with duplicates: 170 Duplicate timestamps: 2 After dedup (keep first): 168 After dedup (mean): 168 Retry duplicate at 2024-06-03 07:00:00: Values: [235.198 234.498] Keep first: 235.198 Mean: 234.848 </code></pre> <p>For most sensor pipelines, keep-first is the right default; the first delivery is the original reading. Mean makes sense when retries come from independent sensors measuring the same quantity.</p> <h2 id="heading-frequency-alignment-and-resampling">Frequency Alignment and Resampling</h2> <p>Real pipelines often mix data at different frequencies. For example, you may need a 1-minute meter reading merged with an hourly weather feed. Before joining them, you need to align frequencies explicitly.</p> <pre><code class="language-python"># 1-minute power draw readings power_1min = pd.Series( 42 + 18 * ((pd.date_range("2024-06-01", periods=1440, freq="T").hour.isin(range(8, 19)))).astype(int) + np.random.normal(0, 2, 1440), index=pd.date_range("2024-06-01", periods=1440, freq="T"), name="power_kw" ) # Downsample to hourly: mean is appropriate for power (average over the hour) power_hourly_mean = power_1min.resample("H").mean().round(2) # Downsample to hourly: max (peak demand within the hour) power_hourly_max = power_1min.resample("H").max().round(2) # Downsample to hourly: sum (total energy = kWh) energy_hourly_kwh = (power_1min.resample("H").sum() / 60).round(3) comparison = pd.DataFrame({ "mean_kw": power_hourly_mean, "peak_kw": power_hourly_max, "energy_kwh": energy_hourly_kwh, }).iloc[7:13] print(comparison) </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext"> mean_kw peak_kw energy_kwh 2024-06-01 07:00:00 42.13 46.28 42.133 2024-06-01 08:00:00 60.56 64.81 60.557 2024-06-01 09:00:00 59.91 64.88 59.912 2024-06-01 10:00:00 60.07 65.16 60.066 2024-06-01 11:00:00 60.08 64.99 60.083 2024-06-01 12:00:00 59.72 63.65 59.724 </code></pre> <p>Which aggregation you choose matters enormously for downstream use. Mean power is right for load profiling. Peak power is right for capacity planning. Sum (converted to kWh) is right for billing. You can probably see why the <em>right</em> answer is domain-specific and not technical.</p> <h2 id="heading-smoothing-noise">Smoothing Noise</h2> <p>Raw sensor data often contains high-frequency noise that obscures the underlying signal. Smoothing before feature engineering prevents the model from fitting to noise, but over-smoothing destroys real variation.</p> <h3 id="heading-exponential-weighted-moving-average">Exponential Weighted Moving Average</h3> <p>Exponential Weighted Moving Average or EWMA gives <em>more weight to recent observations</em> and adapts quickly to level changes. This is better than a simple moving average for non-stationary signals.</p> <pre><code class="language-python"># Noisy temperature sensor (°C) temp_noisy = pd.Series( 3.5 + 1.2 * np.sin(2 * np.pi * np.arange(168) / 24) + np.random.normal(0, 0.8, 168), # high noise index=pd.date_range("2024-06-01", periods=168, freq="H"), name="temperature_c" ) temp_ewma = temp_noisy.ewm(span=6, adjust=False).mean() temp_sma = temp_noisy.rolling(window=6, center=True).mean() comparison = pd.DataFrame({ "raw": temp_noisy, "ewma": temp_ewma.round(3), "sma": temp_sma.round(3), }).iloc[22:30] print(comparison) </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext"> raw ewma sma 2024-06-01 22:00:00 3.212372 2.843 3.035 2024-06-01 23:00:00 3.106840 2.918 3.176 2024-06-02 00:00:00 3.712290 3.145 3.011 2024-06-02 01:00:00 3.344376 3.202 3.294 2024-06-02 02:00:00 2.148946 2.901 3.705 2024-06-02 03:00:00 4.241105 3.284 4.087 2024-06-02 04:00:00 5.677429 3.968 4.381 2024-06-02 05:00:00 5.400083 4.377 4.765 </code></pre> <h3 id="heading-savitzky-golay-filter">Savitzky-Golay Filter</h3> <p>For signals where you need to preserve peak shapes — not just smooth them away — the <a href="https://eigenvector.com/wp-content/uploads/2020/01/SavitzkyGolay.pdf">Savitzky-Golay filter</a> fits a polynomial over a sliding window and is better at maintaining the height of genuine spikes.</p> <pre><code class="language-python">from scipy.signal import savgol_filter temp_savgol = pd.Series( savgol_filter(temp_noisy.values, window_length=11, polyorder=2), index=temp_noisy.index, name="temp_savgol" ).round(3) print(pd.DataFrame({ "raw": temp_noisy, "savgol": temp_savgol, }).iloc[22:30]) </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext"> raw savgol 2024-06-01 22:00:00 3.212372 2.960 2024-06-01 23:00:00 3.106840 2.944 2024-06-02 00:00:00 3.712290 3.114 2024-06-02 01:00:00 3.344376 3.379 2024-06-02 02:00:00 2.148946 3.809 2024-06-02 03:00:00 4.241105 4.288 2024-06-02 04:00:00 5.677429 4.749 2024-06-02 05:00:00 5.400083 5.138 </code></pre> <h2 id="heading-schema-and-sanity-validation">Schema and Sanity Validation</h2> <p>Cleaning without validation is incomplete. You need automated checks that run every time new data arrives — catching problems before they silently corrupt downstream models.</p> <pre><code class="language-python">def validate_time_series(series: pd.Series, config: dict) -&gt; dict: """ Run schema and sanity checks on a time series. Returns a report dict with pass/fail per check. """ report = {} # Frequency check inferred = pd.infer_freq(series.index) report["freq_regular"] = inferred == config["expected_freq"] # Missing value threshold missing_rate = series.isna().mean() report["missing_below_threshold"] = missing_rate &lt;= config["max_missing_rate"] report["missing_rate"] = round(missing_rate, 4) # Value range check in_range = series.dropna().between(config["min_value"], config["max_value"]) report["values_in_range"] = in_range.all() report["out_of_range_count"] = (~in_range).sum() # Duplicate timestamps report["no_duplicates"] = not series.index.duplicated().any() # Monotonic index report["index_monotonic"] = series.index.is_monotonic_increasing return report config = { "expected_freq": "H", "max_missing_rate": 0.05, "min_value": 210.0, "max_value": 250.0, } report = validate_time_series(voltage_outlier_fixed, config) print("=== VALIDATION REPORT ===") for check, result in report.items(): if check in ("missing_rate", "out_of_range_count"): print(f" {check}: {result}") else: status = "✓ PASS" if result else "✗ FAIL" print(f" {status} {check}") </code></pre> <p><strong>Output:</strong></p> <pre><code class="language-plaintext">=== VALIDATION REPORT === ✗ FAIL freq_regular ✓ PASS missing_below_threshold missing_rate: 0.0 ✓ PASS values_in_range out_of_range_count: 0 ✓ PASS no_duplicates ✓ PASS index_monotonic </code></pre> <p>This validator is the kind of function you wrap around every data ingestion step in a production pipeline. Run it before cleaning to know what's broken, and after cleaning to confirm everything passed.</p> <h2 id="heading-the-complete-cleaning-checklist">The Complete Cleaning Checklist</h2> <p>Here's the full sequence to run on any incoming time series dataset:</p> <table> <thead> <tr> <th>Step</th> <th>Technique</th> <th>When to Use</th> </tr> </thead> <tbody><tr> <td><strong>Audit</strong></td> <td>Index check, missing map, value range</td> <td>Always — before anything else</td> </tr> <tr> <td><strong>Reindex</strong></td> <td><code>reindex</code> to canonical frequency</td> <td>When timestamps are absent rather than NaN</td> </tr> <tr> <td><strong>Missing: short gaps</strong></td> <td>Time interpolation</td> <td>Continuous signals, gaps ≤ 3 steps</td> </tr> <tr> <td><strong>Missing: step signals</strong></td> <td>Forward fill</td> <td>Categorical or setpoint data</td> </tr> <tr> <td><strong>Missing: long gaps</strong></td> <td>Seasonal decomposition impute</td> <td>Seasonal signals, gaps &gt; 6 steps</td> </tr> <tr> <td><strong>Outliers: univariate</strong></td> <td>Rolling Z-score or IQR</td> <td>Single sensor, local anomalies</td> </tr> <tr> <td><strong>Outliers: multivariate</strong></td> <td>Isolation Forest</td> <td>Multiple correlated sensors</td> </tr> <tr> <td><strong>Outlier treatment</strong></td> <td>Winsorize or interpolate</td> <td>Depending on whether event is real</td> </tr> <tr> <td><strong>Duplicates</strong></td> <td>Keep first or group mean</td> <td>Pipeline retry duplicates</td> </tr> <tr> <td><strong>Resampling</strong></td> <td><code>.resample()</code> with correct aggregation</td> <td>Frequency alignment before joins</td> </tr> <tr> <td><strong>Smoothing</strong></td> <td>EWMA or Savitzky-Golay</td> <td>Noisy sensors before feature engineering</td> </tr> <tr> <td><strong>Validation</strong></td> <td>Schema + sanity checks</td> <td>After cleaning, and on every new batch</td> </tr> </tbody></table> <h2 id="heading-wrapping-up">Wrapping Up</h2> <p>The order matters. Reindex before imputing. Impute before smoothing. Validate after everything. Skipping steps or doing them out of order compounds errors in ways that are very difficult to trace back once you're looking at model predictions.</p> <p>Time series cleaning isn't glamorous work, but a model trained on clean data and thoughtfully engineered features will almost always outperform a more sophisticated model trained on data that wasn't cleaned properly. Getting this pipeline right is the highest-leverage thing you can do before you try running even the simplest algorithm on your time series data.</p>