Instrumentation & Digital Signal Processing For 3D Pose Coordinates From Video

Programmatic 3D coordinate analysis extracted from video is helping us understand human motion in entirely new and unique ways. With advanced models we can pull a continuous stream of 3D world coordinates from a simple video, but the initial output is less a clean signal, and more a chaotic stream of jitter, noise, and occasional brilliance.

The common first instinct is to apply a simple low-pass or moving-average filter to "smooth it out." This is an approach that often does more harm than good, systematically destroying the very data we seek to measure.

The hallmark of a robust and accurate biomechanical analysis pipeline is not the choice of a single filter, but the implementation of a methodology for signal analysis. This methodology can be broken down into a clear, repeatable process: Instrument, Diagnose, and Apply Layered, Context-Aware Filtering.

This article will detail that process, providing some guidelines for transforming raw, noisy pose data into a source of truth.

Step 1: The First Commandment - Instrument Everything

You cannot fix what you cannot see. Before you write a single line of filtering code, your first task is to build the instrumentation to capture and visualize the raw, unaltered signal. The goal is to log the coordinate data for key landmarks after resampling to a constant frame rate but before any smoothing.

A simple JSON serialization of the vector data per frame is perfect. For a given landmark, your log output should look something like this:

{

  "signal_name": "left_wrist_position",

  "instance": 2,

  "raw_resampled_vectors": [

    [ -0.08, 0.75, -0.21 ],

    [ -0.09, 0.78, -0.23 ],

    [ -0.08, 0.77, -0.22 ],

    [ -0.10, 0.81, -0.25 ],

    ...

  ]

}

This raw log is your ground truth for the debugging process. Plotting this data on a simple graph will immediately reveal the nature of the problems you need to solve.

Step 2: Diagnosing the Signal - A Taxonomy of Pose Estimation Noise

When you visualize the raw signal, you will discover that not all "noise" is the same. Data typically falls into three categories:

Type 1: High-Frequency Jitter

This is the most common artifact. It manifests as small, frame-to-frame oscillations around a true path.

• What it looks like:
  • // Frame-by-frame Y-coordinate of a relatively stable hip
  • [ 0.451, 0.453, 0.452, 0.454, 0.453 ]
• Cause: The pose estimation model making micro-corrections in each frame.
• Problem: When you take the derivative of this signal to find velocity, this jitter creates a massive amount of high-frequency noise, making true acceleration peaks impossible to find.

Type 2: Spurious Spikes (Outliers)

These are single-frame, high-magnitude errors where the model briefly loses track of a landmark.

• What it looks like:
  • // Frame-by-frame Y-coordinate of a hand during a swing
  • [ 0.61, 0.63, 0.98, 0.67, 0.69 ] // The 0.98 is a clear outlier
• Cause: Brief moments of motion blur, occlusion, or model misidentification.
• Problem: A single spike will create catastrophic, physically impossible spikes in the calculated velocity and acceleration, completely invalidating the data.

Type 3: True Biomechanical Peaks

This is the signal, not the noise. It's a sharp, high-magnitude, but legitimate change in position that represents the explosive event we want to measure.

• What it looks like:
  • // Frame-by-frame X-coordinate of a bat barrel at impact
  • [ 0.30, 0.45, 0.62, 0.78, 0.91 ]
• The Challenge: The primary goal of any filtering strategy is to remove Type 1 and Type 2 noise while perfectly preserving the shape and magnitude of the Type 3 signal. A naive filter will treat the true peak like noise and attenuate (flatten) it, rendering the measurement inaccurate.

Step 3: The Layered Filtering Strategy - A Surgical Approach

A single filter cannot solve these distinct problems. A layered approach, applying a sequence of specialized tools, is required.

Layer 1: The Outlier Killer - The Median Filter

The first pass should always be a non-linear filter designed to eliminate spikes. The Median Filter is the ideal tool.

• How it Works: It slides a small window (a kernel_size of 3 is perfect for single-frame spikes) across the signal and replaces the center point with the median of the values in the window.
• Why it's First: It is exceptionally effective at removing Type 2 spikes. The outlier value is discarded, and the point is replaced by one of its neighbors, preserving the signal's integrity. Crucially, it does this with almost no impact on the true peaks, as the median of a sequence like [0.6, 0.8, 0.7] is 0.7, maintaining the general shape.

Implementation: clean_signal = medfilt(raw_signal, kernel_size=3)

Layer 2: The Smoothing Craftsman - The Savitzky-Golay Filter

With the spikes removed, we can now address the high-frequency jitter. The Savitzky-Golay (Savgol) filter is the superior choice over a simple moving average.

• How it Works: Instead of just averaging points, it fits a polynomial of a specified order (polyorder) to a window of data (window_length). This is far better at preserving the underlying shape of the signal.
• The Engineering Trade-off:
  • window_length: A wider window provides more smoothing but increases the risk of attenuating sharp peaks.
  • polyorder: A higher order can fit a more complex shape (like a sharp peak) but is more sensitive to noise.

This is why our layered approach is so effective. The Median filter removed the noise that would have thrown off a high-order polynomial, allowing us to use a polyorder of 3 or 4, which is excellent at matching the true shape of a biomechanical event.

Implementation: final_smooth_signal = savgol_filter(clean_signal, window_length=19, polyorder=3)

Step 4: The Multi-Path Architecture - Tailoring the Strategy to the Goal

Even a layered strategy isn't a one-size-fits-all solution. The optimal filter parameters depend on the measurement goal. A robust physics engine should therefore employ a multi-path architecture. Here is a simple example:

Path A: The Foundational Stream (Full Motion Analysis)

• Goal: Analyze the entire kinematic sequence (e.g., rotation of pelvis, then torso, then arm).
• Signal: The full, multi-second signal for core body landmarks.
• Strategy: Use the full two-stage filter (medfilt -> savgol_filter). The Savgol filter here should have a relatively wide window (e.g., 15-21 frames). This provides excellent general smoothing, creating a stable baseline for calculating the timing and sequence of broad movements.

Path B: The High-Fidelity Stream (Peak Event Measurement)

• Goal: Find the precise peak value of an explosive, short-lived event (e.g., peak hand speed at ball release).
• Signal: A short, "relevant window" sliced from the full signal, containing only the high-velocity phase of motion.
• Strategy: On this short, high-quality slice, use only a single-pass Savgol filter. The key is to use a very narrow window (e.g., 7-9 frames) and a high polynomial order (e.g., 3). The narrow window ensures the filter does not average away the peak, preserving its true magnitude.

Conclusion

Building an accurate biomechanics engine from pose estimation data is a signal processing challenge first and a physics problem second. By treating it as such, we can move beyond naive smoothing and implement a professional-grade pipeline.

The methodology is clear: Instrument to see the truth of your raw signal. Diagnose the distinct types of noise affecting your data. Apply a Layered filtering strategy, using a median filter to kill outliers before applying a Savitzky-Golay filter to smoothly model the true motion. Finally, implement a Multi-Path Architecture that tailors your filtering parameters to the specific goal of the measurement.

This approach is the difference between a system that produces interesting but unreliable estimates and one that delivers verifiable, accurate, and actionable insights.