Case Study: A Defensible Implementation of GenAI for Bounded Observational Tasks in Video Analysis

Architects and engineers building complex systems are navigating a period of intense hype and justifiable skepticism. We are being inundated with the mandate to "put AI on it," often by stakeholders who see Generative AI as a magical black box that can solve any problem. The result, more often than not, is a system that is non-deterministic, unprovable, and fundamentally untrustworthy. We see LLMs being asked to calculate physics, generate metrics from thin air, and make quantitative assessments they are architecturally incapable of performing accurately. These implementations are indefensible.

This trend creates a dangerous skepticism, leading us to believe that GenAI has no place in systems that demand precision and integrity. This is a mistake. The failure is not in the tool, but in the application. The future of robust AI systems lies not in replacing deterministic code with generative models, but in surgically integrating them to solve problems that are, paradoxically, immensely complex for traditional code to handle.

Our implementation of "camera angle detection" is a case study in this approach. While it appears to be a simple query to our powerful, multimodal model, architecturally, it represents a mature and highly defensible implementation strategy.

The "Trivial" Problem That Isn't: Deterministically Coding Camera Angles

Consider the engineering challenge: write a deterministic function that takes a video of an athlete and returns a precise camera angle classification like "Open Side View" or "Down-the-Line (DTL)".

Your first thought might be to use pose estimation data. A simple approach would involve complex geometric calculations based on the 3D coordinates of an athlete's joints. For a baseball hitter, if the vector from the right shoulder to the left shoulder is roughly parallel to the camera's X-axis, you might classify the angle as a "Side View." But which side? You'd then have to factor in handedness and determine if the player's chest or back is more visible.

This complexity explodes when the system must be agnostic to a growing library of sports and actions. The logic for a baseball_hitting_action is fundamentally different from a golf_full_swing_action:

• For a hitter, a "Closed Side View" is defined by "The batter's back is the dominant surface presented to the camera..."
• For a golfer, a "Down-the-Line (DTL)" view is defined by "...their back and rear shoulder primarily facing the camera."
• For a pitcher, an "Arm Side View" depends on which arm—the throwing arm—is closest to the camera.

The deterministic approach would require a hard-coded, multi-level dictionary of geometric rules, mapping each unique action_key to a specific set of anatomical heuristics. Every new sport or action added to our platform would necessitate a new engineering cycle to update this brittle logic.

Now, add the complexity of real-world video. What if the player is twisted at an unusual angle during their follow-through? What if a key landmark like a shoulder is temporarily occluded? What distinguishes a pitcher's "Catcher's View" from a hitter's "Catcher's View" when both can be in the frame?

The deterministic solution quickly devolves into a sprawling, brittle state machine. A complex web of if/else statements and vector math trying to account for every possible sport, action, and player variation. The engineering overhead to build, test, and maintain this is enormous, and its accuracy would still be questionable.

The GenAI Solution: A Surgical Strike

Our solution bypasses this complexity entirely. In the orchestrator/app.py function, after actions have been validated but before they're sent for heavy processing, we make a single, fast, and hyper-focused call to our multimodal model.

• The Function: CAMERA_ANGLE
• The Context: The full video file and a JSON object telling the model the specific action_list to analyze.
• The Question: "For each action instance, assess its viability and execute a strict, landmark-based protocol to classify the camera angle based on the provided rulebook."

The model, with its native understanding of video, doesn't need a complex state machine. It sees the player's orientation, the context of the sport, and the direction of motion all at once. It performs a high-level, observational task—applying a human-readable rulebook to visual evidence—which is the exact kind of task it excels at. It returns a single piece of metadata: camera_angle_detected.

Why This is Architecturally Sound and Defensible

This isn't another "let the AI figure it out" implementation. It is a robust engineering solution precisely because it is bounded, verifiable, and respects architectural separation of concerns.

1. It is Bounded and Specific. The AI's responsibility is microscopic. It is not asked to calculate biomechanics. It is asked to perform a simple classification task from a predefined list. The output is not a complex narrative; it is a single, verifiable string: "Open Side View". This narrow scope, defined in the prompt's mandatory JSON schema, dramatically reduces the "surface area" for hallucination or error.
2. It is Measurable and Verifiable. This is the most critical point for us as engineers. We are not blindly trusting the model. Our architecture has a built-in, deterministic mechanism to audit its accuracy:

• Downstream Consequence Analysis: The aggregator/app.py function contains a process called _apply_angle_gating_to_results. This function takes the AI-generated camera_angle_detected and uses it as a key into a deterministic MASTER_RULEBOOK. This rulebook defines which biomechanical metrics are physically valid from which angles.
• If the angle detection model fails and classifies a clear "Side View" of a pitcher as a "Catcher's View," the angle gating function will deterministically (and incorrectly) remove metrics like lead_leg_block_angle_deg because the rulebook states that this metric is not visible from the front. We can measure our AI's accuracy by the absence of such downstream validation failures. The deterministic code acts as an immediate sanity check on the generative model's observation.

3. It Respects the "Calculator vs. Observer" Paradigm. This is the core architectural principle. Our system is a hybrid model:

• The Calculator (Deterministic Code): The _apply_angle_gating_to_results function is a pure calculator. It takes in a camera angle string and a list of metrics and applies immutable, hard-coded rules from the MASTER_RULEBOOK. It is 100% deterministic and verifiable.
• The Observer (Generative AI): The LLM's role is to provide the configuration for the calculator. It performs the observational, context-aware task of identifying the correct camera angle to use as the key.

We are not using AI to decide if a metric is valid; we are using it to tell our deterministic filter from which perspective to judge the validity. This is a profound difference. It delegates the "fuzzy" observational task to the tool best suited for it, while reserving the "precise" logical task for deterministic code.

Conclusion: The Right Tool for the Right Job

The industry's obsession with using GenAI as a replacement for entire engineering stacks is misguided. Its true value in complex, data-driven systems lies in its surgical application as a specialized component.

By using a powerful multimodal model for a simple, bounded, and verifiable observational task, we eliminated a mountain of complex and brittle computer vision code. We reduced engineering overhead while simultaneously increasing the accuracy and robustness of our data processing pipeline. This is the hallmark of mature AI system design: not a blind faith in a black box, but a deep understanding of the tool's strengths and weaknesses, applied with surgical precision. The most effective use of our AI is as an expert observer, providing the critical context that allows our deterministic code to perform its calculations with integrity.