When you wave your hand in front of a camera, how does a computer know it's a hand — and not a leaf, or a shadow? When you try on a virtual shirt in Lucy AR, how does the app know where your shoulders are, how wide your body is, and which direction you're facing?

The answer is pose estimation — and it's one of the more elegant problems in computer vision. Here's how it works, in plain terms.

What a Landmark Is

A body landmark is a specific anatomical point: the tip of your nose, the centre of your left shoulder, your right wrist. BlazePose — the model we use in Lucy — detects 33 of these points simultaneously in every camera frame.

Each landmark is represented as three numbers: x position, y position (both as fractions of the frame width/height), and a visibility score from 0 to 1. A visibility score of 0.9 means the model is highly confident that landmark is visible. A score of 0.2 means it's probably occluded — hidden behind another body part or out of frame.

33 landmarks, 3 numbers each, 30 times per second. That's 2,970 numbers your phone is computing every second just to know where your body is.

How the Model Finds You

Pose estimation runs in two stages:

Stage 1: Person detection. A lightweight detector scans the full frame to find the bounding box around a person. This runs at lower resolution for speed — it just needs to find where you are, not precise details.

Stage 2: Landmark detection. The region inside the bounding box is cropped and fed to a higher-resolution landmark model. This model outputs the 33 landmarks with their positions and confidence scores.

Running stage 2 only on the cropped region — rather than the full frame — is what makes real-time performance possible on a phone. You're doing expensive computation on a small patch, not the whole image.

The Jitter Problem (and the Fix)

Raw landmark predictions shake. Even if you stand perfectly still, the model's output for your shoulder position will vary by a few pixels frame-to-frame. This is normal — neural networks aren't deterministic at the pixel level.

For most applications, this doesn't matter. For AR, it's fatal. A garment that wobbles by 5 pixels per frame looks broken. Users immediately notice.

The naive fix is a moving average: take the last N frames and average their positions. This smooths the jitter, but introduces lag — fast movements feel sluggish because the average takes time to catch up.

Our fix was velocity-adaptive smoothing. We track how fast each landmark is moving frame-to-frame. When movement is fast, we reduce the smoothing factor — following the motion closely. When movement is slow or stopped, we increase smoothing — averaging more frames to eliminate noise.

In code, it's roughly:

speed = distance(current, previous) / frame_time
alpha = max(0.1, min(0.9, 1 - speed * 0.05))
smoothed = alpha * previous + (1 - alpha) * current

Simple exponential smoothing with a dynamic alpha. The result: garments that hold steady when you're still, and snap responsively when you move.

Garment Occlusion

The final challenge: a realistic garment overlay needs to go behind your arms when you cross them, not on top. This is the occlusion problem.

We solved it with body segmentation — a second model that classifies each pixel as "body" or "background". By rendering the garment layer below the body silhouette, we get natural occlusion: arms in front of the shirt, not behind it.

Computer vision on mobile isn't magic — it's a chain of small, clever approximations, each fast enough to run 30 times per second.

Every technique here — two-stage detection, confidence-based blending, adaptive smoothing, segmentation-based occlusion — is a practical engineering trade-off between accuracy and speed. Real-time AR is the art of being good enough, fast enough, consistently enough.