Most AI features in consumer apps work like this: your photo goes to a server, a GPU cluster processes it, the result comes back. It works, but it introduces latency, requires an internet connection, and sends your data somewhere you can't see.
Lucy AR takes a different approach. Every frame of your camera feed is processed on your device, in real time, with no server involved. The result is instant response, full privacy, and an experience that works even without a connection.
Here's how we built it — and what on-device ML actually involves.
What Is TFLite?
TensorFlow Lite is a trimmed-down version of TensorFlow designed to run on mobile devices. A standard TensorFlow model trained on cloud GPUs can be hundreds of megabytes and require significant compute. TFLite converts those models into a format optimised for mobile CPUs and GPUs — smaller, faster, lower power.
The conversion process involves two key steps:
- Quantisation — replacing 32-bit floating point weights with 8-bit integers. This reduces model size by ~4x and speeds up inference significantly, with minimal accuracy loss for most visual tasks.
- Operator fusion — combining multiple operations (convolution + batch norm + activation) into single optimised kernels that run faster on mobile hardware.
For Lucy, we used a quantised BlazePose model for body landmark detection — 8-bit quantised, running at 15–30 FPS on mid-range Android devices.
Body Landmark Detection
Tracking a body in a video stream requires detecting the same anatomical points — shoulders, hips, wrists, ankles — in every frame, even as the person moves, rotates, or changes lighting conditions.
BlazePose detects 33 body landmarks per frame, each with an x/y coordinate and a confidence score. High confidence means the model is certain that landmark is visible; low confidence means it's occluded or out of frame.
We used confidence scores to handle partial visibility gracefully. If a shoulder landmark dropped below 0.6 confidence, we blended its position from the previous frame rather than snapping to an uncertain prediction. This eliminated a lot of the jitter that makes AR overlays look broken.
The Jitter Problem
Raw landmark predictions are noisy. Even on a stationary person, the detected shoulder position might vary by a few pixels frame-to-frame — enough to make an overlaid garment visibly shake. This is called jitter, and it's the difference between an AR experience that feels real and one that feels like a prototype.
We solved it with velocity-adaptive smoothing. The smoothing factor adapts based on how fast the landmark is moving:
- Fast movement → low smoothing (follow the motion closely, don't lag)
- Slow movement or stationary → high smoothing (average over more frames, eliminate noise)
This is a simple idea but surprisingly effective. The garment overlay moves fluidly with the user, snaps responsively to fast gestures, and holds steady during slow movements.
GPU Acceleration
TFLite supports GPU delegation — offloading inference from the CPU to the device's GPU. On most Android devices, this gives a 2–4x speedup for vision models, which is the difference between 8 FPS (unusable) and 20+ FPS (smooth).
We initialised the TFLite interpreter with the GPU delegate on app launch and fell back to CPU if the delegate wasn't available. This fallback is important: not all devices support GPU delegation, and you don't want the app to crash on older hardware.
On-device AI isn't about replacing the cloud. It's about the class of experiences that are only possible when the latency is zero.
The future of mobile AI is on-device. As models get smaller and mobile chips get faster, more of what currently requires a cloud server will run locally. Privacy-preserving, instant, always-available — that's where the interesting work is happening.