Why Selfvie stopped shearing faces: similarity beats affine

Selfvie's whole job is alignment: take dozens of selfies shot over months, and warp each one so the face sits in exactly the same place, so that played back at speed they become one continuous movie of a face aging through time.

The obvious approach is a 3-point affine transform. Face detection gives you landmarks — two eyes and a nose, say — and mapping three source points onto three target points is a solved linear system. First version did exactly that.

It looked wrong, in a way that took a while to name. Faces came out subtly smeared — one cheek stretched, the jaw skewed — but only in some frames.

The bug is the degrees of freedom

An affine transform fit to three points reproduces those three points perfectly and does whatever it takes to the rest of the plane — including shear. Eyes and nose are only rigid relative to each other while the head faces the camera straight on. Turn the head a few degrees and the nose moves relative to the eye line (that's what 3D projection does), and the affine fit happily absorbs that parallax as a 2D shear of the entire face.

The fix: drop the nose, fit a similarity transform on the eye pair alone. Rotation, uniform scale, translation — four degrees of freedom, and shear is not among them. A similarity transform cannot distort a face, by construction. Head pose now costs a tiny alignment error instead of a warped face, which is the right trade in every single frame.

The background had the same shape of bug

The second-worst artifact had the same root cause — doing the plausible thing instead of the perceptually right one. Filling the empty space around an aligned photo with a blurred copy of itself reads as a creepy double image. What works: smear the photo's own edge pixels outward and fade them toward the mean color. No structure, no ghost face, just ambience.

Both fixes are in the engine today (affine.ts keeps both transforms; renderer.ts owns the background). The pattern that keeps repeating in this project: when output "looks wrong", the fix is rarely more precision — it's a representation that makes the artifact impossible.