Face-aware smart crop reframes a video to a new aspect ratio — a 16:9 interview into a 9:16 vertical short, say — by detecting the subject and the active speaker in the footage and keeping them in frame, instead of slicing a fixed rectangle out of the middle. A blind center-crop assumes the action lives in the center of the picture. It rarely does: speakers stand off to one side, cut to a second person, or move. The result is decapitated heads and off-screen faces. A face-aware crop reads what's actually in the frame and follows it, so the person stays framed the way a human editor would keep them.
This guide explains how smart cropping works — subject and face detection, active-speaker identification, and the intent that drives the crop — and why deriving the crop from the footage beats hand-computing crop rectangles yourself.
The problem with center-crop
Reframing is unavoidable. Footage is shot 16:9 and the feeds it plays in are 9:16, so something has to decide which slice of the wide frame survives the crop. The naive answer is: take the middle. Keep the center 9:16 column, discard the sides.
That works only when the subject happens to sit dead center and never moves — which is almost never. In a real interview the speaker sits off to one side, the shot cuts between two people, a hand enters from the edge, someone leans. A center-crop is blind to all of it. It crops to a coordinate, not to a person, so it clips faces, frames empty background, and lands on the wrong subject the moment the shot changes.
You can fix a center-crop by hand — nudge the crop window per shot, keyframe it to follow the speaker — but that's exactly the manual editing work an API is supposed to remove. Doing it in software at volume means the software has to know where the subject is. That's what "face-aware" means.
What makes a crop "face-aware"
A face-aware smart crop is driven by two things the system perceives in the footage: where the faces are, and which face is the active speaker.
Face and subject detection finds the people in each frame and returns their positions as normalized bounding boxes — coordinates relative to the frame, so they hold up regardless of resolution. This is what turns "keep the person framed" from a wish into a computable target: the crop window can be placed and sized around a box the system actually located, not a guess about where the subject probably is.
Active-speaker identification answers the harder question: when there are two or three faces on screen, which one is talking? A crop that follows the wrong face is as broken as one that clips heads. By identifying the active speaker — and marking the primary speaker in the face roster — the system knows which subject to keep centered as the conversation moves between people. The reframe tracks whoever holds the floor, the way a human editor would cut to the person speaking.
Together these turn reframing from a geometry problem into a content problem. The crop follows the subject, and specifically the right subject, because the system perceives both.
Intent-and-derived vs. hand-computed crops
There are two ways to ask an API to reframe a video, and the difference is the whole game.
The first is to compute the crop yourself and hand the API exact rectangles: for every shot, this pixel offset, this width, this height, changing at these frames. Total control — and total responsibility. You have to locate the speaker across thousands of frames, decide how tightly to frame them, and re-derive all of it whenever the footage changes. The API just draws the rectangles you supply, which means every framing mistake is yours to compute and yours to fix.
The second is intent-and-derived: you express the goal — "reframe this to 9:16 and keep the active speaker framed" — and the API derives the pixel-level crop from the actual content of the footage. This is the approach CueFrame takes. You state what you want; the system inspects the media, finds the speaker, and computes the crop window that keeps them framed, shot by shot. Less brittle, no bounding-box math on your side, and the output adapts to the footage instead of assuming it.
The intent model is what makes smart cropping tractable for automation, and especially for AI agents driving the API. An agent is good at expressing a goal and bad at computing crop coordinates across a timeline. Give it an API that accepts "keep the speaker framed" and it can reframe real footage; force it to emit exact rectangles and it will get them subtly, repeatedly wrong. This is the same intent-and-derived principle that runs through the whole composition model — you declare intent, the API derives the pixels.
The perception that powers it
A smart crop is only as good as what the system can see in the footage. Before it reframes anything, CueFrame analyzes the source: it detects faces and subjects, identifies the active speaker, and transcribes the audio with timing.
That perception is readable. You can pull a piece of media's context before you author anything — the face roster with the primary speaker marked, each face's normalized bounding box and its share of active-speaker time, plus the transcript. So the intent you express is grounded in what's actually in the shot: you can confirm there's a clear primary speaker to follow, see when the conversation hands off to a second face, and reframe against reality instead of assumptions. The crop is derived from the same context you can inspect.
The payoff is that framing decisions become properties of the media, not manual math. Because the system knows where the subject is, it can keep them framed through a reframe — and that same subject detection is what lets captions sit behind the subject rather than covering their face. Perception is the foundation; smart crop and content-aware captions are two things it unlocks.
Smart crop in a composition
Reframing isn't a standalone filter — it's one intent inside a larger edit. In CueFrame the crop is part of the composition: the same structured, JSON-authored edit that carries your clips, captions, and graphics also carries the framing intent. You declare "keep the active speaker framed" alongside "caption this" and "this graphic here," and the API derives all of it from the footage in one render.
That matters because the decisions interact. Captions want to know where the subject is so they don't collide with the crop; the reframe wants to know the aspect ratio the captions are sized for. Authoring them together, as intent over perceived media, is what lets a single render come out looking produced rather than like three automated passes stacked on top of each other. Send the same composition twice and you get the same reframe twice — the crop is a deterministic function of the footage and the intent, which is what makes it safe to run inside an automated pipeline.
Getting started
The fastest way to understand face-aware smart crop is to reframe one clip with it. CueFrame detects the subject and active speaker, reads out the media context, and derives the crop from your intent — over MCP, a CLI, and REST. The Quickstart walks the flow from raw footage to a reframed clip, and the full docs cover the composition model the crop sits inside.
Frequently asked questions
What is smart crop for video? Smart crop reframes a video to a new aspect ratio by detecting the subject in the footage and keeping them in frame, instead of cropping a fixed rectangle out of the center. Face-aware smart crop specifically detects faces and the active speaker, so the crop follows the person who's talking as the shot changes.
How is face-aware crop different from a center-crop? A center-crop keeps the middle of the frame regardless of what's there, so it clips faces and frames empty space whenever the subject sits off-center or moves. A face-aware crop detects where the subject is and follows them, keeping the speaker framed the way a human editor would — no decapitated heads, no off-screen faces.
Do I have to compute the crop coordinates myself? Not with an intent-and-derived API. You express the intent ("reframe to 9:16, keep the active speaker framed") and CueFrame derives the pixel-level crop from the footage using face detection and active-speaker identification. With a coordinate-based API you'd compute and keyframe every crop rectangle yourself.
How does it pick the right person when several are on screen? It identifies the active speaker and marks the primary speaker in the face roster, using each face's share of active-speaker time. The reframe follows whoever holds the floor, so the crop tracks the person talking rather than an arbitrary face — and you can read that roster in the media context before you author the edit.