Adding captions to video via an API means going from an audio track to timed, styled on-screen text without touching an editor. Done well, it's automatic: the API transcribes the speech, aligns the words to time, and burns readable captions into the render — placed so they help the viewer instead of covering the action.
This guide covers the pipeline (transcription → timing → styling → placement), what separates mechanical captions from ones that look hand-edited, and how content-awareness makes placement automatic.
Why captions, and why via API
Captions are no longer optional. Much video today is watched muted by default — especially on mobile and in social feeds — so captions are often the only way the message lands. They also widen accessibility and give search and AI systems text to understand the video by.
Doing it via an API — rather than by hand in an editor — is what makes captions viable at volume: hundreds of clips, personalized variations, or any pipeline where a human editor isn't in the loop. The caption step becomes a property of the composition, not a manual task.
The captioning pipeline
Under the hood, adding captions via an API runs four stages.
1. Transcription. The API transcribes the audio into text. Quality here sets the ceiling for everything downstream — a good transcription handles multiple speakers and languages and produces accurate word timing.
2. Timing (alignment). Words are aligned to timestamps so captions appear in sync with speech. This is what lets captions be broken into readable chunks that change at the right moment, rather than a wall of text.
3. Segmentation and styling. The timed transcript is grouped into caption segments — a few words or a short phrase at a time, sized to be read at a glance — and styled: font, size, weight, color, highlight, and position. Good segmentation respects sentence and phrase boundaries so captions read naturally instead of breaking mid-thought.
4. Placement and rendering. The styled captions are composited into the video and rendered. Where the captions sit — and whether they respect what's behind them — is where most APIs stop and the good ones keep going.
Mechanical captions vs. captions that look edited
A basic captions API drops text in a fixed band at the bottom of the frame. It's legible, but it ignores the picture: it can sit on top of the speaker's chin, cover a lower-third graphic, or fight the composition.
Captions that look hand-edited are placed with awareness of the frame. They're timed to phrase boundaries, sized for the aspect ratio (a 9:16 short needs different caption sizing than a 16:9 video), and positioned so they don't collide with the subject or other on-screen elements. The difference is subtle per frame and obvious across a whole clip — one looks automated, the other looks produced.
Content-aware placement: captions behind the subject
The frontier of programmatic captioning is placement that understands what's in the frame. This requires the API to perceive the footage, not just its dimensions.
With subject and face detection, an API knows where the speaker is. That unlocks a technique that reads as genuinely produced: captions that sit behind the subject rather than covering their face. The text is composited under the person, so the words are present and readable but never obscure the speaker — the kind of touch a careful human editor would do by hand, applied automatically. CueFrame does exactly this: because it detects the subject, captions can be placed behind them by default instead of pasted over the frame.
Content-awareness is the payoff of an intent-based approach. You declare the intent — "caption this" — and because the API has perception (the transcript for timing, the subject for placement), it makes the many small decisions that a coordinate-based API would push back onto you. You don't compute where the captions go; the API derives it from the footage. (See Render Video from JSON for how intent-and-derived composition works generally.)
Reading the media context first
A practical tip that makes captioning line up with reality: read the media context before you author. CueFrame exposes the detected faces (with the primary speaker marked) and the transcript for a piece of media, so your caption timing and placement are based on what's actually in the footage rather than assumptions. Author captions against that context and they land correctly the first time — which, for an AI agent driving the API, is the difference between one pass and a retry loop.
Styling that stays on-brand
Captions are a brand surface. A good API lets you control the type, weight, color, and highlight so captions match the rest of your video — and ideally lets you bind those to a brand kit so every clip is consistent without restating the styling each time. The goal is captions that look like part of the piece, not a generic overlay stamped on top.
Getting started
CueFrame transcribes, times, styles, and places captions as part of a content-aware composition — including captions behind the subject — over MCP (Model Context Protocol), a CLI, and REST. The Quickstart shows the flow end to end, and the pillar guide covers the composition model captions sit inside.
Frequently asked questions
How do you add captions to a video with an API? The API transcribes the audio, aligns the words to time, segments them into readable chunks, styles them, and composites them into the render. With a content-aware API you just express the intent ("caption this") and it derives timing and placement from the footage.
Can captions avoid covering the speaker's face? Yes — an API that detects the subject can place captions behind the speaker, so the text is readable but never obscures their face. CueFrame does this by default because it perceives the subject in the frame.
Do I need to time each caption myself? No. The API aligns the transcript to the audio and segments it, so captions appear in sync and change at natural phrase boundaries without you setting timings by hand.
Can I match captions to my brand? A good captions API lets you control font, size, weight, color, and highlight — and bind them to a brand kit so styling stays consistent across every clip automatically.