"Render video from JSON" means describing an edit as a structured document — clips, tracks, timing, captions, graphics — and sending it to an API that renders it into a finished video. No timeline UI, no manual keyframing. The JSON is the edit, which means software can generate it, diff it, version it, and template it.
This guide covers what goes in that document, the two philosophies for structuring it (the single most important thing to understand), and the loop for turning it into a rendered file reliably.
Why describe video as data
Expressing an edit as JSON unlocks everything you can do with data. You can generate a video from a template plus per-record values (a thousand personalized clips from a spreadsheet). You can diff two versions of an edit to see exactly what changed. You can put a composition in version control, review it, and roll it back. You can have an AI agent author it. None of that is possible when the "edit" lives as clicks in a timeline application.
The composition document is also a contract. It states precisely what the video should be, so the same document renders the same video — a property that makes video safe to automate. (See the pillar guide for why that determinism matters.)
What goes in a composition document
At its core, a composition JSON describes what is on screen and when. The common ingredients:
- Media references — the source footage, images, and audio the edit draws on (usually uploaded or imported first, then referenced by ID).
- Tracks and clips — layers of content over time. A clip points at a piece of media, with in/out trim points and a position on the timeline.
- Captions — text tied to timing, often derived from a transcript, with styling.
- Overlays and graphics — lower-thirds, titles, motion graphics, and other elements composited over the video.
- Output settings — resolution, aspect ratio, and format.
The exact schema varies by API, but the mental model is constant: layered clips on a timeline, plus text and graphics, rendered to a file.
Two philosophies: literal timeline vs. intent-and-derived
Here is the fork that determines everything about how it feels to render video from JSON.
The literal timeline
In a literal model, the JSON specifies every element's exact pixel position, size, start frame, and end frame. You get total control — and total responsibility. You, the caller, compute all of it: where the speaker's face is so a crop doesn't cut them off, how large a caption must be so it doesn't overlap the subject, when to cut. The API is a faithful renderer of coordinates you supply. If your coordinates are wrong, the video is wrong; the API did its job.
This works when you already know the geometry — a fixed template with known slots. It gets brittle fast when the footage varies, because every new video means re-computing the layout.
The intent-and-derived model
In an intent model, you express what you want — "keep the active speaker framed," "caption this," "put this graphic behind the subject" — and the API derives the pixel-level result from the actual content of the footage. This is the approach CueFrame takes. You declare intent; the system inspects the media (detecting faces, the active speaker, and the transcript) and computes the coordinates.
The intent model is less brittle and requires far less math on your side, and the output adapts to the footage instead of assuming it. It's also what makes rendering video from JSON tractable for an AI agent: an agent is good at expressing goals and bad at computing bounding boxes across thousands of frames. (More on that in Video APIs for AI Agents.)
You don't have to choose purity — good APIs let you be explicit where you want control and rely on derivation where you don't. But knowing which model an API defaults to tells you how much work it's pushing onto you.
The author-validate-render loop
Rendering video from JSON reliably follows three steps. Skipping the middle one is the most common and expensive mistake.
1. Author the composition. Build the JSON — reference your media, lay out the clips, add captions and graphics. If you're using an intent-based API, this is where you express goals rather than coordinates.
2. Validate before you render. Rendering is the slow, costly step; validation is where you catch mistakes for free. A good API lets you dry-run a composition — checking that clips fit their tracks, referenced media exists, and parameters are in range — and returns typed, explainable errors. Iterate here until the composition is provably well-formed. Validating first is the difference between paying once and paying for every mistake.
3. Render and retrieve. Submit the validated composition. Rendering is typically asynchronous, since video takes real time — so you either poll the render job or register a webhook to be notified, then fetch the finished file when it's ready.
The shape is deliberate: author cheaply, validate for free, render once. It's the same loop whether a human or an agent is driving.
A concrete example, in plain terms
Suppose you want to turn a 16:9 interview into a captioned 9:16 short. In an intent-and-derived flow, the composition is just data — roughly:
{
"output": { "aspect": "9:16" },
"clips": [{ "media": "med_x9a2", "range": [12, 72] }],
"reframe": { "follow": "active-speaker" },
"captions": { "source": "transcript", "style": "word-pop" }
}
Notice what isn't there: no crop rectangles, no per-word caption timings, no pixel coordinates. You state intent — trim, aspect, follow the speaker, caption from the transcript — and the API derives the rest from the footage.
You don't specify the crop rectangle for every frame — the API reads the footage, finds the speaker, and follows them. You don't hand-time each caption — the API uses the transcript's timing. You validate that composition (does the trim fit? does the media exist?), fix anything the dry-run flags, and render. The output adapts to this interview; the same composition applied to a different interview reframes and captions that one.
For the captioning specifics, see How to Add Captions to Video via API.
Getting started
CueFrame renders production-quality video from a structured, content-aware composition over MCP (Model Context Protocol), a CLI, and REST. The Quickstart walks the whole loop end to end, and the pillar guide covers the concepts behind it.
Frequently asked questions
Can you really make a full video just from JSON? Yes — the JSON describes the edit (clips, timing, captions, graphics, output settings) and the API renders it to a video file. Software can generate that JSON, which is the whole point: templated, personalized, and automatable video.
Do I have to specify exact pixel positions in the JSON? Only with a literal-timeline API. An intent-and-derived API (like CueFrame) lets you express goals — "keep the speaker framed," "caption this" — and derives the coordinates from the footage, so you don't compute geometry yourself.
Why validate before rendering? Rendering is the expensive step. Free validation lets you catch a malformed composition — a clip past the end of its track, a missing media reference — before you pay to render, and iterate until it's correct.
Is rendering synchronous? Usually not — video takes time, so renders run asynchronously. You poll the job or register a webhook and fetch the finished file when it's ready.