The New Paradigm of AI Agents: How video-use Uses Text Instead of Video for Smart Editing
2 months ago
The browser-use team recently open-sourced a project called video-use, which gained nearly a thousand stars in a day.
What it does sounds sci-fi: you throw a raw, unedited video at Claude, and it automatically removes filler words, aligns cut points, adds subtitles, and color-grades, finally outputting a finished video.
But what made me pause is its approach.
AI editing videos is nothing new, but how it edits is interesting.
This AI that edits videos never actually watches the video.
It doesn't understand the video frame by frame, nor does it feed the video into a multimodal model. Most of the time, it only deals with a text file of a few dozen KB.
This isn't cutting corners.
This is the same approach shared by the smartest agent products today.
Understanding this is far more useful than just knowing "there's another AI editing tool."
Suppose you want a model to understand a ten-minute video.
The most intuitive approach is frame extraction: extract a few frames per second, resulting in about 30,000 frames for the whole video. Each frame fed to a multimodal model costs about 1,500 tokens.
Multiply that, and just to look at the video once, you burn through 45 million tokens.
And after all that, what the model remembers is mostly noise—it knows that at frame 12,000 someone is speaking, but what they said or where to cut? The video itself can't tell it.
This isn't hypothetical.
In May this year, a public benchmark found that a visual agent relying purely on screen captures consumed 45 times more tokens to complete the same task compared to directly calling APIs.
The computational cost is one thing, but worse, it performed more poorly—agents that clicked buttons based on screenshots often clicked wrong positions or got misled by pop-ups.
Feeding raw data directly to the model is an expensive, dumb, and error-prone direction.
video-use asks a different question
video-use doesn't ask:
"What is in this frame?"
It asks:
"How is this sentence spoken?"
First, it uses ElevenLabs' speech-to-text to transcribe the entire raw footage into a text file with word-level timestamps, speaker diarization, and even markers for "(laugh)" and "(pause)."
This text file is only a few dozen KB.
That's what Claude receives—a script that can be reasoned about with textual precision.
Editing, after all, is about making decisions on a timeline:
- Which phrase is repeated?
- Where is there a stumble?
- Keep this part or delete it?
These judgments are faster and more accurate when done on text than on video.
When the model reads:
"Um... so, today we're going to talk about..."
It immediately knows this is filler to be removed, without needing to see the face.
So when is the video actually used?
Only at critical decision points where text alone isn't enough.
For example:
- Choosing between two equally good candidate clips
- Determining whether a pause is a mistake or intentional
At these moments, it temporarily generates a composite image made of filmstrip thumbnails, waveform, and text labels, and takes a look.
Expensive perception is kept off most of the time, only turned on at stuck points.
For the entire video, the text is a few dozen KB, plus a few key images—negligible compared to the 45 million token brute-force approach.
This approach was already used by browser-use
If this feels familiar, it's because browser-use itself rose to fame using the same trick.
To let an AI operate a web page, the brute-force method is to take a screenshot and let it figure out:
"Where is the login button? Click there."
browser-use doesn't take screenshots.
Instead, it extracts the page's DOM structure—the tree of tags behind the page—and presents it as text to the model.
The model reads structured data, skipping the image understanding step entirely.
The effect is the same direction:
Research has found that using structured element positioning instead of full-page screenshots saves over 90% of context;
Moreover, the model reads deterministic structures, not misled by visual noise like styles or pop-ups, resulting in much higher first-click accuracy.
Looking back, you'll see that almost all agents that have truly taken off do the same thing:
- Code-writing agents read text and syntax trees, not screenshots of your IDE;
- Browser-operating agents read the DOM, not the web page image;
- Video-editing agents read transcriptions, not video frames.
Their domains are vastly different, but the problem-solving approach is the same.
A judgment framework you can take away
Extracting the common thread gives us a reusable mental model:
Powerful AI agents almost never let the model face the raw world directly.
They first compress the domain into a structure the model is already good at (text, tree, sequence), perform most reasoning on that structure, and only use expensive raw perception at critical decision points.
Video is compressed into transcription text.
Web pages are compressed into DOM.
Code is already text.
Finding this low-cost structural representation is the make-or-break factor for such products.
So next time you see a so-called fully automatic AI agent, instead of just watching the demo, poke it with three questions:
First, does it compress the domain into a structured representation?
Or does it make the model chew on raw data (screenshots, video frames, full HTML)?
The latter almost always means slow, expensive, and unstable—hard to scale.
Second, is expensive raw perception always on, or only invoked on demand at critical points?
If always on, costs explode with task length.
Nice demo, but impractical for real use.
Third, how accurate is that structural representation itself?
video-use's Achilles' heel is transcription—if one word is transcribed incorrectly, all subsequent editing decisions are built on a faulty foundation.
An agent's ceiling often depends not on how smart the model is, but on how much is lost or wrong when translating the world into a structure.
These three questions aren't just for evaluating tools—they can also help you think about your own work:
If you want AI to take over a task, the first step isn't to blindly feed it raw materials, but to think clearly:
Is there a cheap and accurate structural representation for this task that allows the model to reason?
Getting this right is often more effective than switching to a larger model.
video-use appears to be just an editing tool.
But what it reveals is how this generation of agents sees the world:
Instead of staring at raw images, first translate the world into a language the model can read, then start thinking.
