Auto caption videos for free with faster-whisper and ffmpeg

You can auto caption a video for free with two open-source tools: faster-whisper turns the speech into timed captions on your own machine, and ffmpeg burns those captions into the picture in whatever font and style you pick. Nothing gets uploaded anywhere and there's no subscription.

Below I go through how the pipeline works, which Whisper model to use, and a complete Python script that captions a vertical short a few words at a time. The video above covers the same process.

Why I built my own caption tool

I post a lot of Shorts now, and a good Short needs captions. If you scroll Shorts at all, you've seen the style: a few words at a time, big and centered, near the bottom of the frame.

I couldn't find a free tool that does this. The AI video editors online all charge a subscription, and CapCut is a really good editor, but it's paid too. I'm not paying someone every month for something I can build myself, so I built it.

How auto captioning works

You start with the raw video, the file you recorded on your phone or wherever. Then:

1. Generate subtitles. Run the video through a speech-to-text model that gives you every word with a start and end time. I use faster-whisper for this. Group the words into short captions and save them as an .srt subtitle file.
2. Burn the subtitles into the video with ffmpeg, which draws each caption onto the frames at its timestamps. This is also where the styling happens: font family, font size, where on the screen the text sits, borders, backgrounds and shadows.
3. Save the final video file, and you're done.

Burning in means the captions become part of the picture. They show up on every platform and in every player, styled exactly how you set them, and viewers can't turn them off. The cost is that ffmpeg has to re-encode the whole video, which takes longer than copying it.

faster-whisper runs OpenAI's Whisper models on CTranslate2, a faster inference engine, with less memory than the original implementation. It runs fine on a CPU, so you don't need a GPU for any of this.

Which Whisper model to use

Before you write any code, pick the model. faster-whisper downloads it from Hugging Face the first time you use it. These are the three I've used, timed on a 40-second clip of me talking, on my 4-core CPU with no GPU:

I've had good results with small and medium, and luckily they're also the ones my old machine can actually run. Large-v3 gives really good transcriptions, but it takes way too long. I haven't found much difference between medium and large-v3, probably because my videos are in normal English, so there's not a lot of room for error. On my test clip all three got the same words, apart from small details: small.en wrote "gonna" where medium.en wrote "going to", and only small.en put a dollar sign in front of "430,000 Canadian dollars". Large-v3 took two and a half to three times as long as medium, and on this clip it wrote the whole transcript in lowercase with no punctuation, which the smaller models didn't do.

So if you make regular English videos like me, medium is enough. If you want the best transcription there is, or your videos are in other languages, use large-v3, and make sure your machine has the RAM for it.

Install faster-whisper and ffmpeg

You need an ffmpeg built with libass, the library that draws styled subtitles. The ffmpeg in most Linux package managers has it. Check yours:

ffmpeg -hide_banner -filters | grep subtitles

If it lists a subtitles filter, you're set. Then create a virtual environment and install faster-whisper:

python3 -m venv .venv
source .venv/bin/activate
pip install faster-whisper "av<19"

faster-whisper decodes audio with PyAV. Version 1.2.1 passes PyAV an argument that PyAV 19 removed, so without the av<19 pin the first transcription crashes with TypeError: open() got an unexpected keyword argument 'metadata_errors'.

The whole tool is one file:

auto-captions/
└── auto_caption.py

The script

import argparse
import subprocess
import tempfile
from pathlib import Path

from faster_whisper import WhisperModel


def srt_time(seconds):
    ms = round(seconds * 1000)
    h, ms = divmod(ms, 3_600_000)
    m, ms = divmod(ms, 60_000)
    s, ms = divmod(ms, 1000)
    return f"{h:02}:{m:02}:{s:02},{ms:03}"


def transcribe(video, srt_path, model_size, max_words, vad, max_pause=0.5):
    model = WhisperModel(model_size, device="cpu", compute_type="int8")
    segments, info = model.transcribe(
        str(video), beam_size=5, word_timestamps=True, vad_filter=vad
    )
    print(f"Language: {info.language} ({info.language_probability:.2f})")

    captions = []

    def add(chunk):
        text = " ".join(w.word.strip() for w in chunk)
        captions.append((chunk[0].start, chunk[-1].end, text))

    for segment in segments:
        chunk = []
        for word in segment.words or []:
            # Start a new caption when this one is full or the speaker paused.
            if chunk and (len(chunk) == max_words or word.start - chunk[-1].end > max_pause):
                add(chunk)
                chunk = []
            chunk.append(word)
        if chunk:
            add(chunk)

    with open(srt_path, "w", encoding="utf-8") as f:
        for n, (start, end, text) in enumerate(captions, start=1):
            f.write(f"{n}\n{srt_time(start)} --> {srt_time(end)}\n{text}\n\n")
    print(f"Wrote {len(captions)} captions to {srt_path}")


def read_srt(srt_path):
    captions = []
    for block in srt_path.read_text(encoding="utf-8").strip().split("\n\n"):
        lines = block.strip().splitlines()
        if len(lines) < 3:
            continue
        start, end = (t.strip() for t in lines[1].split("-->"))
        captions.append((start, end, " ".join(lines[2:])))
    return captions


def ass_time(srt_stamp):
    # 00:00:01,240 (SRT) -> 0:00:01.24 (ASS uses hundredths of a second)
    hms, ms = srt_stamp.split(",")
    h, m, s = hms.split(":")
    return f"{int(h)}:{m}:{s}.{int(ms) // 10:02}"


def video_size(video):
    out = subprocess.run(
        ["ffprobe", "-v", "error", "-select_streams", "v:0",
         "-show_entries", "stream=width,height", "-of", "csv=p=0", str(video)],
        capture_output=True, text=True, check=True,
    ).stdout
    width, height = out.strip().split(",")
    return int(width), int(height)


def write_ass(captions, ass_path, width, height, font, font_size, margin, uppercase, pop):
    # Colours are &HAABBGGRR. BorderStyle 3 draws an opaque box in OutlineColour
    # behind the text, and Outline sets how far the box reaches past the letters.
    header = f"""[Script Info]
ScriptType: v4.00+
PlayResX: {width}
PlayResY: {height}
WrapStyle: 0

[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: Caption,{font},{font_size},&H00000000,&H00000000,&H00FFFFFF,&H80000000,-1,0,0,0,100,100,0,0,3,{font_size // 4},0,2,80,80,{margin},1

[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
"""
    # Pop-in: start at 95% size, grow to 105% in 100 ms, settle at 100%.
    effect = r"{\fscx95\fscy95\t(0,100,\fscx105\fscy105)\t(100,200,\fscx100\fscy100)}" if pop else ""
    with open(ass_path, "w", encoding="utf-8") as f:
        f.write(header)
        for start, end, text in captions:
            if uppercase:
                text = text.upper()
            f.write(f"Dialogue: 0,{ass_time(start)},{ass_time(end)},Caption,,0,0,0,,{effect}{text}\n")


def burn(video, ass_path, output):
    # Run ffmpeg inside the folder that holds the .ass file, so the subtitles
    # filter only ever sees a plain file name (colons and quotes in a path break it).
    subprocess.run(
        ["ffmpeg", "-y", "-i", str(video.resolve()),
         "-vf", f"subtitles={ass_path.name}",
         "-c:v", "libx264", "-crf", "20", "-preset", "veryfast",
         "-c:a", "aac", str(output.resolve())],
        cwd=ass_path.parent, check=True,
    )


def main():
    parser = argparse.ArgumentParser(description="Burn word-by-word captions into a video.")
    parser.add_argument("video", type=Path)
    parser.add_argument("--model", default="medium.en",
                        help="tiny.en, base.en, small.en, medium.en, large-v3 ... (default: medium.en)")
    parser.add_argument("--max-words", type=int, default=3)
    parser.add_argument("--font", default="Arial")
    parser.add_argument("--font-size", type=int, default=72)
    parser.add_argument("--margin", type=int, default=400, help="distance from the bottom in pixels")
    parser.add_argument("--uppercase", action="store_true")
    parser.add_argument("--pop", action="store_true", help="pop-in animation on each caption")
    parser.add_argument("--no-vad", dest="vad", action="store_false")
    parser.add_argument("--output", type=Path)
    args = parser.parse_args()

    srt_path = args.video.with_suffix(".srt")
    if srt_path.exists():
        print(f"{srt_path} exists, skipping transcription")
    else:
        transcribe(args.video, srt_path, args.model, args.max_words, args.vad)

    output = args.output or args.video.with_name(f"{args.video.stem}_captioned.mp4")
    width, height = video_size(args.video)
    with tempfile.TemporaryDirectory() as tmp:
        ass_path = Path(tmp) / "captions.ass"
        write_ass(read_srt(srt_path), ass_path, width, height, args.font,
                  args.font_size, args.margin, args.uppercase, args.pop)
        burn(args.video, ass_path, output)
    print(f"Saved {output}")


if __name__ == "__main__":
    main()

transcribe loads the model with int8 weights, which keeps it fast and light on a CPU. word_timestamps=True gives every word its own start and end time, and the loop groups the words into captions of at most max_words, so each caption appears when its first word is spoken and disappears when its last word ends. The captions go into a standard .srt file next to the video.

The loop also starts a new caption whenever the speaker pauses for more than half a second. Counting words alone isn't enough. On a test clip with an 8-second pause, a words-only version put "So" from before the pause and "companies and" from after it in one caption, which then sat on screen through the whole silence.

vad_filter=True turns on voice activity detection, which is built into faster-whisper. If you sit in your raw recording thinking about what to say next, it finds that silence and cuts it out before the model sees the audio. Whisper can invent words for stretches with no speech in them, especially with background noise. On my test clip it didn't, with or without VAD, so I keep it on as insurance, and --no-vad turns it off.

write_ass converts the captions to ASS, the subtitle format libass draws, because SRT can't hold any styling. The Style: line is where the look lives. The canvas (PlayResX and PlayResY) is set to the video's own size, so the font size and margin are in real pixels. Uppercase is applied here and not in the .srt, so the .srt stays easy to read and edit. --pop adds the pop-in animation from my own version: each caption starts at 95% size, grows to 105% over 100 ms, then settles at 100%.

burn runs ffmpeg with the subtitles filter and re-encodes the video with x264. It runs ffmpeg from inside the temporary folder that holds the .ass file, because the filter reads its argument as a list of options, and a colon or a quote in the path breaks it. I tried both, and ffmpeg refused to start each time.

Run it

python auto_caption.py clip.mp4 --uppercase --pop

The first run downloads the model. ffmpeg prints its progress as it goes, and the script's own output is:

Language: en (1.00)
Wrote 51 captions to clip.srt
Saved clip_captioned.mp4

The whole run took about a minute for my 40-second clip with the medium model. The .srt looks like this:

12
00:00:07,300 --> 00:00:08,180
thinking yeah maybe

13
00:00:08,180 --> 00:00:08,580
we should

14
00:00:08,580 --> 00:00:09,360
hire this guy.

And clip_captioned.mp4 has each caption in a white box with black text, about a fifth of the way up from the bottom of the frame.

If Whisper gets a word wrong (it will now and then, mostly names and jargon), open the .srt, fix the text and run the same command again. The script sees that the .srt exists, skips the transcription and only does the burn, which took about 10 seconds for this clip.

Change how the captions look

Every option is a flag:

python auto_caption.py clip.mp4 --model small.en --max-words 5 --font "Montserrat" --font-size 90 --margin 300

--model and --max-words only apply when the script transcribes, so delete the .srt before you change either of them.

--max-words matters most for short-form video. Most Shorts show three to five words at a time, and I keep mine at three. --margin moves the captions up from the bottom edge, which you'll want so they don't end up under the platform's buttons and video title. The font has to be installed on your system. If it isn't, libass picks a similar one, and the only sign is a fontselect line in ffmpeg's output naming the file it used instead.

For anything the flags don't cover, edit the Style: line. Colours are written as &HAABBGGRR: alpha, then blue, green and red, with 00 alpha for fully opaque. For white text with a black outline instead of a box, set PrimaryColour to &H00FFFFFF, OutlineColour to &H00000000, BorderStyle to 1 and Outline to 4.

You can vibe code this yourself

The script I showed in the video was about 400 lines, and I'll be honest, I haven't read any of it. I explained what I needed to my coding agent and let it write whatever it came up with. For a little tool that only runs on my machine and never goes near production, I'm completely fine with not reviewing the code. It has a few extras I added for my own convenience, like presets for Shorts and long videos, rounded caption boxes, a check that the font is installed, and a preview mode that only captions the first 5 seconds. It's in my phantom-editor repo along with a lot of other video automations (that link is the last version that runs Whisper locally).

You don't need my code, though. Explain the three steps to whatever agent you code with. For the first step, tell it which Whisper model to use and how many words you want per caption. For the second, describe the look in detail: the font, the size, the position, the background, the shadows. The agent will write the ffmpeg commands for you, and that's the part you most want to hand off, because, honestly, none of us know ffmpeg commands. There's a reason tools like HandBrake exist.

It's a simple process, and you only build it once. After that, every Short you post gets captions for free.