Running FFmpeg by hand works until you have a hundred files. A pipeline is the difference between an afternoon of babysitting and a script you start and walk away from, and the parts that matter are not the FFmpeg commands but the handling around them.
The shape
Four things, in order.
Find the work. Which files still need processing, determined by looking rather than by a list somebody maintains.
Process one. To a temporary name, then rename on success.
Record what happened. So an interrupted run can resume and a failure is visible.
Do the next one. Sequentially, not all at once.
Skip what is already done
This is what makes a pipeline resumable, and it costs one line:
for f in source/*.mp4; do out="output/$(basename "$f")" [ -f "$out" ] && continue ffmpeg -i "$f" -c:v libx264 -crf 23 -c:a aac "$out" done
Run it again after an interruption and it continues from where it stopped rather than redoing everything.
Quote every variable. An unquoted path breaks the moment a filename contains a space, and video files routinely do.
Write to a temporary name
The problem with the version above: if FFmpeg is killed halfway, a partial file exists at the output name, and the next run skips it as done.
tmp="${out}.partial"
if ffmpeg -y -i "$f" -c:v libx264 -crf 23 -c:a aac "$tmp"; then
mv "$tmp" "$out"
else
rm -f "$tmp"
echo "FAILED: $f" >> failures.log
fi
Now the output name only ever exists for a completed file, and failures are recorded rather than silently skipped.
-y matters in any unattended script. Without it FFmpeg prompts before overwriting and the job waits forever for an answer nobody will give.
One at a time
Running several encodes in parallel is slower in total than running them in sequence, because they compete for the same cores, and it makes the machine unusable for anything else while it runs.
If the server also serves a website, lower the priority so the encode yields:
nice -n 19 ffmpeg -i "$f" ...
The encode still uses idle capacity, so it is barely slower, and it steps aside when a visitor arrives.
Check the input before processing it
A pipeline that assumes every file is valid produces confusing failures halfway through a long run.
if ! ffprobe -v error "$f" >/dev/null 2>&1; then echo "INVALID: $f" >> failures.log continue fi
Cheap, and it separates "this file is broken" from "the encode failed", which are different problems with different fixes.
Do not re-encode when you do not have to
The largest saving available in any pipeline. If the codecs already match what you need, a stream copy takes seconds instead of hours:
ffmpeg -i input.mkv -c copy output.mp4
Have the pipeline check with ffprobe and choose. On a library that is already mostly H.264, this turns a day into an hour. There is more on deciding in converting video formats.
Running it unattended
Over SSH, use a terminal multiplexer so a dropped connection does not kill the job:
screen -S encode ./pipeline.sh # Ctrl+A then D to detach
From cron, use absolute paths for the FFmpeg binary and every file, and redirect output to a log. Cron does not start in your directory and does not inherit your shell's PATH:
/usr/bin/ffmpeg -y -i /home/user/in.mp4 ... >> /home/user/encode.log 2>&1
Setting up cron jobs walks through the scheduling side.
Watch the disk
A pipeline that fills the disk halfway through produces failures that look like FFmpeg errors and are not, and on a shared account a full disk takes the website down with it.
Check before starting, and delete or move sources as outputs complete rather than at the end. A batch that needs both the full source library and the full output library present simultaneously needs twice the space, and that is frequently the constraint rather than time.
Read the log afterwards
A pipeline that ran overnight and produced a failures file is doing its job. One that produced no log at all tells you nothing about whether the outputs are correct.
Spot-check a few outputs by playing them rather than trusting exit codes. FFmpeg can exit successfully having produced something you did not intend. A video with no audio because a map was wrong, for instance, is a clean success as far as the exit code is concerned.
When the files come from visitors rather than from you, the input is untrusted and the timing is not yours. How to Handle Video Uploads from a Web Application goes into that case.
A pipeline should also verify what it produced, because an encode can succeed and output a technically valid video of nothing. For catching that, see How to Detect Scenes, Silence and Black Frames with FFmpeg.
How many jobs the queue should run at once is a measured decision in place of the core count. How to Run FFmpeg Jobs in Parallel Without Overloading has the detail.
Make each job idempotent
A pipeline that is interrupted will be run again, and the second run should reach the same result without duplicating work or damage.
That means three things in practice. Skipping inputs whose output already exists and is complete. Writing to a temporary name and moving into place, so a partial file is never mistaken for a finished one. And recording the outcome per input rather than only per run.
Idempotence is what lets you restart a failed batch by simply running it again, which is the difference between a pipeline you can operate and one that requires reasoning each time it stops.
Record failures where they can be read
A batch that processed 900 of 1000 files has told you something, and it is only useful if the 100 are identifiable.
if ffmpeg -nostdin -i "$in" ... "$tmp" 2>>"$log"; then mv "$tmp" "$out"; echo "ok $in" >> results.txt else echo "FAIL $in" >> results.txt; rm -f "$tmp" fi
A results file with one line per input turns a re-run into a targeted operation:
grep '^FAIL' results.txt | cut -d' ' -f2- > retry.txt
Note -nostdin: without it, FFmpeg consumes the loop's input stream and the batch processes only the first file, which is a genuinely confusing failure.
Separate the queue from the worker
A shell loop over a directory works until the list is long or the machine restarts.
The next step is a list of pending work that survives both: a database table, or a directory of job files moved between pending, working and done. The worker takes one item, marks it, processes it, and marks the result.
That arrangement recovers from a reboot without losing its place, allows more than one worker without duplicating jobs, and makes the backlog visible, which is the number you actually need when somebody asks why an upload has not appeared. Running an application as a systemd service goes into keeping the worker alive.
Know what the queue costs before it grows
Two figures make a pipeline predictable, and both come from measurement rather than estimation.
How long one representative file takes, and how much disk one job needs at its peak, which is larger than the output, because the temporary file and the source exist at once.
From those, the arithmetic is simple: how long a backlog will take to clear, and how many concurrent jobs the disk can support. Without them, a queue that was fine at ten files a day fails at a hundred, and the cause is not obvious. Running FFmpeg jobs in parallel deals with the concurrency half.