Every comparison of these codecs leads with compression efficiency, which is the least useful way to choose between them on a server. The three things that actually decide it are who can play the result, what it costs you to produce, and whether you serve the file often enough for the saving to matter.
H.264, and why it is still the answer
ffmpeg -i input.mp4 -c:v libx264 -crf 23 -preset medium -c:a aac -b:a 128k out.mp4
It plays on every browser, every phone, every smart television and every device old enough to be in a drawer. It encodes quickly. Hardware decodes it, so playback costs the viewer almost nothing in battery.
Its files are the largest of the four, and that is the entire disadvantage.
If you are unsure, this is the answer. A file everyone can play is worth more than a smaller one that fails silently for a share of your audience.
H.265, and what the saving costs
ffmpeg -i input.mp4 -c:v libx265 -crf 28 -preset medium -tag:v hvc1 -c:a aac -b:a 128k out.mp4
Roughly a quarter to a third smaller at comparable quality, and several times the encoding effort for it.
Two practical notes. The CRF scale is not shared between encoders, CRF 28 in H.265 is approximately CRF 23 in H.264, and copying the same number across produces a worse-looking file and a confused conclusion. And -tag:v hvc1 matters if Apple devices are in scope; without it the file plays elsewhere and refuses on Safari.
Browser support is real but not universal, which makes H.265 a better fit for downloads and applications than for a video tag with no fallback.
VP9
ffmpeg -i input.mp4 -c:v libvpx-vp9 -crf 32 -b:v 0 -c:a libopus -b:a 96k out.webm
Comparable efficiency to H.265, royalty-free, well supported in browsers and weaker on devices and televisions. Encoding is slow.
The -b:v 0 is not optional: without it, the CRF value is treated as a ceiling on a bitrate-targeted encode and the output is worse than you asked for. It is the single most common VP9 mistake.
AV1
ffmpeg -i input.mp4 -c:v libsvtav1 -crf 35 -preset 6 -c:a libopus -b:a 96k out.mkv
The smallest files of the four, and by a distance the most expensive to produce. Use libsvtav1 instead of the reference encoder unless you have time to spare, and expect the preset number to dominate your encoding time more than any other setting.
AV1 makes sense for a library served thousands of times, where a one-time encoding cost buys a permanent bandwidth reduction. It makes no sense for a user upload watched twice.
The decision, stated plainly
If the same file is served many times, bandwidth dominates. Spend the processor time once and encode with a newer codec. The saving repeats on every view.
If files are encoded constantly and watched rarely, user uploads, in particular, processor time dominates. H.264 keeps the queue moving. A newer codec here means uploads sitting unprocessed while the server works through a backlog.
Handling video uploads from a web application walks through the queue side, and performance optimisation goes into making whichever you choose faster.
Serving more than one
If you want the newer codec's saving without excluding anyone, encode both and let the browser choose:
<video controls> <source src="video.webm" type="video/webm"> <source src="video.mp4" type="video/mp4"> </video>
Browsers pick the first source they can play. Put the efficient one first and H.264 last, and nobody is left without a video.
Audio deserves the same thought, Opus is substantially better than AAC at low bitrates, and pairs naturally with WebM. Serving video efficiently from your own server goes over the delivery side once the encoding is settled.
Compare the codecs on your own material
Published comparisons use standard test clips, which resemble nobody's actual content. Two commands settle it for yours.
for c in libx264 libx265; do /usr/bin/time -f "$c %e s" ffmpeg -nostdin -v error -i sample.mp4 -c:v $c -crf 23 -preset medium -an -y "/tmp/$c.mp4" ls -lh "/tmp/$c.mp4" done
Use a representative clip; a minute of the kind of video you actually publish, not a demo file. Screen recordings, talking heads and fast action compress very differently, and the ranking between codecs changes with them.
Two numbers come out: how long each took, and how large the result is. Combined with how often the file will be served, that is the whole decision, and it takes ten minutes in place of an afternoon of reading.
Encoder settings matter more than the codec
A poorly configured newer codec produces worse results than a well-configured older one, which is how comparisons end up contradicting each other.
Three settings carry most of it. The preset decides how hard the encoder works. The CRF value decides the quality target, and the scale differs between encoders, so the number does not transfer. And for VP9, the -b:v 0 that makes CRF behave as intended.
Compare like with like: the same visual quality, not the same number. Encode at several CRF values, pick the one where each codec looks equivalent, and compare the resulting sizes at that point. There is more on choosing the values in CRF, two-pass and bitrate.
Check what the target devices actually support
Support is not binary; a device may decode a codec in software while lacking hardware support, which plays and drains the battery.
That distinction matters for mobile visitors: hardware-decoded H.264 costs a phone very little, while software-decoded AV1 on an older device is noticeably worse for the viewer even though the video plays.
Where the audience is largely mobile, that argues for H.264 more strongly than any file-size comparison does. Where it is desktop and recent, the newer codecs are a straightforward gain, and serving both removes the question entirely.
Audio deserves its own decision
Video choice dominates the discussion and audio is frequently left at whatever the command produced.
For speech, 96 kbps AAC is generally indistinguishable from 192 and a third of the size: on a talk-heavy library that is a real saving. For music, 192 is the sensible floor.
Opus is meaningfully better than AAC at low bitrates and belongs in WebM rather than MP4, so the audio choice follows the container choice instead of being independent of it.
ffprobe -v error -select_streams a:0 -show_entries stream=codec_name,bit_rate,channels -of csv=p=0 input.mp4
Check what the source actually has before deciding, re-encoding audio that was already suitable loses quality for nothing. Remuxing and stream copy goes into copying it instead.