Demystifying Video Codecs

Today we’re delving into the wonderfully geeky realm of codecs, the fundamentals of usage in post, and best practices. Let’s roll up our sleeves and get ready for some acronyms.

1. What is a codec?

A codec is an abbreviation for “compressor-decompressor”. In the most basic descriptions, it’s the specific algorithm used to encode and decode a video stream. It’s for both creating and playing back a media file. But when working with digital media, the codec is only half of the equation. A codec is delivered in a wrapper – a digital coating that encompasses the codec filling.

That digital coating tells your playback device what application should be used to read the codec inside because that application most likely understands the codec inside. Let’s look at it in another way.

Here we have a can. Of something. Probably delicious. This can is our wrapper, and it contains tonight’s dinner. However, It has no markings – we don’t know what’s inside of it….the essence. We do know, however, that a can needs a can opener to open up the can to see what’s inside. The can opener is the media player. Once we open the can, we can examine the contents -or the codec, and choose the best decoder tool with which to handle what’s in the can. In this case, we have soup. Unfortunately, I don’t have the right tool with which to partake in this feast. Thus, I need to find the right decoder for the codec -in this case, a spoon – so I can consume the soupy goodness.

As you can see, it’s imperative to know both the wrapper and the essence inside to properly handle soup…err, media.

In Post, the terms codec and wrapper are often misused or used interchangeably. How many times has someone asked you for a QUICKTIME? Or an AVI? Again, that’s akin to asking for a can….not what the can contains. As you discuss media decoding at your next party, start to use both terms, like “h.264 Quicktime” or “animation in an AVI wrapper”.

2. What do I need to be aware of when choosing a codec?

Codecs and Wrappers are generally found in 3 places: acquiring, editing, and delivering.

When we are acquiring, you normally don’t initially choose a codec and wrapper. They choose you. Why do you ask? Often, you choose a camera because of the look it has – or because that’s the camera your buddy will lend you. These cameras have encoders built-in, which traditionally only record to one or two different codecs and wrappers. Thus, you need to handle whatever format the chosen camera gives you.

Most consumer or prosumer cameras record in formats that initially look good but perform horribly in post. Like h.264. Small file size, pretty decent image quality, but they break down in post rapidly.
Many NLEs can’t decode several streams of h.264 efficiently, and rendering can be a pain. This is where you have more flexibility to choose your codecs. You can transcode – or flip – these camera original files into more edit-friendly codecs. Like Quicktime wrapper ProRes. Or MXF wrapped DNxHD.

Editing friendly codecs take advantage of larger file sizes by compressing the data less. This means the computer has to work less on interpreting each frame of data. This is what is known as inter vs intraframe encoding. Interframe encoding means the computer needs to look at several neighboring frames to form a complete image of each individual frame. Intraframe means the data for each frame is self-contained within the encoded frame. The latter is obviously easier on the computer – but again, takes up more space. This often times means you need a greater amount of storage and a faster way of getting the data to and from your computer.

Once the project is done, and it needs to be delivered – again, the codec and wrapper choose you. Depending on where you are delivering the file to, you need to hit the deliverable’s specs. If it’s Broadcast, you have Redbook standards to conform to, lest your media fails QC – and you have to regenerate and resubmit. If delivering to an online provider, such as Netflix, Hulu, and to some extent YouTube, they each have specs you must hit, or risk poor image quality, or the file being rejected.

3. What codec should I edit with?

Many NLEs have their favorites. Some are robust enough to handle several different codecs and wrappers, while others are quite a bit finicky. Let’s highlight a few of the best practices in several of the popular NLEs. Let’s start with Avid Media Composer.

Avid’s codec du jour is DNxHD. DNxHD is a lossy, HD frame size codec, developed by Avid. When Media Composer is directly handling DNxHD, Avid wraps the DNxHD in an MXF wrapper.

Avid Media Composer

I’ve often been asked – again, at parties – why an MXF wrapper? The simple answer is that the MXF wrapper – the container – has much more flexibility in the spec. An MXF can contain several different essences, ancillary metadata, etc. So, while it isn’t easily played outside of Avid, the assumption is that if you’re already using the media with Avid – why not use Avid to view it? Avid also excels with SD footage. Old school editors will remember the OMF wrapped media. Avid still handles SD material, and uses the same Avid Meredien codec as it always has – only now it has unified the media to also be an in MXF wrapper. XDCAM is also handled by Avid very well, namely because XDCAM is an older spec, and Media Composer has had many years to perfect its usage and playback.

Final Cut Pro 7 and Final Cut Pro X

Next, we look at Final Cut Pro 7 and Final Cut Pro X.

Final Cut Pro 7 always played with ProRes the best, namely because Apple created the codec. Older codecs like DV25, DV50, and DVCPro HD also worked well but tended to break down more easily in post when doing color and effects. FCP X, given the greatly enhanced playback foundation, can play most media formats much easier than most other NLEs out there. However, Apple will still render in ProRes if you let it, and you’ll get more streams and more real-time effects when using ProRes over, say, H.264.

Adobe Premiere Pro

Lastly, we have Adobe Premiere. Adobe is a bit of a swiss army knife – it has no allegiance to Avid’s DNxHD or Apple’s ProRes. However, Adobe has realized the same thing I talked about earlier: Intraframe codecs perform better that interframe ones. Adobe also is the most open in terms of playing in the same sandbox as the other NLEs. The result is that Premiere works exceptionally well with Avid DNxHD and Apple ProRes. Just be aware that on a PC, you can work with ProRes, but need to render in another codec, like DNxHD.

Thanks, Apple!

There are some basic rules to abide by.

There is almost no reason to ever, ever work with uncompressed HD footage. Can you? Sure you can work with the uncompressed footage. Will you need massive amounts of storage space and very fast storage? Yes. And the visual quality will most likely be imperceivable. Don’t use it unless you have a reason to – like very high-end effects work. Again, mid to high bitrate ProRes and DNxHD will work in almost all cases.

Most camera original codecs suck in post. H.264. Sucks. AVCHD Sucks. All of these look good initially but break down in post, both visually and by system performance. The only real modern exception has been XDCAM. It’s a common acquisition codec and is handled pretty well by modern computers. It’s not as robust as DNxHD or ProRes when it comes to flexibility for effects and color in post, but the file size tradeoff and the ability to go from camera to edit can make up for it in quick turnaround scenarios.

So, when it comes to camera codecs, transcode FROM them, or record directly into an edit-friendly codec – as we’ll talk about right now…..

4.How do I convert from one codec to another?

Earlier, we talked about acquisition codecs, and this is largely being based on the camera you choose. There is a way to hotwire these cameras into recording to formats you want. Often times, cameras have HDMI or HD-SDI outputs, often called “baseband”. Many 3rd party recorders, such as the Ki Pro family from AJA or the field recorders from Atomos, or any one of the Cinedeck recorders can take these uncompressed baseband signals, and record them into edit friendly codecs -often Quicktime wrapped ProRes, or Quicktime or MXF wrapped DNxHD. Not only do these solutions bypass the need to transcode the files into edit friendly codecs later, but as mentioned, the baseband signal is often pre-compression, resulting in a better source image as it’s being recorded.

These types of baseband encoders run anywhere from several hundred dollars to several tens of thousands of dollars. If a baseband recorder is not in your budget, then you’ll need to take the hit in image quality, and use software to transcode the compressed camera footage to make the editing experience a bit more pleasurable. There are dozens of solutions out there – from Apple’s Compressor to Adobe Media Encoder, Telestream’s Episode family – up to enterprise solutions like Telestream’s Vantage, Rhozet Carbon Coder, Root6’s Content Agent – all solutions that not only flip the files for you, but add additional features, such as greater control of the essence of the file, QC, logic-based encoding decisions, and distributed encoding among multiple machines.

Another shortcut is to change the wrapper, but not the codec inside – in the hopes that another player will understand it. MPEG Streamclip is a great tool, as are MP4Tools and AVITools.

A great tool I use daily is “Media Info” – available for Mac or PC. This app digs into the wrapper and codec and gives you all of the dirt on the file. This can be a lifesaver when troubleshooting decoder and codec issues. VLC is a free player that has great compatibility and can usually play most media files you’re likely to run into.

5. How do I get the best look for ______?

Often I’m asked: “When outputting, what is the best possible codec for BLANK”? And there isn’t an easy answer. First and foremost, wherever you are delivering to, follow their format guidelines. Chances are, you are not the only one who has ever had this query. Seriously. Don’t try and out think the distribution hub. You’re at their mercy. Second, I’ve always found the best success has been to base your encoder settings off of existing presets in your software transcoders. No, this isn’t cheating. Often times these profiles produce excellent results or at least get you in the ballpark. Also, not all software transcoders are built equally. You may need to try several solutions to find which one handles your project the best. Something else to keep in mind is that there is also the law of diminishing returns. Simply jacking the data rate of a file up doesn’t mean it will look better. Each codec has their own thresholds

Years ago, there was a job title known as a “Compressionist”.

They melded the science and math of encoding with the art of good visuals. This job has largely been eradicated due to what I perceive to be the public’s tolerance of substandard image quality and the proliferation of template-based encoding applications… However, one size does not fit all, and different stories and shooting styles call for different settings. Experiment and learn, and create the best quality image within the parameters of the deliverables you need to generate.

The realm of codecs is as wide as it is confusing. Have more codec concerns other than just these 5 questions? Ask me in the Comments section. Also, please subscribe and share this tech goodness with the rest of your techy friends….especially at parties.

I plan to be back in 2 weeks with another 5 things. Until the next episode: learn more, do more – thanks for watching.