What Are 3 Levels of Hardware Acceleration?

An overview of 3 different hardware acceleration processes

Questions

• What are the 3 levels of hardware acceleration?
• Which level makes best use of hardware acceleration?

Answers

• Level-1: Hardware encoder
• Level-2: Hardware decoder + hardware encoder
• Level-3: Hardware decoder + hardware process + hardware encoder

Based on the complete loop of video transcoding, hardware acceleration can be divided into three levels as mentioned above, each of which delivers different transcoding speed. Theoretically, only the Level-3 can go all out to accelerate video encoding decoding by leaps and bounds, for all of the three main steps ^[1] can make use of hardware. It used to be an-hour trip to decode and encode videos but can be reached by an 8-minute hop now after turning on Level-3 hardware acceleration.

Level 1 - Hardware Encoder

X264 encoder was the most preferred to encode videos when 1080p/720p HD reigns, which however can't deal with 4K UHD videos easily, despite exhausting an 8-core processor. Nowadays 4K pervades our lives, the encoding of which is much more resource-intensive and time-consuming than ever before, making hardware encoder, a dedicated graphics unit good at parallel computing, a necessity. Nearly 99% of DVD/video converter software claiming to be hardware accelerated in the market is just only able to pass raw data (or processed data) to hardware encoder for compressing, known as the Level-1 hardware encoding. Surely, this is one of the three key steps that can affect overall video transcoding speed, and can be speeded up by utilizing Intel QSV, Nvidia CUDA/NVENC, etc.

1. Software Encoder vs. Hardware Encoder

According to our test on converting a 4K (3840x2160) 30 fps video from H.264 to HEVC without resizing image, X265 encoder transcodes videos at 2.9-5.5, a snail's pace compared to NVENC's speed of 25-33. Clearly, hardware encoder wins.

2. What are the file formats supported by hardware encoder?

• Intel QSV: MPEG-2, H.264, VP8, HEVC (partly), JPEG (partly)
• Nvidia NVENC: H.264, H.265, Lossless

3. How does hardware encoder work?

• Firstly initialize and configure hardware (set the video format, profile or level you want)
• Software will pass the video stream to Graphics Driver via API/SDK.
• Then the QSV or NVENC driver may work alone or together with other drivers if necessary to get ready.
• Next, firmware and hardware are booted to start video encoding.
• At last, the encoded stream of bits will be passed back to software for packing in a container.
• Does hardware acceleration lower video quality? >>

Level 2 - Hardware Decoder + Hardware Encoder

Generally speaking, all videos that you can download online or extract from DVDs are encoded files (probably encrypted). Take MP4 H.264 as an example. H.264 is the codec that your video stream is encoded in while MP4 refers to a media container to integrate videos, audios, subtitles, etc.^[2] To playback, edit or re-encode a video file, you have to decode it into raw data firstly, which can be accelerated by hardware as well. Due to the outstanding computing power of hardware, it is easy to come to a conclusion that hardware decoder + hardware encoder is faster than software decoder + hardware encoder.

1. Hardware Acceleration: Level-1 vs. Level-2

This is the same test of converting a 4K (3840x2160) 30 fps video from H.264 to HEVC without resizing image. This time we use the combination of hardware decoder + hardware encoder. While the Level-1 runs at maximally 25 fps or 33 fps, the Level-2 hardware acceleration reachs up to 43 fps.

2. What are the file formats supported by hardware decoder?

• Intel QSV: MPEG-2, H.264, VP8, HEVC, JPEG
• Nvidia NVDEC: MPEG-2, VC1, VP8, VP9, H.264, H.265, Lossless

3. Do we have to decode video into raw data for processing?

Yes. There are so many types of video encoding methods, each of which adopts different data compression algorithm and formats. Besides, there is strong correlation in the data itself. So if we don't decode video to raw data, there will be a lot of algorithm inventing work to be done.

4. How does hardware decoder work?

• Firstly, software decouple video data streams from audios and other meta-data through a DEMux.
• If you are dealing with an encrypted DVD/video file, there will be a CSS removing process. Note that decoding is not the same thing as decrypting.
• The decrypted video stream will then be passed on to Graphics Driver via NVDEC or QSV API/SDK.
• Next, firmware and hardware are booted to decode video back into raw data.

Level 3 - Hardware Decoder + Hardware Process + Hardware Encoder

Level-3 hardware acceleration actually means that all of the three main parts of video transcoding can be assisted by hardware, not just decoding or encoding, but decoding + processing + encoding. Hardware processing happens after video decoding but before encoding. It plays a big role especially when you need to change image size, change aspect ratio, rotate videos, add subtitles, adjust bit rate, etc. If you have ever used video editors like Adobe Premier Pro, you would know how much fretful it is to suffer through processing 4K videos without turning on hardware acceleration, and sometimes you even can't import 4K HEVC to Premiere Pro. CPU usage increases steeply to 100%, software crashes and even your computer freezes.

Hardware Acceleration: Level-1 vs. Level-2 vs. Level-3

Here is the final test of speed comparison using 3 levels of hardware acceleration. We try to convert a 4K H.264 (3840x2160) 30 fps video to HD H.264 (1920x1080) on i7-4770K 3.5GHz. The Level-3 hardware acceleration amazes us with up to 133 fps.