Insights The LCEVC Coding Standard and 8K Television

[bhobba]

Table of Contents
ToggleIntroductionLCEVC and baseline EVCMy Opinions
Introduction
First, for background, here is an overview of the most common video compression standard in use today, H264: https://www.maketecheasier.com/how-video-compression-works/
H265 has improved on this, but for various reasons, it was a bit of a flop. It is in use and will continue to grow, but royalty issues are a problem. To try and address these, 3 new video standards have been approved by MPEG: https://ottverse.com/vvc-evc-lcevc-mpeg-video-codecs/
A consortium of companies has also endorsed a royalty-free codec called AV1...

Continue reading...

 

[Algr]

Some of the information above reads rather strangely. The stuff about downscaling the video and sending the high frequencies separately is a tautology: We compressed the file by separating the high frequencies and compressing them. But how?

It is a real tragedy that so many companies render all their own work useless by burying their codecs in burdensome licencing snafus.

This is the first I've heard about HEVC being a failure. Is it just Apple using it? Typical.

 

[bhobba]

Some of the information above reads rather strangely. The stuff about downscaling the video and sending the high frequencies separately is a tautology: We compressed the file by separating the high frequencies and compressing them. But how?

Here is a simple explanation. Indeed downscaling means getting rid of the high-frequency information. One way of doing it is to average 4 pixels and send that. That downscales it by 2. Then to upscale it back, duplicate each pixel 4 times. But you have lost the 'high frequency' or 'detail' information. To restore that, you transmit the difference between the average in the downscaled 4 pixels and the original image. It is a better way of doing it since normal compressing methods are not efficient at compressing the high frequency or 'detail' information. An interesting thing about videos is the 'detail' information (i.e. high-frequency information) is sparse - i.e. is mostly zeros. This allows very efficient run-length encoding to be used:
https://en.wikipedia.org/wiki/Run-length_encoding

The above would work, but various tricks can be used to make it even better. One trick is to use bicubic (or others) upscaling and downscaling:
https://en.wikipedia.org/wiki/Image_scaling

The difference information would be smaller, so it is more efficient.

A very sophisticated method would be using AI (in this context, a Convolutional Neural Network). That would be something like Tad-Tau I mentioned before:
https://cv.snu.ac.kr/research/taid/

Still, other tricks can be used. For example, guessing the difference from pixels near it and the previous frame.

All these can be used in LCEVC - the detail can be found in the patent I linked to. As I posted, the bottom line is that using the HEVC codec and LCEVC; one can transmit a virtually identical 8k picture (93 VMAF) at 20mbs. The modern measure of how good a video is perceptually is called VMAF. Anything above 93 is considered imperceptibly different from the original. It is the early stages of LCEVC use, and things will get better. But as of now, 8k is possible at speeds a lot of people have.

HEVC is a 'failure' because of royalty problems, not technical issues:
https://www.streamingmedia.com/Articles/ReadArticle.aspx?ArticleID=122983

Thanks
Bill

 

[Algr]

One way of doing it is to average 4 pixels and send that. That downscales it by 2. Then to upscale it back, duplicate each pixel 4 times. But you have lost the 'high frequency' or 'detail' information. To restore that, you transmit the difference between the average in the downscaled 4 pixels and the original image. It is a better way of doing it since normal compressing methods are not efficient at compressing the high frequency or 'detail' information. An interesting thing about videos is the 'detail' information (i.e. high-frequency information) is sparse - i.e. is mostly zeros. This allows very efficient run-length encoding to be used:
https://en.wikipedia.org/wiki/Run-length_encoding

Yes, but that is what people were doing in the 1990s. The text is written as if this was a new idea for H266.

 

[bhobba]

Yes, but that is what people were doing in the 1990s. The text is written as if this was a new idea for H266.

Many compression ideas have been around for a long time - many over the 20 years royalties apply. The simple compression method I gave is just one of them. That is the idea behind the EVC baseline I mentioned. It uses nothing but ideas whose patent/royalty has expired. The interesting thing about EVC (baseline) is that it encodes very fast and has compression efficiency nearly as good as HEVC, loaded with royalty problems. In fact, perceptually, it could be as good as HEVC:

HEVC bought about the paradigm shift to considering patent/royalty issues. It used a lot of then-new ideas in the standard, which was ratified in 2013. At first, everything was fine, but the patent holders of those ideas were 'sneaky'. What they did is wait until HEVC was widely adapted. Then wham - hit users with royalties for their patent. I have heard in discussions (but do not hold me to it) big companies like Samsung have teams of lawyers just working on that alone. Nobody (except maybe the patent holders) wants a repeat of that debacle. That is not to say HEVC will not be in use for some time. At the moment, X264 is the dominant codec, but it is expected over the next few years, HEVC will be the dominant codec - then it too will fade, being replaced by newer codecs. But the lesson has been learned - royalties must be simplified, and a lot of effort is going into doing just that. VVC is the natural successor to HEVC and builds on it. In doing so, it is loaded with the same royalty issues as HEVC. The codec developers are aware of it and will do all they can to ensure it is manageable. But it may not succeed. EVC handles it differently by having the baseline and the main profile. The main profile uses techniques that can be switched on or off. If the same royalty tactics that were used with HEVC is tried, they switch it off.

Then we have AV1:
https://research.mozilla.org/av1-media-codecs/

Everything in it is royalty-free. It compresses better than the EVC baseline. Its problem is compression time - it is horrid. Work is being done to reduce it, with some success. Many think it will become the dominant codec.

Now let's return to LCEVC. The straightforward compression method I detailed is likely royalty-free. But the tricks developed by V-Nova to make it better are not. V-Nova is very aware of the royalty issue and has released the royalties they will charge, which the industry, in general, seems to be happy with:
https://www.streamingmedia.com/Articles/ReadArticle.aspx?ArticleID=147003

Although I mentioned baseline EVC in my insights article, many think it is a natural fit for AV1. But when operating at 1/4 resolution, the differences in codec compression efficiency not as big an issue - it exists, of course, but the information in the enhancement layer compensates to some extent - it is the same regardless of codec. The big benefit of using LCEVC with the newer codecs is that encoding time is drastically reduced as it operates at lower resolutions. So by using AV1 at 1/4 resolution encoding, AV1 encoding time is manageable. Also, AV1 is a complex codec, where a lot can go wrong and frame dropouts occur. Experiments using it with and without LCEVC has been interesting:
https://www.lcevc.org/wp-content/uploads/AV1-with-LCEVC-pub-1.pdf

With a royalty structure the industry is happy with, the increase in compression efficiency, reduction in encoding time and dropout reduction in complex codecs, LCEVC looks to have a bright future.

Thanks
Bill

 

[Algr]

At first, everything was fine, but the patent holders of those ideas were 'sneaky'. What they did is wait until HEVC was widely adapted. Then wham - hit users with royalties for their patent.

WTF! Isn't that exactly what Compuserve did with the GIF format? The whole point of patent pools is to stop corporations from doing that by having the cumulative rights worked out in advance. Lawyers make communism look good every day.

 

[bhobba]

WTF! Isn't that exactly what Compuserve did with the GIF format? The whole point of patent pools is to stop corporations from doing that by having the cumulative rights worked out in advance. Lawyers make communism look good every day.

I am sure it is a common tactic. The point is new codecs want to avoid it - at least try anyway.

Thanks
Bill

 

[Algr]

This would/should logically result in no one ever trusting an MPEG standard again. I've been frustrated that the internet can't seem to standardize on any standard newer than JPEG and MP3.

 

[bhobba]

There has been an interesting new idea in AI upscaling that would dovetail nicely with LCEVC:
https://arxiv.org/pdf/2012.00650.pdf

I think it could eventually become part of the standard, leading to even greater efficiency gains and bitrate reductions. It is early days yet, and like will be even better with further development, e.g. the idea of adaptive base revolution would be easy to implement in LCEVC.

Thanks
Bill