H.264 encoding - CPU vs GPU: Nvidia CUDA, AMD Stream, Intel MediaSDK and x264 (page 5: Measuring quality: PSRN, SSIM and the pitfalls)

Measuring quality: PSRN, SSIM and their drawbacks

Here at Hardware.fr / Behardware.com, we try and design our tests to be as objective as possible. It's fairly easy to determine performance when the result can be easily measured (processing time for a given task for example). When it comes to video, the question of objective judgement of video quality is unfortunately very… subjective. The true objective criteria is visual quality of the video as perceived by the human eye. This is something that unfortunately can’t be measured or quantified other than by human tests.

Over the years, several tools have been developed to try and compare the quality of one video to another. The basic concept remains the same: frames from the compressed video are compared one by one to the source and this gives us a series of values for each of the frames that make up the video.

The standard measure (or metric) used to compare two frames is PSNR which attempts to determine the level of corrupting noise in a compressed image in comparison to the source. Used above all to judge static frame compression formats, PSNR is considered to be a very statistical meaure depending largely on the compression format chosen or the particularities of the encoder. The other metric used is called SSIM and attempts to determine the structural similarity between frames, with the aim of being a bit more realistic than PSNR.

While in practice SSIM is a better indicator of visual quality, the problem remains relatively complex as human perception is difficult for an algorithm to measure. Our eyes are for example instinctively attracted to faces. This means that the human eye will prefer an image on which the face is sharp but which may be otherwise inaccurate (and which has a low PSNR), to an image with a more even quality across the areas but in which the face isn't as well defined (but ironically a higher PSNR!).

Added to this problematic is the fact that as with any metric for which the algorithm is known, it's very easy to optimise an encoder for one algorithm or another to the detriment of the overall video quality. The x264 encoder illustrates this issue quite well: as well as having options that allow you to optimise the encoder for a film or cartoon, you can also optimise it to get the highest possible PSNR and SSIM values! Here’s a little example of what this can give in a scene from the film Inception. We have calculated the average PSNR and SSIM values with four different x264 optimisations (none, Film, PSNR, SSIM). Note that the PSNR values are expressed in dBs (the higher the value the better the sigal to noise ratio), while the SSIM is a value between 0 and 1 which indicates correlation to the source image, with 1 showing perfect correlation.

Attempting to optimise an algorithm doesn’t always work depending on the scene. The scene chosen here is full of explosions and PSNR optimisation gives the lowest scores. The SSIM metric seems to be best, obtaining the two best scores for the SSIM and PSNR. Is this borne out when we look at the frames? Let’s see what we get on a static frame taken from each video:

[ No Tune ] [ Film ] [ PSNR ] [ SSIM ]
Move the mouse over/click on the links to view the corresponding frame. Inception, Warner Bros

Let’s start with the most obvious. On the PSNR frame, not much is right. Entire parts of the face are blurry and the shading on the right is blocked. Is the SSIM version better than the Film version? No. The eyelashes are sharper on the Film version and there's more detail in the face. So why does it have a lower score when the SSIM version is slightly more blurred? This is because of a parameter which makes PSNR and SSIM comparisons even more complex, the psycho visual optimisations which try to retain a maximum of details in interesting areas. This optimisation can be made out in a static frame but becomes even clearer in a video series: the video quite simply retains more details and what look like artefacts on a static frame (for the PSNR and SSIM metrics) actually give improved quality though with lower metric scores.

For these reasons, while we will give the SSIM/PSNR scores for the various encoders, these scores cannot be considered to be of absolute value!