BBC's Quested talks HD encoder testing
Thursday, December 10 2009, 12:38 GMT
By Andrew Laughlin,
BBC principal technologist Andy Quested has detailed the methods used to test the new encoders at the heart of controversy around BBC HD's picture quality.
After new encoders were introduced by the BBC on August 5, the bitrate on BBC HD dropped from 16Mbs to 9.7Mbs, a reduction of nearly 40%. Viewers subsequently reported various problems with picture quality and sound on the channel.
Quested said that the BBC's picture quality assessment involves using Peak Signal to Noise Ratios (PSNR) in conjunction with more rudimentary "eyeballs" testing.
PSNR is a way of testing the relationship between a signal's strength against the power of corrupting noise levels. The process involves complicated mathematic equations to judge the resulting quality of output on a broadcast channel.
Alongside the process, the BBC also runs "expert" viewing tests involving active comparisons between three 42" TV screens - one carrying a picture from the new encoder, another with the old encoder and a final one showing test material copied to a playout server. Tests are run on a range of broadcast quality receivers and domestic set top boxes to ensure a broad stroke of results.
In the tests, the new encoder was found to have a median PSNR figure 0.5dB greater than the old encoder, which delivered a "just noticeably improvement in perceived picture quality for the majority of the test sequences".
"However, the very easiest, least critical material, where coding artefacts are usually not visible, coded with a better PSNR on the old encoder," said Quested. "We are looking into this at the moment but one explanation could be the new encoder handles image noise differently to the old encoder."
Quested warned that interpreting PSNR curves is "not straightforward" as a 0.3dB difference is "just visible" to an expert viewer but non-experts will only typically notice a difference of 0.5dB or more.
When the new encoder was introduced on BBC HD, there were noticeable problems with the mixes - transitions between two different images. Quested said that the problem arose during initial testing but only affected certain modes and was not deemed to be particularly severe.
However, during a broadcast of West Bromwich Albion's game against Newcastle United, the cameras had to be "racked" over several stops - opening and closing the iris - as bright sunshine turned to deep shadow.
"Coding errors caused by the mix tend to be hidden by the changing images. However racking a camera is actually a mix between two different brightness levels of the same image so there's nowhere for the errors to hide and they become very visible. I apologised and explained we were applying a temporary fix," Quested explained.
"Although the temporary fix is still in place we have now seen an update that improves mixes, fades and lighting changes and are just waiting for it to be incorporated into a software upgrade."
Quested said that comparing PSNR curves showed that the "new encoder is doing better than the old except where the source material has a significant amount of noise".
He added: "To help this we are testing the encoder's noise reduction options to see if adding a small amount improves the look of noisy images. I will update the blog as soon as we have some results."
In terms of expert viewing, Quested said that it is a "tricky business" and one that requires long, dedicated periods of time spent in darkened rooms. Effective overall testing is also about using the subjective viewing results in conjunction with the more scientific PSNR curves.
"The new encoder produces images that correlate quite closely to the PSNR results. Programmes with low or no noise are noticeably better than they were on the old encoder," he said.
"However, where the original images have noise we can see it on the new encoder's output but not on the old, suggesting that the new encoder is attempting to pass on more of the original image and confirming that a bit of noise reduction should be tested. Dark pictures are inherently noisy, either because there is gain in the camera or the signal has been stretched too far in colour grading.
"We actually have a very noisy sequence that has too much camera gain and was stretched too far in post-production. We used it during the tests to push the system, and even turning the bitrate up to just over 16Mbs made no difference to the image. We are trying a few new and different parameters that seem to improve noise handling and reduce the effect on screen. Again I will keep you posted."
In his final post of the series, Quested will cover the specific techniques programme makers can use to improve the perception of picture quality on BBC HD.