Challenges of HTML5 media playback

It’s been almost a year since the BBC first made its HTML5 player available on the desktop via bbc.co.uk/html5. Lloyd Wallis is a Senior Software Engineer in Media Playout and gives an update on the work undertaken in the past twelve months.

Since the HTML5 player was introduced on desktop, we have:

• Expanded the range of content available to include iPlayer Radio and live streams.
• Automatically offered the HTML5 player where possible to browsers without Flash available.
• Worked with device and browser vendors to improve the experience and expand the range of supported devices, including support for Safari 10.

However, there are challenges faced in HTML5 media playback that are either presented in a different way than they are in a Flash space, or don’t exist at all in Flash.

Cross-browser functionality

"Anyone who slaps a 'this page is best viewed with Browser X' label on a Web page appears to be yearning for the bad old days, before the Web, when you had very little chance of reading a document written on another computer, another word processor, or another network."

Tim Berners-Lee 1996

Over the course of the trial, we’ve had more than one user send us feedback including this quote. Now that Internet Explorer is no longer in active development, HTML5 and related technologies are maturing, cross-browser support is definitely far better than it has been in the past. Far fewer CSS hacks or browser-prefixed draft implementations of experimental functionality are needed to create a basic web page.

Once you start stepping into behaviours that are still new, or not commonly used, cross-browser compatibility is something you come across less often.

A code snippet that looks almost like it’s designed to demonstrate how bad the situation is, but in reality most of this now is about support for legacy browsers. Bringing HTML5 to the desktop while still supporting iOS 4.0 and Android 2.2 brings a lot of complexity. Most of these differences don’t exist at all in a Flash world which means we don’t need to spend time debugging how the latest Windows 10 Mobile update behaves in fullscreen when you rotate it clockwise then anti-clockwise then back again and leave and re-enter fullscreen, etc… and more time on providing the best video experience possible.

The APIs we use to enable HTML5 playback are relatively new and are only just beginning to go through the process of becoming formal specifications. While browser support for Media Source Extensions is now relatively widespread, we often see functional regressions in new browser versions. In one case recently, Firefox introduced a bug that broke playback of our live streams; a fix for this is due to go out in a browser update on the 20th September.

Capability detection

Our mandatory requirements for supporting the HTML5 Player on desktop are:

• Media Source Extensions
• Decoder for the AVC3 sample description format

We also make use of the following, where available:

• The responseURL property on the XMLHttpRequest object (XHR)
• Page Visibility API
• Proper capability identification as to whether or not a feature is supported

Some users (and browser vendors!) have pointed to our explicit listing of supported browsers for the trial, rather than using capability detection in the browser. The bbc.co.uk/html5 landing page does in fact use capability detection where possible when choosing whether to display the opt in button, but then applies exceptions or parses the user agent string to advise what action can be taken to enable support.

My personal “favourite” exception can be summarised with the comment above it:

Context isolation

Flash also leaves us mostly isolated from the containing page. The Standard Media Player’s (SMP) “customers” are the dozens upon dozens of different BBC sites that embed it, each with their own design methodology and workflows, be it public facing services such as iPlayer, News, Sport, Weather, iWonder, Bitesize and Radio or our internal tools such as iBroadcast, Castaway or Vivo. To handle all the use cases, SMP has over 150 configurable settings, more than 50 public API methods, and an extensive plugin system with additional APIs.

The HTML5 player is embedded in an iframe to minimise styles or layouts in the player affecting the page and vice versa, but these still happen from time to time. At some point in the future, we may be able to look at using Shadow DOM to make things easier but that is certainly a long way off.

Missing features

There are also things that we can do with media playback in Flash that are still not possible in HTML5.

In Flash, as soon as we start loading a chunk of media, we’ll stream the response into the playback buffer. In a world of HTML5, DASH and XHRs, we need to completely load a response before we can append it to the playback buffer. While there is the draft Fetch API, it’s, which combined with the Streams API and the deprecated MSE appendStream, it is very much experimental. In the meantime, this means a startup delay for users.

Adaptive Bitrate streaming (ABR) is a mechanism for offering the optimal quality based on a range of factors. Every business has different requirements for the rules that apply, but the main rule has traditionally involved measuring the download time of a media chunk to decide whether a higher or lower bitrate would be more suitable for the next request. Research is increasingly moving towards replacing this with a rule based on buffer occupancy: given a target buffer level and the current buffer level, does the client need to load media faster or slower?

The BBC is currently sticking with the first approach, and other than a few edge cases involving calculating latency and detecting cached responses, this isn’t too hard. However, there are a range of other factors we need to consider. One of these is the number of video frames that have been dropped by the client. Dropped frames are a general indicator of the device not having sufficient computational power to display video to a user as well as we’d like, so in Flash we respond to these by switching down to a lower bit rate, which reduces the load on the decoder.

HTML5 does not have an implemented standard for measuring dropped frames. There is a proposal for extracting metrics that has been around for some time, and another one designed to extend the MSE spec, but they have differing levels of support. Even where a dropped frame count is available, we’ve seen cases where browsers have overestimated, and others where the browser hasn’t actually dropped frames, but has stopped presenting them to the user until a later point, when it recovers and plays them faster than real-time.

This means that if you’re using our HTML5 SMP and your device is struggling to play our highest quality stream, chances are we won’t know and as such we won’t be able to switch to a more appropriate stream.

Other general challenges & next steps

Throughout developing the HTML5 player, we’ve tried to help where we can to improve the situation. When we first started the project, Google Chrome was the only browser that was able to play the streams we were producing, but once Safari 10 is released on 20th September, every major browser will support AVC3, which enables us to offer the same DASH streams on desktop as we will with HbbTV, an industry-standard recommendation for how new Smart TVs and set-top boxes should play streaming media, and other similar standard devices. This means that we’ll be able to achieve a core goal of the HTML5 rollout: reducing the size of our working set.

The working set of media is the amount of content we have to physically store. Through previous work within Media Services, we now generate a single output file for every piece of content that goes through our VideoFactory and AudioFactory processes. However, when it comes to distributing this content, CDNs, as well as Radix, our CDN origin service, cache the segmented chunks of media. Currently that makes one copy in the HDS format for use on desktop, and the BBC Media Player app on Android, another copy in HLS for platforms such as TV and iOS. The HTML5 player, along with the new native player in the BBC iPlayer Android app, mean we now are additionally creating DASH versions. We’ve built our infrastructure to support this where needed, but the hope is that in the long term HDS usage will more or less vanish completely, whilst technologies such as fMP4 support in HLS and the CMAF project will reduce the need for a separate HLS version. Unfortunately that’s still many years away from being at a level of penetration such that we could turn off our transport stream HLS media.

As of last month, every browser except IE now has support for the responseURL property on XHRs, allowing CDNs which use redirects in order to “matchmake” a media session to the best possible streaming server to work as well as they can in HTML5.

With Flash, we can use HTTP media chunks on a page served over HTTPS. Whilst of course this is far from ideal and breaks the lock in the browser, this has been useful until HTTPS reached a point where it became cost effective for media delivery. The move to HTML5 will require us to use HTTPS delivered media on a page delivered via HTTPS, and we’re working actively with CDN partners to be ready for this soon.

Of course, we could write a whole post just on why HTTPS is a good idea, even when not protecting user’s privacy. For brevity, it’s sufficient to say we’ve seen cases where networks between us and you will intentionally alter our media streams, and others which unintentionally break them. Both types of problem will become near impossible as we move to HTTPS.

A lot of users will still get a better experience with Flash, and a lot of our archived media is not yet available in a format we can deliver via HTML5. But as Chrome, Safari and Firefox all move to disable Flash by default, prompting the user to enable on a per page basis, we’ll automatically start offering the HTML5 player where possible.

Users without Flash available or opted into the beta, HTML5 will now be offered for all media on bbc.co.uk, including News clips, where it is available in MPEG-DASH. Where we need to use Flash, we’ll make the experience as easy as we can.