Survive CDN failures with Redundant Streams

When you're live streaming, you don't get second chances. Everything has to go off perfectly, every time.

Sometimes though, the internet just doesn't want that to be the case. At Mux we work really hard to make sure that our delivery platform is as reliable as possible, but there exist fundamental risks - BGP leaks, large-scale CDN outages, or even a backhoe through a fiber connection, which present significant problems to even the largest streaming video platforms.

Recently we spent time investigating what the fastest, highest value change we could make to Mux Video to improve our resilience to larger, internet wide service outages.

While today Mux already uses multiple CDNs to deliver your content, selected based on the quality of experience, as of right now, we make that decision when the end viewer starts watching the stream. This is acceptable for short-form VoD content, but starts to run into limitations for long-form content, or live streams.

The solution to this problem is usually referred to "mid stream CDN switching". There's a few approaches that we could have taken to enable mid-stream CDN switching for our customers, but as our first step to get something ready as soon as possible for your high value events, we've chosen to leverage a client side technology known as "HLS Redundant Streams".

HLS Redundant Streams

HLS redundant streams are an often forgotten about part of HLS's functionality. This feature allows you to define multiple HLS renditions in your manifest that have identical characteristics, other than the URL which they point to, allowing the player to treat these as a redundant set, for failover purposes. In the HLS manifest, it looks something like this:

#EXTM3U

#EXT-X-STREAM-INF: BANDWIDTH=2000000, RESOLUTION=1280x720
http://primary-cdn.cdn.mux.com/2mbps/index.m3u8

#EXT-X-STREAM-INF: BANDWIDTH=2000000, RESOLUTION=1280x720
http://secondary-cdn.cdn.mux.com/2mbps/index.m3u8
 

#EXT-X-STREAM-INF: BANDWIDTH=5000000, RESOLUTION=1920x1080
http://primary-cdn.cdn.mux.com/5mbps/index.m3u8

#EXT-X-STREAM-INF: BANDWIDTH=5000000, RESOLUTION=1920x1080
http://secondary-cdn.cdn.mux.com/5mbps/index.m3u8

The redundant renditions in this manifest are identified by the player by checking for an identical EXT-X-STREAM-INF line.

Since failover between the HLS redundant streams is implemented on the client side, the behavior and support does vary between players.

In general, video players look for 4XX or 5XX status codes on either segment or manifest downloads, in order to trigger a failover. Some players will fail over immediately to the same rendition on the alternate CDN, while other players will try other renditions on the primary CDN first, before trying the redundant streams.

Let's take a look at a demo of how one of the most popular video players, HLS.js deals with a major CDN outage when redundant streams are enabled.

Looks great! How do I turn it on?

First, check your player is compatible

We've spent a lot of time testing the compatibility of the most popular HLS players with a variety of failure scenarios, focusing on failures in both manifest, and media delivery. You can check the video players that you're using in the compatibility grid below:

You'll notice there's some missing common players on this list; We're actively working with both open and closed source player vendors to make sure that this functionality is available where you need it.

Add the required playback modifier to your playback URL

Enabling redundant streams is really easy, all you have to do is set a query parameter on your playback URL:

https://stream.mux.com/{PLAYBACK_ID}.m3u8?redundant_streams=true

If you're using signed URLs, you instead need to add a claim inside your JWT body, setting the claim redundant_streams to true - your JWT payload should look something like this:

{
  "exp": 1599583520,
  "aud": "v",
  "sub": "{PLAYBACK_ID}",
  "redundant_streams" : true
}

Check the playback documentation for more details.

Deploy to some users, and monitor for stability

When you deploy redundant streams, you'll want to make sure that your player is behaving as expected with the new manifests we're delivering. We strongly recommend deploying to a subset of users, and using Mux Data's experiment field to run some A/B testing on your viewers. In normal conditions you wouldn't expect to see any change in the viewer experience scores, only in exceptional circumstances, such as a CDN failure, or routing issues to particular CDNs, so don't be surprised if your viewer scores don't immediately improve.

In Summary

Leveraging redundant streams in Mux Video gives you access to a more resilient multi-CDN delivery strategy, allowing you more protection against CDN, or internet infrastructure outages. You can enable redundant streams today for free, on both live and on-demand streams.

This isn't the end for our reliability efforts at Mux, we've got a lot more projects in the pipeline to make sure that your live events are bulletproof - keep your eye on the blog for more updates in the future!