Mux Logo
Survive CDN failures with Redundant Streams
A screenshot showing a failed CDN request

Survive CDN failures with Redundant Streams

09/10/2020September 10, 2020(3 months ago)
4 min read
09/10/2020September 10, 2020(3 months ago)
4 min read

Survive CDN failures with Redundant Streams

When you're live streaming, you don't get second chances. Everything has to go off perfectly, every time.

Sometimes though, the internet just doesn't want that to be the case. At Mux we work really hard to make sure that our delivery platform is as reliable as possible, but there exist fundamental risks - BGP leaks, large-scale CDN outages, or even a backhoe through a fiber connection, which present significant problems to even the largest streaming video platforms.

Recently we spent time investigating what the fastest, highest value change we could make to Mux Video to improve our resilience to larger, internet wide service outages.

While today Mux already uses multiple CDNs to deliver your content, selected based on the quality of experience, as of right now, we make that decision when the end viewer starts watching the stream. This is acceptable for short-form VoD content, but starts to run into limitations for long-form content, or live streams.

The solution to this problem is usually referred to "mid stream CDN switching". There's a few approaches that we could have taken to enable mid-stream CDN switching for our customers, but as our first step to get something ready as soon as possible for your high value events, we've chosen to leverage a client side technology known as "HLS Redundant Streams".

HLS Redundant Streams

HLS redundant streams are an often forgotten about part of HLS's functionality. This feature allows you to define multiple HLS renditions in your manifest that have identical characteristics, other than the URL which they point to, allowing the player to treat these as a redundant set, for failover purposes. In the HLS manifest, it looks something like this:

The redundant renditions in this manifest are identified by the player by checking for an identical EXT-X-STREAM-INF line.

Since failover between the HLS redundant streams is implemented on the client side, the behavior and support does vary between players.

In general, video players look for 4XX or 5XX status codes on either segment or manifest downloads, in order to trigger a failover. Some players will fail over immediately to the same rendition on the alternate CDN, while other players will try other renditions on the primary CDN first, before trying the redundant streams.

Let's take a look at a demo of how one of the most popular video players, HLS.js deals with a major CDN outage when redundant streams are enabled.

Looks great! How do I turn it on?

First, check your player is compatible

We've spent a lot of time testing the compatibility of the most popular HLS players with a variety of failure scenarios, focusing on failures in both manifest, and media delivery. You can check the video players that you're using in the compatibility grid below:

PlayerTested VersionManifest 4XXManifest 5XXMedia 4XXMedia 5XX
Video.js>= 7.6.6
HLS.js>= 0.14.11
JWPlayerProduction Release Channel
Safari iOS (AVPlayer)iOS >= 13.6.1
Safari MacOSSafari > 13.1.2
Video.js
Tested Version
>= 7.6.6
Manifest 4XX
Manifest 5XX
Media 4XX
Media 5XX
HLS.js
Tested Version
>= 0.14.11
Manifest 4XX
Manifest 5XX
Media 4XX
Media 5XX
JWPlayer
Tested Version
Production Release Channel
Manifest 4XX
Manifest 5XX
Media 4XX
Media 5XX
Safari iOS (AVPlayer)
Tested Version
iOS >= 13.6.1
Manifest 4XX
Manifest 5XX
Media 4XX
Media 5XX
Safari MacOS
Tested Version
Safari > 13.1.2
Manifest 4XX
Manifest 5XX
Media 4XX
Media 5XX

You'll notice there's some missing common players on this list, including the most common native player on Android, ExoPlayer. We're actively working with both open and closed source player vendors to make sure that this functionality is available where you need it - we'll post updates in our playback modifiers documentation as we make progress enabling this feature on more platforms

Add the required playback modifier to your playback URL

Enabling redundant streams is really easy, all you have to do is set a query parameter on your playback URL:

If you're using signed URLs, you instead need to add a claim inside your JWT body, setting the claim redundant_streams to true - your JWT payload should look something like this:

You can find more details about using playback modifiers in our documentation.

Deploy to some users, and monitor for stability

When you deploy redundant streams, you'll want to make sure that your player is behaving as expected with the new manifests we're delivering. We strongly recommend deploying to a subset of users, and using Mux Data's experiment field to run some A/B testing on your viewers. In normal conditions you wouldn't expect to see any change in the viewer experience scores, only in exceptional circumstances, such as a CDN failure, or routing issues to particular CDNs, so don't be surprised if your viewer scores don't immediately improve.

Graph showing happy Mux data lines

In Summary

Leveraging redundant streams in Mux Video gives you access to a more resilient multi-CDN delivery strategy, allowing you more protection against CDN, or internet infrastructure outages. You can enable redundant streams today for free, on both live and on-demand streams.

This isn't the end for our reliability efforts at Mux, we've got a lot more projects in the pipeline to make sure that your live events are bulletproof - keep your eye on the blog for more updates in the future!

Subscribe to the Mux newsletter