Tie together your client and CDN logs using Mux Data with CMCD

I have young kids, so I spend a lot of time looking for missing toys. Often we end up finding those toys under a couch or between the bed and a wall. Searching all the nooks and crannies can be hard — squeezing your hand underneath or into the furniture without actually being able to see what you are reaching for. But it's a great satisfaction, seeing the smile on your kid's face, when you find that lost Lego minifig or the missing crayon that the picture just can't be finished without.

You are no doubt asking — What in the world does hunting lost toys have to do with navigating video metrics and CDN logs? Well, they are surprisingly similar experiences. When something goes wrong on the server — a viewer has a bad experience because an error occurred when downloading content — it can feel like you are reaching around in the dark trying to find the cause. Your client analytics, such as Mux Data, provide deep observability into the experience and show you an error occurred. But it's hard to know what happened on the other side of the viewer’s connection. Did the CDN endpoint server fail? Was the content uploaded with an error and not able to be served? Did your origin not respond in time? If you only have server/CDN logs, it's difficult to know which server-side errors actually had an impact on the playback experience, and what the viewer actually experienced, versus what the player buffer and the adaptive algorithm was able to hide.

Before we get into the weeds of troubleshooting viewer experiences, let’s back up for a minute. Why should you care when a viewer has a bad experience or where the issue originated? There are so many options for people’s time now that applications need to ensure they provide reliable experiences. Not every application needs broadcast-level video quality; but every video view session needs to be reliable and deliver the quality that meets viewer expectations. If customers are paying for your content, quality playback experiences are even more important.

Failures will happen. It’s just a matter of how often they occur and what you can do to resolve the issues. Resolving problems quickly allows you to reduce the impact to viewers. Responding quickly to support issues raised by viewers or complaints on social media will reduce subscriber churn and increase the amount of video content viewers will consume in your service, allowing you to maximize your revenue.

OK, back to finding and resolving problems. When errors or poor viewership occur, there are commonly two ways of using data that are useful for investigation. The first is an aggregation of metrics across viewers, which is useful to identify trends, such as a spike in errors or an increase in the time to start a viewer. This approach is straightforward in a tool like Mux Data, which provides aggregated metrics and insight into current performance. The second method focuses on session data, looking specifically at what happens to a single viewer to better understand what caused the viewer’s bad experience. Mux Data provides a session visualization that shows the level of quality throughout the video playback session as experienced from the client.

However, errors or performance issues can also come from other parts of the video workflow. To tie the workflow together and get end-to-end visibility, many customers have tried to ingest and join their client logs with their CDN and origin server logs. This should work, but teams quickly run into a problem. They have client logs and server logs — but no good way to join them. There is no common identifier that is shared across the session the viewer experiences and the session as understood by the CDN or server. Teams will do all sorts of fuzzy matching by approximate time, IP, and device type to try to find the CDN requests that might represent the activity that the viewer experienced. No matter how smart or fuzzy the logic, sometimes the data just doesn't line up quite right.

Enter CMCD.

Thankfully, a solution has finally emerged: Common Media Client Data, usually referred to as CMCD. CMCD is a standard way of passing some standardized data from the player to the CDN. It supports the player sending a number of different client metrics and a variety of session information that the CDN can use to log, measure, and optimize the experience.

The types of data that can be sent include...

• encoded bitrate of the request segment
• buffer starvation level
• content ID
• next object that will be requested

...and a number of other values. The data is sent by the player when making a request during playback, usually either using headers or appending the values to the querystring, and the functionality is largely transparent to the developer using CMCD.

CMCD can enable a lot of use cases, but I'm not going to go into all of the data available today. I want to focus on one of the most basic and also the most powerful item for teams that want to get end-to-end visibility into their viewer experience: transmitting a shared session id.

If your player supports CMCD, you can specify a session id and content id to use during playback. These values will then be sent to the CDN as part of the media request and can be logged by the CDN as part of the normal request logs that are available from providers. You can also specify the same session id and content id to Mux Data, which is made available via our Video View exports. You can now easily match up the client logs with the CDN logs using a direct match rather than a fuzzy matching login.

Show me the code! How do I actually do this?

Here is an example that configures the CMCD session and content ids using HLS.js, which is also set in Mux Data. Mux Data uses slightly different nomenclature, but it captures the same information.

HLS.js has a cmcd object that allows you to set the session id, content id, and whether CMCD data should be transmitted using request headers or appended to the request querystring.

var config = {
  cmcd: {
    sessionId: session_id,   /* session id */
    contentId: video_id      /* content id */
  }
};

var hls = new Hls(config);

...

Dash.js does something similar. You set the session id sid and content id cid and enable the keys to send.

dashjsPlayer.updateSettings({
  streaming: {
    cmcd: {
      enabled: true,        /* enable reporting of cmcd parameters */
      sid: session_id,      /* session id */
      cid: video_id,        /* content id */
      mode: 'query',
      enabledKeys: ['cid', 'sid']
    }
  }
});

Use the same session id and content id when setting the metadata for the Mux Data view. In Mux Data, you set the view_session_id to the session_id set in the player, and video_id is the CMCD content_id.

mux.monitor('#myVideo', {
  data: {
    env_key: 'YOUR ENVIRONMENT KEY',
    ...
    video_id: video_id,              /* content id */
    view_session_id: session_id      /* session id */
  }
});

With this configuration, you now get common session and content ids across your client and CDN logs. You can use your favorite log processing tool to match the session ids from the Mux Data Video View logs with the logs from your CDN(s). Now, when you identify a problem with a viewer experience, you can easily trace that back to the requests on the server and better understand what happened on the CDN and why.

We’ll talk about the actual joining of logs in a future entry.

Where is CMCD supported?

Many major CDNs already support or are building features for CMCD. It’s been embraced by CDNs like Akamai, where the CMCD specification originated, as well as Amazon Cloudfront, Fastly, and others, thanks to its ability to get more useful information from CDN logs and support other advanced CDN features. Ask your CDN about it — there is a good chance support already exists at least for capturing CMCD data.

Players are actively building CMCD support, albeit a little more slowly than CDNs. Web players such as HLS.js, Dash.js, and Shaka Player have CMCD support built in. Roku is the largest and best distributed player currently supporting CMCD. We are waiting for native players such as ExoPlayer and AVPlayer to build in support. But don’t let that stop you from improving your visibility where it is supported now.