TED is a world-renowned conference known for its inspirational and educational talks. Founded in 1984, it’s growth catapulted when they published video recordings of the talks online. When they posted the first 5 videos online in 2006, they were unsure if anyone would watch them, but by 2012, they had reached an incredible one billion views. Mark Bodanoff, Production Engineer, said, “The online platform breathed new life into the organization and it helped advance our motto: ‘Ideas worth spreading.’”
To power their online videos and platform, TED’s engineering team built the entire video1 production pipeline themselves. “We own everything from recording through encoding through publishing. But we never had any outside validation, other than people viewing our content, of whether what we were doing was good, where it was good, or where we could get better.”
Because they didn’t have a system that provided ongoing measurement, their experiments ran the risk of being inefficient. When they wanted to run an experiment, they would decide what to measure on an ad hoc basis and experienced setbacks if they realized they were measuring the wrong thing or needed more than one metric.
When Mark and his colleague Joe Bartlett, Video Developer, learned about Mux Data, they realized it could provide them with the outside validation they needed plus greatly improve the efficiency and accuracy of their experiments.
In India, a phone called the JioPhone came out and pretty much took the country by storm. The JioPhone, which runs KaiOS (which is a fork of Firefox OS), was not only a phone but provided free internet. What felt like overnight, the TED team started seeing a lot of requests coming in. At the time, the team was using Google Analytics and they were alerted to an issue with dropped video views coming from Android tablets running Firefox.
“It took us a long time to work through that data and realize that Google was miscategorizing the devices and that they were these KaiOS feature phones trying to play video,” says Joe.
In addition, they were able to identify thse KaiOS devices and tag them appropriately which you can’t do in Google Analytics. This enabled them to isolate this segment and look at the traffic separately so they could make adjustments to their default viewing experience based on the specific needs of users. In terms of Quality of Experience (QoE) metrics, their viewer experience score started at 48 and is now 77.
“Mux gives us confidence when we roll out new changes and it’s game changing. I can hit publish after making changes to the player and A) know that I’m not breaking anything and B) see when it has a positive impact,” said Mark.
Currently, Mark and the team are in the midst of a very large rearchitecture of their video production pipeline. As they prepare to publish the first videos out of the new system, Mux Data is providing the crucial metrics so the team can compare the video performance of the new system with the old one.
“Mux Data caught a number of issues we didn’t expect to find along the way. Actually, issues we would never have found otherwise. And beyond issues, the data helps us identify where there are opportunities for improvement and enables us to create quantifiable goals for the video team,” said Mark.