Making Storyboards Fast!

*This is part II in the journey Mux software developer Josh Allmann had with creating Storyboards. Check out the product announcement here!

On making Storyboards fast

On a personal note, this was my first major feature after joining Mux as an engineer, but I can’t take all the credit for this - storyboards have been in the works here, on and off, since we launched Mux Video. The sticking point with storyboards has always been performance. Why’s that? We’ll start with JIT.

JIT Storyboards

Storyboards are generated at the first request, in a process that we call just-in-time (JIT). Because of this, the initial request for storyboards may take a few seconds. After that first request completes, the image is cached, and follow-on requests are very quick. JIT storyboards bring a lot of benefits for Mux and our developers:

• Storyboards are immediately available to all assets, without any explicit opt-in or further configuration. This includes any back catalog that users have accumulated.
• We do not have to pre-process or store storyboards that would never be used.
• If we need to make improvements, we can mass invalidate cached storyboards, and follow-on requests will generate fresh storyboards.

However, we are under tight time constraints to ensure a responsive experience with JIT. Video is computationally heavy, and Mux processes videos with arbitrary resolutions (some very large), frame rates (some very high), and durations (some very long). For storyboards, we have to retrieve the video, extract dozens of thumbnails at various points within the video, and assemble those thumbnails into a final image. This has to be consistently fast, regardless of the video, and done within a few seconds.

Collecting Performance Wins

High performance is typically gained through a collection of incremental improvements. (Word to the wise: speed can just as easily be eroded away from the other direction, which is why we also closely monitor storyboard performance along with hundreds of other metrics.) Here are some of the techniques that were used to improve performance.

Speeding up thumbnail extraction

We use ffmpeg for this, with some careful tuning - rescaling early on in the process, avoiding pixel-format conversions if possible, and generally minimizing the amount of work done. Proportionally, probably the largest win.

Pipelining work

Internally, our assets are split into chunks. We need to download these chunks, extract the thumbnails from chunks, and composite the thumbnails into the final image. Rather than downloading all the thumbnails as one stage - which is only as fast as downloading the slowest chunk - we pipeline the download and thumbnailing. As soon as a chunk is downloaded, it is sent to the thumbnailer. If all thumbnailers are busy, the downloaded chunk waits. This smooths over fluctuations and allows us to use resources effectively - for example, if we have N thumbnailers, we probably don’t need more than 2N downloads in-flight.

Modulating concurrency

We run on big, fast machines, so their utilization should be maximized, but not saturated. Overloading the machine makes things slower, but so does under-utilization.

Coalescing work as necessary

For assets that may have several thumbnails in the same chunk, especially shorter assets, we extract all thumbnails in a single run, rather than taking several passes over the same chunk.

Cheating

For larger assets, we take the very first frame of the chunk, rather than decoding into the chunk to extract a frame at a specific timestamp. As a result, storyboard thumbnails may not always be spaced within precise intervals, but this is an acceptable tradeoff to achieve good performance. Can you notice?

Now that you have learned how Mux Video's Storyboards were created to work quickly, incorporate them into your video projects!