This is going to be the second blog post I've made with this observation, but: COVID is weird. And while it's not one of the biggest ways COVID has turned over our collective applecart, it has now been a solid eight months since the majority of us have gone to a conference and exchanged air (hopefully only full of your usual con crud) with other humans.
One of our favorite conferences for the last 5 years has been Demuxed. Yes, sure, one of the people that started Mux also started the conference, and he’s quick to tell people that Demuxed came first (yeah, we get it, Matt), but seriously, it’s a fantastic community. As the organizing sponsor of the event, our team always takes care of production, which most years involves obvious things like getting a venue and someone making sure the next speaker has a mic in time. This year, however...
I'm writing this as I drink from my Demuxed mug, one notably not decomposed into a complimentary bonus set of ceramic knives, so you know where this is going. That's right: we did Demuxed and we did it Over The Internet and, to toot our own horn only modestly, the reviews from virtual attendees were--brace for it, Phil--fantastic.
So, as the goon behind the boards for three days, I figured it might be nice to discuss how we did this and to walk you through the production side of Demuxed 2020 at a high level. (If you're interested in the nuts and bolts of the stream side of things, be it the web application Dylan hammered together or the redundant data flows and closed captioning that Phil and Ward masterminded, that'll be another day.)
Pre-Production
If you fail to plan, you plan to fail. That's one of those homespun lines to which we actually can attribute an author--that Ben Franklin guy, the one with the print shop and the kite and the vanishing to France for a decade--and it's as true now as it was in seventeen dickety-six. We took it to heart.
The Look
Some of this was put together before I started in August, so to me this is a bit misty, but one of our outside designers, Ollie, put together a design language and a set of comps in Photoshop for us to riff off of. This included the "paint swoosh" nameplates, the scribbly borders around camera viewports, and the like--which, I think, gave Demuxed a real sense of character that your average "rectangles, rectangles, and BTW more rectangles" show doesn't have. (This is not to say we didn't have a lot of rectangles. But most of them were my fault. More on that later.)
The Talks
The core of Demuxed 2020 was, as in previous years, the talks. Live talks were right out; we weren't going to have the entire show dependent on the vagaries of presenters' internet connections. (We'd have the entire show dependent on the vagaries of my internet connection, but we got away with that one.) To that end we reached out to our presenters and asked them to pre-record their talks two ways: give us a screen with your presentation, and give us your beautiful mugs in a face camera. Matt and Phil put together instructions to embed together both feeds in a single recording via a double-width OBS canvas, so we could avoid everyone's least favorite Olympic sport of video synchronization, and we were off to the races.
Once a recording came in, we shipped them off to our editor, Tim, who carved up each talk and slotted it into our existing designs from Ollie to provide a video for day-of shooting. This video had no textual captions--that way we had more flexibility when managing the run-of-show. We'd then play those videos during the show. (Our YouTube uploads are not livestream cuts of the talks; Tim has also put together titled versions of each talk for upload.)
On balance, I think this went really well. Great content and engaging presentations--and talking into a void like that is actually really hard, so when we're grading on that curve the talks were nothing short of fantastic. Where we could do better was that audio quality was a little hit or miss. We’d planned to ship out standard microphones to presenters (not that we have some firm opinions on the topic, oh no) but time pressures made that impractical and so we relied on presenters' own setups for audio--which can be a bit of a grab bag. In the future, should we need to go down this road again, getting known-good gear into speakers’ hands is a high priority.
The Rig
Let’s talk tech a bit. This will be on the production side of things more than the host side. Zoom Like You Mean It, Part 3 will be covering video, and we’ll talk about the setups that Matt and Phil (and to a lesser extent, I) used during the show.
With all the pieces in place, we then had to rig them for day-of broadcast. As mentioned before and a few other times in ye olde blog, I came on at Mux in late August, and I brought with me a certain amount of livestream experience. I'm not sure if I ever actually volunteered to run Demuxed or if once they saw the downright foolish amount of hardware in the office Matt and Phil and I all kind of assumed I would be running the show, but, surprise! I'm running the show, so let's figure out how to do it.
Unlike much of the streaming community, I don't use OBS for my core mixer. Which is not, I stress, a knock against OBS; my shows run towards a pretty tightly connected experience, with a lot of data-driven systems and a lot of I/O going back and forth in terms of video content, video streams from other sources, and the like. OBS can technically do a lot of it but audio mixing is a bear and there's a lot of duct tape you have to bring yourself. Instead I use vMix, which provides a decent audio mixer (it's no DAW, but it's tightly integrated and doesn't require a bunch of virtual audio cables to get anything!) and a bunch of built-in features to help make me look way better at this than I actually am. We used a lot of them for Demuxed.
Titling
For the uninitiated: titling, in this context, refers to the creation, the display, and the management of, well, titles-lower thirds and other textual elements of the show. OBS users often call these "overlays".
In ages past (like, 2018), vMix's titling solution was...well, it was fun, I'll give them that. They provided a boatload of pre-designed titles, but who wants those? And the vMix title designer that they shipped wasn't super impressive. If you wanted a quality, animated title, you had to go hunt down Microsoft Expression Blend, packaged with Visual Studio, and learn that whole nightmare. (Former WPF developer checking in. It's not fun, folks!) But with more recent versions of vMix they package the GT Designer, which finally brought making good-looking, professional titles within reach of mere mortals.
I took Ollie's comps for our titles, scribbly borders and all, and started knocking together our final look-and-feel. For the most part, this was straightforward to do (the quick way, if not the optimal way; that we used dark backgrounds let me hide a lot of sins with this approach): export all of the overlay elements into a full-screen PNG, load it into the GT Designer, and then add to it text fields in the appropriate places. We'd then later wire up those text fields to our data backend--more on that in a second.
Not all was sunshine and roses, though, and we did run into some issues borne of this-or-that unfamiliarity with the video process when making some early decisions. Nothing earth-shattering, but little stuff: text doctored with Photoshop's font-twiddling tools that couldn't be replicated, camera viewports that weren't 16:9 so the crop was different from scene to scene, that sort of thing. They're the sort of thing which, in the future, we'll nail down, and it's unlikely to be something that a casual viewer would notice (save for Matt's occasionally disappearing chin), but it's an area for improvement and a tightening-up of our review process now that we know that this can happen.
Data-Driven Run of Show
If you've ever run a live event through the more conventional stream tools out there, you've probably run into the situation where you're hunting through the interface looking for particular text labels and scrambling to get them updated, either on the fly and "showing your work" on the program feed or by duplicating the universe and switching from A to B. (Or you've built some big HTML5 overlay monster. Which--I can empathize. Been there too.)
One of the biggest reasons I use vMix is that it does not leave me holding the bag like that. Internally it has the concept of data sources: row-based chunks of data. You can fetch them from an HTTP API (just return an array of objects!), from a local CSV or Excel spreadsheet...or from Google Sheets.
Google Sheets is a superpower here. It allows for a multi-user interface with a surprising amount of data validation and management and you don't have to do anything to make it work. I run fighting game tournaments on the side and my personal mess of VLOOKUPs and some Google App Script baling wire regularly handle 50+ attendees and 80+ games in tournament brackets, with as many assistants as I need partying on different functions of the system at the same time. We're updating it during events sometimes multiple times per minute. By comparison? Loading into Demuxed the run of show--our hosts, our guests, the talks and the details about them--was as easy as it gets. And once loaded, it's trivial to tell vMix "hey, map Caster1Name to this field you're getting out of Google Sheets". Minimal context switching, no surprises.
And, with everything rigged and the talks in place, we were ready to go live.
Doing It Live
This was the easy part. It was also the exhausting part; I felt like I had jumper cables on my brain for three days. But it was the easy part.
With the amount of pre-production work we had at hand, and the other secret weapon I haven't mentioned yet, we came into the first day of Demuxed in a really good spot. We had planned out a workflow for the show:
- Matt, Phil, or both would hop on for some yep-we're-definitely-human banter while I cued up a talk.
- Matt or Phil would throw to the talk and we'd fire off a sick stinger transition (🤘 extremely not optional) and start the talk video.
- While the talk was running, our behind-the-scenes team (shout out to Banks and to Victoria) would collect questions and load them into Google Sheets.
- Also while the talk was running, Phil would coordinate with the speaker to get them into our voice conference stuff (more on that in a minute, it's a game-changer) and I would do a little producer magic off-stream: make sure they were on the correct microphone, level them out, do a little EQ if it sounded particularly rough, the usual.
- As the talk ended, we'd return to live with a host, either Matt or Phil, with the speaker. They'd ask questions, I'd hopefully have the question ready to bring up on screen and the speaker would, naturally, be brilliant and incisive about it.
- Matt or Phil would wrap the Q&A and either throw to another talk or to a break.
This is a multi-step process, but from my perspective behind the boards it's not a super complicated one. We didn't really have time to rehearse ahead of time, but the whole day-of team was in a Slack channel; I felt pretty confident that we'd pull it off okay.
Believe it or not, this is confidence.
My background, after all, is in devops.
And so, we hit "go", and...
...and...
...everything pretty much worked as planned. Hosts did host things, guests came in and did a fantastic job, Nick Chadwick asked somewhere between thirty and seven thousand questions, and the show went off with only one hitch. We'll talk about that at the end, because it's silly, and there's fun stuff to go into first.
Q&A Was Awesome
One of the things we've heard a lot in feedback from attendees was that the mix of live and recorded segments was clean and really engaging. Much of that was thanks to the ease of use, and the ease of Getting It Right, that is vMix Call. vMix provides a WebRTC platform that slots directly into the mixer.
I say it a lot, but video is really about audio and vMix Call, for its imperfections (and it's got plenty), gives you a lot of control of audio. It's not like Zoom or Discord capture, where you've got somebody else's audio mix and you're carving up a window capture to get your face cameras. Somebody hops into vMix Call and they're just there, they're a separate and composable audio and video input. Regularly during Demuxed I'd solo a guest into my headphones and just ask them to say a few words to make sure we didn't have to level in master--which, for those of you from a broadcast background is "duh?" and for those of us from a streaming background is "shut up and take my money."
Between Q&A and both Matt and Phil coming in through vMix Call for the entire show, I feel pretty comfortable asserting that we couldn’t have done this nearly as well without vMix Call and, even had we spent a stack of cash on another solution, it wouldn’t have been nearly as straightforward on my end of things.
Lightning Talks: The Dark Side of Doing It Live
A conference without lightning talks is barely a conference at all. Might not even be legal in some jurisdictions. They’re often the most highly rated part of a conference and they’re a ton of fun. (I’m new in town, so I wasn’t there for this, but legends tell of Phil emceeing lightning talks--and announcing wrong names from time to time…)
So of course we had to have lightning talks. Like this one!
I've been doing video nonsense for a while now and something that I've learned and taken to heart is this: when your show looks professional, audiences interpret jank as comedy. This is a powerfully liberating feeling; you can just sort of squint and YOLO something out there and if it works you're a genius and if it doesn't--clearly you're just being intentionally bad. There's a line there--having your stream, ahem, crash is not jank, it's just bad--but you can be a bit more risk-friendly and give new stuff the old college try.
Like being told on day two that, by the way, we want to do lightning talks. Half of them are pre-recorded and half of them are live! How should we do this?
We should do it by breaking out the duct tape, that's how. I'm used to running multiple computers as intakes to the stream kit, and that's how we went ahead here. The inimitable Matt Ward took the reins of scheduling and selection and parked himself in an empty co-working space, then fired up a Zoom call. I grabbed that Zoom call on one of my satellite machines and shoveled it over to the mixer via SRT at about a 250ms delay--hopefully enough, I figured, to be able to lip-read what Ward was saying a little early. This was a thing because Ward couldn't see nor hear me, and I couldn't hear him until the SRT feed hit the mixer and my headphones. Even a little head start would help.
And it did help. Didn't mean we weren't ping-ponging at high speed between recorded lightning talks on my end and live talks on Ward's end, and I think I cut off Ward's countdowns more often than I actually got them on time, but you know what? Swooshing around with cube zooms hide a lot of mistakes.
Lightning talks are supposed to be janky, and these, finishing up with a high-speed tour of the stream rig and how Demuxed was happening while it was happening, were my favorite part of the week.
A Few F's In The Chat
Near the end of the first day, we encountered a really spectacular crash of the video mixer. And I don't mean "vmix dot exe has encountered and error and needs to close", I mean "and as I clicked close program the computer locked up, called me names, and spit rivets all over the floor."
Which I had, to date, never seen before, but the box came back up just fine. Fortunately enough, because as mentioned on Twitter--
--we didn't really have a great backup plan for "the mixer melted down and caught fire."
Fortunately, the crash didn't repeat itself. I couldn't investigate it day-of, though. Most ways I had to debug were a bit destructive, after all, and we had a show to do. So we did a lot of breath-holding as we eased our way through the last segment of the day, which thankfully did not explode. Nor did the next two days, though I had on the sly repurposed another machine I had as a hot spare just in case.
So What Actually Happened?
The F's in the chat during Demuxed proved to be the start of some actual problems for the box, though. Day-of, it looked like a vMix problem, as it happened just as somebody hopped into vMix Call, and I was suddenly awfully suspicious, but soon enough there were stability problems even when not streaming. A couple weeks ago I finally and conclusively nailed down the problem. I feel a little silly about it, but nobody learns without falling on their face, so!
Halfway through reassembly. Please enjoy some randomly scattered SSDs. I do.
This here is the stream kit that, back when going outside was allowed, I'd take on-site for events. My "hat" is being able to run a four-camera show anywhere on the planet where I can get power, and this box here is why. One of our speakers at Demuxed opined that one shouldn't invest their time in building PCs for video processing these days, but nobody will sell me a 12-core, 24-thread, at-least-three-PCIe x16 slots 4U PC that fits in a shorty Gator case so I can throw it and a ten-gigabit switch in the back of a car along with a brace of microphones, lights, stands, and the like when I'm going somewhere. So we make do.
After that first crash, and a couple of really unfortunate ones since, I found myself doing the usual dance: replace a component, see if it still has issues. Swapped the GPU, no good. Swapped the RAM, no good. Swapped the Ryzen 3900X for a Ryzen 2600X--suddenly OK.
A clue.
Long story short: that box was doing the Little Engine That Could through some accidentally awful problems. Ryzen's "turbo boost", Precision Boost Overdrive, will enthusiastically boost its precision in an overdriven manner straight up to the point where it will happily overdraw the PSU given the opportunity. And, beyond that, under load the machine was hitting very high ambient temperatures; while the CPU would be happy at a nice toasty 95 deg C, it stands to reason that the rest of the hardware likely wasn't super happy at temperatures approaching "YOU DID WHAT?", too.
So, one new PSU and one competent fan configuration later, we've gone from baking the machine by looking at it funny to full load not even approaching the thermal envelope. Rookie mistake, but one easily rectified. And now I own a FLIR camera, so it's a win-win all around.
That's A Wrap
So there you have it -- that's how, from the perspective of sitting behind this desk, we produced Demuxed. I'm actually looking forward to the rest of our articles on this one, 'cause I was heads-down on all of this and I'm not even rightly sure how the rest of the show went down. So when that comes out, I'll be learning along with you.
Thanks for giving this a read, and I'll see you next time.
