For certain reasons, I had to implement HLS Push for web delivery at Akamai in the past few days, and I’m so annoyed that I have to blog about it.

What is this HLS stuff?

For those of you who don’t know what HLS is, the format has become the de facto standard for videos and livestreams on the web. It usually consists of several media playlists (like Winamp back in the day), one for each video resolution. And there is the so-called master playlist, which links the other playlists. The master playlist is only read once by the video player.

The media playlist for the currently used resolution is called up regularly at intervals of several seconds during livestreams to see whether new media segments - i.e., audio and video in short clips of 1 to 60 seconds in length - have been added. So much for the theory.

How does the retrieval normally work?

Imagine you have a server that produces this livestream. It works reasonably well so far, but you don’t want all your viewers to access this server via HTTPS. After all, it has other things to do than deliver data to the internet. So you could put a CDN in front of it.

This retrieves the playlists and video segments from your server and distributes them to the browsers out there, taking the load off your server. This is the usual HLS pull and actually, all CDNs do exactly this.

But then Akamai came along and thought, „Hold my beer; we’re going to break that!“

What is Akamai doing differently now?

Akamai probably thought, „Why don’t we just have the data pushed directly into our system? Then it’s already available as soon as the user requests come in, and we don’t have to retrieve it anywhere else first“.

So far, the theory is certainly good and reasonably well thought out. But only in theory; just because the segments are in the system doesn’t mean that Akamai will deliver them.

The requirement was to send an audio stream with 2 different bitrates in HLS format to Akamai. This means that we have 3 playlists here, so 1x master + 2x media playlists for the different bitrates.

As soon as new segments are produced, they are uploaded to the Akamai endpoint, and then the media playlists are updated and uploaded. Also, straightforward so far. It’s just a pity that the player keeps giving up when playing these streams because the segments return an Error 404 Not Found when called up.

The first thought: Maybe it hasn’t been distributed internally yet. Could happen. So I add a delay before adding the segment to the playlist. This gives Akamai a little time to distribute it internally. The attempts fail, of course. Even after a 5-second delay for a 2-second segment, it’s a few KB of data, and we get many 404 Not Found responses.

The alternative to the delay

Holy shit! Let’s not give it a fixed time, but simply make a HEAD request to the segment URL. So we don’t retrieve the segment completely, but only the HTTP header, which already tells us whether the file is available.
And now guess!

You’re right, it works! At least for a few seconds. Because just because your server that pushes the segments receives the status 200 OK does not mean that the other CDN endpoints (PoPs) worldwide will also deliver this segment. But instead of taking a bit of time internally to pull the segment from their own caches, these Akamai idiots deliver a 404 Not Found status to the user client.

If you try to get it a second later, it is also delivered perfectly, but unfortunately, in this case, I have no influence on the player configurations to configure the retry.

So, let’s be honest: how did this bunch of idiots become one of the biggest CDNs? I wonder if they ever use their own stuff.