I wonder if there will be any long term reputational repercussions for Netflix because of this. Amongst SWEs, Netflix is known for hiring the best people and their streaming service normally seems very solid. Other streaming services have definitely caught up a bit and are much more reliable then in the early days, but my impression still has always been that Netflix is a step above the rest technically.
This sure doesn't help with that impression, and it hasn't just been a momentary glitch but hours of instability. And the Netflix status page saying "Netflix is up! We are not currently experiencing an interruption to our streaming service." doesn't help either...
Not the same demographic but their last large attempt at live was through a Love is blind reunion. It was the same thing, millions of people logging in, epic failure, nothing worked.
They never tried to do a live reunion again. I suppose they should have to get the experience. Because they are hitting the same problems with a much bigger stake event.
yup wanted to say that live stream stuttering has happened before on Netflix - I don't think the reputation is deserved.
From a livestreaming standpoint, netflix is 0/x - for many large events such as love is blind, etc.
From a livestreaming standpoint, look to broadcast news, sports / Olympics broadcasters, etc and you'll see technology, equipment, bandwidth, planning, and professionalism at 1000x of netflix.
Heck, for publicly traded quarterly earnings livestream meetings, they book direct satellite time in addition to fiber to make sure they don't rely only on terrestrial networks which can fail. From a business standpoint, failure during a quarterly meeting stream can mean the destruction of a company (by making shareholders mad that they can't see and vote during the meeting making them push for internal change) - so the stakes are much higher than live entertainment streaming.
Netflix is good at many things, livestreaming is not one of those things.
for livestreams, individual events like the Olympics probably has a surge audience of 10x of netflix events.
Netflix events is small potatoes compared to other livestream stalwarts.
Imagine having to stream a cricket match internationally to UK / India / Australia with combined audience that crushes the Superbowl or a football match to all of Europe, or even something like livestreaming F1 racing that has multiple magnitudes of audience than a boxing match and also has 10x the number of cameras (at 8K+ resolution) across a large physical staging arena (the size of the track/course) in realtime, in addition to streaming directly from the cockpit of cars that are racing 200mph++.
Livestream focused outfits do this all day, everyday.
Netflix doesn't even come close to scratching the "beginner" level of these kinds of live events.
It's a matter of competencies. We wouldn't expect Netflix to be able to serve burgers like McDonald's does - Livestreaming is a completely different discipline and it's hubris on Netflix's part to assume just because they're good at sending video across the internet they can competently do livestreaming.
the point i’m making is that the netflix live streaming timeline didn’t go
chris rock -> love is blind -> mike tyson
they have had other, successful executions in between. the comment i was replying to had cherry picked failures and i’m trying to git rebase them onto main.
From what I've heard, Netflix has really diluted the culture that people know of from the Patty McCord days.
In particular, they have been revising their compensation structure to issue RSUs, add in a bunch of annoying review process, add in a bunch of leveling and titles, begin hiring down market (e.g. non-sr employees), etc.
In addition to doing this, shuffling headcount, budgets, and title quotas around has in general made the company a lot more bureaucratic.
I think, as streaming matured as a solution space, this (what is equivalent to cost-cutting) was inevitable.
If Netflix was running the same team/culture as it was 10 years ago, I'd like to say that they would have been able to pull of streaming.
Combination of 2 and 3. The business changed. Streaming was more or less a solved problem for Netflix. They needed money for content, not expensive engineers. Ted is co-ceo… you can see where the priority is.
So the issue is that Netflix gets its performance from colocating caches of movies in ISP datacenters, and a live broadcast doesn't work with that. It's not just about the sheer numbers of viewers, it's that a live model totally undermines their entire infrastructure advantage.
Correct, this is not Netflix’ regular cup of tea, and it’s a very different problem to solve. They can probably use their edge caches, but it’s challenging.
My wild assed guess is the differences in the edge nodes.
Netflix's edge nodes are optimized for streaming already encoded videos to end users. They have to transcode some number of formats from the source and send them all to the edge nodes to flow out. It's harder to manage a ton of different streams flowing out to the edge nodes cleanly.
I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients. Only one stream from source to edge to worry about and is much simpler to support and reason about.
> I would guess YouTube, being built on google's infrastructure , has powerful enough edge nodes that they stream one video stream to each edge location and the edges transcode for the clients.
Ha, no, our edge nodes don't have anywhere near enough spare CPU to do transcoding on the fly.
We have our own issues with livestreaming, but our system's developed differently over the past 15 years compared to Netflix's. While they've historically focused on intelligent pre-placement of data (which of course doesn't work for livestreaming), such an approach was never feasible for YT with the sheer size of our catalog (thanks to user-generated content).
Netflix is still new to the space, and there isn't a good substitute for real-world experience for understanding how your systems behave under wildly different traffic patterns. Give them some time.
It also helps that youtube serves shit tier quality videos more gracefully. Everyone is used to the step down to pixel-world on youtube to the point where they don’t complain much.
And decent part of these users are on free tier, so they are not paying for it. That alone gives you some level of forgiveness. At least I am not paying anything for this experience.
Live streams have different buffering logic to video on demand. Customers watching sports will get very upset if there is a long buffer, but for a VOD playback you don't care how big the buffer is. Segment sizes are short for live and long for VOD because you need to adapt faster and keep buffers small for Live, but longer download segments are better for buffering.
In my experience even YouTubeTV has problems sometimes. I'll have the 1080p (and enhanced mode also I think) quality set and still deal with a lot of compression artifacts.
Not sure how Netflix does it. But this is not very time sensitive, and I would have delayed the stream by 15 to 30 seconds to cache it and then deliver to everyone.
Not sure I fully buy that. The “live” stream is rarely “live”. It’s often a highly cached buffer that’s a few mins from latest. Those in isp caches can still help here.
Yep. Having actually worked on this sort of stuff I can confirm.
Your ISP doesn't have enough bandwidth to the Internet (generally speaking) for all users to get their feed directly from a central location. And that central location doesn't have enough bandwidth to serve all users even if the ISP could. That said, the delay can be pretty small, e.g. the first user to hit the cache goes upstream, the others basically get the stream as it comes in to the cache. This doesn't make things worse, it makes them better.
I don't bet so I have no clue, but why is that? Are people able to place bets in the middle of the match or something? I would have assumed bets get locked in when the fight starts
This is kind of silly because the delay between actual event happening to showing up on OTA TV or cable TV to showing up on satellite TV can already be tens of seconds.
Or, hear me out here, it's a wild concept, just work.
You know, like every other broadcaster, streaming platform, and company that does live content has been able to do.
Acting like this is a novel, hard problem that needs to be solved and we need to "upsell" it in tiers because Netflix is incompetent and live broadcasting hasn't been around for 80+ years is so fucking stupid.
I don't think that live doesn't work with caches. No one watching live would care about a O(s) delay, which is highly amenable to caching at ISPs and streaming changes from there to downstream clients. Offhand I'd say that would support O(ms) delay but no less.
That model still works for streaming. You have a central source stream only to the distributed edge locations, then have clients only stream from their local edge location. Even if one region is overwhelmed, the rest can still work. Load on the central source is bounded.
Likely these devices use different media formats and/or quality levels. And yes, it's possible one device buffers more than the other. Infinite freezes sounds like some routing issues or bugs.
When I was watching the behavior on the tv, was wondering if buffering sends some separate, non-business-as-usual requests, and that part of Netflix's delivery architecture was being overloaded.
E.g. "give me this previous chunk" vs "send me the current stream"
Buffering typically just consumes the same live stream until there's enough in the buffer. No difference other than the request rate being potentially higher. At least I can confidently say that for the standard players/video platforms. NetFlix could be doing something different. I'm not sure if they have their own protocols. But I'd be very surprised if the buffering used a completely different mechanism.
Damn that sucks. I wonder if they could have intentionally streamed it 5 min late? I don’t have all the context around the fight though — maybe a competing service would win if Netflix intentionally induced delay?
they were introducing 5 minute delays on some of the clients. I noticed my ipad was always live and the smart tv had a 5 minute delay but you could fast forward to live.
If Netflix still interviews on hacker rank puzzles I think this should be a wake up call. Interviewing on irrelevant logic puzzles is no match for systems engineering.
I did a round of netflix interviews, didn't get an offer (but passed the technical coding rounds) they absolutely had the best interview process of any company I've interviewed at my entire career.
They do make you code but the questions were
1. Not on hacker rank or leetcode
2. Pratical coding questions that didn't require anything more than basic hashmaps/lists/loops/recursion if you want. Some string parsing, etc.
They were still hard, you had to code a fast, but no tricky algorithms required. It also felt very collaborative, it felt like you were driving pair programming. Highly recommended even though didn't get an offer!
For systems design and engineering, absolutely this. I expected the very highest standards and upmost uptime from Netflix, similar to Google and Amazon.
Tells you the uselessness of their engineering blogs.
If places like Paramount+ can figure it out, Netflix, given their 10+ year head start on streaming and strong engineering culture, should also have been able to. And if you don't like my example, literally every other streaming service has streamed live sports without issue. YT TV, Hulu, Paramount+, Amazon Prime, Peacock, even Apple TV streams live sports.
It may be "new" to them, but they should have been ready.
I won’t argue that they shouldn’t have done better, I’m only pointing out that this is fairly different from their usual product. Amazon, YouTube, and Hulu all have a ton of experience with live streaming by now. Apple has live streamed wwdc for several years.
I did expect that Netflix would have appropriately accounted for demand and scale, though, especially given the hype for this particular event.
Has Netflix ever live streamed something before? People on reddit are reporting that if you back up the play marker by about 3 minutes the lag goes away. They've got a handle on streaming things when they have a day in advance to encode it into different formats and push it to regional CDNs. But I can't recall them ever live streaming something. Definitely nothing this hyped.
I don't spend much time streaming, but I got a glimpse of the Amazon Prime catalog yesterday, and was surprised at how many titles on the front page were movies I'd actually watch. Reminded me of Netflix a dozen years ago.
Amazon Prime isn't so great. Lots of for rent/purchase content or content with ads these days. And they end up repeating slots of content in all the rows in their UI, so I end up seeing the same suggestions everywhere rather than much that's new (other than first party productions).
To me they're basically padding their front page.
But honestly that's most of the major streaming platforms these days. I recently cancelled Disney Plus for similar reasons. The only reasons I don't cancel prime or Netflix are because I have family members I split the memberships with to share.
I recently found a lil dvd rental place in my city. It’s a non-profit, they also do archivals and stuff.
It’s pretty much a two-story townhouse packed head to toe with DVDs (lots of blu rays!)
You don’t realize how limited the streaming collection is until you’re back in a movie store, looking through thousands and thousands of movies you would never find otherwise.
Since I found it, I’ve started doing movie night every week with my friends. It’s such an absolute blast. We go to the store, each pick out a random movie that looks good (or bad, in a good way) or just different.
That's an excellent option. I think it'd be remiss not to mention local libraries. Of course, your mileage may vary, but the ones I've gone to do seem to have adequate selections. I just don't often make time to go there and browse like I would have at traditional video rental places back in the day.
Heck, mine even have some video games; though from when I've checked they're usually pretty back-reserved.
I was in high school in the early 00s, and going to the movies was such a big part of my life. Now, I never even know what's out.
I suspect life stage is a factor, but it does feel like there are many classes of entertainment (cinema and standup come to mind) that don't resonate like they used to.
Back in the day everyone was watching the same thing.
The choices for entertainment were limited to whatever was showing in movie theatres, whatever was on TV and whatever record stores were selling.
I've given Netflix a lot more money than I've gotten value out of. I've had an account for ~15y and only really use it for airplanes unless there's a specific thing I'm excited to watch.
I'm in the same boat where as soon as they make it too hard to share, I'll probably cancel it. I think the main reason their sharing crackdown hasn't been a problem so far is that I use it so seldomly, it thinks the "main" address is my parents, which makes it easy for me to pass the "are you traveling" 2FA on my own phone when I do want to watch something.
> And they end up repeating slots of content in all the rows in their UI, so I end up seeing the same suggestions everywhere rather than much that's new
All of the streaming services do this and I hate it. Netflix is the worst of the bunch, in my experience. I already scrolled past a movie, I don't want to watch it, don't show it to me six more times.
Imagine walking through a Blockbuster where every aisle was the same movies over and over again.
There's also the "FreeVee" items, which have ads regardless of whether you're a prime subscriber or not. And it feels like a lot of their catalog has been transferred over to FreeVee.
It's been pretty rough the last few years. So many great films and series, not to mention kids programming, removed to make way for mediocre NeTfLiX oRiGiNaLs and Bollywood trash.
Prime Video has to be the worst of all major streaming services. The video quality is horrible, its crippled with ads (3 not skippable ads for an episode of 45 minutes, lastly), and a lot of interesting titles are behind a "partner paywall".
I have prime and my shopping experience is crippled with ads too.
I think it got worse for sellers recently too. If I search for something, like a specific item using its description, sometimes the only result for it shows "sponsored".
It used to show up as sponsored and also unsponsored below.
If this changed, I assume it is bad for the seller. Either they pay for all search results, or their metrics are skewed because all searches were helped by "sponsorship" (and there are no longer unsponsored hits)
I was watching the rings of power and it started with a "Commercial free experience provided by so and so" with a long ad at the start of the episode, and then a third of the way into the episode, at a critical action part, it broke in the middle of the actor's sentence to a 6 minute ad block.
I exited playback and haven't gone back to finish it. I'll wait for it eventually to make it to a Blu-ray release someday.
> ut my impression still has always been that Netflix is a step above the rest technically.
I always assumed youtube was top dog for performance and stability. I can’t remember the last time I had issues with them and don’t they handle basically more traffic than any other video service?
Maybe a client issue, but i've got a low-end smart tv which handles netflix fine, but youtube is unwatchable due to buffering and failed cuts to adverts
I think Netflix will have even more sw engineers looking to work there once they notice even for average quality of work they can get paid 3 times more than their current pay.
Is it really that big a deal if you are watching a few minutes behind?
I've watched ball games on streaming networks where I can also hear a local radio broadcast, and the stream is always delayed compared to the radio, sometimes by quite a lot. But you'd never know it if you were just watching the stream.
I remember a few years ago reading about a scam at the Australian Open Tennis where there were people inside the stadium who were betting on individual points as they happened.
I guess they could bet before the betting streams caught up.
It seems ridiculous to me that you can bet on individual points, but here we are.
The issue is that most people are trying to watch live which is what it's advertised as. And until they figure out that they need to watch X minutes behind, it is unwatchable. Many will not figure that out.
So for the first hour it was just total frustration until I stopped trying to go back to live mode.
Internet streams are not real-time even in the best case. There is always a few seconds of delay, often quite a bit more than that depending on number of hops and link speeds, congestion, etc.
Most people pay Netflix to watch movies and tv shows, not sports. If I hadn't checked Hacker News today, I wouldn't even know they streamed sports, let alone that they had issues with it. Even now that I do, it doesn't affect how I see their core offering, which is their library of on-demand content.
Netflix's infrastructure is clearly built for static content, not live events, so it's no shock they aren't as polished in this area. Streaming anything live over the internet is a tough technical challenge compared to traditional cable.
I think why I will remember about this fight is not the (small) streaming issue I encountered as much as the poor quality of the fight itself. For me that was the reputational loss. Netflix was touting “NFL is coming to Netflix”. This fight did not really make me want to watch that.
I don't care about boxing or UFC or the grade-A douchebags that are the Paul brothers, but I tuned in just because I had the time and a Netflix subscription.
It was actually great that the fight itself was so boring because it justifies never having to spend time / money on that kind of bullshit. It was a farce. A very bright, loud, sparkly, and expensive (for some people) farce.
The value I got from it was the knowledge that missing out on that kind of thing isn't really missing out on anything at all.
Not really a joke, though? VOD has obvious methods to cheat a bit. Redundancy abounds and you can even network shape for costs. Could probably get even better compression for clear reasons.
Live, not so much. One source that you have to fanout from and absolutely no way to get cheap redundancy. Right?
I don't think it'll be long-term. Most people will forget about this really quickly. It's not like there will be many people saying "Oh, you don't want to sign up for Netflix, the Tyson fight wasn't well streamed" in even 6 months nevermind 10 years.
Most third-party internet-based streaming solutions are overlaid on top of a point-to-point network, while broadcast is one-to-many, and even cable tends to use multicast within the cable provider's network.
You have potentially different problems, e.g. limited bandwidth / spectrum. If, say, there are multiple games going on at the same time, you can only watch whichever feed the broadcaster decides to air. And, of course, regardless of the technology in use, there are matters of acquiring rights for various events. One benefit of internet-based streaming is that one service can acquire the rights and be able to reach everyone, whereas an individual cable provider might only reach its direct subscribers.
On cable(terrestrial is entirely different) even the bandwidth or spectrum is less limiting for broadcasting multiple games. Hard thing is the other parts of production, like cameras, live-directing and live commentary. Adding new channels is less challenging than actual producing content at expected level there.
Based on this I'm wondering whether it was straight up they did not expect it to be this popular?
> Some Cricket graphs of our #Netflix cache for the #PaulVsTyson fight. It has a 40 Gbps connection and it held steady almost 100% saturated the entire time.
I don't think Netflix is even designed to handle very extreme multi-region live-streaming at scale as evidenced in this event with hundreds of millions simultaneously watching.
YouTube, Twitch, Amazon Prime, Hulu, etc have all demonstrated to stream simultaneously to hundreds of millions live without any issues. This was Netflix's chance to do this and they have largely failed at this.
There are no excuses or juniors to blame this time. Quite the inexperience from the 'senior' engineers at Netflix not being able to handle the scale of live-streaming which they may lose contracts for this given the downtime across the world over this high impact event.
Very embarrassing for a multi-billion dollar publicly traded company.
The assumption that it was related to insufficient investment isn’t supported by any evidence. Flawed technical decisions can be made by the most expensive engineers too.
Other potential and future entertainment partners Netflix will be working with e.g. WWE, will certainly see my view as they will be questioning Netflix's capability after that major streaming issue we both saw.
This isn't Netflix's first time they had this live streaming problem.
People will see this as an underinvestment from Netflix's part and they will reconsider going to a different streaming partner.
Yea, it’s a bad look. But I switched to watching some other Netflix video and it seemed fine. Just this event had some early issues. Looks fine now though.
Streamed glitch free for me both on my phone and Xbox. The fight wasn’t so great though, but still a fun event. Jake Paul is a money machine right now.
Static files have been pretty much the standard streaming protocols for both VOD and live for the last 15 years ago. Before, it was Adobe Flash (RTMP).
With the way that they are designed, you can even use a regular CDN.
You can push these files to all the edges before you release the content which will protect your origin. Livestream all your edge servers are grabbing content from the origin unless you have another tier of regional servers to alleviate load.
Sure but that’s why your edge servers do request collapsing. And there are full blown CDN companies that will write an enterprise contract with you that can do this stuff with ease. Akamai is like 25 years old now.
Scale has increased but the techniques were figured out 20 years ago. There is not much left to invent in this space at the current moment so screwing up more than once is a bit unacceptable.
Two games actually, both on Christmas Day. A day when most people are at home or the home of family or friends, and they are both pretty good late-season matchups (Chiefs-Steelers and Ravens-Texans) so I imagine viewership will be high.
If they botch the NFL games, it will surely hurt their reputation.
Yeah, the funny part is that Hulu, Amazon Prime, and Peacock have all demonstrated the ability to handle an event of this caliber with no issue. Netflix now may never get another opportunity like this again.
This sure doesn't help with that impression, and it hasn't just been a momentary glitch but hours of instability. And the Netflix status page saying "Netflix is up! We are not currently experiencing an interruption to our streaming service." doesn't help either...