Why Bandwidth Tells You Nothing About Video Streaming Performance

Gigabit and, in some places, multi-gigabit access has become ordinary. On most fixed-line networks a speed test now reports hundreds of megabits per second, often more, and for years that headline number was a fair proxy for “good internet”. In fact, many benchmarking results still put that number on the top of their results. We believe that era is ending. The speed test number doesn’t tell you what a subscriber actually experiences when they press play on a video.

Prompted by a recent rollout of more Surfmeter clients in the field, we looked at this directly. Across four fixed-line access networks we ran continuous video measurements for a week, recording how long each session took to start — “Initial Loading Delay”, i.e., the wait from pressing play to the first frame. We paired it with a standard download speed test on the same connection.

To avoid some blame discussion, the networks are anonymized here as A through D — let’s focus on the relationship between speed and video QoE.

When more speed doesn’t help

Our first chart puts the two metrics (speed and video loading delay) side-by-side: download speed on the horizontal axis, loading delay on the vertical one. Each point is a network’s typical (median) loading delay; the bar shows the range that most sessions fall into.

Each network’s typical video loading delay against its download speed. The bar spans the middle half of sessions.

If bandwidth actually drove the experience, the points would slope downward — more speed, faster starts, simple, right? However, you can see that Network A moves data more than five times faster than Network B, roughly 520 against 100 Mbps, yet both begin playing a video in about nine-tenths of a second.

Network D has more than twice Network B’s bandwidth and is the slowest of the four to start. The quickest-starting network happens to be the fastest link, but the second-quickest start comes from the slowest link of the set.

This just shows that above the modest speed a video stream actually needs to buffer the first few video segments, adding megabits simply does not grant you a faster start.

Why speed and loading delay don’t correlate

Download speed describes how much data a link can move per second once a transfer is running in full speed. Loading delay is set by the opening moments of a session instead: resolving the service’s name in DNS, opening the connection through the TCP and TLS handshakes (or establishing a QUIC connection to the server), loading the website and all its scripts, and the round trips needed to fetch the first segments of video from a content delivery network.

All the above steps are governed by two aspects:

• raw latency
• responsiveness (i.e., latency under load)

These again depend on two other aspects:

• the technology used (e.g., WiFi equipment, although our probes are connected via LAN)
• how close and how well-connected the serving CDN is to that particular network

The first few seconds of an adaptive video stream need only a few megabits, and every network here had 10x–100x of that available.

The spread matters as much as the average

Averages (or medians) only tell us half the story. So, the next chart shows the full distribution of loading delay within each network.

Distribution of video loading delay per network. The box covers the middle half of sessions, the line is the median, and the whiskers reach the fast and slow extremes.

Even on the quickest network, loading delay ranges between <1s and >2s, depending on the session. Networks with similar advertised speeds can deliver different video metrics!

Not all video is the same

Different services start at very different speeds by their own design. This includes the complexity of the website and the video that is being streamed. A short social clip is often low-resolution, so there is very little data to fetch before the first frame can appear. A service like Netflix, by contrast, loads a far more complex page, with much more to set up before playback even begins.

When we split up loading delay by service, you can see that the range is quite broad.

Median initial loading delay per service. Each network’s download speed is shown in the legend.

Here, the low figures for Facebook and TikTok can be explained by the low-resolution video they serve by default. 3sat is just a fast site. YouTube, ARD and Netflix require more data to be fetched before playback can start.

One caveat here is that it is tricky to determine the point at which loading starts, so these charts don’t necessarily prove that one service is always faster than another. It depends on which video is measured.

But that’s beside the point. Notice where the network has the most impact! On the lightweight services everyone is fast, but on the heavier ones the 100 Mbps connection is among the quickest to start, while the one with more than twice that bandwidth is among the slowest. The direct correlation — expected by conventional wisdom — between bandwidth and startup time is not there!

What this means for monitoring

People are still talking about bandwidth a lot. Sure, it’s what you pay for, what’s on your monthly bill, and what you can complain to your ISP about. Several countries have rules in place that let customers (rightfully so, we think!) be compensated for lack of performance in that regard. However, beyond raw speed, who ensures you get the right streaming experience?

What will separate competing networks is how well they can serve content: latency, routing, peering, CDN proximity, DNS health, … Bandwidth alone is losing its role as Value Proposition. Let’s face it, for most people, increasing bandwidth by 50% or even 100% would barely make a difference. Choosing the best ISP would.

Of course, we wouldn’t be showing you this if we didn’t also think we have a solution. We run real video sessions against the services people actually use. We capture what matters — key metrics like loading delay, stalling, video resolution and bitrate. On top of that, our standardized quality score based on ITU-T recommendations. But we know that these metrics are less useful without context, so we conduct the detailed network tests in addition.

Contact us to find out more about how we can help you monitor your network and the video experience of your subscribers!