How to do browser automation for videos? Insights from the developer

(This post is part 2 of a small series on our development work.)

If you read our previous post, it should have become clear that it’s not trivial to build automation tests involving video. As we indicated in the article, one of the main issues developers normally stumble upon is gathering the current playout state of a video. If you are an automation engineer or a web developer, you’ve probably already come across a situation where you had to develop a website with video embedded. In this post, we’re going to show you how to radically simplify that process.

If you are doing browser test automation, you can check a lot of things:

• Is the logo rendered correctly?
• Are all text labels displayed?
• Does the right image load?
• Is it displayed in the right corner?
• Does this particular link work?
• Does the video play? … wait! How can we check if the video plays?

Stack Overflow is the place to go for such problems. But it usually delivers only a limited set of answers: in general, we can split them in two categories – if you’re building a test for YouTube, they have an API for that. If you try to build it for another site, you need to know JavaScript and use the W3C API for HTML5 video. Okay, so far, so good. What if you run the site yourself? Does the generic answer really help in every case where we might find a video on a website?

The tale of the playing video

So, let’s assume we want to know if the video is playing. That should be simple, right? You can check if the video is playing by asking if it is e.g. not paused, i.e. !video.paused. Great!

const videoElement = await driver.findElement(By.id('videoPlayer'));
await sleep(5000);
bool testPassed = !(await driver.executeScript(`return arguments[0].paused`, videoElement));

Code Example of a common, non-accurate approach to detect if a video is playing in Selenium

But then your tests might still fail for the wrong reasons. If the video is paused, you probably don’t know why it is not paused: because another helpful colleague has temporarily disabled the autoplay option and accidentally committed that change? Or maybe the content is not available and the player is just waiting for a video to load? Or is it paused because the content is still loading and the test server’s connection is just a bit slow today – and it would have passed your play check if you only had waited a bit longer than usual? Oh, and by the way, even if the video API says that it’s not paused, is the content then really playing or is it still loading some information in the background? And what if you want to know that the video really played in 1080p HD? And that there was no stalling in the middle?

It turns out that the W3C HTML5 API for media playback is quite complex. Really, all you want is this:

const videoAnalytics = new surfmeter.VideoAnalytics();
await sleep(5000);
expect(videoAnalytics.isVideoPlaying(video)).toBe(true);

const { qualityLevel } = videoAnalytics.getQualityStats(video);
expect(qualityLevel).toEqual("excellent");

Video test automation gets easy with Surfmeter!

Our Surfmeter test automation library does just that. It provides a single entry point for all relevant video statistics. Just let it do its measurements; it works with any service.

How our library can help

We strive to build solutions to monitor playbacks from the outside, not being part of the actual service’s production code. So for us, the services are sometimes black boxes. At AVEQ we’ve seen our fair share of inconvenient behaviors of different video APIs. We’ve had plenty of opportunities to experience that during our recent long-term research projects. We compared over 50 different video services – widely known ones such as YouTube and Netflix, but also smaller ones such as the ones used by regional news portals. We’ve inspected hls.js and dash.js, and all the other players.

After finishing all our implementations, we said “No more!” to custom logic for each service, so we put a lot of effort into designing a unique solution that is capable of handling all of these different behaviors. For the external user, this means you don’t have to do any service-specific adaptations and fiddle around with the player states. No more of this:

const currentUrl = await driver.getCurrentUrl();
if (currentUrl.match(/galleryPage/)) {
  detectPlayingVideo();
  doCustomPlaybackStateAnalysisForGallery();
} else if (currentUrl.match(/singleVideoPage/)) {
  doCustomPlaybackStateAnalysisForSingleVideo();
} else {
  // 1240957 more lines of confusing code
}

Code Example of non-stable custom logic approach – as soon as the website changes, you need to refactor all of this

What else can you measure?

You might ask why it’s better to use the external solution from AVEQ when you could just do the job with two or three lines of JavaScript? Our solution enables you to do even more with video! 

Fetching playout states is a nice feature, but we can also monitor the whole playout in a timeline, where you can see:

• every resolution and buffer change
• quality degradations
• stalling events
• initial loading delay
• performance statistics like framerate, bitrate or playback rate, and, last but not least,
• a single score for streaming quality: a rating for video, audio and the overall user’s impression

The last part is what you saw above in the code example: you can add a check that the playout quality was “excellent” – and this reflects the quality that a user would have rated if he/she had seen the video. Here we rely on an international standard for video quality measurement which delivers a Mean Opinion Score – a rating between 1 and 5 that reflects customer experience.

So to summarize, you can monitor different playouts directly from your test and let them pass or fail if a certain quality level was (not) achieved. You can do this for a full playback from the beginning until the end of the video, or for different time periods using regular checks (which can be useful during live-streaming).

Useful analytics

Just getting the statistics is not all we can do: All videos monitored by our solution – whether you checked them actively or not – are monitored in the background and connected to our analytics dashboard, from where you can perform additional deeper analyses. You can even do this a long time after your test has already finished. This also applies to videos that just pop up somewhere on the web page while you are performing the test.

For example, you can track overall quality and loading delay over time. The left chart shows the Mean Opinion Score over time, with the red line marking a threshold between good/excellent. The right chart shows the average initial loading delay over time. See that spike there? You should investigate that.

You can also directly visualize and compare the quality of different playouts on fully customizable dashboards:

Summing up

You should use Surfmeter for your test automation if you want to:

• assure a certain quality level of your video playout
• detect start failures or unexpected interruptions
• check if the correct videos are playing, or whether a certain video is playable at all
• control an ad to be played at the correct time and if the actual videos resumes afterwards
• compare the loading performance across different browsers or geographical regions
• perform load tests with multiple simultaneous connections/website visits
• find out the effects on your pages’ play-outs, when you do network throttling

Found yourself in one of these scenarios? Contact us – we are happy to help and find the best solution for you.

Finally, we are always investigating new – upcoming – services and checking their compatibility to our technologies, to keep it up-to-date. This ensures high reliability for running our solution in your environments.

Epilogue: did the playing video finally end?

Coming back to the example from the beginning of our story, you might object that the video could have been also paused because the content has already ended. Yes, that could be a possibility. There’s an attribute for that in the W3C API. Here, again, you can’t always rely on the `ended` attribute. There are players which do not set this parameter properly and return to some other state when finishing a video. Or they might set it only from time to time. We’ve seen this and had our own challenges with it when trying to find a universal solution. Save essential development time and headaches for custom efforts by relying on our technology. When will we welcome you?

Contact us to find out more!