Now arriving in your Test Suite: Automation Testing for Video Playback

(This post is part 1 of a small series on our development work.)

Imagine you are developing a video platform – you want to ensure that your platform works well for all users and that all assets are playable at their highest quality. Normally you’d do this via integration testing of your website – a rather complex task involving tools like Selenium and writing a lot of custom code.

AVEQ’s Surfmeter solution has always enabled video providers to measure the quality of their playouts. Now we’re proud to present our newest solution: video analytics for browser automation testing! With zero configuration needed, all you need to do is add our Surfmeter Test Automation plugin to your Selenium-based tests.

Why can’t you “just” measure a video session?

Why is this a novelty for the video industry? All you want is an overview of what really happened during a session – that shouldn’t be too hard. But if you’ve ever tried searching for common methods to track a video’s playout state during an automation test with frameworks like Selenium, Stack Overflow and other relevant Internet Q&A sites usually deliver vague answers such as “Are you doing the test for YouTube? They have some API for that.” or “There is an API from W3C; you need Javascript knowledge for that!”. Such responses aren’t entirely helpful.

So what makes our solution unique? Playout states are one crucial aspect of video playback – is the video playing or not? –, but we noticed that gathering the state sometimes fails, for instance if the player blocked some important events. In that case, you need to rely on additional information and heuristics. At AVEQ we’ve dug into the inner workings of dozens of video services over the past projects to ensure that we can reliably estimate playout state. If you’re interested in our lessons-learned, this summer we gave a talk on such common pitfalls in gathering quality data for browser-based video sessions.

With Surfmeter, you can get a good overview of what happened during the video session, like shown in the below image:

You will know exactly how long it took for a video to load or what resolution was played for how long.

Reliable Monitoring with our unique Streaming Quality Score

With our test automation tool, we’re proud to present a unique solution that can monitor all playout states reliably, and deliver all details you’d want to know about the overall video streaming performance and quality: You want to detect start-up failures? Surfmeter can do it. You want to check if the right ad is playing and the video resumes afterwards? We got you covered. You want to find out how a real-world user would have rated the overall playout quality? Our single score for streaming experience – the Mean Opinion Score (MOS) – is the answer.

The below graphic shows you the direct output of a video quality test, with all relevant key performance indicators and the overall MOS score.

Since the results are directly available in the test environment, you can make decisions on successful/failing tests right away. Like all of our Surfmeter products, the test automation library also comes with a connection to our analytics dashboard, from where you can perform deeper analyses.

Would you like to see more?

If you’re interested in the details, our CEO Werner Robitza made a video about the whole process here.

Get in touch with us if you’re interested in doing test automation with video. We’re happy to talk with you about your use cases and find potential for improving your test coverage.

Stay tuned for part 2, in which we’ll explain in more detail what happens under the hood of our library!