(This post is part 1 of a small series on our development work.)
Imagine you are developing a video platform – you want to ensure that your platform works well for all users and that all assets are playable at their highest quality. Normally you’d do this via integration testing of your website – a rather complex task involving tools like Selenium and writing a lot of custom code.
AVEQ’s Surfmeter solution has always enabled video providers to measure the quality of their playouts. Now we’re proud to present our newest solution: video analytics for browser automation testing! With zero configuration needed, all you need to do is add our Surfmeter Test Automation plugin to your Selenium-based tests.
So what makes our solution unique? Playout states are one crucial aspect of video playback – is the video playing or not? –, but we noticed that gathering the state sometimes fails, for instance if the player blocked some important events. In that case, you need to rely on additional information and heuristics. At AVEQ we’ve dug into the inner workings of dozens of video services over the past projects to ensure that we can reliably estimate playout state. If you’re interested in our lessons-learned, this summer we gave a talk on such common pitfalls in gathering quality data for browser-based video sessions.
With Surfmeter, you can get a good overview of what happened during the video session, like shown in the below image:
You will know exactly how long it took for a video to load or what resolution was played for how long.
With our test automation tool, we’re proud to present a unique solution that can monitor all playout states reliably, and deliver all details you’d want to know about the overall video streaming performance and quality: You want to detect start-up failures? Surfmeter can do it. You want to check if the right ad is playing and the video resumes afterwards? We got you covered. You want to find out how a real-world user would have rated the overall playout quality? Our single score for streaming experience – the Mean Opinion Score (MOS) – is the answer.
The below graphic shows you the direct output of a video quality test, with all relevant key performance indicators and the overall MOS score.
Since the results are directly available in the test environment, you can make decisions on successful/failing tests right away. Like all of our Surfmeter products, the test automation library also comes with a connection to our analytics dashboard, from where you can perform deeper analyses.
If you’re interested in the details, our CEO Werner Robitza made a video about the whole process here.
Get in touch with us if you’re interested in doing automation tests with video. We’re happy to talk with you about your use cases and find potential for improving your test coverage.
Stay tuned for part 2, in which we’ll explain in more detail what happens under the hood of our library!