Testing software is hard. Testing software and hardware together is even trickier. In this article, we’ll discuss the various approaches to quality control (QC) that we’ve tried at Screenly over the years. It has been an interesting journey, and each iteration of our setup has taught us new things.

QC Rig v1

Back in 2018, we built our first dedicated test rig. When we took this live, we wrote an article with photos and details. The setup was very primitive, and we used one of the “tiles” from Balena’s The Beast to fit as many Pi 3s as possible into a small amount of space.

This setup served us well for a while, but we identified two issues with the setup:

• The screens were not connected to any displays, making it hard to diagnose visual issues.
• We weren’t able to easily add more devices to the setup physically.

As we started to move to Pi 4 in 2020, we decided it was time to overhaul our QC setup.

QC Rig v2

In the next iteration of our QC Rig, we took an entirely new approach. Instead of running it ourselves, we reached out to the fine folks over at Mythic Beasts to see if they could host this for us.

For those of you not familiar with Mythic Beasts, these are the guys who host the Raspberry Pi website (on Raspberry Pis), among other things. What they do is pretty impressive. They’ve managed to build a custom rackmount case (3U) that fits no fewer than 48 Raspberry Pis. In addition, they also allow you to individually power cycle devices through their web interface.

After a number of conversations with Mythic Beasts, we decided to work with them to deploy a custom ‘Screenly’ cluster that sat directly in one of their data centers.

With almost 4x the number of Raspberry Pis as the previous version, we were able to catch a number of issues before they affected customers.

While the setup certainly helped us scale up the number of devices we ran on, we still had an issue with a certain category of bugs that was hard to catch without a physical screen.

QC Rig v3

In 2023, it was time again to revisit our QC rig. By now, we had added yet another hardware player (Screenly Player Max) that needed to be covered in our test rigs.

This time, we decided that we really wanted a real physical screen for each device. Given how much space this took up, we had to rent an office space just to host all these screens (remember, we’re a remote-only company). The initial deployments consist of 12 screens, spread across various brands of monitors and resolutions. To increase the test surface, we also decided to distribute the devices equally between Ethernet and WiFi.

Now, setting screens that show content on a loop is a great test, but we wanted to push our devices even further. To do that, we created automated tests that randomly change the content for all of these screens every hour using our GitHub Actions plugin.

In order to get even more insights, we are also using Prometheus’ Node Exporter to expose various metrics from the devices to better monitor changes in both network and performance over time.

Finally, we’re also using our PagerDuty Integration to raise alerts (via email, mobile and Slack) if one of our test devices goes offline.

Going forward

Having 12 screens allows us to have multiple monitors for each of our supported hardware platforms (Pi 3, Pi 4 and Screenly Player Max) for both staging and production environments. When building the setup, we ensured that we had enough physical real estate to scale up our QC Rig further with additional hardware platforms, or just adding more of the existing devices.