Improving customer experience in the Internet of Things with end-to-end testing
If you’re developing an Internet of Things device, then you’ll appreciate the challenge of ensuring product features across hardware, firmware, networking, and cloud software. Your customer cares about a working feature regardless of outstanding unit test coverage or the most agile of development efforts.
In this post, we’ll cover end to end testing for Internet of Things and how to adapt this method to make your customer’s experience better. Real world interfaces such as sensors and actuators are key missing ingredients to the software-centric end to end testing methodology.
Let’s consider an Internet of Things device with radio connectivity that can actuate a relay. Such devices are pervasive across industries, for example the Smart Home vertical includes thermostats, irrigation controllers, and security panels. A relay is one of the most basic components in control systems and the Internet of Things.
Now, consider the most basic functional validation: did the relay turn on? Simple, right? A control command starts at an app, or API, and traverses a network then down through electronics to eventually apply a control voltage to actuate the relay. Before marking that smoke test a “pass”, consider a few real world aspects to up level your end to end testing.
Include physical interfaces in end to end testing
Logical state of physical output in the firmware or software is just that, unless the electronics have separate, dedicated sensing. Test results using logical state for real world interactions, including during end to end testing, can be misleading or downright inaccurate especially in embedded, resource constrained devices.
Let’s look at the relay trigger voltage as example. Yes, welcome back to the analog world. An electromechanical relay has a design specification for minimum input voltage to physically actuate the components.
Hopefully the hardware is designed properly to handle sagging bus voltage when that OLED is shiny bright and the radio is talking loud. Or robust power management when batteries are running low and nearing threshold voltages. The real world is full of ways that relay may receive insufficient voltage.
So what? Logical and physical states may be different under just the right conditions where relay trigger voltage is influenced by real world product usage. This is why end to end testing that uses dedicated sensing for real world interfaces is important for Internet of Things.
Leverage end to end testing throughout development
If we look to software testing best practices, end to end testing includes beginning to end application flow, and examines real world scenarios.
The continuum of test methods is well documented where testing progresses through unit, integration, smoke, system, end to end, and user acceptance testing. Smoke testing is often contrasted to end-to-end testing in terms of thoroughness, where smoke testing often contains fewer less formal tests. And, in contrast to user acceptance testing, end to end testing is often performed by the development team rather than the customer.
While intuitive, this progression becomes less effective in the context of Internet of Things.
Let’s come back to the example, and consider a common issue of dueling subsystems in the hardware.
Did the smoke test exercise the functionality in a real world manner to build confidence for user acceptance, ensuring logical relay state and physical relay state are indeed the same? An operating system may not even be viable, instead relying on loop controllers which are notoriously difficult to debug. A common design issue is control logic dependent on timing rather than state which can lead to inconsistent output, and in this case, divergent logical and physical states of the relay.
Did unit testing give you the confidence of subsystem interactions representative of a misbehaving user (whom we love, of course) banging on the device? Radio stacks, and networks, do not always play nicely with host processors. And users sure do love to bang on a device to get it to work, potentially leading to radio buffer issues and timeouts. Often you’ll find designs that strive for good user experience under these conditions by providing logical responses quickly, but where the physical response may be delayed or in the worst case get dropped when the buffer overflows.
Building in end to end testing earlier is an effective way to mitigate trouble areas at the intersection of hardware and software.
The cost of not doing real world end to end testing
In this post, we’ve looked at end to end testing in the context of Internet of Things.
Hardware deployed at scale in the field is awfully difficult to fix even if it is remotely upgradeable, unlike pure software development.
During your Internet of Things development project, consider adapting the software-centric methodology for end to end testing in order to improve product quality and customer experience.
Extend end to end testing to the real world interfaces of your devices.
Leverage end to end testing throughout the testing process.
If you’re already running end to end tests, then consider test automation. Knilb is a software platform that automates end to end testing for Internet of Things devices by using connected test equipment.
Interested in end to end test automation with Knilb? Send us an email for a 1:1 demo.