Skip to content

Physical interface verification on a real bench

A connected product reports its own state, and most of the time nobody checks that report against the physical interface. Doing it by hand is slow, and doing it automatically usually means a custom harness that one person understands. So the software is taken at its word, until the field says otherwise.

Four short videos show what checking it looks like on a real bench. The first sets the method: one device, one piece of test equipment on its physical interface, and one test. The other three keep the method and change the test equipment: swapping it, adding a second, and building it yourself.

One test, a real device, and an off-the-shelf monitor on its physical interface

The first bench, filmed in real time with no cuts.

The bench

Two parts, each reached through its own API contract. Neither is special to Knilb: each is registered as a participant and cast into a role, and the plan never names the hardware.

  • The device under test: a Wi-Fi thermostat, chosen because it switches a fan relay and reports the fan's state over its API
  • The test equipment: an off-the-shelf networked I/O monitor, with an input wired to the device's fan terminal, that reports over its own API whether the circuit actually closed

The test

One case in three ordered clauses, seven steps in all.

  • A known starting state: set the thermostat to off with the fan on auto, and confirm the fan circuit is open at the wire
  • The fan call and the wire: turn the fan on, confirm the thermostat reports it running, and confirm the circuit closed at the wire
  • Back to off: return the thermostat to off with the fan on auto, and confirm the circuit opened again

Two runs, one change

The same test ran twice. Between the runs one thing changed on the bench: the sense wire from the fan terminal to the monitor was pulled. Nothing changed in the software.

With the wire connected With the sense wire pulled
The device accepts the fan call Yes Yes
The device reports the fan running Yes Yes
The monitor sees the circuit close Yes No
The run Passes, all seven steps Fails at "Fan relay closed at the wire"

Software alone would call both runs a pass. The only witness to the difference is the input light on the monitor, which the test reads through the monitor's API.

The monitor during the first run: the green input light is on, because the fan circuit closed.
First run: the input light is on.
The monitor during the second run, with the sense wire pulled: the input light stays dark.
Second run: the input light stays dark.

The answer

The result names the failing step, what was expected, and what was observed. Every step is badged with the layer that answered it, here the real device rather than a mock. The first failure ends the case, so the cleanup steps are recorded as not reached instead of adding noise, and the record is frozen, so the same answer is there when you reopen it.

The Knilb result for the second run: setup passed, the step "Fan relay closed at the wire" failed with the expected and observed values, and the teardown steps were not reached.

Swap the instrument, keep the test

Test scripts are often tied to one instrument, so moving to another bench or to less expensive equipment means rewriting them. Here the same thermostat has a second instrument on its fan wire: a small single-board computer that answers the API contract of the off-the-shelf monitor from the first bench, a web relay, exactly as it does.

The test has already passed with the web relay in its monitor role. The computer is given the web relay's interface specification, the web relay is removed from the role, and Knilb offers the computer for it, because both keep the same contract. Nothing is rewired and no step changes. The same test runs again and passes, and each result keeps the address of the device that filled every role, so it's clear which instrument answered.

A second bench can use less expensive equipment, and a failed instrument can be replaced without touching the test plan.

Two instruments, one test, one answer

Many tests need more than one instrument, and each usually comes with its own software, so someone puts the answers together by hand after the run. Here the thermostat has two pieces of test equipment on different wires: the web relay on the fan wire, and a small single-board computer on the heat and cool wires.

The test has three roles, one for the thermostat and one for each instrument, and every step names a role rather than a device. The test turns the fan on, and the web relay confirms the fan wire closed. It then calls for heat, and the computer confirms the heat wire closed while the cool wire stayed open. At the end, the computer has counted exactly two changes on the heat wire, one clean heat cycle, and the run passes.

The result is one answer for the whole bench, with the steps from all three devices in one list and the address of the device behind each role.

One small computer runs the test and watches the wires

Some behavior only shows up over time. This thermostat protects its compressor: after cooling stops, it holds off for four minutes before cooling again. Checking that by hand means watching a clock and a wire at the same time, often more than once.

Here one small single-board computer with an I/O board is wired to the thermostat's heat, cool, and fan wires, and it is also the client that runs the test, so there is no other computer on the bench. The test ends a cooling call and asks for cooling again at once. While the protection holds, the computer checks once a minute that the cool wire stays open; once it releases, the same check finds the wire closed. The computer also counted three changes on the cool wire, so the cooling call never came early. The run took about fifteen minutes on the bench, and the long waits are sped up in the video.

When the instrument a bench needs doesn't exist, a small computer with the right inputs can be built to answer the test, and Knilb runs it as one more role in the plan. The relay monitor used here and the web relay stand-in from the swap are public examples you can rebuild.

What this shows

Test equipment takes part in a test like any other participant, so the question of whether it all works together is answered at the wire rather than inferred from what the software reports. How to test a connected product end to end lays out the method behind these benches, part by part. The device happens to be a thermostat. Across the four benches, the test names roles, not boxes: an instrument can be swapped without changing a step, two instruments can watch one device in one run with one answer, and an instrument you build yourself is one more role. Test equipment that exposes its own API is cast into a role the same way, and the same model takes in any other part of the system that has an API.

Work with us

If your product has a gap between what the software says and what the hardware does, tell us what you are building.

Become a design partner