One test, a real device, and an off-the-shelf monitor on its physical interface¶
A connected product reports its own state, and most of the time nobody checks that report against the physical interface. Doing it by hand is slow, and doing it automatically usually means a custom harness that one person understands. So the software is taken at its word, until the field says otherwise.
This is what checking it looks like on a real bench, filmed in real time with no cuts.
The bench¶
Two parts, each reached through its own API contract. Neither is special to Knilb: each is registered as a participant and cast into a role, and the plan never names the hardware.
- The device under test: a Wi-Fi thermostat, chosen because it switches a fan relay and reports the fan's state over its API; any device with a physical output works the same way
- The test equipment: an off-the-shelf networked I/O monitor, with an input wired to the device's fan terminal, that reports over its own API whether the circuit actually closed
The test¶
One case in three ordered clauses, seven steps in all.
- A known starting state: set the thermostat to off with the fan on auto, and confirm the fan circuit is open at the wire
- The fan call and the wire: turn the fan on, confirm the thermostat reports it running, and confirm the circuit closed at the wire
- Back to off: return the thermostat to off with the fan on auto, and confirm the circuit opened again
Two runs, one change¶
The same test ran twice. Between the runs one thing changed on the bench: the sense wire from the fan terminal to the monitor was pulled. Nothing changed in the software.
| With the wire connected | With the sense wire pulled | |
|---|---|---|
| The device accepts the fan call | Yes | Yes |
| The device reports the fan running | Yes | Yes |
| The monitor sees the circuit close | Yes | No |
| The run | Passes, all seven steps | Fails at "Fan relay closed at the wire" |
Software alone would call both runs a pass. The only witness to the difference is the input light on the monitor, which the test reads through the monitor's API.
The answer¶
The result names the failing step, what was expected, and what was observed. Every step is badged with the layer that answered it, here the real device rather than a mock. The first failure ends the case, so the cleanup steps are recorded as not reached instead of adding noise, and the record is frozen, so the same answer is there when you reopen it.
What this shows¶
Test equipment takes part in a test like any other participant, so the question of whether it all works together is answered at the wire rather than inferred from what the software reports. This bench has two parts, and the device happens to be a thermostat; the pattern is the same for any device with a physical output. The same model puts a cloud service and a gateway in the same plan, alongside the device and the equipment that watches it.
Work with us¶
If your product has a gap between what the software says and what the hardware does, tell us what you are building.