How to test a connected product end to end¶

Testing a connected product end to end means one test that reaches every part of the product through its API contract, from the cloud service to the device, with test equipment watching the device's physical outputs at the wire, and one answer for the run. End to end, here, runs from the first API call the test makes to the last physical output it can observe.

Most teams reach this question from the same place. Every part passes its own tests: the firmware has its unit tests, the cloud service has its API tests, the app has its UI tests. What's left is the question no single suite owns: does it all work together?

What component tests can't tell you¶

Component testing checks each part against its own contract, and each team signs off on its own part. System behavior is what happens where the parts meet: the cloud service, the gateway, the device, and the physical interface the device drives. None of the component suites is looking there.

The failures that live at those boundaries tend to look like this:

• A device reports a state its physical output never reached. The command was accepted and the API says the fan is running, but the fan circuit never closed.
• Steps pass on their own and fail in sequence, because one part depends on a state another part left behind.
• A test quietly stops touching real hardware, because a substitute answered in its place and nothing said so.

The bench story shows the first of these on a real bench. The same test ran twice, and between the runs one sense wire was pulled from the device's fan terminal. Both times the device accepted the fan call and reported the fan running. Software alone would call both runs a pass. Only the test equipment on the wire saw the difference.

In many teams this check is still manual. Someone sets up the bench before a release, works through a checklist, and reads the results by eye, and one person knows how the rig works. That isn't carelessness. The parts belong to different teams, every bench is custom, and a harness that spans all of it is real work to build and keep running.

Draw the system you're testing¶

Before writing a step, decide what's in the test and how the test reaches each part.

Start by listing the parts. A typical connected product has a cloud service, often a gateway, the device itself, and, for anything the device does physically, test equipment that can observe it. Each part needs a way in, and the most useful one is its API contract: an OpenAPI file that says which operations the part offers, what they take, and what they return. A part that already has a contract can be tested from it. For a part that doesn't, writing one is usually the first task.

Then decide where the test reads each answer. Some answers can be read at a part's API: the command was accepted, the device reports its new state. Others can only be read at the wire, by test equipment observing a physical output. Naming that boundary before the first step is what keeps "end to end" from meaning whatever the team happened to check.

Write the test against roles, not machines¶

A test that names a device's address is welded to one bench. When the hardware moves, multiplies, or isn't there yet, the test needs rewriting.

In Knilb, a plan declares a named role for each part instead, each an alias paired with a contract, such as dut for the device under test and ate for the test equipment. Steps target roles, never devices. A deployment then casts a concrete participant into each role, a real device or a mock that answers from the contract, and picks the runner that executes it. Because no step names hardware, the same plan runs against mocks before the hardware exists and against the real bench later, without editing a step. The testbed model explains the separation in full.

A mock earns its place in a system test only if it can't leak. In Knilb, a mocked role never reaches the real device: even when no mock response matches, or its mock profile has been deleted, the step fails visibly instead of falling through to the hardware. A mock answers what the contract declares. It stands in for the device's API, not for how the device behaves.

Build each step from the part's contract¶

Steps typed by hand from an API's documentation drift from the API: a path changes, a parameter is renamed, and the test fails for reasons that have nothing to do with the product.

In Knilb, a step is picked from what the contract declares: the operation, its parameters, the expected status, the response fields, and the headers. Authoring becomes selection rather than transcription, so a whole class of typo-shaped failures rarely reaches a run. Authoring a plan walks through it.

Checks follow the field's declared type. Numbers compare as numbers, so 3, 3.0, and "3" all match 3. Strings compare as exact text, and true or false match only real JSON booleans.

Start from a known state, and stop at the first failure¶

A system test is only as readable as its starting point. Organize each case into three ordered clauses:

The bench story's case is a small example: seven steps that set a known starting state and confirm the fan circuit is open at the wire, turn the fan on and check both the device's report and the wire, then return the device to off.

Then stop at the first failure. In Knilb, the first failing step ends the case, the steps after it are kept in the record as "Not reached", and the result is saved even when the case fails. You read one cause instead of a cascade of failures that all follow from it. Because a failure skips the rest of the case, Teardown included, the next run can't rely on Teardown having happened: Setup has to do the full reset. Runs and results covers how a run moves through a case.

Check the physical outputs with test equipment¶

Some answers aren't in any API the device offers. A device reports what its software believes, and the only way to check what actually happened is to observe it.

So make the test equipment a participant like any other. Test equipment that exposes its own API contract is registered and cast into a role beside the device, and it translates between the wire and the API: a contact closing, a line changing state. The step asserts on what the equipment observed, not on what the device says about itself.

See the test on a real bench: a Wi-Fi thermostat as the device under test, an off-the-shelf networked I/O monitor wired to its fan terminal, and one test that fails at exactly the step where the two disagree. The thermostat is the example, not the point. It was chosen because it switches a relay and reports the relay's state over its API.

Read one answer, and what answered each step¶

With each part reached through its own contract in one plan, the run gives one result: a pass, or a failure that names the step, what was expected, and what was observed.

Every step that ran also carries a badge for what answered it: DEVICE for the real device, or the mock layer that stood in for it, CLOUD PROXY, CLOUD DOWNLOAD, or LOCAL FILE. A run that was meant to touch the hardware and didn't is visible on the result rather than silent. Per-step badges need knilb 0.8.0 and knilb-cli 0.5.0 or newer.

Each result freezes what ran: the outcome, the step log, the case as it stood when the run started, and the deployment as cast, filed together or not at all. The plan can be edited and the bench re-cast afterwards, and the record of that run doesn't change. It freezes the record, not the bench.

Run it the same way every time¶

"I can't reproduce the failure the developer is asking about" is one of the most expensive sentences in system testing. A run can be reproduced when its starting state, its plan, and its cast are the same each time, and when the record of the last run is exact enough to compare against.

Each of those has a place to live. The Setup clause sets the known state. The deployment records which participant or mock fills each role. The frozen result shows exactly what ran and what each step saw. A failure becomes something you can reopen and run again, rather than something to argue about. How a run stays the same shows the pieces together.

Runs start when someone runs knilb run on a runner. The cloud directs each step and the runner reports each outcome, so a runner can be a bench machine or a CI container, with its token and the device credentials supplied through environment variables. Setting up a runner covers both kinds of machine.

The plan, the deployment that says which participant fills each role, and the frozen result of every run are kept in Knilb, rather than in one person's head. Regression testing a connected product covers what that record gives a release decision.

Do you need an IoT test automation framework?¶

Many teams start by building their own: scripts around each part, a script to run them in order, and a shared folder of results. The scripts are the cheap part. What surrounds them isn't: a known starting state, a record of which device sat on which bench, and a record of exactly what ran. That's the part that decays when the person who built it moves on.

So the useful question is less which framework to adopt and more where those things will live. In Knilb they live with the test, as the sections above describe.

The tools you already run keep their jobs. An API client is still the fastest way to try one endpoint. A protocol analyzer still owns the radio, and Knilb checks what the system did once the command got through: it validates behavior, not protocols. Test management still holds the plan of record. Where Knilb sits beside your other tools goes through each.