Foundry — home

Resources

What a useful software behavior report should contain

The sections, evidence labels and level of detail that make a behavior report usable by engineers, not just readable by managers.

A behavior report answers one question: what does this system do, and how sure are we? Many documents called behavior reports answer a different question, such as how the system is structured or what its authors intended. Those are useful. They are not the same thing.

A report is useful when an engineer can act on it the day it arrives, and when a second engineer, a year later, can tell which parts to trust. This article lists what that takes.

Open with the question and the scope

The first page states what the report was commissioned to answer and what was in scope. Without this, readers cannot tell whether a missing topic is a gap or a boundary.

  • The decision the report supports. “Can we replace the pricing service without changing order totals?”
  • The systems, versions and environments examined. A commit hash or build identifier, and whether the running system was staging or production.
  • What access the investigation had. Source code, a running instance, traffic captures, database read access, none of these.
  • The dates of the work. Behavior changes; the report describes a moment.

One finding per line, each with its evidence

The body is a list of findings, not prose. Each finding is a single testable statement with a label and a pointer to its evidence.

A finding reads like this:

POST /v2/orders returns 409 when the ref field repeats within 24 hours. Verified. tests/orders/test_ref_dedup.py.

Not like this:

The orders endpoint has duplicate protection built around a reference field, which appears to be time-boxed.

The second version buries three claims, labels none of them, and cannot be checked.

The labels are the same four used throughout the investigation:

Label Meaning What the evidence pointer should be
Verified A repeatable test confirms it The test, and where it ran
Observed Seen directly, not yet tested The request, log, trace or query
Inferred Reasoned from evidence The observations it rests on
Open Unknown, and said so What was tried, and what would resolve it

A report with no open findings is a warning sign, not a comfort.

Group by behavior, not by file

Readers come to the report with a behavior in mind: “what happens when a coupon is applied”, “what gets written when an order is cancelled”. Organize by those behaviors, or by interface, so that everything about one topic sits together. A structure that mirrors the repository layout forces readers to assemble the picture themselves.

For each interface, a consistent set of sub-sections helps:

  • Inputs the system actually requires, versus what the documentation claims.
  • Outputs and their shape, including error responses.
  • Side effects: tables written, messages published, external calls made.
  • Ordering and timing: what happens first, what is asynchronous, what retries.
  • Limits: sizes, rates, truncation, timeouts.

Separate the documented from the discovered

Mark each finding as documented or not. The gap between the two is often the most valuable page in the report. A field that the documentation never mentions but the system requires is a trap for the next integrator. A documented behavior that the system no longer has is a trap for anyone relying on the docs.

Where documentation exists, quote it next to the finding rather than paraphrasing. Readers can then judge the discrepancy for themselves.

Include the dependency map

A behavior report without a dependency section leaves out half of what readers need. For each dependency, record:

  • Direction: does the system call it, or does it call the system?
  • Protocol and interface: HTTP, queue topic, shared table, file drop.
  • How it was discovered, with a label. A configuration value is observed; a hostname pattern in a log is inferred.
  • What happens when it is unavailable, if that was tested.

A diagram helps, but the table is the source of truth. Diagrams drop detail.

State limitations plainly

A section titled “What we could not confirm” belongs in every report. It lists:

  • Behaviors that were out of scope.
  • Behaviors that could not be tested safely, and why.
  • Environments not examined. Behavior in production can differ from staging.
  • Assumptions about data, configuration or traffic that the findings depend on.

This section protects the reader more than any other. It also protects the author.

End with next steps, tied to findings

Recommendations should reference findings by identifier. “Add a conformance test for finding 14 before migrating” is actionable. “Improve test coverage” is not. Keep recommendations short and separate from the findings; a reader who disagrees with a recommendation should still be able to trust the evidence.

A short checklist

Before a report is handed over, check that:

  • The question and scope are on the first page.
  • Every finding is one statement, labeled, with an evidence pointer.
  • Documented and discovered behavior are distinguished.
  • Dependencies are listed with direction and discovery method.
  • Open items and limitations are explicit.
  • The tests referenced exist and can be run by the reader.

Where Foundry fits

Foundry Intelligence delivers a behavior report in exactly this form: each finding labeled verified, observed, inferred or open, a dependency map, a reconstructed interface specification, the conformance tests behind every verified item, and a written list of limitations. The package is described on the Foundry Intelligence page.