Foundry — home

Resources

How to define an effective software intelligence engagement

What to pin down before anyone analyzes a system for you: the decision, the authorized target, data handling, success criteria, deliverables and limits.

Analysis of a system you depend on is only useful if it changes a decision. An engagement defined as “help us understand the platform” produces a long document and a longer invoice. One defined as “tell us which behaviors of the platform must survive the migration, with evidence” produces something a team can act on. The difference is in the definition, and the definition is written before any analysis starts.

What follows is the set of things to settle, in the order they tend to depend on each other.

Start from the decision

Name the decision the analysis will inform, and who will make it. Examples that work:

  • Whether to wrap, replace or keep the claims engine, by the end of the quarter.
  • Whether an internal agent may be granted write access to the scheduling API.
  • Whether the SDK we generate for a partner’s API can be released to customers.
  • Which behaviors of the order service must be preserved in the rewrite.

Each of these has an owner, a deadline and a shape the answer must take. “Understand the system” has none of those. If a decision cannot be named, the engagement is premature. Do the measuring first.

Define the authorized target precisely

Write down exactly what may be analyzed, in terms that leave no room for interpretation.

  • Systems and components. By name, environment and version. “The order service in staging, build 4.12, and the read replica of its database” rather than “the order platform”.
  • Boundaries. Which upstream and downstream systems may be observed, called or exercised, and which may not be touched at all. A third-party API that charges per call, or a partner system you do not own, needs explicit treatment.
  • Authority. Who, in your organization, has the right to authorize analysis of each item, and confirmation that they have. For anything owned by another party, their written permission.
  • Methods permitted. Reading source, observing traffic, sending test requests, executing code, load. Each is a different level of intrusion and some will not be acceptable on some targets.

Nobody should analyze a system on the basis that it was probably fine to.

Settle access, sensitivity and data handling

This is the part most often left to the week before work starts, and the part most likely to delay it.

Access. What credentials, network paths, source repositories and environments will be provided, by whom, with what scope and for how long. Prefer dedicated, revocable credentials with the minimum scope the methods require.

Sensitivity classification. Classify what will be exposed: source code, production data, personal data, credentials, trade secrets, regulated records. The classification drives everything that follows.

Data handling. Agree in writing, before anything is received:

  • What is received, and in what form. Source, binaries, logs, traffic captures, database snapshots.
  • Where it is processed. Which environment, which jurisdiction, under whose control.
  • Who can see it. Named roles, and whether subcontractors are involved.
  • How long it is kept, and what “kept” includes: copies, caches, derived artifacts, test fixtures built from real data.
  • How and when it is deleted, and how deletion is confirmed.
  • What the analyst may retain afterwards. Usually the deliverables and nothing else, but say so.

If the answer to “where is it processed” is unacceptable for the sensitivity involved, that is the moment to find out. Some organizations resolve it by running the analysis inside their own environment rather than sending anything out.

Write success criteria the deliverable can be checked against

Success criteria turn a decision into a test for the deliverable. They should be specific enough that two people could disagree about whether they were met and settle it by looking at the output.

  • Every public endpoint of the target is listed with method, parameters, response shape and status codes.
  • Every behavior marked relied-upon in the inventory has a conformance test that passes against the current system.
  • Every outbound dependency is named with owner, protocol and observed failure mode, or marked open with a reason.
  • Open items are enumerated, not summarized.

Avoid criteria that depend on completeness that cannot be demonstrated. “All behavior is documented” cannot be checked. “All behavior reachable through the listed interfaces under the listed inputs is documented, and the scope section states those limits” can.

Agree the deliverables and how findings are labeled

List the artifacts by name and form. A typical set, adjusted to the decision:

Deliverable Form Answers
Behavior report Document, finding by finding What the system does
Interface specification OpenAPI or equivalent What it exposes
Dependency map Diagram plus structured data What it relies on
Conformance tests Runnable suite Which findings are verified
Limitations Document What could not be established
Recommendations Document What to do next about the decision

Then agree the labeling rule, because it determines how much of the report you can act on. Every finding should carry one of four states: observed directly, inferred from evidence but not tested, verified by a repeatable test, or open. The engagement should state which findings must reach verified to meet the success criteria, and accept that some will stay open.

State the limits up front

An engagement that does not list what it will not do will be assumed to have done it. Write the exclusions into the definition.

  • It is not a security assessment and will not certify the system as secure.
  • It covers the environments, inputs and time windows listed, and nothing else.
  • It does not establish whether behavior is correct or desirable, only what it is.
  • It does not grant or imply authorization for any system to act on the target.

These protect both sides. They also make the report more credible to the people who have to rely on it.

Decide whether the analysis should recur

Some decisions are made once. Many are not. If the target changes regularly, or if an integration, SDK or agent will depend on its behavior over time, a single analysis goes stale at the first deployment. Agree at the outset whether there will be re-analysis, on what trigger, and what it produces: typically a change report naming what moved, what is new, and which verified behaviors were re-tested. A recurring arrangement with a fixed scope is a product. A series of one-off requests is a queue of bespoke projects.

Keep it from becoming open-ended consulting

Four rules hold an engagement to its definition.

  1. Scope changes are written, priced and dated. New questions are welcome. They become a new scope line, not a quiet extension.
  2. Expert time is bounded. Where human investigation is needed beyond the automated analysis, cap it, and report how much was used.
  3. The deliverable is checked against the criteria, not against satisfaction. Review the output against the success criteria agreed at the start. Disagreements about whether it is good enough should be resolved by the criteria.
  4. Open items end the engagement, not extend it. Anything still open at the end is listed, with what it would take to close it. Closing it is a decision for you, made with the report in hand.

Where Foundry fits

Foundry Intelligence is built around this definition: a scoped objective and authorized target, data-handling terms agreed in writing first, and a defined package of behavior report, interface specifications, dependency map, conformance tests, limitations and next steps, with every finding labeled observed, inferred, verified or open. Where the work has to happen inside the customer’s own infrastructure, it is scoped as a controlled-environment engagement once feasibility and data flows are confirmed. How an engagement runs sets out the steps.