# Why generated SDKs need conformance tests

> Why an SDK generated from a specification can still be wrong, and what a conformance suite must cover before anyone builds on it.

Published 2026-10-24 by Maxint. Written for: Engineers, AI agent builders.
Canonical: https://foundry.maxint.com/resources/why-generated-sdks-need-conformance-tests

Generating a client library from a specification is fast and, for the parts the specification gets right, reliable. The problem is the phrase "the parts the specification gets right". A generated SDK is only as truthful as its input, and the input is usually a document written by people describing what they meant to build.

The SDK compiles. The types look sensible. The first call works. None of that tells you whether the SDK matches the system. Conformance tests do.

## What a generated SDK inherits

An SDK generator takes a specification and produces code that trusts it completely. Every assumption in the specification becomes a hard-coded assumption in the client:

- A field marked required is validated before the request is sent. If the server does not actually require it, callers are blocked for no reason. If the server requires a field the spec omits, the SDK sends invalid requests and the error arrives from the server, late and unclear.
- An enum with four values rejects the fifth value the server started returning last quarter.
- A response schema that says `amount` is an integer breaks when the server returns `"12.50"`.
- An error model with three documented codes gives callers no way to handle the undocumented fourth.

None of these are generator bugs. They are specification errors made executable.

## What the specification usually gets wrong

Specifications drift from systems in predictable ways, and each one is a test case.

- **Required versus optional.** The most common discrepancy. Documentation marks fields required because the designer intended them to be; the server accepts their absence because validation was never added.
- **Nullability.** Fields that are "always present" until a particular code path leaves them null.
- **Numeric types and precision.** Money as float, IDs as numbers that overflow, timestamps in three formats across endpoints.
- **Pagination.** The cursor field name, what an empty page looks like, whether the last page includes a cursor.
- **Error shapes.** The documented error envelope, and the HTML page the load balancer returns on a timeout.
- **Idempotency and retries.** Whether a repeated request is safe, which header controls it, and for how long.
- **Rate limits.** The documented limit per key, and the actual limit per account.

A generated SDK makes a confident decision about each of these. The question is whether the decision is right.

## What a conformance suite is

A conformance suite is a set of tests that run the SDK against the real system, or a faithful recording of it, and assert that the SDK's behavior matches the system's. It is not the SDK's unit tests, which test the SDK against itself, and it is not the server's tests, which test the server against its own intentions.

For each operation the suite should establish:

1. **Request shape.** The SDK sends what the server accepts. Minimal requests, maximal requests, and each optional field alone.
2. **Response parsing.** Real responses, including unusual but valid ones, deserialize without loss. Fields the SDK does not know about do not cause failures.
3. **Error handling.** Each error the server actually produces maps to something the caller can catch and distinguish.
4. **Edge semantics.** Pagination to the end, empty collections, idempotent retries, timeouts.
5. **Negative cases.** Requests the SDK's types permit but the server rejects. These reveal constraints the specification left out.

Each passing test turns a specification claim into a verified finding. Each failing test is a discrepancy to resolve, either in the specification or in the understanding of the system.

## A small example

Suppose the specification says:

```yaml
CreateOrderRequest:
  required: [customer_id, items, currency]
  properties:
    coupon: { type: string }
```

A conformance test sends a request without `currency`. If the server accepts it and defaults to USD, the specification is wrong and the generated SDK has been rejecting valid calls. A second test sends `coupon` with a non-USD currency. If the coupon is silently ignored, that is a behavior the SDK should surface, perhaps as a documented limitation, perhaps as a warning. Neither fact was in the specification. Both now are, with a test behind them.

## Keeping the suite honest

- Run it against the system the SDK targets, in an environment that behaves like production. Mocks test your beliefs.
- Record responses so the suite can also run offline, but refresh the recordings on a schedule. Recordings age the same way specifications do.
- Treat an unknown field in a response as a finding, not noise. The server is telling you something.
- Version the suite with the SDK. A conformance result without the commit it applies to is a rumor.
- When the server changes, run the suite before anything else. A failing conformance test is the cheapest possible way to learn about an upstream change.

## When the SDK is for an agent

If the consumer of the SDK is an AI agent rather than a developer reading documentation, conformance matters more, not less. A developer notices when a call behaves oddly and reads the source. An agent retries, reinterprets, or proceeds. The SDK is the agent's only description of the world, so every discrepancy between SDK and system is a discrepancy between what the agent believes and what happens. The conformance suite is how you know those are the same.

## Where Foundry fits

Foundry reconstructs the interface from how a system really responds, generates the SDK from that reconstruction, and ships the conformance tests that prove each behavior, so anything untested is marked observed, inferred or open rather than assumed. How verification works is on the [approach page](/approach).