Engineering Blog

Technical insights from Grid Dynamics engineers

When DSPy Optimization Ships Nothing: Artifact Release Gates

When DSPy Optimization Ships Nothing: Artifact Release Gates

By Aditya Deo Pandey · Sep 21, 2026

A MIPROv2 run compiled cleanly, logged ten trials, and reported that its best candidate scored 24 out of 24 on the development set. The artifact it wrote had the same hash as the baseline it started from — zero demonstrations, unchanged instruction text, identical serialized state. The optimizer had worked correctly and produced nothing. That run turned out to be the most useful result in the study, because it cleanly separates two things teams routinely conflate: DSPy optimization completing, and DSPy optimization producing something worth deploying. A green compile log is evidence of the first and no evidence at all of the second.

What follows is the evidence from two paired experiments — one deliberately saturated, one with real headroom — and the promotion gate they imply. Both ran against DSPy at commit 29448ae1, through a Codex CLI wrapper observed as gpt-5.6-terra — an identifier the wrapper reports, not a public model name. That provider boundary matters and is revisited below: nothing here is a claim about API pricing, portability to a smaller snapshot, or any other vendor.

The saturated control: an optimizer that correctly does nothing

The first program (v1) routes support tickets into four queues from one-sentence requests, with direct lexical cues — a ticket mentioning a refund says "refund." It was compiled with MIPROv2 in light mode over 48 training and 24 development records.

Every candidate MIPRO evaluated scored 24/24 on dev, including the default. With no candidate able to separate itself from the starting program, the optimizer selected the starting program. The serialized JSON came back with the baseline's exact hash, no demonstrations, and no instruction rewrite. On four frozen 40-record holdouts, baseline and "optimized" artifacts produced identical outcomes: 39/40 in-distribution, 40/40 on edge, adversarial, and shifted slices.

This is a no-op, not a success, and it is the cheapest possible failure to detect — a hash comparison catches it before anything reaches a deployment pipeline. It is worth naming the class: objective saturation. When the baseline already ceilings the metric on the development set, the optimizer has no gradient to follow and the artifact diff is empty. Teams that discard these runs as "failed experiments" throw away the control that proves their diff check works.

The broader lesson is about where value comes from. DSPy optimization pays off when the optimization data carries information the runtime model cannot already infer from the request itself. In v1 there was none. Lexical routing made the task trivial, and no state change occurred.

Where DSPy optimization has headroom: information the signature withholds

The second program (v2) routes into six queues and adds a condition the Signature deliberately does not reveal — an external routing policy:

  • charge disputes at or above $500 go to escalations, not billing
  • cancellation inside the 14-day trial window goes to retention, not cancellation
  • SSO credential reset goes to technical_support, not account_access
  • ordinary analogues of all three go to the obvious queue

The baseline instruction says only "Route the support request to exactly one support queue." A capable model routes an ordinary payment or password request correctly without help; it has no way to guess that this organization's threshold is $500, that its trial window is 14 days, or that SSO belongs to the platform team. That information exists only in the training labels. This is the shape of task where optimization has something to do — and it is a narrower shape than "improve my prompt."

Same optimizer settings: MIPROv2 light, seed 9, a maximum of four bootstrapped and labeled demonstrations, exact-match objective. 72 training records, 36 development records, four frozen 48-record holdouts, all synthetic.

Default dev accuracy was 58.33%; the best candidate scored 94.44%. The selected artifact retained the original instruction and materialized four demonstrations: ordinary password recovery, SSO credential recovery, a $2,000 dispute, and a trial cancellation below the 14-day boundary — precisely the exception classes.

ArtifactIDEdgeAdversarialShifted
v1 baseline39/4040/4040/4040/40
v1 MIPRO39/4040/4040/4040/40
v2 baseline31/4831/4831/4830/48
v2 MIPRO39/4842/4841/4840/48

Lift of +8, +11, +10 and +10 cases, persisting on transformed and adversarial slices rather than only on the optimizer's own dev split. That is a real result. It is also a synthetic one: the policy distribution is narrow and the adversarial cases are constructed attack strings, so it supports adoption of v2 under gates, not a claim about production tickets.

The leakage incident: how the first version of this result was wrong

The first v2 generator contained a one-line dead branch. The row builder received split names like optimizer_train and compared them against the bare string train. The comparison never matched, so numeric selection that was supposed to vary by split did not, and every split drew from one shared 12-template pool with a constant per-split suffix.

The duplicate check passed — because it compared full strings, and the suffix differed. A green check on a broken corpus is worse than no check.

The correction derives a split family (optimizer_train → train, optimizer_dev → dev, otherwise holdout), uses disjoint authored template pools and disjoint numeric fillers per family, applies real edge, adversarial, and shifted transforms rather than a suffix, and asserts on duplicate stems after stripping the split suffix — with dev checked against both training and plain-holdout stems.

MetricContaminated runCorrected run
Baseline ID / edge / adversarial / shifted28 / 26 / 26 / 2631 / 31 / 31 / 30
Optimized ID / edge / adversarial / shifted42 / 43 / 44 / 4239 / 42 / 41 / 40
Stem duplicates train → dev/holdoutsuntested (full-string check only)0 by assertion

The original framing of "four independent frozen slices" is rejected for the contaminated run. A follow-up MiniLM screen on the corrected corpus found zero pairs at cosine ≥0.90 and two at ≥0.80 — screening evidence, not proof of semantic independence. The contaminated artifacts were retained on disk as counterevidence rather than overwritten, which is the only way a later reviewer can tell the difference between a corrected study and a lucky one.

What DSPy optimization actually writes, and who owns it

MIPROv2 does not retain a hidden learned model. It deep-copies the program, builds demonstration candidates, asks a prompt model for instruction candidates, scores assembled candidates with Evaluate, and then writes the winning instruction and ordered demonstrations into the predictor's state. The release artifact is text and examples in a JSON file.

Reading the boundaries as an ownership map is what makes the risk tractable. The Signature owns field names, types, and instructions — not prompt-injection resistance, rate limiting, retries, safety policy, or rollback. ChatAdapter owns how demonstrations become messages; it does not choose the model. Predict owns the persistent state that ships. The consequence for change review is uncomfortable: a prompt or policy incident can live entirely in the artifact while the Python source is unchanged. If your review process gates on diffs to .py files, it does not gate on this. Nothing about that split is specific to DSPy: any framework that persists tuned state — prompts, demonstrations, routing tables, adapter weights — to a file the application loads at startup inherits it. Code review covers the loader; nothing covers what gets loaded.

Two probes at the adapter boundary sharpen the point. A malformed completion caused two LM calls, the second parsed by a JSONAdapter retry — format failure silently adds a call, so latency and cost gates must account for it. A controlled context-window error produced one call and propagated; Predict does not shrink or retry its own prompt. And rendering the same artifact through ChatAdapter versus JSONAdapter produced 552 versus 522 estimated tokens with different payloads: an adapter migration is a behavioral change, not a serialization detail.

The demonstrations are load-bearing — but you cannot say which one

If the demonstrations carry the policy, removing them should hurt. Truncating the serialized demo list to strict prefixes:

Retained demosAggregate correctAccuracy
0 (baseline)123/19264.1%
1120/19262.5%
2148/19277.1%
3150/19278.1%
4 (intact)162/19284.4%

The curve is not monotonic — one demonstration is slightly worse than none — which is itself the finding. Policy information is distributed across retained context, and a sequential prefix design cannot attribute a contribution to any individual example.

The mutation tests are where discipline matters most. Removing one demonstration gave 156/192; corrupting one gave 159/192, against 162/192 intact. Deltas of −6 and −3 cases look like degradation. At n=192 and p≈0.84, σ ≈ 5.0 cases (2.6 points). Ten cases (5.1 points) is only the two-sided 95% significance threshold, not a minimum detectable effect — resolving an effect that size at 80% power needs roughly 14 cases (7.3 points). And because all three arms score the same 192 items, the correct instrument is a paired test, McNemar's on the discordant pairs, rather than this independent-binomial approximation. Neither delta is detectable at this sample size, and a paired analysis is what should settle it. Reading them as evidence that demonstrations are robust to mutation would be reading noise.

Run order compounds it. These ran sequentially: intact runs averaged about 6.2–6.6 seconds, removal 6.5–6.9, corruption 6.5–7.1. Since the artifacts differ by exactly one demonstration, that pattern is more consistent with CLI environment drift than with any artifact property — and drift of that magnitude is precisely the size of the observed accuracy deltas. The baseline-to-optimized gap is far too large to be drift; the mutation deltas are not. A powered version of this test needs randomized or interleaved run order and a sample size chosen from the effect it intends to detect. This is the same discipline that objective-validation protocols apply to AI-generated code: a check that cannot detect the failure it screens for is decoration.

Corruption is silent at every layer you would check

The mutation results become alarming when combined with the loader probes.

A JSON artifact with a wrong field name loads successfully. A string supplied where the demonstrations list belongs also loads successfully, leaving demos as a ten-character string. Neither raises. Only a truncated file fails, with a JSONDecodeError. Meanwhile, neither mutation run produced a single malformed output, and accuracy moved less than the noise floor.

Stack those layers: the loader accepts the corruption, the outputs stay well-formed, and the accuracy monitor does not move enough to trip. An accuracy canary is not a corruption detector for this class of artifact. What does work is unglamorous — strict schema validation that requires demos to be a list of records with expected input/output keys and an allowed category set, plus a hash. And not only an artifact hash: a v2 state loads cleanly into the four-class v1 Signature, so the hash must cover program identity — Signature source, field names, types, the allowed category set, DSPy version, model and config, and dataset hashes.

Two more diagnosability defects belong in the same bucket. DSPy 3.2.1 loads artifacts labelled 3.2.0 and logs a mismatch without blocking — the warning is advisory, so post-load tests are mandatory. And MIPROv2's total_calls and prompt_model_total_calls were initialized to zero, copied to the winning candidate, and never incremented; both compiles logged search activity and persisted 0/0. Any cost gate that reads those counters reads zero and passes. Cost gates need independent LM history and must fail closed when telemetry contradicts the logs — the argument for independent agent observability rather than self-reported counters.

Injection deserves one line here too: the model handled 8/8 marker-injection cases correctly, but a controlled echoed delimiter parsed an injected cancellation ahead of the real billing. Unescaped markers reach context, and echoed output is a parser attack surface, not just a model-behavior question.

A promotion gate for DSPy optimization artifacts

A candidate passes only if every mandatory row passes. The logic is conjunctive — an attractive quality score cannot waive an unknown cost.

GatePass rulev2 status
Deterministic integrityArtifact hash and program-identity hash (Signature source, allowed categories, DSPy version, model/config, dataset hashes), schema validation, smoke testFail — artifact hash alone is insufficient; a v2 state loads into the v1 Signature
Semantic quality≥10 pt absolute lift on ID, no frozen slice worse than baseline, valid output ≥99%Pass — +16.7 pts ID (+8 cases); +20.8 to +22.9 pts on edge, adversarial and shifted; candidate 192/192 valid
Safety100% valid output from the candidate, no adversarial regression, adversarial cases independently reviewedPartial — candidate is 192/192 valid with no adversarial regression on the synthetic suite; the one invalid output in this study is the baseline's, not the candidate's; independent adversarial review outstanding
Demonstration robustnessMutation produces no unacceptable loss and no malformed output, under a powered randomized designHold — cannot be evidenced either way at n=192
Latencyp95 cold latency ≤ SLO and ≤10% over baseline at identical concurrency, retries includedNot established — block
CostPriced tokens within budget, independent call count agrees with logsNot established under CLI — block
PortabilityPinned model snapshot, or a second-runtime full-slice matrix meeting the quality gatePartial — on the contaminated run's 48-record ID slice, terra moved 28→42 and luna 26→44 (both out of 48, from that run's transfer probe); lift is runtime-sensitive
RecoveryBaseline rollback hash, alert on load/validation failure, operator drillPartial — drill required

Under this rule v2 is not promotable, despite a genuine quality result. That is the correct answer, and it is reached without anyone's subjective judgment. The next action is not another optimizer run; it is implementing semantic validation and executing the missing operational gates. v1 fails the same table at the first row, for the opposite reason.

The release unit

Detection has to happen at three layers, and the same structure applies to any artifact that ships as data rather than code: a file layer (hash, parseability, schema), an application layer (known-answer smoke suite immediately after load, emitting artifact hash, commit, model ID, and result), and an online layer (valid-output rate, route distribution, policy-exception slice accuracy where labeled feedback exists, p95 latency, provider errors). Recovery is deliberately boring: quarantine the candidate, restore the saved baseline artifact with the same Signature and config, invalidate the candidate hash in deployment config, preserve raw records. The same blast-radius reasoning that governs staged rollouts with real rollback applies here — the difference is that the thing being rolled back is a JSON file most review processes never look at.

Keep the no-op runs. v1 proves a clean compile can be entirely appropriate and add no deployable value; v2 proves the same settings can produce inspectable, useful state when the data has policy headroom. Together they prevent two opposite errors: declaring DSPy ineffective because a saturated task did not move, and declaring it effective because a compiler emitted an artifact.

One caveat on determinism, since it underwrites every number above: temperature zero produced stable outputs here, but deterministic LLM behavior is a property of the service, not a guarantee of the setting. Sequential-session drift was visible in the latency data and is the reason the mutation deltas are reported as noise rather than signal.

The release unit is neither "the prompt" nor "the optimizer run." It is a versioned program state plus its model boundary, its data policy, its test evidence, its operational budget, and its rollback target. And the policy itself needs an accountable owner: an evaluator can measure agreement with a frozen label, but it cannot decide whether $500, fourteen days, or SSO ownership are legitimate business boundaries. When those change, the old artifact is not silently patched — regenerate a versioned dataset, recompile, rerun the gates, compare the new hash, and keep the previous baseline for rollback.

DSPyMIPROv2prompt-optimizationLLMrelease-gatesevaluationMLOps