How to Measure Conformance Between an Operational Specification and Its Execution

<p>In <a href="https://www.freecodecamp.org/news/separate-operational-intent-from-executor/">my previous article</a>, I explored executor independence: the idea that different execution mechanisms should be able to satisfy the same operational specification.</p> <p>By <strong>executor independence</strong>, I mean that the operational specification remains stable even when the mechanism performing the work changes.</p> <p>That gives us a useful architecture:</p> <pre><code class="language-text">Specification ↓ Executor ↓ Evidence </code></pre> <p>But it leaves an important question unanswered: how do we decide whether the evidence produced by an execution actually satisfies the specification?</p> <p>That's a conformance problem.</p> <p>A deployment can finish successfully from the executor's point of view and still violate the operational contract.</p> <p>A remediation agent can complete every action it planned and still leave the system outside acceptable limits.</p> <p>A restore operation can finish without errors and still fail to restore the required data.</p> <p>This is why a generic status like:</p> <pre><code class="language-text">success = true </code></pre> <p>isn't enough.</p> <p>We need a way to compare what should have happened with what was actually observed and produce a result that's explicit, explainable, and useful.</p> <p>In this article, I'll show you how to model operational conformance, evaluate individual constraints, distinguish pass, fail, and unknown outcomes, handle tolerances, work with missing evidence, assign severity, calculate a useful conformance summary, and avoid turning conformance into a misleading score.</p> <p>The examples use TypeScript and a deployment scenario, but the same approach applies to infrastructure changes, incident response, backup restoration, data pipelines, and AI-operated systems.</p> <p>The central idea is this: conformance isn't whether the executor says it succeeded. Conformance is whether observed evidence satisfies the operational specification.</p> <h2 id="heading-prerequisites">Prerequisites</h2> <p>You should be comfortable with:</p> <ul> <li><p>key software architecture concepts</p> </li> <li><p>TypeScript or a similar language</p> </li> <li><p>basic testing</p> </li> <li><p>observability</p> </li> <li><p>CI/CD concepts</p> </li> <li><p>operational specifications</p> </li> <li><p>executor independence</p> </li> </ul> <p>You don't need a particular deployment platform, and the examples are deliberately small so the evaluation model stays visible.</p> <h2 id="heading-table-of-contents">Table of Contents</h2> <ul> <li><p><a href="#heading-prerequisites">Prerequisites</a></p> </li> <li><p><a href="#heading-what-conformance-means-in-this-context">What Conformance Means in This Context</a></p> </li> <li><p><a href="#heading-start-with-a-specification-and-evidence">Start with a Specification and Evidence</a></p> </li> <li><p><a href="#heading-evaluate-constraints-individually">Evaluate Constraints Individually</a></p> </li> <li><p><a href="#heading-dont-reduce-everything-to-pass-or-fail">Don't Reduce Everything to Pass or Fail</a></p> </li> <li><p><a href="#heading-treat-missing-evidence-as-unknown">Treat Missing Evidence as Unknown</a></p> </li> <li><p><a href="#heading-use-tolerances-where-exact-comparison-is-wrong">Use Tolerances Where Exact Comparison Is Wrong</a></p> </li> <li><p><a href="#heading-add-severity-to-constraints">Add Severity to Constraints</a></p> </li> <li><p><a href="#heading-separate-hard-constraints-from-advisory-constraints">Separate Hard Constraints from Advisory Constraints</a></p> </li> <li><p><a href="#heading-build-a-reusable-conformance-evaluator">Build a Reusable Conformance Evaluator</a></p> </li> <li><p><a href="#heading-calculate-a-conformance-summary-carefully">Calculate a Conformance Summary Carefully</a></p> </li> <li><p><a href="#heading-why-a-single-score-can-be-dangerous">Why a Single Score Can Be Dangerous</a></p> </li> <li><p><a href="#heading-preserve-the-evidence-behind-the-verdict">Preserve the Evidence Behind the Verdict</a></p> </li> <li><p><a href="#heading-conformance-should-be-reproducible">Conformance Should Be Reproducible</a></p> </li> <li><p><a href="#heading-how-this-applies-to-multiple-executors">How This Applies to Multiple Executors</a></p> </li> <li><p><a href="#heading-how-this-applies-to-ai-agents">How This Applies to AI Agents</a></p> </li> <li><p><a href="#heading-a-small-end-to-end-example">A Small End-to-End Example</a></p> </li> <li><p><a href="#heading-a-practical-conformance-workflow">A Practical Conformance Workflow</a></p> </li> <li><p><a href="#heading-what-conformance-cant-prove">What Conformance Can't Prove</a></p> </li> <li><p><a href="#heading-from-execution-success-to-verifiable-operations">From Execution Success to Verifiable Operations</a></p> </li> <li><p><a href="#heading-conclusion">Conclusion</a></p> </li> </ul> <h2 id="heading-what-conformance-means-in-this-context">What Conformance Means in This Context</h2> <p>In ordinary language, conformance means complying with a rule, standard, or requirement.</p> <p>Here, I use the term more specifically:</p> <blockquote> <p>Operational conformance is the evaluation of whether, and where, observed execution evidence satisfies the requirements defined by an operational specification.</p> </blockquote> <p>In this context, <strong>evidence</strong> is the set of observable facts collected from the system that can be compared against the specification. It's not a success verdict. It's the data used to produce that verdict.</p> <p>That definition has three parts.</p> <h3 id="heading-1-the-specification">1. The Specification</h3> <p>This defines what should be true.</p> <p>For example:</p> <pre><code class="language-text">version = v42 available replicas &gt;= 3 error rate &lt;= 1% p95 latency &lt;= 400 ms </code></pre> <h3 id="heading-2-the-evidence">2. The Evidence</h3> <p>This describes what was actually observed.</p> <p>For example:</p> <pre><code class="language-text">version = v42 available replicas = 3 error rate = 0.4% p95 latency = 280 ms </code></pre> <h3 id="heading-3-the-evaluation">3. The Evaluation</h3> <p>This compares the evidence with the specification.</p> <p>For example:</p> <pre><code class="language-text">version PASS available replicas PASS error rate PASS p95 latency PASS </code></pre> <p>Conformance is therefore not an executor property. It's the result of evaluating evidence against a specification.</p> <p>Conceptually:</p> <pre><code class="language-text">Specification + Evidence ↓ Conformance Evaluation ↓ Result </code></pre> <h2 id="heading-start-with-a-specification-and-evidence">Start with a Specification and Evidence</h2> <p>Suppose we have this deployment specification:</p> <pre><code class="language-typescript">type DeploymentSpec = { version: string; minReplicas: number; maxErrorRate: number; maxP95LatencyMs: number; }; const spec: DeploymentSpec = { version: "v42", minReplicas: 3, maxErrorRate: 0.01, maxP95LatencyMs: 400, }; </code></pre> <p>Now define the evidence:</p> <pre><code class="language-typescript">type DeploymentEvidence = { version: string; replicas: number; errorRate: number; p95LatencyMs: number; }; const evidence: DeploymentEvidence = { version: "v42", replicas: 3, errorRate: 0.004, p95LatencyMs: 280, }; </code></pre> <p>The simplest possible evaluator might return a boolean:</p> <pre><code class="language-typescript">function conforms( spec: DeploymentSpec, evidence: DeploymentEvidence ): boolean { return ( evidence.version === spec.version &amp;&amp; evidence.replicas &gt;= spec.minReplicas &amp;&amp; evidence.errorRate &lt;= spec.maxErrorRate &amp;&amp; evidence.p95LatencyMs &lt;= spec.maxP95LatencyMs ); } </code></pre> <p>This works, but it has a problem.</p> <p>If the result is:</p> <pre><code class="language-text">false </code></pre> <p>you don't know why. That's not enough for operational work.</p> <h2 id="heading-evaluate-constraints-individually">Evaluate Constraints Individually</h2> <p>Instead of one boolean, evaluate each constraint separately.</p> <pre><code class="language-typescript">type ConstraintResult = { name: string; passed: boolean; }; function evaluateDeployment( spec: DeploymentSpec, evidence: DeploymentEvidence ): ConstraintResult[] { return [ { name: "version", passed: evidence.version === spec.version, }, { name: "replicas", passed: evidence.replicas &gt;= spec.minReplicas, }, { name: "error-rate", passed: evidence.errorRate &lt;= spec.maxErrorRate, }, { name: "latency", passed: evidence.p95LatencyMs &lt;= spec.maxP95LatencyMs, }, ]; } </code></pre> <p><code>ConstraintResult</code> gives each check a name and a boolean result. The <code>evaluateDeployment()</code> function compares one observed value at a time against the corresponding rule in the specification. It checks whether the deployed version matches exactly, whether the replica count is high enough, and whether error rate and latency stay below their limits.</p> <p>Instead of collapsing all of those checks into one <code>true</code> or <code>false</code>, the function returns one result per constraint. That lets the caller see exactly which requirement passed or failed.</p> <p>Now the output can be:</p> <pre><code class="language-text">version PASS replicas PASS error-rate PASS latency PASS </code></pre> <p>or:</p> <pre><code class="language-text">version PASS replicas FAIL error-rate PASS latency FAIL </code></pre> <p>This is much more useful. A failed operation becomes explainable.</p> <h2 id="heading-dont-reduce-everything-to-pass-or-fail">Don't Reduce Everything to Pass or Fail</h2> <p>Binary evaluation is still sometimes too simple.</p> <p>Suppose the specification requires:</p> <pre><code class="language-text">error rate &lt;= 1% </code></pre> <p>but your observability system is unavailable.</p> <p>What should the result be?</p> <p><code>PASS</code> is clearly wrong.</p> <p><code>FAIL</code> may also be misleading, because that mixes two different situations:</p> <pre><code class="language-text">the system violated the constraint </code></pre> <p>and:</p> <pre><code class="language-text">we do not know whether the system violated the constraint </code></pre> <p>A better model uses at least three states:</p> <pre><code class="language-text">PASS FAIL UNKNOWN </code></pre> <p>For example:</p> <pre><code class="language-typescript">type ConstraintStatus = | "PASS" | "FAIL" | "UNKNOWN"; </code></pre> <p>Unknown evidence should never silently become success.</p> <h2 id="heading-treat-missing-evidence-as-unknown">Treat Missing Evidence as Unknown</h2> <p>Suppose evidence becomes partial. In other words, the system may return some observations but not all of them. A deployment record might tell us which version is running and how many replicas are available, while the error-rate metric is temporarily unavailable.</p> <p>We can represent that by making each evidence field optional:</p> <pre><code class="language-typescript">type DeploymentEvidence = { version?: string; replicas?: number; errorRate?: number; p95LatencyMs?: number; }; </code></pre> <p>The evaluator then needs to distinguish between a real violation and a missing observation:</p> <pre><code class="language-typescript">function evaluateErrorRate( spec: DeploymentSpec, evidence: DeploymentEvidence ): ConstraintStatus { if ( evidence.errorRate === undefined ) { return "UNKNOWN"; } return ( evidence.errorRate &lt;= spec.maxErrorRate ) ? "PASS" : "FAIL"; } </code></pre> <p>The first branch asks whether an error-rate value was observed at all. If it was not, the evaluator returns <code>UNKNOWN</code> instead of guessing. If the value exists, the function compares it with the maximum allowed error rate and returns <code>PASS</code> or <code>FAIL</code>.</p> <p>This produces:</p> <pre><code class="language-text">errorRate = 0.004 → PASS errorRate = 0.03 → FAIL errorRate missing → UNKNOWN </code></pre> <p>That distinction matters. <strong>Missing telemetry</strong>, here, means that a metric or operational signal we expected to observe wasn't available. For example, because the monitoring system didn't report an error-rate value for that execution window. If missing telemetry is treated as success, conformance becomes dangerously optimistic.</p> <h2 id="heading-use-tolerances-where-exact-comparison-is-wrong">Use Tolerances Where Exact Comparison Is Wrong</h2> <p>Some constraints should use exact equality.</p> <p>For example:</p> <pre><code class="language-text">deployed version must equal v42 </code></pre> <p>Others should not.</p> <p>Numerical systems may need tolerances because measured or computed values can differ by tiny amounts even when they're operationally equivalent. Floating-point arithmetic, sampling, rounding, and measurement noise can all produce values that shouldn't fail a constraint just because they're not exactly equal.</p> <pre><code class="language-typescript">function withinTolerance( observed: number, expected: number, tolerance: number ): boolean { return ( Math.abs(observed - expected) &lt;= tolerance ); } </code></pre> <p>Then:</p> <pre><code class="language-typescript">withinTolerance( 1.9999999, 2, 0.0001 ); </code></pre> <p>returns <code>true</code>.</p> <p>The function computes the absolute difference between the observed and expected values, then checks whether that difference stays within the allowed tolerance. This is useful when tiny numeric differences are expected and exact equality would create false failures.</p> <p>But tolerances should come from the operational domain. They shouldn't be introduced simply to make failures disappear.</p> <h2 id="heading-add-severity-to-constraints">Add Severity to Constraints</h2> <p>Not every constraint has the same operational importance.</p> <p>Suppose two constraints fail:</p> <pre><code class="language-text">p95 latency = 405 ms replicas = 0 </code></pre> <p>Both are failures, but they're not equivalent.</p> <p>A small latency miss may deserve attention without requiring an immediate rollback. Zero available replicas, on the other hand, may mean the service is unavailable and should trigger urgent recovery. Severity lets the evaluation preserve that operational difference instead of treating every failure as equally important.</p> <pre><code class="language-typescript">type Severity = | "INFO" | "WARNING" | "CRITICAL"; type ConstraintResult = { name: string; status: ConstraintStatus; severity: Severity; }; </code></pre> <p>Here, <code>status</code> tells us whether the constraint passed, failed, or couldn't be evaluated. <code>severity</code> tells us how important that result is operationally.</p> <p>The two dimensions are separate: a constraint can fail with <code>WARNING</code> severity or fail with <code>CRITICAL</code> severity.</p> <p>Then:</p> <pre><code class="language-text">latency FAIL WARNING replicas FAIL CRITICAL </code></pre> <p>This matters when deciding whether to continue, pause, rollback, or escalate.</p> <h2 id="heading-separate-hard-constraints-from-advisory-constraints">Separate Hard Constraints from Advisory Constraints</h2> <p>Some requirements define whether the operation is acceptable at all. Others are useful signals but don't necessarily block execution.</p> <p>This is different from severity. Severity describes the impact of a result, while constraint mode describes whether that rule participates in the final conformance decision. A <code>REQUIRED</code> constraint must pass for the operation to be conformant. An <code>ADVISORY</code> constraint can fail and still leave the operation conformant overall, although the failure should still be reported.</p> <pre><code class="language-typescript">type ConstraintMode = | "REQUIRED" | "ADVISORY"; </code></pre> <p>A result can now include:</p> <pre><code class="language-typescript">type ConstraintResult = { name: string; status: ConstraintStatus; severity: Severity; mode: ConstraintMode; expected?: unknown; observed?: unknown; }; </code></pre> <p>For example:</p> <pre><code class="language-text">version REQUIRED PASS latency-target ADVISORY FAIL </code></pre> <p>The operation may still be conformant overall, but with a warning. That's more expressive than one global boolean.</p> <h2 id="heading-build-a-reusable-conformance-evaluator">Build a Reusable Conformance Evaluator</h2> <p>So far, the examples have been tied to one deployment and a handful of hard-coded checks. The next step is to separate the evaluation mechanism from the individual rules so the same engine can evaluate different operational specifications.</p> <p>To do that, we represent each rule as a <code>Constraint&lt;T&gt;</code>. Each constraint carries its own name, severity, mode, and evaluation function. An <code>OperationalSpec&lt;T&gt;</code> then becomes a collection of those constraints.</p> <pre><code class="language-typescript">type Constraint&lt;T&gt; = { name: string; severity: Severity; mode: ConstraintMode; evaluate( evidence: T ): ConstraintStatus; }; type OperationalSpec&lt;T&gt; = { name: string; constraints: Constraint&lt;T&gt;[]; }; </code></pre> <p>Then:</p> <pre><code class="language-typescript">function evaluateConformance&lt;T&gt;( spec: OperationalSpec&lt;T&gt;, evidence: T ): ConstraintResult[] { return spec.constraints.map( (constraint) =&gt; ({ name: constraint.name, status: constraint.evaluate(evidence), severity: constraint.severity, mode: constraint.mode, }) ); } </code></pre> <p>The specification owns what should be evaluated.</p> <p>The evaluator collects the results. It doesn't need to know the meaning of each constraint. It simply executes every constraint's evaluation function and preserves the resulting status, severity, and mode. That keeps the evaluation engine generic while the specification remains responsible for the operational rules.</p> <p>The <code>expected</code> and <code>observed</code> fields are optional in this minimal example, but in a production system I would populate them, or store references to the underlying evidence, so the final verdict remains explainable and auditable.</p> <h2 id="heading-calculate-a-conformance-summary-carefully">Calculate a Conformance Summary Carefully</h2> <p>Individual results are useful, but sometimes you also need a summary.</p> <pre><code class="language-typescript">type ConformanceSummary = { total: number; passed: number; failed: number; unknown: number; conformant: boolean; }; </code></pre> <p>Then:</p> <pre><code class="language-typescript">function summarize( results: ConstraintResult[] ): ConformanceSummary { const required = results.filter( (result) =&gt; result.mode === "REQUIRED" ); const passed = results.filter( (result) =&gt; result.status === "PASS" ).length; const failed = results.filter( (result) =&gt; result.status === "FAIL" ).length; const unknown = results.filter( (result) =&gt; result.status === "UNKNOWN" ).length; const conformant = required.every( (result) =&gt; result.status === "PASS" ); return { total: results.length, passed, failed, unknown, conformant, }; } </code></pre> <p>Overall conformance now requires all required constraints to pass.</p> <p>Notice what that means for <code>UNKNOWN</code>: because <code>every()</code> requires a required constraint to have status <code>PASS</code>, a required constraint with missing evidence makes the overall result non-conformant rather than silently passing.</p> <p>Advisory constraints still appear in the report. They just don't determine the final verdict.</p> <h2 id="heading-why-a-single-score-can-be-dangerous">Why a Single Score Can Be Dangerous</h2> <p>It's tempting to create one single score:</p> <pre><code class="language-text">conformance = 92% </code></pre> <p>But that can be misleading.</p> <p>Suppose:</p> <pre><code class="language-text">9 low-risk constraints pass 1 critical security constraint fails </code></pre> <p>A naïve score says:</p> <pre><code class="language-text">90% conformant </code></pre> <p>Operationally, that may be completely unacceptable.</p> <p>If you calculate a score, you should preserve:</p> <pre><code class="language-text">constraint-level results severity required/advisory status unknown evidence failure reasons </code></pre> <p>The score can summarize, but it shouldn't erase the evidence.</p> <h2 id="heading-preserve-the-evidence-behind-the-verdict">Preserve the Evidence Behind the Verdict</h2> <p>A conformance result shouldn't only say <code>FAIL</code>. It should preserve what produced that verdict.</p> <pre><code class="language-json">{ "constraint": "minimum-replicas", "expected": "&gt;= 3", "observed": 2, "status": "FAIL", "severity": "CRITICAL" } </code></pre> <p>This lets someone answer:</p> <pre><code class="language-text">What did we expect? What did we observe? Which rule was applied? Why did it fail? </code></pre> <p>That is especially important when conformance decisions trigger automated recovery.</p> <h2 id="heading-conformance-should-be-reproducible">Conformance Should Be Reproducible</h2> <p>Suppose an operation executes at 10:00.</p> <p>At 10:01, the error rate is:</p> <pre><code class="language-text">0.6% </code></pre> <p>At 14:00, the dashboard shows:</p> <pre><code class="language-text">2.1% </code></pre> <p>If you reevaluate the old operation using current telemetry, you may get a different result.</p> <p>Conformance should therefore be evaluated against evidence associated with the execution window. By <strong>execution window</strong>, I mean the period during which the operation ran and its immediate effects were being verified. For a deployment that finished at 10:00, for example, you might evaluate health using the telemetry collected between 10:00 and 10:05 rather than whatever the dashboard happens to show several hours later.</p> <p>The point is to bind the verdict to the evidence that was available for that specific execution. Otherwise, later changes in the system can retroactively change the apparent result of an operation that already happened.</p> <p>A reproducible record might preserve:</p> <pre><code class="language-text">specification version execution ID evidence snapshot evaluation rules timestamp </code></pre> <p>For example:</p> <pre><code class="language-json">{ "executionId": "deploy-812", "specVersion": "3", "evaluatedAt": "2026-10-06T10:01:00Z", "evidence": { "version": "v42", "replicas": 3, "errorRate": 0.006, "p95LatencyMs": 290 } } </code></pre> <p>Now the decision can be reconstructed later.</p> <h2 id="heading-how-this-applies-to-multiple-executors">How This Applies to Multiple Executors</h2> <p>Suppose two executors run the same specification.</p> <p>Executor A:</p> <pre><code class="language-text">version PASS replicas PASS error-rate PASS latency PASS </code></pre> <p>Executor B:</p> <pre><code class="language-text">version PASS replicas FAIL error-rate PASS latency PASS </code></pre> <p>The question is no longer:</p> <blockquote> <p>Which executor is better?</p> </blockquote> <p>The immediate question is:</p> <blockquote> <p>Which execution satisfied the specification?</p> </blockquote> <p>You answer that by running the same conformance evaluator against the evidence produced by each executor. The specification and evaluation rules stay the same; only the evidence changes.</p> <p>In the example above, Executor A satisfies the specification because every required constraint evaluates to <code>PASS</code>. Executor B does not, because its replica constraint evaluates to <code>FAIL</code>. That gives us a common basis for comparison without requiring the two executors to use the same internal steps.</p> <p>Over time, executor performance can be compared using conformance rate, failure categories, latency, cost, recovery frequency, and unknown evidence rate. But conformance remains the contract-level evaluation.</p> <h2 id="heading-how-this-applies-to-ai-agents">How This Applies to AI Agents</h2> <p>AI agents make explicit conformance especially important.</p> <p>An agent may choose different plans for the same objective.</p> <pre><code class="language-text">Agent execution 1: scale deploy observe route Agent execution 2: deploy canary observe expand Agent execution 3: create parallel environment switch traffic </code></pre> <p>Because the agent can choose a different plan each time, comparing executions step by step isn't very useful. One run might use a canary rollout while another creates a parallel environment, and both can still be valid ways to reach the same objective.</p> <p>What stays stable is the specification: the objective, constraints, and evidence requirements don't change just because the agent chooses a different path.</p> <p>After each run, the system collects evidence about the resulting state. The conformance evaluator then compares that evidence with the same specification used for every other run. This lets the agent vary the execution plan while keeping the definition of success outside the agent itself.</p> <pre><code class="language-text">Specification ↓ AI Agent ↓ Dynamic Plan ↓ Execution ↓ Evidence ↓ Conformance </code></pre> <p>The agent can propose actions. It shouldn't be the only authority deciding whether those actions achieved the intended outcome.</p> <h2 id="heading-a-small-end-to-end-example">A Small End-to-End Example</h2> <p>Start by defining the evidence:</p> <pre><code class="language-typescript">type DeploymentEvidence = { version?: string; replicas?: number; errorRate?: number; p95LatencyMs?: number; }; </code></pre> <p>Now define the constraints:</p> <pre><code class="language-typescript">const constraints: Constraint&lt;DeploymentEvidence&gt;[] = [ { name: "version", severity: "CRITICAL", mode: "REQUIRED", evaluate: (evidence) =&gt; { if (!evidence.version) { return "UNKNOWN"; } return ( evidence.version === "v42" ) ? "PASS" : "FAIL"; }, }, { name: "replicas", severity: "CRITICAL", mode: "REQUIRED", evaluate: (evidence) =&gt; { if ( evidence.replicas === undefined ) { return "UNKNOWN"; } return ( evidence.replicas &gt;= 3 ) ? "PASS" : "FAIL"; }, }, { name: "error-rate", severity: "CRITICAL", mode: "REQUIRED", evaluate: (evidence) =&gt; { if ( evidence.errorRate === undefined ) { return "UNKNOWN"; } return ( evidence.errorRate &lt;= 0.01 ) ? "PASS" : "FAIL"; }, }, { name: "latency-target", severity: "WARNING", mode: "ADVISORY", evaluate: (evidence) =&gt; { if ( evidence.p95LatencyMs === undefined ) { return "UNKNOWN"; } return ( evidence.p95LatencyMs &lt;= 350 ) ? "PASS" : "FAIL"; }, }, ]; </code></pre> <p>Create the specification:</p> <pre><code class="language-typescript">const deploymentSpec: OperationalSpec&lt;DeploymentEvidence&gt; = { name: "deploy-orders-v42", constraints, }; </code></pre> <p>Evaluate:</p> <pre><code class="language-typescript">const evidence: DeploymentEvidence = { version: "v42", replicas: 3, errorRate: 0.004, p95LatencyMs: 370, }; const results = evaluateConformance( deploymentSpec, evidence ); const summary = summarize(results); </code></pre> <p>The result might be:</p> <pre><code class="language-text">version PASS REQUIRED replicas PASS REQUIRED error-rate PASS REQUIRED latency-target FAIL ADVISORY </code></pre> <p>And:</p> <pre><code class="language-text">conformant = true </code></pre> <p>Every required constraint passed.</p> <p>The advisory latency target failed, so the operation should still surface a warning.</p> <h2 id="heading-a-practical-conformance-workflow">A Practical Conformance Workflow</h2> <p>If you want to introduce conformance evaluation into an existing system:</p> <ol> <li><p>Pick one operation.</p> </li> <li><p>Define the specification.</p> </li> <li><p>Classify constraints as required or advisory.</p> </li> <li><p>Assign severity.</p> </li> <li><p>Define the evidence model.</p> </li> <li><p>Decide how missing evidence becomes <code>UNKNOWN</code>.</p> </li> <li><p>Evaluate constraints individually.</p> </li> <li><p>Produce a summary.</p> </li> <li><p>Preserve the evidence snapshot.</p> </li> <li><p>Version the specification.</p> </li> <li><p>Revisit rules that produce unexpected failures or unknowns.</p> </li> </ol> <p>Don't simply weaken the specification to make results pass.</p> <h2 id="heading-what-conformance-cant-prove">What Conformance Can't Prove</h2> <p>Conformance can tell you:</p> <blockquote> <p>The observed evidence satisfies the specification.</p> </blockquote> <p>It can't automatically tell you:</p> <blockquote> <p>The specification is correct.</p> </blockquote> <p>The specification may be wrong, and the evidence may also be wrong.</p> <p>So conformance depends on:</p> <pre><code class="language-text">specification quality evidence quality evaluation correctness </code></pre> <p>It's a verification mechanism, not absolute truth.</p> <h2 id="heading-from-execution-success-to-verifiable-operations">From Execution Success to Verifiable Operations</h2> <p>Traditional automation often ends here:</p> <pre><code class="language-text">execute ↓ success / failure </code></pre> <p>A conformance-based model adds more structure:</p> <pre><code class="language-text">Specification ↓ Executor ↓ Execution ↓ Evidence ↓ Constraint Evaluation ↓ Conformance </code></pre> <p>Now you can ask:</p> <blockquote> <p>Which requirements were satisfied?</p> </blockquote> <blockquote> <p>Which failed?</p> </blockquote> <blockquote> <p>Which could not be evaluated?</p> </blockquote> <blockquote> <p>What evidence supports the verdict?</p> </blockquote> <p>If the system is conformant immediately after execution, you now have a baseline.</p> <p>From that baseline, you can keep observing the system over time.</p> <p>And then ask:</p> <blockquote> <p>What happens when reality starts to diverge from the specification after the operation has already succeeded?</p> </blockquote> <p>That's operational drift.</p> <h2 id="heading-conclusion">Conclusion</h2> <p>A successful executor isn't enough. Neither is a green pipeline or a completed agent plan.</p> <p>What matters is whether the resulting system satisfies the operational specification.</p> <p>That requires three things:</p> <pre><code class="language-text">Specification Evidence Evaluation </code></pre> <p>A useful conformance model should preserve:</p> <pre><code class="language-text">constraint-level results PASS / FAIL / UNKNOWN severity required vs advisory rules observed values expected values evidence snapshots specification version </code></pre> <p>The executor performs the work, evidence describes what happened, and the specification defines what should have happened.</p> <p>Then the conformance evaluator compares the two.</p> <p>That gives us a stronger foundation for traditional automation and for increasingly autonomous systems.</p> <p>But conformance at one point in time isn't the end of the story. A system can conform now and drift later.</p> <p>The next question is:</p> <blockquote> <p>How do we detect when observed reality stops matching operational intent?</p> </blockquote> <p>That's where I want to go next.</p>