How to Build an AI Agent Harness from Scratch in TypeScript

<p>Many AI agent tutorials start with a language model, give it a few tools, and run a loop until it produces an answer. That's enough to demonstrate tool calling, but it leaves the interesting engineering questions unanswered.</p> <p>What happens when a tool fails? Who validates the arguments? What prevents an agent from running indefinitely, executing unsafe commands, or exhausting its context window? And how do you test any of this without relying on unpredictable model responses?</p> <p>These are responsibilities of the <strong>agent harness</strong>: the software that sits between a language model and the environment it operates in.</p> <p>In this tutorial, we'll build a small, provider-independent harness in TypeScript, using a coding agent as our example. We'll start with a basic execution loop and progressively add tool validation, workspace isolation, execution budgets, context management, observability, and deterministic failure testing.</p> <h2 id="heading-what-well-cover">What We'll Cover</h2> <ul> <li><p><a href="#heading-prerequisites">Prerequisites</a></p> </li> <li><p><a href="#heading-1-build-the-basic-agent-loop">1. Build the Basic Agent Loop</a></p> <ul> <li><a href="#heading-test-without-a-real-model">Test Without a Real Model</a></li> </ul> </li> <li><p><a href="#heading-2-validate-tool-calls-before-executing-them">2. Validate Tool Calls Before Executing Them</a></p> <ul> <li><p><a href="#heading-a-registry-with-explicit-contracts">A Registry with Explicit Contracts</a></p> </li> <li><p><a href="#heading-not-every-error-means-the-same-thing">Not Every Error Means the Same Thing</a></p> </li> </ul> </li> <li><p><a href="#heading-3-give-the-agent-tools-without-giving-it-the-machine">3. Give the Agent Tools Without Giving It the Machine</a></p> <ul> <li><p><a href="#heading-confine-file-operations-to-a-workspace">Confine File Operations to a Workspace</a></p> </li> <li><p><a href="#heading-let-the-model-choose-commands-not-invent-them">Let the Model Choose Commands, Not Invent Them</a></p> </li> </ul> </li> <li><p><a href="#heading-4-put-limits-on-execution">4. Put Limits on Execution</a></p> <ul> <li><p><a href="#heading-count-turns-and-tool-calls-separately">Count Turns and Tool Calls Separately</a></p> </li> <li><p><a href="#heading-propagate-cancellation-and-deadlines">Propagate Cancellation and Deadlines</a></p> </li> <li><p><a href="#heading-test-the-limits-without-waiting-for-real-failures">Test the Limits Without Waiting for Real Failures</a></p> </li> </ul> </li> <li><p><a href="#heading-5-keep-the-context-under-control">5. Keep the Context Under Control</a></p> <ul> <li><p><a href="#heading-bound-tool-output-and-model-facing-history">Bound Tool Output and Model-Facing History</a></p> </li> <li><p><a href="#heading-dont-split-a-tool-call-from-its-result">Don't Split a Tool Call From its Result</a></p> </li> </ul> </li> <li><p><a href="#heading-6-make-agent-execution-observable">6. Make Agent Execution Observable</a></p> <ul> <li><p><a href="#heading-emit-events-at-execution-boundaries">Emit Events at Execution Boundaries</a></p> </li> <li><p><a href="#heading-account-for-model-usage-without-inventing-it">Account for Model Usage Without Inventing it</a></p> </li> </ul> </li> <li><p><a href="#heading-7-test-the-failure-paths-not-just-the-happy-path">7. Test the Failure Paths, Not Just the Happy Path</a></p> <ul> <li><p><a href="#heading-validate-the-model-response-before-trusting-it">Validate the Model Response Before Trusting it</a></p> </li> <li><p><a href="#heading-keep-rejected-batches-out-of-the-transcript">Keep Rejected Batches Out of the Transcript</a></p> </li> <li><p><a href="#heading-use-reproducible-variation">Use Reproducible Variation</a></p> </li> </ul> </li> <li><p><a href="#heading-where-to-go-next-connect-a-real-language-model">Where to Go Next: Connect a Real Language Model</a></p> </li> </ul> <h2 id="heading-prerequisites">Prerequisites</h2> <p>Familiarity with TypeScript and basic knowledge of AI agents will be helpful. We'll use Node.js 24, TypeScript, and its built-in test runner, with minimal dependencies.</p> <p>The complete implementation is available in the companion repository: <a href="https://github.com/gkoos/agent-harness-from-scratch">agent-harness-from-scratch</a>. Each section corresponds to a Git tag, so you can follow the project as it develops instead of starting with the final code.</p> <h2 id="heading-1-build-the-basic-agent-loop">1. Build the Basic Agent Loop</h2> <p>To follow along, check out the initial milestone:</p> <pre><code class="language-shell">git clone https://github.com/gkoos/agent-harness-from-scratch.git cd agent-harness-from-scratch git switch --detach 01-model-loop npm ci npm run typecheck npm test npm run demo </code></pre> <p>The detached checkout puts you at the exact snapshot discussed here without moving a branch. When you reach the next section, switch to its tag. You'll need Node.js 24 and npm installed.</p> <p>The most important architectural decision is that <strong>the model proposes actions while the harness executes them</strong>. When a model requests a tool call, it hasn't actually executed anything. It has generated a structured request containing a tool name and arguments. The harness must interpret that request, invoke the corresponding implementation, and return the result to the model.</p> <p>The interaction looks like this:</p> <img src="https://cdn.hashnode.com/uploads/covers/68b08746916c71e1ed2db58e/8fa4527d-9ca0-43a9-baa7-0bd3b37d4674.svg" alt="Agent execution loop: the harness sends the conversation to the model, executes requested tools, and feeds results back until the model returns a final answer." style="display: block;" width="600" height="400" loading="lazy"> <p>We'll start by defining the contracts in <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/01-model-loop/src/core/types.ts"><code>src/core/types.ts</code></a>.</p> <pre><code class="language-typescript">export type ToolCall = { id: string; name: string; arguments: unknown }; export type Message = | { role: 'user'; content: string } | { role: 'assistant'; content: string; toolCalls?: ToolCall[] } | { role: 'tool'; callId: string; content: string; isError: boolean }; export interface ToolDefinition { name: string; description: string; parameters: Record&lt;string, unknown&gt;; // JSON Schema for the model } export interface Tool extends ToolDefinition { execute(argumentsValue: unknown, signal: AbortSignal): Promise&lt;unknown&gt;; } export interface Model { generate(input: { messages: readonly Message[]; tools: readonly ToolDefinition[]; signal: AbortSignal; }): Promise&lt;{ content: string; toolCalls?: ToolCall[] }&gt;; } export type AgentResult = { status: 'completed' | 'step_limit' | 'aborted'; answer?: string; messages: Message[]; steps: number; }; </code></pre> <p>The <code>Model</code> interface deliberately contains nothing specific to OpenAI, Anthropic, or any other provider. A provider adapter can translate between its API and these types without changing the harness.</p> <p>Notice that tool arguments are <code>unknown</code>, not <code>any</code>. Model-generated data is untrusted input. We'll validate it before execution in the next section.</p> <p>The agent loop in <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/01-model-loop/src/core/agent.ts"><code>src/core/agent.ts</code></a> follows a simple sequence:</p> <ol> <li><p>Send the conversation and available tool definitions to the model.</p> </li> <li><p>Append the model's response to the conversation.</p> </li> <li><p>If it requested tools, execute them and append their results.</p> </li> <li><p>Repeat until the model returns a final answer or the harness reaches its step limit.</p> </li> </ol> <p>Here is the complete initial loop in <code>src/core/agent.ts</code>. The model receives tool definitions, while the harness retains the executable implementations:</p> <pre><code class="language-typescript">import type { AgentResult, Message, Model, Tool } from './types.ts'; export async function runAgent(options: { model: Model; tools: readonly Tool[]; prompt: string; maxSteps?: number; signal?: AbortSignal; }): Promise&lt;AgentResult&gt; { const { model, tools, prompt, maxSteps = 10 } = options; if (!Number.isSafeInteger(maxSteps) || maxSteps &lt; 1) { throw new RangeError('maxSteps must be a positive safe integer'); } const signal = options.signal ?? new AbortController().signal; const messages: Message[] = [{ role: 'user', content: prompt }]; const registry = new Map(tools.map(tool =&gt; [tool.name, tool])); if (registry.size !== tools.length) throw new Error('Duplicate tool names'); const definitions = tools.map(({ name, description, parameters }) =&gt; ({ name, description, parameters })); for (let step = 0; step &lt; maxSteps; step++) { if (signal.aborted) return { status: 'aborted', messages, steps: step }; let response; try { response = await model.generate({ messages: [...messages], tools: definitions, signal }); } catch (error) { if (signal.aborted) return { status: 'aborted', messages, steps: step }; throw error; } if (signal.aborted) return { status: 'aborted', messages, steps: step }; const calls = response.toolCalls ?? []; messages.push({ role: 'assistant', content: response.content, ...(calls.length ? { toolCalls: calls } : {}) }); if (!calls.length) return { status: 'completed', answer: response.content, messages, steps: step + 1 }; for (const call of calls) { if (signal.aborted) return { status: 'aborted', messages, steps: step + 1 }; const tool = registry.get(call.name); if (!tool) { messages.push({ role: 'tool', callId: call.id, content: `Unknown tool: ${call.name}`, isError: true }); continue; } try { const result = await tool.execute(call.arguments, signal); if (signal.aborted) return { status: 'aborted', messages, steps: step + 1 }; messages.push({ role: 'tool', callId: call.id, content: JSON.stringify(result) ?? 'null', isError: false }); } catch (error) { if (signal.aborted) return { status: 'aborted', messages, steps: step + 1 }; messages.push({ role: 'tool', callId: call.id, content: error instanceof Error ? error.message : String(error), isError: true }); } } } return { status: 'step_limit', messages, steps: maxSteps }; } </code></pre> <p>A response without tool calls ends the run. Otherwise, the harness looks up each requested tool and executes it.</p> <p>Tool failures don't automatically terminate the agent. Instead, the harness records an error message associated with the original call ID. This allows the model to react to a missing file or failed operation on its next turn.</p> <p>We also impose a maximum number of model turns. A model can keep requesting work, but it doesn't get to decide how much execution time or how many iterations it's entitled to.</p> <h3 id="heading-test-without-a-real-model">Test Without a Real Model</h3> <p>Rather than making nondeterministic API calls in tests (and burning tokens or GPU power), we introduce a scripted fake model in <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/01-model-loop/src/models/fake.ts"><code>src/models/fake.ts</code></a>. It returns predefined responses: first a tool request, then a final answer.</p> <p>The fake model advances through an array of responses. Exhausting the script is an error, so an unexpected extra model turn won't silently pass:</p> <pre><code class="language-typescript">// src/models/fake.ts import type { Model } from '../core/types.ts'; export function scriptedModel( responses: Array&lt;Awaited&lt;ReturnType&lt;Model['generate']&gt;&gt;&gt; ): Model { let index = 0; return { async generate() { const response = responses[index++]; if (!response) throw new Error('Scripted model exhausted'); return response; } }; } </code></pre> <p>Here's a minimal test for <code>tests/agent.test.ts</code>. Run it with <code>npm test</code>:</p> <pre><code class="language-typescript">import { test } from 'node:test'; import assert from 'node:assert/strict'; import { runAgent } from '../src/core/agent.ts'; import { scriptedModel } from '../src/models/fake.ts'; test('executes a tool and returns the final answer', async () =&gt; { const result = await runAgent({ prompt: 'Say hello', tools: [{ name: 'hello', description: 'Return a greeting', parameters: { type: 'object' }, async execute() { return 'Hello'; } }], model: scriptedModel([ { content: '', toolCalls: [{ id: '1', name: 'hello', arguments: {} }] }, { content: 'Done' } ]) }); assert.equal(result.status, 'completed'); assert.equal(result.answer, 'Done'); assert.deepEqual(result.messages[2], { role: 'tool', callId: '1', content: '"Hello"', isError: false }); }); </code></pre> <p>The <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/01-model-loop/src/demo.ts">companion demo</a> uses the same pattern. Additional <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/01-model-loop/tests/agent.test.ts">tests</a> cover unknown tools, step limits, and cancellation without API keys, network access, or token costs.</p> <p>At this point, we have a working agent loop, but it still trusts model-generated tool arguments and has no meaningful restrictions on what registered tools can do.</p> <h2 id="heading-2-validate-tool-calls-before-executing-them">2. Validate Tool Calls Before Executing Them</h2> <p>Our first loop lets the model request a tool by name, then passes its arguments directly to the implementation. That works for a demo, but the tool description is only a <em>request</em> to the model, it isn't a guarantee that the model will return valid arguments.</p> <p>For this section, check out the second milestone of the <a href="https://github.com/gkoos/agent-harness-from-scratch">companion repository</a>:</p> <pre><code class="language-shell">git switch --detach 02-tool-dispatch </code></pre> <p>This tag adds <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/02-tool-dispatch/src/core/registry.ts"><code>src/core/registry.ts</code></a>, which handles tool registration and runtime argument validation. It also changes the failure behavior in <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/02-tool-dispatch/src/core/agent.ts"><code>src/core/agent.ts</code></a>.</p> <h3 id="heading-a-registry-with-explicit-contracts">A Registry with Explicit Contracts</h3> <p>The <code>ToolRegistry</code> replaces the initial loop's plain <code>Map</code>. It rejects invalid or duplicate tool names and exposes only the definitions the model needs:</p> <pre><code class="language-typescript">import type { Tool, ToolDefinition } from './types.ts'; export class ToolRegistry { private readonly tools = new Map&lt;string, Tool&gt;(); constructor(tools: readonly Tool[]) { for (const tool of tools) { if (!/^[a-zA-Z][a-zA-Z0-9_-]*$/.test(tool.name)) throw new Error(`Invalid tool name: ${tool.name}`); if (this.tools.has(tool.name)) throw new Error(`Duplicate tool name: ${tool.name}`); this.tools.set(tool.name, tool); } } definitions(): ToolDefinition[] { return [...this.tools.values()].map(({ name, description, parameters }) =&gt; ({ name, description, parameters })); } get(name: string): Tool | undefined { return this.tools.get(name); } } </code></pre> <p>Create it with <code>const registry = new ToolRegistry(tools)</code> and obtain the model-facing definitions with <code>registry.definitions()</code>. When a response requests a tool, validate the supplied arguments <strong>before</strong> calling <code>execute</code>:</p> <pre><code class="language-typescript">const tool = registry.get(call.name); if (!tool) { messages.push({ role: 'tool', callId: call.id, content: `Unknown tool: ${call.name}`, isError: true }); continue; } const validationErrors = validate(call.arguments, tool.parameters); if (validationErrors.length) { messages.push({ role: 'tool', callId: call.id, content: `Invalid arguments: ${validationErrors.join('; ')}`, isError: true }); continue; } </code></pre> <p>This is a critical boundary. <code>arguments: unknown</code> remains untrusted until the harness checks it. If validation fails, the tool never runs. The model instead receives an error result associated with its original call ID.</p> <p>The tutorial's <code>validate</code> function supports a deliberately small JSON Schema subset: primitive and container <code>type</code> checks, object <code>properties</code>, <code>required</code> fields, <code>additionalProperties: false</code>, array <code>items</code>, and <code>enum</code>. It reports paths such as <code>$.flags[0]</code> to help identify malformed input. It's <strong>not</strong> a complete JSON Schema implementation, and it doesn't validate tool schemas at registration time.</p> <p>Here's the validator used above. It first checks the declared type, then recursively checks object properties and array items:</p> <pre><code class="language-typescript">export function validate(value: unknown, schema: Record&lt;string, unknown&gt;, path = '$'): string[] { const errors: string[] = []; const type = schema.type; if (typeof type !== 'string') return [`${path}: schema must declare a type`]; const matches = type === 'object' ? value !== null &amp;&amp; typeof value === 'object' &amp;&amp; !Array.isArray(value) : type === 'array' ? Array.isArray(value) : type === 'integer' ? typeof value === 'number' &amp;&amp; Number.isInteger(value) : type === 'number' ? typeof value === 'number' &amp;&amp; Number.isFinite(value) : type === 'null' ? value === null : type === 'string' || type === 'boolean' ? typeof value === type : false; if (!matches) return [`${path}: expected ${type}`]; if (Array.isArray(schema.enum) &amp;&amp; !schema.enum.some(item =&gt; Object.is(item, value))) { errors.push(`${path}: not an allowed value`); } if (type === 'object') { const obj = value as Record&lt;string, unknown&gt;; const properties = (schema.properties ?? {}) as Record&lt;string, Record&lt;string, unknown&gt;&gt;; for (const key of (schema.required ?? []) as string[]) { if (!Object.hasOwn(obj, key)) errors.push(`${path}.${key}: required`); } for (const [key, item] of Object.entries(obj)) { if (Object.hasOwn(properties, key)) errors.push(...validate(item, properties[key]!, `${path}.${key}`)); else if (schema.additionalProperties === false) errors.push(`${path}.${key}: unexpected property`); } } if (type === 'array' &amp;&amp; schema.items &amp;&amp; typeof schema.items === 'object') { (value as unknown[]).forEach((item, index) =&gt; { errors.push(...validate( item, schema.items as Record&lt;string, unknown&gt;, `${path}[${index}]` )); }); } return errors; } </code></pre> <h3 id="heading-not-every-error-means-the-same-thing">Not Every Error Means the Same Thing</h3> <p>Milestone 2 also makes an important distinction between <strong>tool failure</strong> and <strong>model failure</strong>.</p> <p>An unknown tool, invalid arguments, or an exception thrown by a tool becomes a <code>role: 'tool'</code> message with <code>isError: true</code>. The model can inspect that result and try something else on its next turn. By contrast, if <code>model.generate()</code> itself throws, the harness terminates with <code>status: 'model_error'</code> and records the error message.</p> <p>This isn't just bookkeeping. A missing file might be recoverable, but an unavailable model provider means there's no model response to guide recovery. The harness shouldn't pretend these are the same situation.</p> <p>In the loop's existing <code>model.generate()</code> catch block, replace the rethrow with a structured outcome, and add <code>'model_error'</code> and an optional <code>error?: string</code> field to <code>AgentResult</code>:</p> <pre><code class="language-typescript">if (signal.aborted) return { status: 'aborted', messages, steps: step }; return { status: 'model_error', messages, steps: step, error: error instanceof Error ? error.message : String(error) }; </code></pre> <p>The tool execution catch block still records an error message and continues the loop.</p> <p>The deterministic tests make that distinction concrete. In <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/02-tool-dispatch/tests/agent.test.ts"><code>tests/agent.test.ts</code></a>, an invalid <code>echo</code> argument is rejected <strong>without invoking the tool</strong>, while a thrown tool exception still allows the scripted model to return a final answer. <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/02-tool-dispatch/tests/registry.test.ts"><code>tests/registry.test.ts</code></a> checks nested argument validation and unexpected properties.</p> <h2 id="heading-3-give-the-agent-tools-without-giving-it-the-machine">3. Give the Agent Tools Without Giving It the Machine</h2> <p>So far, our agent can request tools and the harness validates their arguments. But validation only establishes that a request has the <em>right shape</em>. A valid request to read <code>../../secrets.txt</code> is still a request we should refuse.</p> <p>Check out <a href="https://github.com/gkoos/agent-harness-from-scratch/tree/03-workspace-tools">Milestone 3</a> in the companion repository:</p> <pre><code class="language-shell">git switch --detach 03-workspace-tools </code></pre> <p>This milestone adds <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/03-workspace-tools/src/tools/workspace.ts">workspace file tools</a> and a <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/03-workspace-tools/src/tools/commands.ts">restricted command tool</a>. Neither depends on a particular model provider.</p> <h3 id="heading-confine-file-operations-to-a-workspace">Confine File Operations to a Workspace</h3> <p>A coding agent needs to read and edit files, but it shouldn't automatically receive access to the entire filesystem. The <code>Workspace</code> class resolves a canonical root when it's created:</p> <pre><code class="language-typescript">const workspace = await Workspace.open('/path/to/project'); const tools = workspaceTools(workspace); </code></pre> <p>The resulting <code>read_file</code> and <code>write_file</code> tools accept paths relative to that root. Before opening a file, <code>Workspace</code> rejects absolute paths, paths escaping the root, and symbolic links in existing path components. Writes require the parent directory to exist. The tool doesn't create arbitrary directory trees.</p> <p>These two methods inside <code>Workspace</code> enforce the path checks. The file imports <code>isAbsolute</code>, <code>relative</code>, and <code>resolve</code> from <code>node:path</code>, and <code>lstat</code> and <code>realpath</code> from <code>node:fs/promises</code>. <code>Workspace.open()</code> stores <code>await realpath(root)</code> as <code>this.root</code>:</p> <pre><code class="language-typescript">private lexical(path: string): string { if (!path || isAbsolute(path)) throw new Error('Expected a relative workspace path'); const full = resolve(this.root, path); const rel = relative(this.root, full); if (!rel || rel === '..' || rel.startsWith(`..${process.platform === 'win32' ? '\\' : '/'}`) || isAbsolute(rel)) { throw new Error('Path is outside the workspace or refers to its root'); } return full; } /** Reject symlinks in every existing component, not just the final file. */ private async check(path: string, allowMissingFinal: boolean): Promise&lt;string&gt; { const full = this.lexical(path); const rel = relative(this.root, full); const parts = rel.split(/[\\/]/); let current = this.root; for (let i = 0; i &lt; parts.length; i++) { current = resolve(current, parts[i]!); try { const stat = await lstat(current); if (stat.isSymbolicLink()) throw new Error('Symbolic links are not permitted'); if (i &lt; parts.length - 1 &amp;&amp; !stat.isDirectory()) throw new Error('Parent is not a directory'); } catch (error) { if (allowMissingFinal &amp;&amp; i === parts.length - 1 &amp;&amp; (error as NodeJS.ErrnoException).code === 'ENOENT') break; throw error; } } return full; } </code></pre> <p><code>read</code> calls <code>this.check(path, false)</code> before opening an existing file. <code>write</code> calls <code>this.check(path, true)</code>, which allows only the final file to be missing. A missing parent directory or any symlink still fails the check.</p> <p>The boundary is enforced by the <strong>tool implementation</strong>, not by asking the model to behave. A model can request <code>../secret.txt</code>, but the workspace refuses to open it. The harness then reports the failure as a recoverable tool result, just as it did for invalid arguments in Milestone 2.</p> <p>For an additional check, the implementation opens files with <code>O_NOFOLLOW</code> where supported, to avoid following a symbolic link at the final path component. That doesn't eliminate every race: directory components can change between checking and opening them. This is a useful educational restriction, <strong>not</strong> an OS-level sandbox.</p> <h3 id="heading-let-the-model-choose-commands-not-invent-them">Let the Model Choose Commands, Not Invent Them</h3> <p>Running commands is even more sensitive. Rather than accept a shell string generated by the model, we register named command presets:</p> <pre><code class="language-typescript">const tool = commandTool(projectRoot, { test: { executable: process.execPath, args: ['--test', 'tests/example.test.js'] } }); </code></pre> <p>The model can request <code>{ "command": "test" }</code>, but it can't supply a different executable, extra arguments, or shell syntax. The command tool's dispatch boundary looks like this (inside its <code>execute(value, signal)</code> method):</p> <pre><code class="language-typescript">signal.throwIfAborted(); const name = (value as { command: string }).command; if (!Object.hasOwn(presets, name)) throw new Error('Command not permitted'); const preset = presets[name]!; const timeout = preset.timeoutMs ?? 10_000; if (!Number.isSafeInteger(timeout) || timeout &lt; 1) { throw new Error('Invalid timeout'); } // Inside the Promise that collects the command's result: const child = spawn(preset.executable, [...preset.args], { cwd, shell: false, signal, windowsHide: true, stdio: ['ignore', 'pipe', 'pipe'] }); const timer = setTimeout(() =&gt; child.kill(), timeout); </code></pre> <p>Here <code>spawn</code> comes from <code>node:child_process</code>, <code>cwd</code> is the configured project root, and <code>presets</code> is the developer-supplied map above. The result handlers collect stdout and stderr, kill the child if either captured string exceeds 65,536 characters, and clear <code>timer</code> on error or close. The implementation measures JavaScript string length, rather than an exact byte count.</p> <p>This reduces the number of commands the model can select, but it doesn't make the selected commands harmless. A test script can itself read files, access the network, or launch child processes. Killing a timed-out child also doesn't reliably terminate its entire process tree. Running untrusted repositories still requires process isolation, such as a properly configured container or OS sandbox.</p> <p>The <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/03-workspace-tools/tests/workspace.test.ts">Milestone 3 tests</a> cover normal reads and writes, traversal attempts, symbolic links, recoverable access failures, and rejected command names.</p> <h2 id="heading-4-put-limits-on-execution">4. Put Limits on Execution</h2> <p>Our agent can now call validated tools, including workspace-aware file operations and predefined commands. But a valid operation isn't necessarily one we should allow indefinitely. A model could request hundreds of tool calls, or a command could hang while the agent waits for its result.</p> <p>In this milestone, we make <strong>execution limits a harness responsibility</strong> rather than something we ask the model to respect.</p> <p>Follow along with the exact repository version:</p> <pre><code class="language-shell">git switch --detach 04-execution-limits </code></pre> <p>If you haven't cloned the project yet, start with <code>git clone https://github.com/gkoos/agent-harness-from-scratch.git</code> and <code>cd agent-harness-from-scratch</code>.</p> <h3 id="heading-count-turns-and-tool-calls-separately">Count Turns and Tool Calls Separately</h3> <p>The original <code>maxSteps</code> setting limits model responses, not the number of tool calls. One response can request several tools, so a turn limit alone doesn't bound the amount of work.</p> <p>We extend <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/04-execution-limits/src/core/agent.ts"><code>runAgent</code></a> with two additional options:</p> <pre><code class="language-typescript">const result = await runAgent({ model, tools, prompt: 'Check the project', maxSteps: 10, maxToolCalls: 30, timeoutMs: 15_000 }); </code></pre> <p>Every requested call consumes the tool-call budget, including unknown tools and calls with invalid arguments. This matters because even failed requests consume harness work. Initialize <code>let toolCalls = 0</code> before the loop, then increment it immediately before looking up each admitted call's tool. That placement counts rejected arguments and unknown tools too.</p> <p>The model may also return multiple tool calls at once. Instead of executing the first few and then discovering that the budget has run out, the harness checks whether <strong>the entire batch</strong> fits:</p> <pre><code class="language-typescript">if (calls.length &gt; maxToolCalls - toolCalls) { return { status: 'tool_limit', messages, steps, toolCalls }; } </code></pre> <p>This prevents <em>budget-driven</em> partial execution. It doesn't make the batch transactional: a tool can still fail, or execution can be cancelled partway through.</p> <h3 id="heading-propagate-cancellation-and-deadlines">Propagate Cancellation and Deadlines</h3> <p>A caller can supply an <code>AbortSignal</code>, while <code>timeoutMs</code> creates an internal deadline. The harness combines these through an <code>AbortController</code> and passes its signal to the model and tools.</p> <p>The setup replaces the initial loop's directly selected signal:</p> <pre><code class="language-typescript">const controller = new AbortController(); let timedOut = false; const onAbort = () =&gt; controller.abort(options.signal?.reason); if (options.signal?.aborted) onAbort(); else options.signal?.addEventListener('abort', onAbort, { once: true }); const timer = timeoutMs === undefined ? undefined : setTimeout(() =&gt; { timedOut = true; controller.abort(new Error('Agent deadline exceeded')); }, timeoutMs); const signal = controller.signal; </code></pre> <p>Pass this <code>signal</code> to <code>model.generate()</code> and <code>tool.execute()</code>, as in the original loop. Wrap the loop in <code>try</code>/<code>finally</code> so completion and early returns both release the timer and listener:</p> <pre><code class="language-typescript">try { // The execution loop goes here, using controller.signal. } finally { if (timer !== undefined) clearTimeout(timer); options.signal?.removeEventListener('abort', onAbort); } </code></pre> <p>At execution boundaries, check <code>signal.aborted</code> and return <code>timedOut ? 'timeout' : 'aborted'</code> as the status. The result also reports <code>steps</code> and <code>toolCalls</code>, so callers can see how much work happened before termination.</p> <p><strong>This is cooperative cancellation, not forced termination.</strong> If an adapter ignores its signal, a pending operation may outlive the deadline. Likewise, killing a child process doesn't necessarily stop its descendants. A hard execution boundary requires process or container isolation.</p> <h3 id="heading-test-the-limits-without-waiting-for-real-failures">Test the Limits Without Waiting for Real Failures</h3> <p>The <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/04-execution-limits/tests/budgets.test.ts"><code>budget tests</code></a> use the scripted model to request an over-budget batch and verify that <strong>zero tools execute</strong>. Other tests check that invalid requests consume budget, a pre-aborted signal prevents model invocation, and cooperative models and tools stop on deadlines.</p> <p>The result types in <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/04-execution-limits/src/core/types.ts"><code>types.ts</code></a> now distinguish <code>completed</code>, <code>step_limit</code>, <code>tool_limit</code>, <code>aborted</code>, <code>timeout</code>, and <code>model_error</code>. These outcomes are deliberately different: a run can stop correctly without completing the user's task.</p> <h2 id="heading-5-keep-the-context-under-control">5. Keep the Context Under Control</h2> <p>The harness now has limits on how long it can run, but its conversation keeps growing. Every model response and tool result becomes another message sent back to the model. A single file read could return thousands of lines. Several iterations can exhaust a model's context window long before the step budget is reached.</p> <p>Milestone 5 introduces a distinction between <strong>the transcript we retain</strong> and <strong>the messages we send to the model</strong>. Check out the implementation:</p> <pre><code class="language-shell">git switch --detach 05-context </code></pre> <p>The relevant files are <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/05-context/src/core/context.ts"><code>src/core/context.ts</code></a>, <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/05-context/src/core/agent.ts"><code>src/core/agent.ts</code></a>, and <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/05-context/tests/context.test.ts"><code>tests/context.test.ts</code></a>.</p> <h3 id="heading-bound-tool-output-and-model-facing-history">Bound Tool Output and Model-Facing History</h3> <p>The new <code>context</code> option sets two separate limits:</p> <pre><code class="language-typescript">const result = await runAgent({ prompt: 'Inspect the project', model, tools, context: { maxContextChars: 32_000, maxToolResultChars: 8_000 } }); </code></pre> <p><code>maxToolResultChars</code> bounds each tool result <strong>before</strong> it enters the transcript. Oversized results retain their beginning and end, with an omission marker in the middle. The discarded portion isn't available later, even in <code>result.messages</code>.</p> <p>The truncation helper reserves room for the marker, then divides the remaining space between the beginning and end:</p> <pre><code class="language-typescript">export function boundToolResult(content: string, limit: number): string { if (content.length &lt;= limit) return content; const marker = `\n...[${content.length - limit} or more characters omitted]...\n`; if (marker.length &gt;= limit) return marker.slice(0, limit); const remaining = limit - marker.length; const front = Math.ceil(remaining / 2); const back = Math.floor(remaining / 2); return content.slice(0, front) + marker + (back ? content.slice(-back) : ''); } </code></pre> <p>Apply it when recording both successful results and error strings: <code>content: boundToolResult(content, context.maxToolResultChars)</code>. This compact version explicitly handles a zero-length tail so <code>slice(-0)</code> can't accidentally retain the whole result.</p> <p><code>maxContextChars</code> limits the serialized message array passed to the model. The harness retains older messages in its returned transcript, but <code>selectContext()</code> sends the initial user request plus as many recent turns as fit. There's no summarization: omitted history is simply unavailable to the model on that turn.</p> <p>These are character limits, not token limits. They measure JavaScript string length after JSON serialization, excluding tool definitions and provider-specific overhead. A real provider adapter would need token-aware accounting and space reserved for the response.</p> <h3 id="heading-dont-split-a-tool-call-from-its-result">Don't Split a Tool Call From its Result</h3> <p>Context trimming isn't as simple as removing the oldest messages. An assistant response may request several tools, each identified by a call ID. Sending only some of their results would create an invalid or misleading conversation.</p> <p>The selector therefore treats the assistant message <strong>and all results from that batch</strong> as an indivisible group. It either includes the complete group or omits it. It also rejects orphaned results and incomplete batches rather than quietly constructing inconsistent context.</p> <p>Here's <code>selectContext()</code> from <code>src/core/context.ts</code>, using the <code>Message</code> type defined in Section 1. The first loop validates and groups the transcript. The second walks backward through whole groups until the budget is full:</p> <pre><code class="language-typescript">import type { Message } from './types.ts'; const size = (messages: readonly Message[]) =&gt; JSON.stringify(messages).length; export function selectContext(messages: readonly Message[], maxChars: number): Message[] { if (!Number.isSafeInteger(maxChars) || maxChars &lt; 1) { throw new RangeError('maxContextChars must be a positive safe integer'); } if (!messages.length) return []; const groups: Message[][] = []; for (let i = 0; i &lt; messages.length;) { const message = messages[i]!; if (message.role === 'assistant' &amp;&amp; message.toolCalls?.length) { const ids = new Set(message.toolCalls.map(call =&gt; call.id)); if (ids.size !== message.toolCalls.length) throw new Error('Duplicate tool call IDs in assistant turn'); const group: Message[] = [message]; i++; while (i &lt; messages.length &amp;&amp; messages[i]!.role === 'tool') { const result = messages[i]!; if (result.role === 'tool' &amp;&amp; !ids.delete(result.callId)) throw new Error('Orphan or duplicate tool result'); group.push(result); i++; } if (ids.size) throw new Error('Incomplete tool result batch'); groups.push(group); } else { if (message.role === 'tool') throw new Error('Orphan tool result'); groups.push([message]); i++; } } const first = groups[0]!; if (first[0]?.role !== 'user') throw new Error('Conversation must start with a user message'); const recent: Message[][] = []; for (let i = groups.length - 1; i &gt;= 1; i--) { const group = groups[i]!; const next = [...first, ...[group, ...recent].flat()]; if (size(next) &gt; maxChars) { if (recent.length === 0) throw new Error('Context budget too small for latest turn'); break; } recent.unshift(group); } const selected = [...first, ...recent.flat()]; if (size(selected) &gt; maxChars) throw new Error('Context budget too small for required messages'); return selected; } </code></pre> <p>Before each model call, use <code>selectContext(messages, context.maxContextChars)</code> as its <code>messages</code> input. If the initial prompt or the most recent required group can't fit, the selector throws and the harness returns <code>context_limit</code>. That's an execution decision, not a tool error the model can recover from.</p> <p>The deterministic tests exercise batch preservation, orphan detection, output truncation, and the model-facing projection. They verify not merely that the context is smaller, but that it still represents a coherent conversation. Next, we'll make these decisions observable.</p> <h2 id="heading-6-make-agent-execution-observable">6. Make Agent Execution Observable</h2> <p>An agent can return a correct answer while doing something unexpected along the way: retrying a failing operation, making unnecessary model calls, or spending most of its time in one tool. A final answer alone doesn't tell us what happened. We need execution events, without turning the harness into a logging framework.</p> <p>Check out this milestone in the <a href="https://github.com/gkoos/agent-harness-from-scratch">companion repository</a>:</p> <pre><code class="language-shell">git switch --detach 06-observability </code></pre> <p>The implementation lives in <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/06-observability/src/core/events.ts"><code>src/core/events.ts</code></a>, with integration in <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/06-observability/src/core/agent.ts"><code>src/core/agent.ts</code></a>.</p> <h3 id="heading-emit-events-at-execution-boundaries">Emit Events at Execution Boundaries</h3> <p>We add an optional <code>onEvent</code> callback to <code>runAgent</code>. The harness emits <code>model_start</code> and <code>model_end</code> around model calls, <code>tool_start</code> and <code>tool_end</code> around tool dispatch, <code>model_error</code> when a model operation fails, and <code>run_end</code> when the run terminates.</p> <p>Each event has a run ID, an incrementing sequence number, and a timestamp. Model and tool completion events also report duration, measured using a monotonic clock. That gives us enough information to reconstruct the order of operations and identify slow calls.</p> <pre><code class="language-typescript">const events: AgentEvent[] = []; const result = await runAgent({ prompt: 'Inspect the project', model, tools, onEvent: event =&gt; events.push(event), }); console.log(result.runId, events); </code></pre> <p>Import <code>AgentEvent</code> from <code>src/core/events.ts</code>. The callback is optional: callers that don't need telemetry can omit it.</p> <p>One subtle requirement is that <strong>observability must not control execution</strong>. An exception in an event listener shouldn't make a successful tool call fail. The event emitter catches listener exceptions and optionally reports them through <code>onObserverError</code>, and failures in that handler are suppressed too. Callbacks are synchronous, though, so slow listeners still delay the agent.</p> <p>The emitter creates <code>runId</code> once with <code>randomUUID()</code> from <code>node:crypto</code>, and initializes <code>sequence</code> to zero. Its <code>emit(event)</code> method enriches the event and isolates both callbacks:</p> <pre><code class="language-typescript">const enriched = { ...event, runId, sequence: ++sequence, timestamp: new Date().toISOString() }; try { options.onEvent?.(enriched); } catch (error) { try { options.onObserverError?.(error); } catch { /* Observer failures cannot change the run outcome. */ } } </code></pre> <p>At a model or tool boundary, save <code>const start = performance.now()</code> before execution and emit <code>durationMs: performance.now() - start</code> afterward. A shared <code>finish()</code> helper emits <code>run_end</code> before returning each terminal result, so early exits remain observable too.</p> <h3 id="heading-account-for-model-usage-without-inventing-it">Account for Model Usage Without Inventing it</h3> <p>We can also extend the model response with optional provider-reported token usage:</p> <pre><code class="language-typescript">return { content: 'Done', usage: { inputTokens: 125, outputTokens: 20 }, }; </code></pre> <p>The harness accumulates reported counts in <code>result.usage</code>, alongside <code>modelCallsWithUsage</code>. This last field matters: zero reported tokens with zero usage-reporting calls means <strong>unknown usage</strong>, not a free run. Invalid counts are rejected rather than silently corrupting totals.</p> <p>The accumulator uses optional input and output counts and validates both individual values and the resulting totals:</p> <pre><code class="language-typescript">export type Usage = { inputTokens?: number; outputTokens?: number }; export type UsageTotals = { inputTokens: number; outputTokens: number; modelCallsWithUsage: number }; export function addUsage(totals: UsageTotals, usage: Usage | undefined): void { if (!usage) return; for (const value of [usage.inputTokens, usage.outputTokens]) { if (value !== undefined &amp;&amp; (!Number.isSafeInteger(value) || value &lt; 0)) { throw new RangeError('Model usage must contain nonnegative safe integers'); } } if (!Number.isSafeInteger(totals.inputTokens + (usage.inputTokens ?? 0)) || !Number.isSafeInteger(totals.outputTokens + (usage.outputTokens ?? 0))) { throw new RangeError('Accumulated model usage exceeds safe integer range'); } totals.inputTokens += usage.inputTokens ?? 0; totals.outputTokens += usage.outputTokens ?? 0; totals.modelCallsWithUsage++; } </code></pre> <p>Initialize the totals to zero at the start of the run. Add <code>usage?: Usage</code> to the model response contract, call <code>addUsage(totals, response.usage)</code> after validating each response, and include a copy of the totals in the result. Missing usage leaves the counters unchanged.</p> <p>The event payloads deliberately omit prompts, tool arguments, and tool results. Error strings may still contain sensitive data, so exporting them requires care. This is lightweight, in-memory instrumentation, not distributed tracing or a durable audit log.</p> <p>The tests in <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/06-observability/tests/events.test.ts"><code>tests/events.test.ts</code></a> verify event order, correlation, usage accounting, observer isolation, and terminal events.</p> <h2 id="heading-7-test-the-failure-paths-not-just-the-happy-path">7. Test the Failure Paths, Not Just the Happy Path</h2> <p>Our harness now has tool validation, execution limits, context management, and observability. But those mechanisms interact. A model might return malformed tool calls, cancellation might arrive halfway through a batch, or a rejected batch might leave the conversation in a state that can't be sent back to a model. The final milestone tests those boundaries directly.</p> <p>Check out the final milestone in the <a href="https://github.com/gkoos/agent-harness-from-scratch">companion repository</a>:</p> <pre><code class="language-shell">git switch --detach 07-testing </code></pre> <p>The new tests are in <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/07-testing/tests/failures.test.ts"><code>tests/failures.test.ts</code></a>. They use the same scripted model as our first milestone, so failures are reproducible without an API key.</p> <h3 id="heading-validate-the-model-response-before-trusting-it">Validate the Model Response Before Trusting it</h3> <p>Validating tool <em>arguments</em> isn't enough. A model adapter can also return a malformed response: missing <code>content</code>, non-array <code>toolCalls</code>, missing arguments, or duplicate call IDs. The new <a href="https://github.com/gkoos/agent-harness-from-scratch/blob/07-testing/src/core/response.ts"><code>validateModelResponse()</code></a> checks the response envelope before the harness records it or executes anything.</p> <pre><code class="language-typescript">import type { Model, ToolCall } from './types.ts'; /** Validate the adapter boundary before accepting a model turn or running side effects. */ export function validateModelResponse(value: unknown): asserts value is Awaited&lt;ReturnType&lt;Model['generate']&gt;&gt; { if (value === null || typeof value !== 'object' || Array.isArray(value)) { throw new TypeError('Model response must be an object'); } const response = value as Record&lt;string, unknown&gt;; if (typeof response.content !== 'string') throw new TypeError('Model response content must be a string'); if (response.toolCalls === undefined) return; if (!Array.isArray(response.toolCalls)) throw new TypeError('Model toolCalls must be an array'); const ids = new Set&lt;string&gt;(); for (const call of response.toolCalls as unknown[]) { if (call === null || typeof call !== 'object' || Array.isArray(call)) { throw new TypeError('Tool call must be an object'); } const item = call as Partial&lt;ToolCall&gt;; if (typeof item.id !== 'string' || !item.id.trim()) throw new TypeError('Tool call id must be a nonempty string'); if (typeof item.name !== 'string' || !item.name.trim()) { throw new TypeError('Tool call name must be a nonempty string'); } if (!Object.hasOwn(item, 'arguments')) throw new TypeError('Tool call arguments are missing'); if (ids.has(item.id)) throw new TypeError(`Duplicate tool call id: ${item.id}`); ids.add(item.id); } } </code></pre> <p>Call <code>validateModelResponse(response)</code> and then <code>addUsage(usage, response.usage)</code> inside a <code>try</code> block immediately after <code>model.generate()</code>, before appending the assistant message or running tools. If either throws, emit <code>model_error</code> and return <code>finish('model_error', { error: message })</code>, where <code>message</code> is the exception's message.</p> <p>Malformed responses terminate as <code>model_error</code>, with no tool side effects. That's different from a valid tool call with invalid arguments: the latter becomes a recoverable tool error, allowing the model to try again.</p> <h3 id="heading-keep-rejected-batches-out-of-the-transcript">Keep Rejected Batches Out of the Transcript</h3> <p>The tests also caught a real integration bug. Previously, the harness appended an assistant message containing tool calls <em>before</em> checking whether the batch exceeded <code>maxToolCalls</code>. Rejecting that batch left unanswered tool calls in the transcript.</p> <p>The fix is small but important: <strong>check the budget before recording the assistant turn</strong>.</p> <pre><code class="language-typescript">if (calls.length &gt; maxToolCalls - toolCalls) { return finish('tool_limit'); } messages.push({ role: 'assistant', content: response.content, ...(calls.length ? { toolCalls: calls } : {}) }); </code></pre> <p>This makes budget rejection atomic with respect to the transcript. Cancellation during an already admitted batch can still leave an incomplete terminal transcript. Resuming such a run would require reconciliation.</p> <h3 id="heading-use-reproducible-variation">Use Reproducible Variation</h3> <p>Alongside targeted tests for malformed responses, cancellation, and observer failures, we'll run <strong>150 seeded tool-batch scenarios</strong> with different batch sizes and budgets. A fixed pseudo-random seed generates the same sequence on every run, making failures reproducible.</p> <p>The following examples belong in <code>tests/failures.test.ts</code>. They share a no-op tool and a helper that creates well-formed calls:</p> <pre><code class="language-typescript">import { test } from 'node:test'; import assert from 'node:assert/strict'; import { runAgent } from '../src/core/agent.ts'; import { scriptedModel } from '../src/models/fake.ts'; import type { Tool } from '../src/core/types.ts'; const noop: Tool = { name: 'noop', description: 'Return a fixed result', parameters: { type: 'object' }, async execute() { return 'ok'; } }; const call = (id: string) =&gt; ({ id, name: 'noop', arguments: {} }); </code></pre> <p>The first test varies budgets from 0 to 8 and batch sizes from 0 to 12. It checks execution counts, termination status, and whether rejected calls were kept out of the transcript:</p> <pre><code class="language-typescript">test('seeded batches respect tool budgets', async () =&gt; { let seed = 0x12345678; const next = () =&gt; { seed ^= seed &lt;&lt; 13; seed ^= seed &gt;&gt;&gt; 17; seed ^= seed &lt;&lt; 5; return seed &gt;&gt;&gt; 0; }; for (let i = 0; i &lt; 150; i++) { const limit = next() % 9; const batch = next() % 13; let executed = 0; const result = await runAgent({ prompt: 'test', maxToolCalls: limit, tools: [{ ...noop, async execute() { executed++; return 'ok'; } }], model: scriptedModel([ { content: '', toolCalls: Array.from({ length: batch }, (_, j) =&gt; call(`call-${j}`)) }, { content: 'done' } ]) }); const admitted = batch &lt;= limit; assert.equal(executed, admitted ? batch : 0, `case ${i}`); assert.equal(result.toolCalls, admitted ? batch : 0, `case ${i}`); assert.equal( result.status, admitted ? 'completed' : 'tool_limit', `case ${i}` ); if (!admitted) { assert.deepEqual(result.messages, [{ role: 'user', content: 'test' }]); } } }); </code></pre> <p>An admitted batch executes all requested tools, while an over-budget batch executes zero. The <code>case</code> labels identify which generated scenario failed, and the fixed seed lets us reproduce it.</p> <p>Another 60 runs check that tool exceptions remain recoverable errors. These repeat the same failure path with different error messages. They don't use seeded variation:</p> <pre><code class="language-typescript">test('tool exceptions remain recoverable', async () =&gt; { let executed = 0; for (let i = 0; i &lt; 60; i++) { const result = await runAgent({ prompt: 'test', tools: [{ ...noop, async execute() { executed++; throw new Error(`failure ${i}`); } }], model: scriptedModel([ { content: '', toolCalls: [call('1')] }, { content: 'recovered' } ]) }); assert.equal(result.status, 'completed'); assert.equal(result.answer, 'recovered'); assert.ok(result.messages.some(message =&gt; message.role === 'tool' &amp;&amp; message.isError &amp;&amp; message.content === `failure ${i}` )); } assert.equal(executed, 60); }); </code></pre> <p>The seeded scenarios are property-style checks, while the exception test checks a specific recovery path repeatedly. Neither proves correctness or replaces operating-system sandboxing, but both run without a live model and give us repeatable evidence about the harness's behavior.</p> <p>We've now built the harness from a simple model/tool loop into an execution layer that owns validation, permissions, limits, context, and telemetry. The model still proposes actions while <strong>the harness remains responsible for deciding what actually happens</strong>.</p> <p>Run these examples with <code>npm test</code>. The repository includes additional targeted cases for the boundaries discussed above, alongside the complete milestone implementations.</p> <h2 id="heading-where-to-go-next-connect-a-real-language-model">Where to Go Next: Connect a Real Language Model</h2> <p>Throughout this tutorial, we've used a scripted model instead of calling a real LLM. That was intentional: deterministic responses let us test the harness, including its failure paths, without network calls, API keys, or unpredictable model behavior.</p> <p>But the <code>Model</code> interface was designed to make providers interchangeable. Connecting OpenAI, Anthropic, or another provider doesn't require rewriting the agent loop. We just need an adapter that translates between the provider's API and our existing types.</p> <p>For example, an OpenAI adapter using the Responses API could look something like this:</p> <pre><code class="language-typescript">// src/models/openai.ts import OpenAI from 'openai'; import type { Model } from '../core/types.ts'; export function openAIModel(client: OpenAI): Model { return { async generate({ messages, tools, signal }) { const response = await client.responses.create( { model: 'YOUR_MODEL', input: toOpenAIInput(messages), tools: toOpenAITools(tools), }, { signal } ); return { content: extractText(response), toolCalls: extractToolCalls(response), usage: extractUsage(response), }; }, }; } </code></pre> <p>This is an architectural sketch, not a complete adapter. The conversion functions would translate our messages and tool definitions into the provider's format, then normalize its response. In particular, they must preserve tool-call IDs, parse tool arguments, and associate tool results with the correct calls on subsequent requests.</p> <p>The adapter also forwards the harness's <code>AbortSignal</code>, allowing cancellation to propagate to the API request. Provider errors become exceptions, which the harness already handles as <code>model_error</code> outcomes.</p> <p>Once the adapter implements <code>Model</code>, the rest of the system stays unchanged. The same tool registry, argument validation, execution budgets, context management, and observability work regardless of which provider generates the next response.</p> <p>There are further improvements worth exploring: provider-specific token accounting, API retry and rate-limit handling, persistent conversation history, and stronger process isolation for running untrusted code. These are substantial engineering problems in their own right, but they fit into the boundaries we've established rather than requiring a different architecture.</p> <p>The complete implementation is available at the <a href="https://github.com/gkoos/agent-harness-from-scratch/tree/07-testing"><code>07-testing</code> milestone</a>.</p> <p>We've built more than a loop that sends prompts to a model and executes functions. We've built the machinery that decides <strong>which actions are permitted, how much work can happen, what context the model receives, and how failures are handled and observed</strong>.</p> <p>That's the essential distinction between a language model with tools and an agent harness.</p>