Start With the Responsibility Boundary
OpenAI's current Agents guide positions the Agents API as the managed option for long-running tasks: OpenAI runs a Codex harness and saves session configuration, turns, and items. The nearby alternatives keep more orchestration in your application. That is an architectural choice, not evidence that an existing agent will behave the same after migration.
The architecture guide draws the line more precisely. The hosted harness runs the model and tool loop and maintains the session. Your application server still submits tasks, receives events, and handles function tools. If you provide the execution environment, your code also manages its lifecycle. A remote MCP tool may be called by the harness, while a function call comes back to your application for execution.
With environment.type: "none", the hosted path has no built-in Bash, apply-patch, workspace files, or executor MCPs. If the current agent uses any of those, either configure an equivalent environment or remove that capability from both test paths. A prettier answer produced with fewer permissions is not behavioral equivalence.
Freeze One Portable Fixture
Choose one task your current agent completes often enough that you know its failure modes, but keep side effects harmless. A good fixture is a small repository with a failing test, a short issue, and one function tool called record_candidate. The accepted result is a patch that makes the test pass plus one structured record containing the changed file, test command, and final status. The tool writes to a disposable evidence store.
Pin the same model identifier, instructions, input files, tool names, JSON schemas, time limit, retry limit, and acceptance test in both paths. Give each run a fresh workspace and a unique fixture ID. Record the exact configuration rather than saying “same settings”; defaults on either side are part of the harness and can differ.
{
"fixture_id": "hb-001",
"task": "Fix src/parse.js so npm test passes",
"accepted_when": [
"process_exit_code == 0",
"tests_failed == 0",
"record_candidate_calls == 1",
"recorded_status == 'passed'"
],
"limits": { "wall_seconds": 300, "agent_retries": 0 }
}The prompt and score now ask for the same thing: a passing patch and one valid record. Do not award points for eloquence or for a particular chain of thought. Compare observable artifacts. If stochastic variation matters, repeat the paired run later; first establish that one completely documented pair can be reproduced.
Capture Five Columns of Comparable Evidence
| Field | Current harness | Hosted Agents API | Acceptance rule |
|---|---|---|---|
| Accepted output | Patch hash, exit code, test count | Same three values | Both satisfy every fixture predicate |
| Tool calls | Name, arguments hash, result hash, executor | Saved item/event plus application log | Exactly one valid side-effecting call |
| Context | Inputs supplied and retained state | Session ID, turns, items, supplied files | No forbidden file and no missing required input |
| Latency | Monotonic start/end timestamps | Same client-side timestamps | Within your declared task budget |
| Cost basis | Model tokens, tools, compute | Recorded usage plus tools, sandbox, retries | Estimated with one dated rate card |
The events and items guide gives the hosted path two useful evidence streams. Events report live activity; items preserve messages and tool calls for later retrieval. Store both, but treat your application log as the authority for a function that your application executed. It can bind the call ID, canonicalized arguments, result, fixture ID, and timestamp into one record.
For a side-effecting call, also carry an idempotency key derived from the fixture ID and operation. This comparison checks whether each harness made the expected call. The deeper design for retries belongs in the idempotent tool-call guide; use its pattern rather than rebuilding retry safety inside this test.
Probe Context Across a Second Turn
A one-turn patch can pass even when the migration breaks continuity. After the patch is accepted, send the same follow-up in each path: “Report the fixture ID, the file changed, the exact test command, and whether the record tool succeeded. Do not call tools.” Score four literal fields. This checks whether the context needed by your workflow remains available without rewarding extra prose.
On the hosted path, save the session ID and retrieve the relevant turns and items. On the current path, save the history or checkpoint identifier your harness actually uses. Do not require identical serialized state; require the same accepted facts and no additional tool call. If your existing design must recover after a process crash, keep that as a separate gate using the checkpoint and replay procedure. Hosted session persistence does not erase your application's recovery responsibilities around external work.
Measure Latency and Cost Without False Precision
Measure wall time at the same boundary: immediately before the client submits the initial task and immediately after the final accepted artifact is available. Also record time to first event and time spent waiting for application-handled tools. One run cannot establish a service-level objective, but it can expose a boundary mistake such as a function handler adding most of the delay.
OpenAI's observability guide says session and turn usage is best-effort: it may be null, may change as accounting arrives, and is not a final bill. Missing usage therefore means unknown, not zero. An agent may make several model calls, and total cost can also include tool charges, sandbox compute, third-party services, retries, and subagents.
Estimate both paths with the same rate card captured on the test date. Keep input, cached input, output, and reasoning-token categories separate when the selected model exposes them. Add external charges as named line items. If a value is unavailable, mark the paired comparison incomplete. Do not turn an incomplete denominator into a claim that the hosted path is cheaper.
Write a Go/No-Go Record
The decision record should fit on one page. Name the two harness versions and every pinned input. Link the raw evidence bundle. For each of the five columns, write pass, fail, or unknown, followed by the observed value and threshold. Then list responsibilities that moved to OpenAI and responsibilities your application retains.
Approve GO when:
accepted_output == pass
tool_call_evidence == pass
context_follow_up == pass
latency_budget == pass
cost_basis != unknown
retained_owners_are_named == trueA no-go is useful when it names the boundary that failed. “Function result was returned but not correlated to the accepted artifact” points to application evidence. “Required shell unavailable with no environment” points to configuration. “Second turn omitted the test command” points to context behavior. Fix one cause, preserve the fixture, and rerun the pair.
Even a go decision is fixture-specific. Retest after changing the model, instructions, tool schema, environment type, acceptance rule, or the application handler. The goal is not to prove that two implementations are identical. It is to show that the hosted boundary preserves the behavior and evidence your particular workflow requires—and to make every responsibility that remains yours impossible to overlook.