OpenAI Evals Shutdown: Move to Promptfoo by November 30

What the Deprecations page says

OpenAI's Deprecations page lists three dates for the Evals platform. On June 3, 2026, OpenAI announced the deprecation. On Oct 31, 2026, "Existing evals become read-only." On Nov 30, 2026, "The Evals dashboard and API are scheduled to shut down." The entry adds that graders documented for eval workflows are part of the same transition.

Two things are missing from that entry. It does not describe an export feature, and it does not say what happens to stored eval runs, datasets or scores after November 30. Promptfoo's command-line reference documents one partial path: promptfoo import accepts an OpenAI Evals dashboard JSONL export (eval_items_*.jsonl) as historical results. Those files hold output-item rows, not the eval definition, data-source config, run config or testing criteria, so they can keep old outputs on record but cannot replace the rebuild. Treat the read-only month as your window to copy out everything you need to rebuild and compare, and assume that whatever you have not recorded by November 30 may be gone.

Inventory every eval while it is still readable

Before you write any Promptfoo config, make one inventory row per eval that still feeds a decision, such as a release gate or a model comparison. For each one, record the prompt templates and their variables, the model and any sampling settings, the test rows with their expected outputs, every testing criterion with its pass threshold, and the scores from the most recent run you trust. That last item is the baseline you will compare against later; without it you cannot tell whether the rebuild drifted.

Prioritize by consequence, not by count. An eval that blocks deploys deserves a full rebuild this month; an abandoned experiment from spring may just need its dataset saved. If you inherited evals without an owner, the read-only dashboard is the last convenient place to see which criteria they used.

Recreate one eval in promptfooconfig.yaml

OpenAI's migration guide to Promptfoo covers manually recreating an evaluation and says it "does not require an OpenAI Evals export feature." You rebuild the eval's prompts, provider, test cases and assertions in a promptfooconfig.yaml file, with an API key for the model provider such as OPENAI_API_KEY. The guide maps OpenAI's graders, configured as testing criteria, to Promptfoo assertions and metrics.

A small ticket classifier shows the shape. The prompt asks for nothing but a label, and the assertion checks that exact label, so the grader measures what the prompt requests. The keys follow Promptfoo's configuration guide, and equals is the documented "output matches exactly" assertion.

# promptfooconfig.yaml
prompts:
  - |
    Classify the support ticket as billing, bug or other.
    Reply with the label only.

    Ticket: {{ticket}}
providers:
  - openai:gpt-6-luna   # use the model your OpenAI eval ran
tests:
  - vars:
      ticket: I was charged twice for October.
    assert:
      - type: equals
        value: billing
        metric: label_match
  - vars:
      ticket: The export button returns a 500 error.
    assert:
      - type: equals
        value: bug
        metric: label_match

For a real eval, load the inventoried rows from a file with tests: file://tests.csv instead of listing them inline. Criteria that a model judged map to model-graded assertions such as llm-rubric, and Promptfoo's assertion reference lets you set the judge with a provider on the assertion. Pin it to the judge model your old grader used, or the comparison changes two variables at once. The guide also flags tool, agent and custom provider workflows as cases you must configure and test yourself.

Validate, run and view

The commands come from the OpenAI guide and match the Promptfoo command-line reference:

promptfoo validate config -c promptfooconfig.yaml
promptfoo eval -c promptfooconfig.yaml --no-cache
promptfoo view

Run validation first so that a misspelled key fails before you spend model calls. Promptfoo documents --no-cache as "Do not read or write results to disk cache", which keeps a baseline run from reusing an earlier response. promptfoo view opens a browser UI for the results. The new runs live in Promptfoo and are separate from your earlier OpenAI Evals history.

Compare old and new scores before switching

The OpenAI guide tells you to compare the recreated prompts, assertions and outputs before using the eval in an ongoing workflow, and warns that similarity-based scores may differ numerically between systems. The guide does not prescribe a comparison method; this is the one we suggest. Run the rebuilt eval on the same rows as your saved baseline, then put both results side by side per criterion. Matching aggregate pass rates are not enough: look at the individual rows where one system passed and the other failed.

A mismatch usually points to a difference you can name, such as a changed model, a different threshold or a template variable that lost its whitespace. Fix the config so that it reproduces the old behavior, rather than moving a threshold until the totals line up. If a gap remains, write it down with the baseline so the next regression decision accounts for it.

Validate recreated LLM-as-a-judge graders

The guide's sharpest warning is about judges: "Any manually recreated grader, especially an LLM-as-a-judge grader, should be validated" before you rely on it for regression decisions. A rewritten rubric can pass the same outputs for different reasons. Take a few dozen rows that a person has already labeled, run the new judge on them and count where it agrees. If the judge compares two answers, also test it with the answer order swapped; our position-bias article explains why order alone can move a verdict. For the wider method of keeping evals honest in production, see the AI evals guide.

Two other OpenAI shutdowns on November 30

The same Deprecations page schedules two related shutdowns for Nov 30, 2026. The v1/prompts API and reusable prompt objects go away, and OpenAI's prompt object migration guide moves that content into application code. Agent Builder also shuts down, while ChatKit remains available; the Agent Builder migration guide covers exporting workflows as Agents SDK code. If an eval used a stored prompt, rebuild its template in the Promptfoo config rather than referencing an object that will no longer exist. The earlier Assistants API shutdown report covers a separate OpenAI retirement.