OpenAI Evals Turns Read-Only Before November Shutdown | Zeniteq
Loading
Loading article
Zeniteq is now available on mobile for iPhone, iPad and Apple Watch.Download nowLearn more
OpenAI Evals Turns Read-Only Before November Shutdown
Teams using the legacy Evals dashboard or API have until November 30 to preserve their evaluation records, rebuild workflows, and validate OpenAI’s recommended Promptfoo replacement.
OpenAI is transitioning its legacy Evals platform to a read-only state on October 31, 2026, followed by a full dashboard and API shutdown on November 30, 2026. Teams must manually migrate their evaluation workflows to the recommended Promptfoo system before these deadlines, as no automated transfer tool exists.
This migration requires more than just moving configurations; teams must carefully validate that recreated graders produce consistent, trustworthy results. Users should prioritize preserving historical data and testing new workflows against known benchmarks to ensure that scoring differences do not inadvertently disrupt release processes or quality standards.
OpenAI’s legacy Evals platform becomes read-only on October 31, 2026, ahead of the scheduled shutdown of its dashboard and API on November 30, 2026. Teams that depend on the service for model-quality checks now need to remove write dependencies and complete their migration during November. OpenAI lists Promptfoo as the recommended migration path in its deprecation schedule.
October 31 is an access-change milestone in an existing deprecation schedule. The immediate job is to preserve what the old system knows and make sure its replacement catches the same failures.
OpenAI’s official migration guide describes manually recreating evaluations in Promptfoo. That distinction matters: moving the configuration is only part of the work. Teams also need to check whether recreated graders still make trustworthy release decisions.
What Changes on October 31
OpenAI’s Evals documentation confirms that existing content remains available during the transition window, with read-only access beginning October 31 and shutdown scheduled for November 30.
Milestone
Documented change
Practical response
October 31, 2026
Existing Evals become read-only
Stop depending on operations that create or change platform state.
November transition window
Existing content remains available ahead of shutdown
Capture definitions, data, and useful historical results while rebuilding workflows.
November 30, 2026
Evals dashboard and API are scheduled to shut down
Complete the cutover before this date.
The published dates do not specify a cutoff time or time zone. Neither midnight UTC nor access through the end of November 30 should be treated as guaranteed.
Read-only access also changes how teams should think about the remaining month. It is a retrieval and migration window, not a period for continuing normal development on the legacy platform. Review anything that creates evaluations, modifies criteria, or launches and records new runs. The cited notice does not provide an endpoint-by-endpoint account of write failures, so avoid building a workaround around assumed error behavior.
My recommendation is to separate two deadlines internally: preserve essential records immediately, then finish validation and cutover comfortably before November 30. Otherwise, an unexpectedly difficult grader migration could consume the time needed to recover historical context.
Inventory and Preserve the Existing Evals
OpenAI’s migration guide says:
“This guide covers manually recreating an evaluation in Promptfoo.”
The guide does not document a complete history-export procedure or an automated transfer. That is narrower than saying no export capability exists anywhere. Teams should use available retrieval or export mechanisms where supported, then manually capture whatever the replacement needs.
Create an inventory with an owner and migration status for each evaluation. Include:
Evaluation purpose: The behavior being checked, the application it protects, and whether it blocks a release.
Historical evidence: Useful outputs, per-case scores, known failures, run identifiers, and dates.
Dependencies: CI jobs, scheduled scripts, dashboards, and deployment checks that consume the results.
Store enough material outside the retiring platform to explain a past pass or failure. A copied dashboard URL will have limited value after shutdown.
For example, a historical score of 92% is hard to interpret without its dataset, prompt, model, and grading criteria. Preserving that context makes the score useful as a baseline. Preserving only the percentage leaves future maintainers guessing.
Prioritize release-blocking suites and evaluations that contain hard-to-recreate reference data. Large collections of experimental runs may deserve a lighter archive, depending on their value and your retention requirements.
Map Each Evaluation Into Promptfoo
Promptfoo is an open-source CLI and library for evaluating and red-teaming AI applications and agents. OpenAI’s guide describes a configuration-centered workflow that can run locally or in continuous integration.
The documented migration mapping preserves familiar building blocks, but changes where they live and how they run:
Legacy Evals component
Promptfoo destination
What to verify
Evaluation definition
Portable configuration, such as promptfooconfig.yaml
The same task and cases are represented.
Prompts and input variables
Prompt configuration and test cases
Roles, substitutions, and reference fields retain their meaning.
Provider and model
Provider configuration
The intended model and required settings are selected.
Graders
Assertions and metrics
Rubrics, thresholds, and pass/fail behavior remain appropriate.
Platform-managed runs
CLI or CI execution
Results reach the people and systems that need them.
OpenAI explicitly distinguishes this workflow from reproducing the old dashboard in another tool. Evaluation configuration can sit alongside application code, while execution becomes part of development and deployment processes.
That is useful for reviewing prompt and grader changes together. It also means teams must decide where credentials are supplied, who maintains the configuration, and how results affect a build.
Tool-using applications need particular attention. OpenAI’s guide says agent, tool, or custom-provider workflows require additional configuration when they are not represented in the recreated suite. A successful migration of a plain-text prompt does not demonstrate that a tool-enabled workflow has been preserved.
OpenAI also says platform features such as dataset review and prompt iteration remain in OpenAI. Those features should not be treated as proof that every legacy Evals dashboard or API dependency has a direct replacement.
Validate Behavior, Not Only the YAML
OpenAI lists three prerequisites for its migration walkthrough: an installed Promptfoo environment, an existing evaluation, and a model-provider API key, such as OPENAI_API_KEY.
After recreating the evaluation in promptfooconfig.yaml, the guide documents this configuration-validation command:
promptfoo validate config -c promptfooconfig.yaml
The command checks the configuration. It is a useful preliminary step, but a valid configuration cannot tell you whether a recreated rubric measures the intended behavior.
The guide then provides commands for a fresh evaluation and the local results viewer:
The first command creates a new Promptfoo evaluation run using the specified configuration without using cached results. The second opens the results locally. OpenAI notes that this run is separate from previously completed OpenAI Evals runs.
Separate Model Changes From Grader Changes
A migration can change two things at once: the outputs being evaluated and the mechanism scoring them. If the overall score moves, that alone does not identify the cause.
OpenAI warns that similarity-based scoring may produce different numerical results across systems. It specifically advises validating manually recreated graders, especially LLM-as-a-judge graders, before relying on them for regression decisions.
A practical validation plan follows from that warning:
Preserve representative old outputs. Include clear passes, clear failures, and cases close to a release threshold.
Check recreated grading against those outputs where your setup permits. This helps isolate grading differences from newly generated responses.
Run fresh outputs through the replacement workflow. Compare individual cases as well as the aggregate score.
Investigate changed verdicts. Look for altered prompts, reference mappings, rubrics, thresholds, or model settings.
Document accepted differences. Record why the new behavior is suitable before making it a release gate.
This is a validation approach derived from the documented scoring warning, not a claim that OpenAI supplies an automated comparison tool.
I would put the most effort into borderline cases. Clear failures are useful sanity checks, but cases near a threshold reveal whether the migration has quietly changed which builds ship. A similar average can conceal a different set of failures.
Do not adjust a threshold solely to recover a familiar headline score. Any recalibration should be tied to examples of acceptable and unacceptable behavior.
Remove Legacy Dependencies Before Cutover
A working local run is one migration milestone. It does not show that the production development workflow has stopped relying on Evals.
Search repositories and operational configuration for legacy eval identifiers, API calls, dashboard links, scheduled jobs, and scripts that publish results. For each dependency, decide whether to replace it, archive it, or remove it.
Use this final cutover checklist:
Disable legacy write paths. Retire jobs that expect to create or modify Evals state.
Connect the replacement to CI. Confirm that the intended suite runs in the environment used for release decisions.
Configure credentials and permissions. Avoid relying on a developer’s local setup.
Define failure handling. A missing or broken evaluation should be distinguishable from a passing evaluation.
Make results accessible. Reviewers need enough output and scoring detail to diagnose failures.
Record baseline differences. Explain any accepted changes in scoring or coverage.
Assign an owner and completion date. Leave time before November 30 for configuration and access problems.
A temporary interruption to a release gate deserves an explicit decision. Silently skipping a broken eval can turn migration trouble into an unnoticed loss of coverage.
Keep checking OpenAI’s deprecation page for schedule updates. The strongest completion criterion is operational: the replacement runs reliably, its scoring has been checked, essential history is preserved, and no release-critical workflow needs the retiring dashboard or API.
Frequently Asked Questions
3 questions
1
When Does OpenAI Evals Shut Down?
The legacy Evals dashboard and API are scheduled to shut down on November 30, 2026. Existing Evals become read-only on October 31. OpenAI does not specify a cutoff time or time zone.
2
Does OpenAI Provide an Automatic Migration?
The official guide describes manually recreating evaluations in Promptfoo. It does not document an automated transfer of complete evaluations and historical runs.
3
Will Promptfoo Produce the Same Scores?
Identical scores are not guaranteed. OpenAI warns that similarity-based scoring can differ and recommends validating recreated graders, especially LLM-as-a-judge graders, before using them for regression decisions.
