Meta’s WildArtifactBench Tests AI Agents on Real-World Deliverables
The new benchmark uses human and agentic preference judging to evaluate complex multimodal work that cannot be reduced to one correct answer.
Last updated on
Meta is previewing WildArtifactBench, an internal evaluation framework built around a difficult question: how do you measure an AI agent when its job is to…
AI Summary
Meta is previewing WildArtifactBench, an internal benchmark that evaluates AI agents by the quality of finished, real-world deliverables rather than short text responses.
Its public viewer includes 10 tasks across audio, medical imaging, engineering, machine learning, 3D modeling, design, and video editing. Outputs range from CSV, JSON, PNG, SVG, and MP4 files to watertight meshes and interactive browser applications. Examples include matching 46 bird recordings to species, building eight parallel DQN driving environments, extracting cardiac ultrasound contours, and editing a 25-to-45-second trailer. Human and agentic judges compare artifacts pairwise, producing win rates and Elo scores, while deterministic checks can verify validity.
Meta has not yet published rankings, confidence intervals, judge calibration, prompts, inspection tools, or detailed execution conditions. Those disclosures will determine whether results meaningfully distinguish model ability from agent tooling, planning, memory, and validation—and whether Elo scores reflect trustworthy performance rather than opaque internal methodology.
Meta is previewing WildArtifactBench, an internal evaluation framework built around a difficult question: how do you measure an AI agent when its job is to produce a finished piece of work rather than a short text response?
The public WildArtifactBench task viewer currently exposes 10 tasks spanning audio analysis, medical imaging, engineering, machine learning, 3D modeling, digital design, and video editing. Depending on the assignment, an agent may need to return a CSV file, analytical plots, segmentation masks, an interactive browser application, a watertight 3D mesh, a laser-cutting design, or an edited MP4.
Meta’s more consequential choice is how those artifacts are compared. WildArtifactBench uses win rates and Elo scores derived from human and agentic preference judges instead of depending entirely on rigid ground-truth scoring. That expands the range of workflows the Benchmark can cover, but it also makes judge quality and methodological transparency central to whether its rankings can be trusted.
WildArtifactBench Evaluates Finished Work, Not Chat Responses
Most LLM benchmarks end with text, code, a multiple-choice answer, or a deterministic state change. WildArtifactBench asks an agent to assemble deliverables that may require perception, planning, tool use, file manipulation, programming, visual judgment, and validation in the same run.
The preview’s tasks illustrate how far Meta wants to move beyond conventional question answering:
Turn source footage into a coherent trailer with an intentional audio edit
A 25-to-45-second landscape MP4
Other tasks involve mapping coronary stenoses, creating a low-poly cat mesh, integrating furniture into a realistic 3D kitchen, and designing an SVG plywood whale that can be laser-cut and assembled without glue.
These are not isolated perception tests. The bird task, for example, requires an agent to inspect dozens of audio files, interpret textual species descriptions, make repeated classifications, and package the results under an exact filename. A capable audio model can still fail if the surrounding agent loses track of files or produces a malformed CSV.
Preference Judging Makes More Tasks Measurable
WildArtifactBench compares outputs through preferences. Judges are shown competing artifacts and decide which one better satisfies the task. The resulting wins, losses, and ties can be reported directly as win rates or combined into Elo ratings.
A win rate describes performance against a particular opponent or reference set. Elo attempts to produce a broader relative ranking from many pairwise matchups. Similar systems have been used for open-ended LLM evaluation, most visibly in Chatbot Arena, because selecting the better of two responses is often easier and more consistent than assigning each response an absolute score.
Artifact evaluation raises the difficulty. A text-only judge cannot properly determine whether an OBJ file is watertight, a browser simulator runs smoothly, an SVG can be fabricated, or a trailer has coherent visual and audio timing. An agentic judge can potentially use tools to open files, render models, execute applications, inspect metadata, run validation scripts, and gather evidence before choosing a winner.
That approach resembles the broader Agent-as-a-Judge idea, where the evaluator is itself an agent capable of multi-step inspection rather than an LLM making a single judgment from a prompt.
Meta has not yet explained the exact inspection tools, prompts, or decision process used by WildArtifactBench’s agentic judges. Those details will matter as much as the task collection.
Ground Truth Works Until Quality Has Several Valid Forms
Some requirements in WildArtifactBench remain objectively testable. A video can be checked for duration and resolution. A CSV can be parsed. An SVG can be inspected for overlapping paths. A mesh can be tested for non-manifold edges, while masks and coordinates can be compared with held-back annotations.
The public task pages therefore include evaluation criteria even though the overall framework emphasizes preference judging. This suggests a useful division of labor: deterministic checks establish whether an artifact is valid, while preference judges compare the quality of artifacts that pass those checks.
That distinction is essential for open-ended work. There is no single correct low-poly cat model or Big Buck Bunny trailer. Two kitchen renders may contain every requested object but differ substantially in lighting, composition, realism, and faithfulness to the reference. A DQN demonstration may technically run while remaining too slow or confusing to be useful in a classroom.
A strict rubric either misses these differences or requires hundreds of narrowly specified rules. Pairwise judging lets the Benchmark ask a more practical question: which deliverable would a person prefer to use?
The Agent Stack Matters More Than the Base LLM
WildArtifactBench should not be interpreted as a pure test of an LLM’s internal intelligence. Performance will depend on the complete agent system surrounding the model.
An agent may need to:
understand text, images, audio, video, and technical file formats;
select and operate domain-specific tools;
write scripts for processing or validation;
maintain state across dozens of inputs;
recover from failed commands or invalid outputs;
inspect its own work before submission;
organize multiple related artifacts under precise filenames.
This makes the framework relevant to developers building production agents. A stronger foundation model may lose to a smaller model paired with better tools, planning, memory, and verification. Conversely, an impressive LLM can score poorly if its harness cannot open the source material or validate the final files.
A single overall Elo number could hide those distinctions. WildArtifactBench will be more informative if Meta reports task-level results, modality breakdowns, failure categories, completion costs, and the performance of different agent scaffolds using the same underlying model.
Meta Still Needs to Document the Evaluation Pipeline
The 10-task preview demonstrates scope, but it is not yet a complete benchmark report. The task viewer does not provide model rankings, confidence intervals, judge calibration results, or a detailed evaluation protocol.
A credible release should document several elements:
Execution conditions: available tools, hardware, internet access, time limits, token budgets, retry policies, and maximum tool calls.
Pairing methodology: which agents compete, how matchups are selected, how ties are handled, and how Elo ratings are estimated.
Judge configuration: the models, prompts, tools, reasoning budgets, and artifact viewers given to agentic judges.
Human evaluation: evaluator qualifications, domain expertise, agreement rates, adjudication procedures, and compensation.
Bias controls: output-order randomization, anonymized model identities, repeated comparisons, and checks for verbosity or presentation bias.
Statistical reporting: uncertainty intervals, per-task sample sizes, repeated runs, and sensitivity to changes in the competitor pool.
Automated judges are known to exhibit position, verbosity, and self-preference biases. Those problems become harder to detect when the judge is navigating files and tools across a long evaluation trajectory. Meta will need to measure agreement between agentic and human judges rather than treating either group as unquestionable ground truth.
The high-stakes tasks deserve additional care. A strong preference score on cardiac ultrasound or coronary angiography would demonstrate performance within a controlled benchmark environment. It would not establish clinical validity or make an agent suitable for independent medical decision-making.
Elo Measures Relative Preference, Not Real-World Readiness
An Elo score only has meaning relative to the evaluated systems, task distribution, judging process, and agent harness. Adding stronger competitors or changing judge behavior can alter ratings even when an agent itself has not changed.
The same caution applies to win rates. A 60% win rate against one baseline may be impressive or weak depending on the baseline, the number of comparisons, and the uncertainty around the estimate.
WildArtifactBench results should therefore be read as evidence that one system produced more frequently preferred artifacts under a defined setup. They should not be presented as a universal percentage of jobs automated, proof of professional equivalence, or a guarantee that an agent will remain reliable on unfamiliar inputs.
Final Thoughts
WildArtifactBench’s most valuable idea is not Elo scoring by itself. It is treating the artifact as the unit of AI evaluation.
That shift exposes failures hidden by text benchmarks: invalid files, incomplete deliverables, weak visual execution, broken applications, poor tool choices, and results that satisfy instructions mechanically without being useful. It also introduces a new dependency on the evaluator. If Meta publishes judge calibration, execution settings, task-level results, and uncertainty alongside future rankings, WildArtifactBench could become a useful model for assessing multimodal agents. Without that transparency, its Elo table would be difficult to separate from another internal marketing score.
Frequently Asked Questions
4 questions
1
What is Meta WildArtifactBench?
WildArtifactBench is an internal Meta evaluation framework for testing AI agents on complex, real-world artifact creation. Its public preview contains 10 tasks involving audio, medical images, engineering analysis, browser software, 3D models, vector designs, and video. Agents must produce usable files and finished deliverables rather than only generate text answers.
2
How does WildArtifactBench use Elo scores?
WildArtifactBench derives Elo scores from pairwise preference judgments between agent outputs. Human or agentic evaluators choose which deliverable better satisfies the task, producing wins, losses, and ties. Elo combines those outcomes into a relative ranking, but the score depends on the competitors, tasks, judging method, and evaluation conditions.
3
What deliverable formats does WildArtifactBench cover?
The preview covers formats including CSV, JSON, PNG, SVG, OBJ or other 3D files, browser-based applications, analytical reports, segmentation masks, and MP4 video. Several tasks require packages containing multiple connected artifacts, such as a machine-readable analysis, visual overlays, plots, and supporting documentation.
4
Does WildArtifactBench evaluate an LLM or a complete AI agent?
WildArtifactBench primarily evaluates the complete AI agent system. The base LLM is important, but results also depend on multimodal perception, tool access, planning, memory, coding ability, file handling, recovery behavior, and self-validation. Comparing models fairly will therefore require Meta to standardize or clearly document the surrounding agent harness.
