OperateBench / Field notes

OPERATEBENCH / RESEARCH PREVIEW 2026

AI can answer.
Can it operate?

The real-world benchmark for AI agents executing complex service operations.

Not another multiple-choice exam. We test whether models can navigate evolving information, tools, approvals and exceptions to complete entire operational workflows—correctly.

Explore benchmark How it works
28models observed
44synthetic cases / campaign
02metrics · goal & quality
OPERATIONS / SIMULATION● PREVIEW
01 / INTAKERequest receivedContext & constraints
02 / REASONPlan & verifyPolicies · authority
03 / EXECUTETools & actionsApprovals · exceptions
04 / EVALUATEDid it really work?Independent checks
END-TO-END RELIABILITY≠ SELF-REPORTED SUCCESS

01 / THE CORE QUESTION

Passing a prompt is easy.
Running operations is hard.

Real work takes dozens of decisions. A successful agent must identify the right information, respect permissions, handle exceptions and leave the system in the right state. OperateBench evaluates outcomes—not just plausible answers.

The initiative · Proposed 2026 expansion

Can AI agents complete real operational work?

Beyond single prompts and isolated tasks: multi-step service workflows with changing evidence, approvals, policies and real-world constraints.

OperateBench is building toward a cross-industry evaluation of end-to-end agent performance, developed with service companies contributing realistic, privacy-safe synthetic scenarios. Our ambition is to learn which systems reliably complete the job, not just report success.

Benchmark design

Real service operations. Different industries.

Operational workflows

Evaluate multi-step processes involving decisions, tool use, approvals, exceptions and changing information—not one-shot prompts.

Cross-industry cases

Invite service operators and AI-native companies across property management, logistics, banking, legal services and other sectors. Potential contributors are not yet confirmed.

Public + held-out evaluation

Publish a clear methodology and suitable synthetic examples while designing private, held-out tests to reduce benchmark contamination. The held-out program is proposed, not live.

Methodology & governance

Independent scrutiny is part of the plan.

We intend to establish a small methodology board, bringing together project leads and independent researchers. Its remit would include reviewing scenario design, evaluation criteria, scoring, reproducibility and how to communicate limitations. We are seeking an academic or research expert to help validate the approach and potentially co-author a methodological paper.

No independent methodology board, validation or research publication is being claimed for the current preview. Current results are descriptive observations under heterogeneous conditions.

Early contributors · Invite only

Help make the benchmark representative.

We are preparing an invitation-only first cohort of companies transforming service delivery with AI. The goal is a broad range of genuinely complex operational workflows, not a list of company endorsements.

What contributors provide

Representative workflow descriptions and privacy-safe synthetic cases, with expected outcomes and checks. Please do not submit real customer data, secrets or proprietary records to a public repository.

What we provide

Submission guidance and a proposed contribution format, followed by methodology review. Exact requirements, workload and access rules are still being developed.

How to participate

Initial introductions are coordinated privately with the project team. Participation, public credit and publication timing will be confirmed with each company before launch.

Roadmap · Indicative

Working toward a December launch.

Now · OctoberRefine the methodology, identify independent reviewers and shortlist founding contributors.

NovemberOnboard participating teams, expand cases across industries, define held-out checks and prepare a coordinated communications plan.

Target · Early December 2026Publish the expanded benchmark with contributors, methodology and clearly stated limitations, if validation and collection are ready.

Target is the first or second week of December, subject to contributors and methodological readiness—not a committed release date.

02 / THE DATA

Benchmark leaderboard.

Experimental results across 28 models. Quality and goal achievement are separate measurements; neither is a universal capability score.

◎
Strict quality

Percentage of scored cases satisfying all reported quality checks.

↗
Goal achieved

Percentage of scored cases reaching the intended outcome, even if some checks fail.

$
Run cost estimate

Estimated campaign costs—not production pricing or invoices.

Unweighted average of one or two original per-observation rates, with heterogeneous runtime/evaluator versions and provider settings: descriptive, not a controlled ranking. For the original 21, two-rate means are retained; the six additional models have one observation each. Rates are conditional on scored coverage; technical exclusions and interrupted aborts are not quality failures. Corrected technical attempts replace only excluded or aborted original cells within the same observation; source revisions and admission registry mappings differ across original and correction attempts. Compliance 0.3.0 applies to the additional observations; historical grades remain frozen. Original reported grades and the known correction are explained by the pinned observation study.

Quality, tasks solved & cost
Strict qualityTasks solved28 models · sorted by quality

Mint shows cases passing all quality checks. The blue endpoint shows tasks solved (goal achieved), including those passing all checks. Both are average percentages across one or two observations.

  1. gpt-6-astra
    Strict quality96.6%Tasks solved100.0%Est. run cost$238.13
  2. gemini-4-argon
    Strict quality95.5%Tasks solved100.0%Est. run cost$94.57
  3. gpt-6.1-sol
    Strict quality95.5%Tasks solved100.0%Est. run cost$52.67
  4. gpt-6.1-terra
    Strict quality86.4%Tasks solved100.0%Est. run cost$6.11
  5. claude-fable-5-1
    Strict quality80.5%Tasks solved100.0%Est. run cost$192.63
  6. claude-opus-5-5
    Strict quality54.5%Tasks solved98.9%Est. run cost$153.68
  7. gemini-3.8-flash
    Strict quality45.3%Tasks solved100.0%Est. run cost$29.09
  8. claude-sonnet-5-5
    Strict quality44.3%Tasks solved92.0%Est. run cost$20.31
  9. gpt-6-sol
    Strict quality44.3%Tasks solved89.8%Est. run cost$74.55
  10. grok-4.7
    Strict quality33.3%Tasks solved100.0%Est. run cost$77.66
  11. muse-wonder-1.0
    Strict quality26.5%Tasks solved56.4%Est. run cost$52.06
  12. GLM-5.3
    Strict quality16.7%Tasks solved88.9%Est. run cost$7.82
  13. zai-org/GLM-5.3-Flash
    Strict quality14.8%Tasks solved77.3%Est. run cost$1.21
  14. nvidia/GLM-5.2-NVFP4-262K
    Strict quality13.6%Tasks solved71.6%Est. run cost$1.73
  15. muse-spark-1.3
    Strict quality10.2%Tasks solved42.0%Est. run cost$46.82
  16. Qwen/Qwen3.8-27B-FP8
    Strict quality3.4%Tasks solved3.4%Est. run cost$2.52
  17. deepseek-ai/DeepSeek-V4.1-Flash
    Strict quality2.3%Tasks solved18.4%Est. run cost$2.90
  18. deepseek-ai/DeepSeek-V4-Flash-0731
    Strict quality2.3%Tasks solved5.7%Est. run cost$2.54
  19. moonshotai/Kimi-K3
    Strict quality0.0%Tasks solved43.2%Est. run cost$28.98
  20. gpt-6-luna
    Strict quality0.0%Tasks solved21.5%Est. run cost$1.42
  21. Qwen/Qwen3.6-35B-A3B-FP8
    Strict quality0.0%Tasks solved17.0%Est. run cost$1.96
  22. nvidia/Kimi-K2.6-NVFP4
    Strict quality0.0%Tasks solved16.1%Est. run cost$1.00
  23. Qwen/Qwen3.5-397B-A17B-FP8
    Strict quality0.0%Tasks solved6.8%Est. run cost$1.36
  24. claude-haiku-5-5
    Strict quality0.0%Tasks solved0.0%Est. run cost$3.88
  25. mistral-large-4
    Strict quality0.0%Tasks solved0.0%Est. run cost$10.37
  26. mistral-medium-3-5
    Strict quality0.0%Tasks solved0.0%Est. run cost$25.77
  27. nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
    Strict quality0.0%Tasks solved0.0%Est. run cost$2.26
  28. Qwen/Qwen3.5-9B
    Strict quality0.0%Tasks solved0.0%Est. run cost$1.27

November costs average reported-usage estimates for one 44-case observation at official current tariffs; stand-alone diagnostics are excluded from costs and quality; missing usage is excluded. These are estimates, not invoices. The original 21 retain historical campaign tariffs and River owner scenarios, including known retries. Some estimates have incomplete recorded usage. The six additional estimates use official current API rates as of 2026-10-30; gpt-6.1-terra covers complete recorded usage. claude-haiku-5-5, mistral-large-4, grok-4.7, GLM-5.3, moonshotai/Kimi-K3 cover retained recorded usage only; missing usage is excluded. All costs are estimates, not invoices. For the original 21 only: River uses owner-selected $0.12/$0.01/$0.50 per million input/cached-input/output tokens; ranges compare hypothetical all-cached versus no-cache usage. Other estimates retain campaign tariffs.

Download model averages (JSON) ↓

About OperateBench

OperateBench explores whether agents can keep operations safe and progressing as time, evidence and authority change. These exploratory observations cover related synthetic cases; the official Lifecycle Track remains a technical preview. Read about the project ↗