Scale AI
IncumbentScale AI is the data-labeling and AI-data incumbent that has extended into RL environments (simulated web apps, macOS/Windows-like desktop VMs and MCP-tool environments with expert rubrics and automated verifiers). It runs a large public evaluation program through Scale Labs (SWE-Bench Pro, MCP Atlas, Humanity's Last Exam, Remote Labor Index). After Meta's ~$14.3B deal in June 2025 (~49% non-voting stake) and Alexandr Wang's move to Meta, several frontier labs reportedly scaled back work. In 2026 Scale grew mainly in government and enterprise: the Pentagon CDAO agreement ceiling rose to $500M, it acquired ICG Solutions, and Francis deSouza (ex-Google Cloud COO) became CEO in August 2026.
What's new
- 22 Sep 2026benchmark
SWE-Bench Pro v2: 642 public tasks, a 51-task Hard split, a 272-task private set and network-locked evaluation
89 invalid instances were removed from the original 731. The release adds anti-cheating measures (sanitized git history, network isolation during the agent phase, re-grading on pristine images, verifier hardening). 23 contracted engineers spent 1,897 hours on cleanup. Claude Opus 5 resolved 638/642 public tasks (99.4%) but only 222/272 private tasks (81.6%).
Why it matters: A widely used coding-agent benchmark. The roughly 18-point public/private gap is evidence of training-time contamination, which matters to anyone using SWE-Bench Pro for evaluation or as an RL training signal.
- 22 Sep 2026partnership
Scale and Google Cloud publish a reference architecture for deploying Scale GenAI Portfolio (SGP) agents on Google Cloud and Gemini Enterprise
Announced at the Google Cloud Doha Summit. Agents are built and evaluated in SGP, deployed on GKE with GPUs/TPUs inside customer VPCs, and surfaced through Gemini Enterprise, with A2A/MCP interoperability and an Agent Registry.
Why it matters: A cloud and go-to-market partnership with Google after Google reportedly cut data-labeling ties in 2025. It does not show that Google DeepMind is a data customer.
- 3 Sep 2026research
Scale Labs releases READY ('READY or Not'), enterprise agent-deployment benchmarks
Produces deployment profiles covering reliability, human-oversight rate and cost for human-plus-agent workflows. It starts with healthcare (CliniCARE-Bench, PSEBench), with financial services announced as coming soon. Built on Inspect AI; paper at arXiv 2609.02095.
Why it matters: Shifts agent evaluation from capability scores to deployment-level metrics such as review burden.
- 2 Sep 2026research
Scale Labs 'Who Grades the Graders?': study of verifier reliability for computer-use agent tasks
The team generated 603 programmatic verifiers with coding models. When the single most critical number in each of 38 deliverables was corrupted, none of the 38 verifiers failed the output: 25 still scored it 1.0, and 13 docked only 0.4-10.7%. The post recommends deterministic checks combined with vision-capable judges and mutation ('corruption') testing of verifiers. Programmatic verifiers agreed with VLM judges on 99.5% of the items the judge passed.
Why it matters: Directly relevant to reward computation in computer-use RL environments, and a view into Scale's own verifier practice.
- 30 Jul 2026hire
Scale AI appoints Francis deSouza as CEO (effective 2026-08-10); interim CEO Jason Droege to assist the transition
deSouza was most recently COO and President of Security Products at Google Cloud, and before that President and CEO of Illumina. Alexandr Wang, quoted as 'Founder and Chairman', endorsed the appointment. The announcement names BP and Mayo Clinic as enterprise customers, plus U.S. and international governments.
Why it matters: Ends the interim period that followed the Meta deal. Hiring an enterprise and security operator as CEO is consistent with a focus on enterprise and government growth. That reading is our inference, not a company statement.
- 16 Jul 2026partnership
Scale joins the DOE Genesis Mission Consortium as a formal member
Builds on the May MOU. Scale joins working groups on scientific GenAI benchmarks and evaluation, research agents, and computer-vision and robotics data pipelines tied to data from the 17 national labs.
Why it matters: Formalizes Scale's role in federal scientific-AI evaluation.
- 15 Jul 2026customer
Mayo Clinic collaboration on clinical AI applications built on Scale GenAI Platform
Focus areas are reducing clinical documentation-review time, automated detection of patient-safety events, and reducing administrative burden. Patient data stays in Mayo's HIPAA-compliant environment. The post includes a quote from Matthew R. Callstrom, M.D., Ph.D., Medical Director of Mayo's AI Program.
Why it matters: A named healthcare customer. The announcement was published by Scale but quotes a named Mayo executive.
- 7 Jul 2026partnership
Scale and Singapore's IMDA sign an MOU on joint AI evaluation research, starting with a Singapore-law legal benchmark
The agreement was formalized in April 2026 and announced 2026-07-07. Planned work: a large-scale legal benchmark for Singapore law, public release of a subset of the benchmarks where possible, and exploration of agentic AI, evaluation methods and multilingual AI safety.
Why it matters: Scale's evaluation business is expanding into international government partnerships.
- 6 Jul 2026research
Scale Labs publishes VeRO, a framework in which an optimizer agent edits a target agent
VeRO (Versioning, Rewards, and Observations) lets an optimizer agent edit any part of a target agent. It reported gains of up to 19 points on multi-step benchmarks, mostly from changes to how the agent interacts with its environment rather than from reasoning. The paper is listed on labs.scale.com/papers as 2026-02-25; the blog post is dated 2026-07-06.
Why it matters: Relevant to harness and scaffold optimization and to avoiding overfitting during agent-improvement loops.
- 30 Jun 2026benchmark
DrugDiscoveryBench released with Phylo
82 problems written by drug-discovery experts. Leading agents (OpenAI, Anthropic and Google models) solved about half. Expert step-by-step plans lifted some previously unsolved tasks to near-perfect scores, and harness optimization added up to 17 points for some models.
Why it matters: A domain-expert agent benchmark that shows Scale's push into scientific and pharma evaluation.
- 24 Jun 2026partnership
Scale and Circadence announce a collaboration on AI-driven cyber readiness
Combines Circadence's cyber readiness platform with Scale's agentic AI infrastructure. Initial priorities are AI-enabled cyber mission environments, workflow automation for cyber operations, and analyst decision support for defense, intelligence and critical infrastructure.
Why it matters: Adds cyber mission environments to Scale's defense portfolio, next to its RL-environment work.
- Jun 2026research
Scale Labs publishes rubric-based RL and agent-environment papers (Reward Hacking in Rubric-Based RL, 2026-05-12; Not Every Rubric Teaches Equally, 2026-05-19; Rubric-Guided Self-Distillation, 2026-06-10; ChainWorld, 2026-06-19; SWE-INTERACT, 2026-06-29)
A run of papers on rubric rewards for RL post-training (reward hacking, policy-aware rubric quality, distillation without rubric verifiers) and on long-horizon agent environments: ChainWorld composes long desktop workloads from atomic OSWorld tasks, and SWE-INTERACT covers user-driven coding sessions. Spine-Branch Coordination for Multi-agent Computer Use followed on 2026-08-27.
Why it matters: These methods matter for RL environments with partially verifiable tasks, and the papers show how Scale designs rewards.
- 20 May 2026product
Scale's GenAI Platform, T&E and services added to the GSA Multiple Award Schedule through Carahsoft
Available through Carahsoft's GSA MAS (47QSWA18D008F), and also through SEWP V, ITES-SW2, NASPO ValuePoint and OMNIA Partners contracts.
Why it matters: Gives federal, state and local buyers a simpler procurement path.
- 14 May 2026other
'Ten Years of Scale': company reports 700+ customers, 15B human decisions and $1B+ paid to contributors
Anniversary post by interim CEO Jason Droege with company-stated cumulative figures, including coverage of 150 languages and locales.
Why it matters: The latest scale figures from the company. They are self-reported and not audited.
Founding team
What practitioners should know
For researchers
- Scale Labs (labs.scale.com, launched 2026-03-09) extends SEAL and keeps a dated papers index. Its 2026 output centers on rubric rewards for RL (reward hacking, policy-aware rubrics, rubric-guided self-distillation), long-horizon agent environments (ChainWorld from OSWorld tasks, SWE-INTERACT, Spine-Branch multi-agent computer use), agent optimization (VeRO) and deployment evaluation (READY).
- Open releases: SWE-Bench Pro (HF ScaleAI/SWE-bench_Pro; GitHub scaleapi/SWE-bench_Pro-os), MCP Atlas (HF ScaleAI/MCP-Atlas; GitHub scaleapi/mcp-atlas) and RSI Bench (GitHub scaleapi/rsi-benchmark). SWE-Bench Pro and MCP Atlas keep held-out private splits.
- SWE-Bench Pro v2 shows a large public vs private gap (Claude Opus 5: 99.4% vs 81.6%), which Scale attributes mainly to training-time exposure. Treat public-split scores as likely contaminated.
- The 'Who Grades the Graders?' study (2026-09-02) found that programmatic verifiers missed critical-number corruption in all 38 tested deliverables. It recommends deterministic checks plus vision-capable judges and corruption testing, which is a useful reference for reward design in computer-use environments.
For program managers
- Leadership changed: Francis deSouza (ex-Google Cloud COO, ex-Illumina CEO) has been CEO since 2026-08-10, replacing interim CEO Jason Droege. Alexandr Wang (Meta Chief AI Officer) is still on the board and is called 'Founder and Chairman' in the announcement. Meta holds a ~49% non-voting stake from June 2025.
- Momentum in the window is mostly government and enterprise: ICG Solutions acquisition (2026-04-20), DOE Genesis MOU (2026-05-01) and consortium membership (2026-07-16), Pentagon CDAO OTA ceiling raised to $500M (2026-05-06), GSA MAS listing (2026-05-20), Circadence cyber partnership (2026-06-24), Singapore IMDA evaluation MOU (2026-07-07), Mayo Clinic (2026-07-15) and the Google Cloud reference architecture (2026-09-22).
- Company-stated scale: 700+ customers to date, 15B human decisions, $1B+ paid to contributors (2026-05-14 post), plus 1,000+ employees and a $29B valuation (scale.com/about). All self-reported.
- No new primary funding round, tender or secondary was found for 2026-04 to 2026-10.
- No public 2026 confirmation was found that OpenAI, Google DeepMind, Anthropic or xAI buy RL environments from Scale. Labs worried about Meta's stake should do their own diligence.
For data ops
- Security: SOC 2 Type II, ISO/IEC 27001:2022, FedRAMP High Authorized, DoD IL4 Provisional Authorization (DISA). Trust Center at trust.scale.com. HIPAA is not listed on the security page; the Mayo work runs inside Mayo's own HIPAA-compliant environment.
- RL Environments are a managed offering (simulated apps, desktop VMs, MCP servers) that claims 'standard environment interfaces' with trajectory tracking and state resets. No public API docs, self-host option or published interchange format was found.
- Scale's open benchmarks ship as Docker-image environments with verifiers (SWE-Bench Pro) or containerized MCP servers (MCP Atlas). READY is built on Inspect AI.
- Procurement vehicles through Carahsoft: GSA MAS 47QSWA18D008F, SEWP V, ITES-SW2, NASPO ValuePoint and OMNIA Partners. DoD components can use the CDAO production OTA without a new competition.
Benchmarks & research
Products
Scale & velocity
Capital
Security & compliance
Product
Buyer analysis
How we verified this
Related vendors
Sources
- scale.com/blog/scale-appoints-new-ceo · 2026-10-05, CEO announcement 2026-07-30, effective 2026-08-10; deSouza background; Droege transition; Wang called 'Founder and Chairman'; BP and Mayo named; deSouza photo
- scale.com/blog/scale-ai-pentagon-cdao-500-million-agreement · 2026-10-05, CDAO production OTA raised from $100M to $500M, 2026-05-06
- scale.com/blog/scale-acquires-icg-solutions · 2026-10-05, ICG Solutions acquisition, 2026-04-20; terms and location not disclosed
- scale.com/blog/scale-labs · 2026-10-05, Scale Labs launch, 2026-03-09, extends SEAL
- scale.com/blog/ten-years-of-scale · 2026-10-05, 2026-05-14 anniversary post by Jason Droege: 700+ customers, 15B decisions, $1B+ paid to contributors
- scale.com/blog/deploying-enterprise-ai-agents-with-scale-and-google-clou · 2026-10-05, Google Cloud reference architecture, 2026-09-22, Doha Summit
- scale.com/blog/mayo-clinic-scale · 2026-10-05, Mayo Clinic collaboration, 2026-07-15, quote from Matthew Callstrom
- scale.com/blog/scale-ai-gsa-schedule-available · 2026-10-05, GSA MAS through Carahsoft, plus SEWP V, ITES-SW2, NASPO, OMNIA; 2026-05-20
- scale.com/blog/scale-ai-doe-genesis-mission-mou · 2026-10-05, DOE Genesis Mission MOU, 2026-05-01
- scale.com/blog/scale-ai-joins-genesis-mission-consortium · 2026-10-05, Genesis Mission Consortium membership, 2026-07-16
- scale.com/blog/singapore-imda-mou · 2026-10-05, Singapore IMDA MOU (formalized April 2026), announced 2026-07-07
- scale.com/blog/scale-circadence-ai-driven-cyber-readiness · 2026-10-05, Circadence collaboration, 2026-06-24
- scale.com/blog/vero · 2026-10-05, VeRO agent-optimization framework, blog 2026-07-06
- scale.com/blog/swe-atlas-complete · 2026-10-05, SWE Atlas completion, 2026-05-07
- scale.com/blog/drugdiscoverybench · 2026-10-05, DrugDiscoveryBench with Phylo, 2026-06-30
- scale.com/blog/scale-ai-announces-next-phase-of-company-evolution · 2026-10-05, June 2025: Meta investment at $29B+ valuation; Wang stays as director; Droege background (CSO since Sept 2024, ex-Uber Eats)
- scale.com/blog/sitemap.xml · 2026-10-05, Index of 2026 blog posts (lastmod), used to find posts in the window; each post's date was confirmed on the post itself
- labs.scale.com/leaderboard · 2026-10-05, All Scale Labs leaderboards and current leaders
- labs.scale.com/papers · 2026-10-05, Scale Labs paper list with dates
- labs.scale.com/blog/swe-bench-pro-v2 · 2026-10-05, SWE-Bench Pro v2, 2026-09-22; Opus 5 638/642 public vs 222/272 private
- labs.scale.com/leaderboard/swe_bench_pro_public_v2 · 2026-10-05, SWE-Bench Pro v2 leaderboard
- labs.scale.com/blog/ready · 2026-10-05, READY benchmark, 2026-09-03
- arxiv.org/abs/2609.02095 · 2026-10-05, READY or Not: Reliable Enterprise Agent Deployment (paper)
- labs.scale.com/blog/verifier-design-for-cua · 2026-10-05, 'Who Grades the Graders?' verifier study, 2026-09-02
- labs.scale.com/leaderboard/humanitys_last_exam · 2026-10-05, HLE leaderboards, with Center for AI Safety
- labs.scale.com/leaderboard/hil · 2026-10-05, HiL-Bench details and leader
- labs.scale.com/rsi-benchmark · 2026-10-05, RSI Bench; GitHub scaleapi/rsi-benchmark
- labs.scale.com/leaderboard/mcp_atlas · 2026-10-05, MCP Atlas details
- huggingface.co/datasets/ScaleAI/MCP-Atlas · 2026-10-05, MCP Atlas dataset (resolves)
- labs.scale.com/leaderboard/rli · 2026-10-05, Remote Labor Index details
- arxiv.org/abs/2505.24760 · 2026-10-05, Reasoning Gym paper; authors listed without Scale affiliation; repo open-thought/reasoning-gym
- scale.com/about · 2026-10-05, CEO deSouza, 1,000+ employees, $29B valuation, logos (Meta, Cohere, Adept, Cisco, TIME)
- scale.com/rlenvironments · 2026-10-05, RL Environments product page; no named customers or figures
- scale.com/security · 2026-10-05, SOC 2 Type II, ISO 27001:2022, FedRAMP High, DoD IL4 PA
- outlier.ai · 2026-10-05, 'Outlier is a platform operated by Scale AI'
- en.wikipedia.org/wiki/Alexandr_Wang · 2026-10-05, Wang background; infobox photo
- en.wikipedia.org/wiki/Lucy_Guo · 2026-10-05, Guo background; infobox photo
- en.wikipedia.org/wiki/Scale_AI · 2026-10-05, Confirms deSouza as CEO (July 2026) and Scale Labs; infobox valuation is stale ($14B, May 2024)
- news.google.com/rss/search?q=%22Scale+AI%22+when:180d&hl=en-US&gl=US&cei · 2026-10-05, Press index only, used to look for missed items. It shows Reuters and Axios coverage of the CEO appointment and Bloomberg and Forbes coverage of the $500M DoD deal. Individual articles were not opened and are not cited for any claim.
- scale.com/blog/rl-environments · 2026-06-07, Blog: The Next Frontier of Data Training: RL Environments
- scale.com/blog/soc2-hipaa · 2026-06-07, SOC 2 Type II and HIPAA compliance announcement
- techcrunch.com/2025/06/13/scale-ai-confirms-significant-investment-from- · 2026-06-07, Meta ~$14.3B, ~49% stake, ~$29B valuation, Wang departure
- techcrunch.com/2025/09/21/silicon-valley-bets-big-on-environments-to-tra · 2026-06-07, Industry context on RL environments; Scale among vendors
- www.cnbc.com/2025/07/16/scale-ai-cuts-14percent-of-workforce-after-meta- · 2026-06-07, 14% layoff (~200 FTE) from ~1,400 workforce; OpenAI/Google/xAI scaling back
- techcrunch.com/2024/05/21/data-labeling-startup-scale-ai-raises-1b-as-va · 2026-06-07, Series F $1B May 2024 at $13.8B valuation
- news.crunchbase.com/ai/scale-holistic-raise-big-accel-nvda-amzn/ · 2026-06-07, Series F investors: Accel, Amazon, Nvidia, Meta etc.
- www.linkedin.com/company/scaleai · 2026-06-07, Public LinkedIn company page (headcount band varies by source)
- fortune.com/2025/06/14/self-made-billionaire-college-dropout-alexandr-wa · 2026-06-07, Meta $14.3B deal context