# rl-list.com, RL Environment Vendor Directory, full dataset
Generated 2026-10-05. 42 vendors. Source: rl-list.com

## What this is
A machine-readable export of an independently-compiled directory of vendors building reinforcement-learning environments, agent evals, and human training data for AI labs. Paste it into an LLM to build a shortlist, compare vendors, or pressure-test a procurement decision.

## Read this before analysing
- FIRST-PASS overview only. Good for a shortlist / market map, NOT a final decision.
- Every figure comes from PUBLIC sources or data shared by vendors directly, so coverage is limited and uneven.
- For a real selection, request WORK SAMPLES from vendors and let those be the deciding factor.
- "unknown" means we could not source it publicly. It is NOT zero and NOT negative, do not infer a value.
- Per-field confidence tags: confirmed (primary/official source) | reported (credible third party) | estimated (clearly-labelled inference) | unknown (not found).
- Customers tagged "self-claimed" come from the vendor's own site (a claim, not proof); "verified" = corroborated by a third party.
- RFI internals (task/sample counts, pass@1, complexity tiers, harness, data format, unit/total pricing) are DELIBERATELY EXCLUDED, they are not public and only come from direct vendor engagement. Do not estimate them.
- Ranking: only dedicated commercial vendors are ranked, by the "RL List score" (scale & traction + signals + verification), with a few editorial placements. Incumbents, infrastructure and open-source projects are listed but NOT ranked.

## Suggested prompt
> "Here is a sourced, confidence-tagged directory of RL-environment vendors. Shortlist the best 3–5 for <our use case, budget, security/compliance needs, deployment model>. Prefer higher data confidence, explicitly flag where key fields are 'unknown', and list what work samples I should request to decide."

---
## AfterQuery, rank #1
- slug: afterquery
- segment: Commercial vendors
- website: https://www.afterquery.com
- focus_areas: Coding, Computer Use, Enterprise Workflows
- positioning: AfterQuery is a San Francisco applied-research lab and expert-data company (YC W25). It supplies AI labs with expert-generated SFT and RL data, rubrics, agent environments and computer-use trajectories, drawn from a stated network of nearly 100,000 verified professionals. Forbes reported it was raising at a $3.2B valuation in Sept 2026. NVIDIA's Nemotron 3 Ultra report names AfterQuery tasks in its GDPval training recipe. The company publishes or co-builds real-task benchmarks (FinanceQA, VADER, IDE-Bench, SpreadsheetBench 2, Legora BAR) and is adding forward-deployed enterprise work.
- best_fit: A frontier or enterprise AI team needing expert-authored RL environments, post-training data, and realistic real-task benchmarks across code, finance, and professional workflows.
- overall_confidence: medium
- founded_year: 2025 [confirmed], source: https://www.ycombinator.com/companies/afterquery (accessed 2026-06-07); https://siliconangle.com/2026/04/10/ai-training-data-startup-afterquery-nabs-30m-investment/ (accessed 2026-06-07), founded Jan/Feb 2025, YC W25
- status: active [confirmed], source: https://www.afterquery.com (accessed 2026-06-07); https://www.businesswire.com/news/home/20260409469482/en/AfterQuery-Raises-$30-Million-Series-A-Round-at-$300-Million-Valuation (accessed 2026-06-07)
- hq_location: San Francisco, USA [confirmed], source: https://www.ycombinator.com/companies/afterquery (accessed 2026-06-07); https://www.afterquery.com/careers (accessed 2026-06-07)
- other_locations: unknown [unknown]
- distributed_remote: no [reported], source: https://www.afterquery.com/careers (accessed 2026-06-07), open roles listed as San Francisco; note core expert-data workforce (~100k practitioners) is distributed but corporate roles are SF-based
- current_headcount: ~126-140 employees (Tracxn 126 as of 2026-04-30; LinkedIn/aggregators ~136-140) [reported], source: https://tracxn.com/d/companies/afterquery (accessed 2026-06-07); https://www.linkedin.com/company/afterquery (public snippet, accessed 2026-06-07)
- headcount_band: 51-200 [reported], source: https://www.linkedin.com/company/afterquery (accessed 2026-06-07); https://tracxn.com/d/companies/afterquery (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: 39 [reported], source: https://www.afterquery.com/careers (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: https://arxiv.org/abs/2505.19395 (accessed 2026-06-07); https://arxiv.org/abs/2601.11868 (accessed 2026-06-07), published arXiv benchmarks (VADER, Terminal-Bench) and Research roles on careers page
- researcher_count: unknown (2 open Research roles; benchmark-paper authors exist but team count not enumerated) [unknown], source: https://www.afterquery.com/careers (accessed 2026-06-07)
- researcher_backgrounds: Founders: Spencer Mateega (CEO; Wharton/Penn; ex-Silver Lake, Morgan Stanley, Meta, Google), Carlos Georgescu (CTO; UBC CS; ex-Citadel Securities, Meta, Google; earlier founded an acquired ed-tech startup); Early co-author Danny Tang (FinanceQA, VADER); current role unverified; Team cites prior roles at Goldman Sachs, McKinsey, Jane Street, Palantir, NVIDIA, Google [reported], source: https://www.ycombinator.com/companies/afterquery (accessed 2026-10-05)
- published_papers_or_benchmarks: FinanceQA (arXiv:2501.18062); VADER (arXiv:2505.19395); UI-Bench (arXiv:2508.20410); App-Bench (appbench.ai); Market-Bench (arXiv:2512.12264); IDE-Bench (arXiv:2601.20886); SpreadsheetBench 2 (arXiv:2606.29955; AfterQuery partner org); Legora BAR (co-created with Legora, 2026) [confirmed], source: https://www.afterquery.com/research (accessed 2026-10-05)
- total_raised: $30.5M disclosed ($0.5M pre-seed + $30M Series A), plus an undisclosed, reportedly unclosed 2026 round [reported], source: https://techfundingnews.com/afterquery-reportedly-hits-3-2b-valuation-becoming-y-combinators-fastest-ever-unicorn/ (accessed 2026-10-05)
- last_round: Series B (reported, in progress as of 2026-09), amount undisclosed, $3.2B valuation; lead not named [reported], source: https://decrypt.co/377231/afterquery-ai-tradining-data-unicorn-y-combinator (accessed 2026-10-05)
- notable_investors: Altos Ventures (lead, Series A); The Raine Group; Y Combinator; BoxGroup; Latitude Capital; Angel investors from Google DeepMind, OpenAI, Anthropic, Meta Superintelligence Labs, Microsoft AI (per baseline BusinessWire release); 2026 round lead: not named [confirmed], source: https://www.afterquery.com/blog/human-expertise-reimagined (accessed 2026-10-05)
- valuation: $3.2B (reported by Forbes, Sept 2026; round reportedly not yet closed and not announced by the company). Previously $300M post-money at Series A (Apr 2026) [reported], source: https://techcrunch.com/2026/09/01/afterquery-reportedly-becomes-y-combinators-fastest-ever-unicorn-now-valued-at-3-2b/ (accessed 2026-10-05)
- revenue_signals: $100M+ run rate (Apr 2026, company-stated). CEO posted on X in July 2026 that recurring revenue was in the 'hundreds of millions' (founder self-claim relayed by Forbes). Reportedly profitable (anonymous Forbes source) [reported], source: https://www.forbes.com.au/news/innovation/afterquery-becomes-ycs-fastest-unicorn-at-us3-2-billion/ (accessed 2026-10-05)
- soc2: unknown [unknown], source: https://www.afterquery.com (accessed 2026-06-07), no trust/security page or SOC2 claim found
- other_certifications: unknown [unknown]
- security_page: unknown [unknown], source: https://www.afterquery.com (accessed 2026-06-07), no dedicated security/trust page found
- what_they_sell: mixed [confirmed], source: https://www.afterquery.com (accessed 2026-06-07), expert human data (SFT, RL rubrics), agent/RL environments, computer-use trajectories, and benchmarks
- open_source: partial [reported], source: https://github.com/AfterQuery (accessed 2026-10-05)
- license: unknown (the AfterQuery/harbor repo is no longer public) [unknown], source: https://github.com/AfterQuery/harbor (accessed 2026-10-05)
- deployment_model: managed-hosted (data + RL environments delivered as a service to AI labs); open-source frameworks available (Harbor) [estimated], source: https://www.afterquery.com (accessed 2026-06-07); https://github.com/AfterQuery/harbor (accessed 2026-06-07)
- maturity: GA [estimated], source: https://www.afterquery.com (accessed 2026-06-07); https://www.businesswire.com/news/home/20260409469482/en/AfterQuery-Raises-$30-Million-Series-A-Round-at-$300-Million-Valuation (accessed 2026-06-07), commercially selling to AI labs with stated $100M+ run rate; product maturity not formally labeled by vendor
- notable_customers: NVIDIA (Nemotron 3 Ultra; AfterQuery tasks named in NVIDIA's technical report) (verified, frontier-lab tie); Legora (BAR benchmark; credited on legora.com/bar) (verified); Thinking Machines Lab (reported by Forbes; not confirmed by TML) (self-claimed, frontier-lab tie); Motif Technologies (Motif 3; 'sole data partner' per AfterQuery, also named by TechCrunch; not credited on Motif's model card) (self-claimed); The Raine Group (Raine Search; Raine exec quoted in AfterQuery post; also an investor) (self-claimed); Unnamed Chinese AI labs (reported by Forbes) (self-claimed); Frontier AI labs (company claims every leading lab) (self-claimed, frontier-lab tie) [reported], source: https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf (accessed 2026-10-05)
- founding_team:
  - Spencer Mateega, Co-founder & CEO, Wharton BS (Finance and Statistics) and Penn MS in CS. Earlier worked at Silver Lake, Morgan Stanley, Meta and Google (YC page). Age 23 per Forbes (Sept 2026). Co-author of FinanceQA, VADER, UI-Bench, Market-Bench and IDE-Bench.
  - Carlos Georgescu, Co-founder & CTO, UBC BS in Computer Science. Two internships at Citadel Securities, plus software engineering at Meta and Google. Earlier founded an ed-tech startup that was acquired (YC page). Age 22 per Forbes.
  - Danny Tang, Early co-founder, Co-author of FinanceQA (2025) and VADER (2025) with the two founders. The YC page's team section mentions a 'Danny' who met Spencer in high school, but he is not in the YC founder list or 2026 press.
- funding_history:
  - 2026-09-01 Series B undisclosed, valuation $3.2B, source: https://techcrunch.com/2026/09/01/afterquery-reportedly-becomes-y-combinators-fastest-ever-unicorn-now-valued-at-3-2b/
- recent_developments (last 6 months):
  - 2026-09-01 [funding] Forbes reports AfterQuery is raising at a $3.2B valuation, YC's fastest launch-to-unicorn, source: https://techcrunch.com/2026/09/01/afterquery-reportedly-becomes-y-combinators-fastest-ever-unicorn-now-valued-at-3-2b/
  - 2026-08-30 [benchmark] AfterQuery co-builds Legora's BAR legal-agent benchmark, source: https://www.afterquery.com/blog/building-a-frontier-legal-evaluation-in-partnership-with-legora
  - 2026-08-23 [customer] AfterQuery says it was the sole data partner for Motif Technologies' Motif 3, source: https://www.afterquery.com/blog/afterquery-celebrates-the-release-of-motif-3
  - 2026-07-07 [research] Qwen3.5-9B improved on τ³-Banking with PivotRL using fewer than 250 tasks, source: https://www.afterquery.com/blog/improving-qwen35-9b-on-t3-banking
  - 2026-07-02 [customer] NVIDIA's Nemotron 3 Ultra technical report names AfterQuery tasks in its GDPval training recipe, source: https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf
  - 2026-06-29 [benchmark] SpreadsheetBench 2 released with AfterQuery as a partner organization, source: https://spreadsheetbench.github.io/
  - 2026-06-08 [research] +21.4% net win-loss margin on GDPval using on-policy distillation, source: https://www.afterquery.com/blog/on-policy-distillation-gdpval
  - 2026-06-03 [other] Essay on the enterprise 'last mile' (OpenAI DeployCo, Anthropic ServiceCo), source: https://www.afterquery.com/blog/deployco
  - 2026-04-28 [partnership] Raine Search built for The Raine Group (also a Series A investor), source: https://www.afterquery.com/blog/solving-the-last-mile-problem-in-partnership-with-the-raine-group
  - 2026-04-09 [funding] $30M Series A at $300M valuation led by Altos Ventures, source: https://www.afterquery.com/blog/human-expertise-reimagined
- benchmarks_and_research:
  - Legora BAR (Benchmark for Agentic Reasoning) (benchmark), 2026-08-30, https://legora.com/bar
  - SpreadsheetBench 2 (benchmark), 2026-06-29, Best model reaches 34.89% overall task accuracy; debugging accuracy is only 12.00%, https://spreadsheetbench.github.io/
  - IDE-Bench (paper), 2026-01-28, https://arxiv.org/abs/2601.20886
  - Market-Bench (benchmark), 2025-12-13, LLMs can scaffold basic trading infrastructure but struggle to reason robustly about prices, inventory and risk, https://arxiv.org/abs/2512.12264
  - App-Bench (leaderboard), 2025-10-25, Orchids leads at 76.8%, followed by Claude Code (Opus 4.5) at 67.5% (leaderboard snapshot 2026-10-05), https://appbench.ai/
  - UI-Bench (benchmark), 2025-08-28, https://arxiv.org/abs/2508.20410
  - VADER (benchmark), 2025-05-26, OpenAI o3 scored 54.7% overall, https://arxiv.org/abs/2505.19395
  - FinanceQA (benchmark), 2025-01-30, LLMs fail about 60% of realistic financial analysis tasks, https://arxiv.org/abs/2501.18062
- products:
  - Supervised fine-tuning (SFT) data: Expert prompt-response pairs and reasoning traces., https://www.afterquery.com
  - RL + rubrics: Expert-designed prompts with grading rubrics for reinforcement learning., https://www.afterquery.com
  - Agent environments (API/MCP): Custom RL and agent environments across APIs, tools and services., https://www.afterquery.com
  - Computer-use trajectories: Human-demonstrated interactions in browser and desktop environments., https://www.afterquery.com
  - Off-The-Shelf Office Agent Training Dataset: GDPval-style, file-grounded professional task dataset. NVIDIA used AfterQuery tasks for Nemotron 3 Ultra. Other off-the-shelf GDPval and agentic post-training datasets are available on request via research@afterquery.com., https://www.afterquery.com/blog/how-afterquery-helped-nvidia-hill-climb-gdpval
  - Atlas platform / forward-deployed enterprise work: AfterQuery's annotation platform and forward-deployed team, used to build enterprise AI tooling such as Raine Search for The Raine Group., https://www.afterquery.com/blog/solving-the-last-mile-problem-in-partnership-with-the-raine-group
  - Custom evaluations: Frontier evaluations co-built with partners (e.g. Legora BAR, SpreadsheetBench 2). Contact: research@afterquery.com., https://www.afterquery.com/research
- notes_for_researchers:
  - Publishes methods posts with numbers (vendor-reported): on-policy distillation on GDPval (+21.4% net win-loss margin for Qwen3.5-9B, +20.9% for Nemotron-3-Nano), PivotRL on τ³-Banking (pass@5 11.3% to 17.5%), and a reported 5x Terminal-Bench 2.0 gain using Tinker and Harbor (March 2026).
  - Authored or co-authored FinanceQA, VADER, UI-Bench, Market-Bench and IDE-Bench (arXiv) and App-Bench (appbench.ai). Partner on SpreadsheetBench 2 and Legora BAR. Not an author on the Terminal-Bench paper (arXiv:2601.11868).
  - NVIDIA's Nemotron 3 Ultra technical report (2026-06-09) names 'AfterQuery (AQ) tasks' as the training distribution for its GDPval-style office-task teacher.
  - The GitHub org (github.com/AfterQuery) showed zero public repos on 2026-10-05, and earlier repos such as harbor now return 404. The FinanceQA dataset remains public on Hugging Face. Other datasets are offered on request via research@afterquery.com.
- notes_for_program_managers:
  - Reported $3.2B valuation (Forbes, Sept 2026), up from $300M in April. The round was reportedly not closed and the lead unnamed. Profitability comes from an anonymous Forbes source. 'Hundreds of millions' in recurring revenue is the CEO's own claim on X (July 2026).
  - Named customers: NVIDIA (confirmed in NVIDIA's own report) and Legora (Legora credits AfterQuery on its BAR page). Thinking Machines Lab and Motif Technologies are press-reported or vendor-claimed. The company also claims every leading AI lab as a customer (self-claimed).
  - Forbes reports that AfterQuery 'works closely with some Chinese AI labs'. This may matter for buyers with data-governance or geopolitical sourcing constraints.
  - Engagement model: custom data and environment programs for labs, plus a forward-deployed enterprise team (e.g. The Raine Group, also an investor). States a network of nearly 100,000 verified professionals.
  - 39 open roles on the careers page as of 2026-10-05 (38 SF, 1 remote LatAm), including project leads for healthcare, legal, AI safety, STEM and consulting domains.
- notes_for_data_ops:
  - No public security or trust page (afterquery.com/security returns 404) and no SOC 2 claim found. The careers page lists Senior SWE (Security/Infrastructure) and ML Engineer (Data Privacy & Anonymization) roles, which suggests this function is being built but is not evidence of certification.
  - Methods posts mention Harbor, Tinker and a Dockerized harness (IDE-Bench). No public API or docs found. The Harbor fork is no longer public.
  - Contact for datasets and evaluations: research@afterquery.com.
- sources:
  - https://techcrunch.com/2026/09/01/afterquery-reportedly-becomes-y-combinators-fastest-ever-unicorn-now-valued-at-3-2b/ (accessed 2026-10-05)
  - https://www.forbes.com/sites/annatong/2026/09/01/afterquery-becomes-ycs-fastest-unicorn-at-32-billion/ (accessed 2026-10-05)
  - https://www.forbes.com.au/news/innovation/afterquery-becomes-ycs-fastest-unicorn-at-us3-2-billion/ (accessed 2026-10-05)
  - https://decrypt.co/377231/afterquery-ai-tradining-data-unicorn-y-combinator (accessed 2026-10-05)
  - https://techfundingnews.com/afterquery-reportedly-hits-3-2b-valuation-becoming-y-combinators-fastest-ever-unicorn/ (accessed 2026-10-05)
  - https://www.afterquery.com (accessed 2026-10-05)
  - https://www.afterquery.com/blog (accessed 2026-10-05)
  - https://www.afterquery.com/research (accessed 2026-10-05)
  - https://www.afterquery.com/blog/how-afterquery-helped-nvidia-hill-climb-gdpval (accessed 2026-10-05)
  - https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf (accessed 2026-10-05)
  - https://www.afterquery.com/blog/afterquery-celebrates-the-release-of-motif-3 (accessed 2026-10-05)
  - https://huggingface.co/Motif-Technologies/Motif-3 (accessed 2026-10-05)
  - https://www.afterquery.com/blog/building-a-frontier-legal-evaluation-in-partnership-with-legora (accessed 2026-10-05)
  - https://legora.com/bar (accessed 2026-10-05)
  - https://www.afterquery.com/blog/solving-the-last-mile-problem-in-partnership-with-the-raine-group (accessed 2026-10-05)
  - https://www.afterquery.com/blog/deployco (accessed 2026-10-05)
  - https://www.afterquery.com/blog/on-policy-distillation-gdpval (accessed 2026-10-05)
  - https://www.afterquery.com/blog/improving-qwen35-9b-on-t3-banking (accessed 2026-10-05)
  - https://www.afterquery.com/blog/human-expertise-reimagined (accessed 2026-10-05)
  - https://www.ycombinator.com/companies/afterquery (accessed 2026-10-05)
  - https://www.afterquery.com/careers (accessed 2026-10-05)
  - https://www.afterquery.com/security (accessed 2026-10-05)
  - https://github.com/AfterQuery (accessed 2026-10-05)
  - https://github.com/AfterQuery/harbor (accessed 2026-10-05)
  - https://huggingface.co/datasets/AfterQuery/FinanceQA (accessed 2026-10-05)
  - https://spreadsheetbench.github.io/ (accessed 2026-10-05)
  - https://arxiv.org/abs/2606.29955 (accessed 2026-10-05)
  - https://arxiv.org/abs/2601.20886 (accessed 2026-10-05)
  - https://arxiv.org/abs/2512.12264 (accessed 2026-10-05)
  - https://arxiv.org/abs/2508.20410 (accessed 2026-10-05)
  - https://appbench.ai/ (accessed 2026-10-05)
  - https://arxiv.org/abs/2505.19395 (accessed 2026-10-05)
  - https://arxiv.org/abs/2501.18062 (accessed 2026-10-05)
  - https://arxiv.org/abs/2601.11868 (accessed 2026-10-05)
  - https://www.afterquery.com/blog/terminal-bench-improvement (accessed 2026-06-07)
  - https://techfundingnews.com/afterquery-gets-30m-from-altos-ventures-to-help-ai-understand-real-world-problems-better/ (accessed 2026-06-07)
  - https://startupintros.com/orgs/afterquery (accessed 2026-06-07)
  - https://www.linkedin.com/company/afterquery (accessed 2026-06-07)
  - https://tracxn.com/d/companies/afterquery (accessed 2026-06-07)
  - https://sacra.com/c/afterquery/ (accessed 2026-06-07)

## Bespoke Labs, rank #2
- slug: bespoke-labs
- segment: Commercial vendors
- website: https://www.bespokelabs.ai/
- focus_areas: Long-Horizon
- positioning: Bespoke Labs is an applied AI research lab in Mountain View, founded in 2024 by Mahesh Sathiamoorthy (ex-Google DeepMind) and UC Berkeley professor Alex Dimakis. It builds RL environments and infrastructure for training and evaluating reliable agents, and is backed by $40M from Wing VC, 8VC, Mayfield and others. It combines commercial environment and post-training work, such as a jointly published project with Intuit Credit Karma, with a large open-research output: OpenThoughts, Curator, GEPA, Terminal-Bench, AutoResearchExam and ConjectureBench.
- best_fit: Buyers needing reasoning-focused data curation, open reproducible datasets/recipes, and custom RL-environment/eval data delivery from a research-led team.
- overall_confidence: high
- founded_year: 2024 [reported], source: https://tracxn.com/d/companies/bespokelabs (accessed 2026-06-07); SEC Form D filed 2024-06-04 https://www.formds.com/issuers/bespokelabs-ai-inc (accessed 2026-06-07)
- status: active [confirmed], source: https://www.bespokelabs.ai/about-us (accessed 2026-06-07); active OSS releases and hiring on https://jobs.ashbyhq.com/bespokelabs (accessed 2026-06-07)
- hq_location: Mountain View, California, USA [confirmed], source: https://www.linkedin.com/company/bespokelabsai (public snippet: 800 W El Camino Real, Mountain View, CA 94040, accessed 2026-06-07); SEC Form D lists Santa Clara, CA
- other_locations: unknown [unknown]
- distributed_remote: yes [reported], source: https://jobs.ashbyhq.com/bespokelabs (some roles open to remote in the US, accessed 2026-06-07)
- current_headcount: ~40-48 employees (40 as of 2026-04-30 per Tracxn; ~48 profiles listed on LinkedIn page) [reported], source: https://www.linkedin.com/company/bespokelabsai (48 employees listed, accessed 2026-06-07); https://tracxn.com/d/companies/bespokelabs (40 as of 2026-04-30, accessed 2026-06-07)
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/bespokelabsai (accessed 2026-06-07); Tracxn 40 employees as of 2026-04-30
- headcount_growth: unknown (LinkedIn growth posts reference recent hiring but no formal % disclosed) [unknown], source: https://www.linkedin.com/company/bespokelabsai (accessed 2026-06-07)
- open_roles_count: several open roles (e.g. RL Environments, Research Engineer, Data Operations Manager, DevOps Engineer, Technical Recruiter) on Ashby; exact count not reliably extractable from public page [reported], source: https://jobs.ashbyhq.com/bespokelabs (accessed 2026-06-07)
- has_researchers: yes [confirmed], source: https://www.bespokelabs.ai/about-us (self-described 'applied AI research lab'; team from Google/DeepMind, UC Berkeley, Stanford, NYU; multiple published works, accessed 2026-06-07)
- researcher_count: unknown (research-led team; exact count not extractable from public team page) [unknown], source: https://www.bespokelabs.ai/about-us (accessed 2026-06-07)
- researcher_backgrounds: Co-founder/CEO Mahesh Sathiamoorthy: ex-Google Brain/DeepMind staff engineer; B.Tech IIT Kharagpur, MS/PhD USC; Co-founder/Chief Science Officer Alex Dimakis: Professor, UC Berkeley EECS (formerly UT Austin, USC); IEEE Fellow; co-director, NSF AI Institute for Foundations of ML; Advisors (vendor-listed): Joseph Gonzalez (UC Berkeley), Greg Durrett (NYU), Tasso Argyros (Databricks; ex-ActionIQ CEO) [reported], source: https://bespokelabs.ai/about (accessed 2026-10-05)
- published_papers_or_benchmarks: AutoResearchExam: 24-hour ML research-agent benchmark, 29 tasks (2026-09); ConjectureBench: 14,865 open math problems (2026-07); data-eng-bench (Snowflake Labs; Bespoke co-development reported by Teradata, 2026-07/08); OpenThoughts datasets (ICLR 2026); Terminal-Bench 2.0 (ICLR 2026; multi-institution, Dimakis co-author); GEPA (ICLR 2026 oral; Dimakis co-author); Dynamic Semantic Tags technical report with Intuit (2026-09); Bespoke-Nimble-9B model (2026-09); Bespoke Curator; Evalchemy; Bespoke-MiniCheck [reported], source: https://www.bespokelabs.ai/blog (accessed 2026-10-05)
- total_raised: $40M ($8.25M seed + $31.75M Series A) [confirmed], source: https://bespokelabs.ai/blog/bespoke-labs-raises-40m-to-build-environments-that-enable-reliable-agents (accessed 2026-07-15)
- last_round: Series A led by Wing VC (SiliconANGLE reports $31.75M), announced 2026-07-06 together with an $8.25M 8VC-led seed; $40M total [reported], source: https://siliconangle.com/2026/07/06/ai-post-training-startup-bespoke-labs-raises-40m-funding/ (accessed 2026-10-05)
- notable_investors: Wing VC (Series A lead); 8VC (Seed lead); Mayfield; The House Fund; Jeff Dean (angel); Tristan Handy (angel); Spiros Xanthos (angel); Dheeraj Pandey (angel); Unnamed angels from Anthropic, OpenAI and Meta [confirmed], source: https://bespokelabs.ai/blog/bespoke-labs-raises-40m-to-build-environments-that-enable-reliable-agents (accessed 2026-10-05)
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: unknown [unknown]
- what_they_sell: RL environments and infrastructure for agent post-training (company-scale simulations, sandboxing), agent evaluation and optimization (GEPA), custom post-training of specialized models, plus open-source data tools, datasets and benchmarks [reported], source: https://www.bespokelabs.ai/ (accessed 2026-10-05)
- open_source: yes [confirmed], source: https://github.com/bespokelabsai (accessed 2026-06-07)
- license: Apache-2.0 (Curator, SkyRL fork, Harbor fork, AutoResearchExam, AutoResearchExam-Terminus, conjecture-bench, bespokelabs-python, Bespoke-Nimble-9B weights per HF card); MIT (gepa-artifact fork; Verifiers fork per baseline); OpenThoughts artifacts Apache-2.0 [confirmed], source: https://github.com/bespokelabsai (accessed 2026-10-05)
- deployment_model: mixed, open-source libraries/datasets (self-hosted) plus custom data/RL-environment delivery via direct engagement; Curator integrates with provider APIs (OpenAI/Anthropic, etc.) [reported], source: https://www.bespokelabs.ai/; https://github.com/bespokelabsai/curator (accessed 2026-06-07)
- maturity: GA (open-source tools publicly available; commercial data delivery via direct contracts) [estimated], source: https://github.com/bespokelabsai/curator (Apache-2.0, 1.6k+ stars; PyPI package published, accessed 2026-06-07)
- notable_customers: Intuit / Credit Karma (verified); Fortune 500 enterprises (unnamed) (self-claimed); Frontier labs (unnamed) (self-claimed, frontier-lab tie); Model builders using OpenThoughts datasets (unnamed) (self-claimed) [reported], source: https://files.bespokelabs.ai/ck-bespoke-tr.pdf (accessed 2026-10-05)
- founding_team:
  - Maheswaran (Mahesh) Sathiamoorthy, Co-founder & CEO, Previously Staff Software Engineer at Google (Brain, then DeepMind), where he worked on Semantic IDs and generative retrieval for recommender systems. B.Tech IIT Kharagpur; MS/PhD USC.
  - Alexandros G. (Alex) Dimakis, Co-founder & Chief Science Officer, Professor of EECS at UC Berkeley (previously UT Austin and USC). Co-director of the NSF National AI Institute for Foundations of Machine Learning. IEEE Fellow (2022). Co-author of GEPA, Terminal-Bench and OpenThoughts.
- funding_history:
  - 2026-07-06 Series A $40M led by Wing VC (Series A), 8VC (Seed), source: https://bespokelabs.ai/blog/bespoke-labs-raises-40m-to-build-environments-that-enable-reliable-agents
- recent_developments (last 6 months):
  - 2026-09-19 [customer] SeeWhy: small specialized model post-trained with Intuit Credit Karma, source: https://www.bespokelabs.ai/blog/post-training-seewhy-with-intuit-credit-karma
  - 2026-09-18 [open-source] Bespoke Nimble / Bespoke-Nimble-9B released: local typed-decision model built with contrastive data curation, source: https://github.com/bespokelabsai/nimble
  - 2026-09-10 [benchmark] AutoResearchExam: benchmark for agents doing 24-hour open-ended ML research, source: https://www.bespokelabs.ai/blog/introducing-autoresearchexam
  - 2026-09-03 [research] Personalizing the Inkling open coding model for a code repository with SFT + RL (GRPO), source: https://www.bespokelabs.ai/blog/personalizing-inkling-for-your-code-repository-with-post-training
  - 2026-08 [partnership] data-eng-bench (Snowflake Labs) reported as co-developed with Bespoke Labs, source: https://www.teradata.com/press-releases/2026/tera-an-agentic-coworker
  - 2026-08 [other] CEO Mahesh Sathiamoorthy talk 'RL Environments are all you need' at TestMu 2026, source: https://smahesh.com/
  - 2026-07-31 [benchmark] ConjectureBench: 14,865 open math problems linked to their sources, source: https://www.bespokelabs.ai/blog/conjecturebench-15k-open-math-problems-for-you-to-solve
  - 2026-07-06 [funding] Bespoke Labs announces $40M (Wing VC-led Series A plus an 8VC-led $8.25M seed) to build environments for reliable agents, source: https://bespokelabs.ai/blog/bespoke-labs-raises-40m-to-build-environments-that-enable-reliable-agents
- benchmarks_and_research:
  - Dynamic Semantic Tags Reduce Hallucinations in Small-LLM Post-Training (Intuit + Bespoke Labs technical report) (paper), 2026-09-19, 100% card-name, 99.2% fee and 99.9% rate accuracy for Llama 3.1 8B SeeWhy, at 15-20x lower serving cost (authors' report)., https://files.bespokelabs.ai/ck-bespoke-tr.pdf
  - AutoResearchExam (benchmark), 2026-09-10, Leaderboard (results as of 2026-09-09): Claude Fable 5.1 narrowly leads hidden-test AUARC by hour 22. Astra leads for roughly the first 3 hours. Next on the leaderboard: Claude Opus 5, Qwen 3.8 Max, Kimi K3 and Gemini 3.8 Flash., https://benchmarks.bespokelabs.ai/autoresearchexam/
  - ConjectureBench (dataset), 2026-07-31, https://github.com/bespokelabsai/conjecture-bench
  - data-eng-bench (Snowflake Labs; Bespoke Labs role reported) (benchmark), 2026-07-29, https://github.com/Snowflake-Labs/data-eng-bench
  - Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (Terminal-Bench 2.0) (benchmark), 2026-01-17, Frontier models and agents score below 65%., https://www.tbench.ai/
  - GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning (paper), 2025-07-25, Beats GRPO by about 6% on average (up to 20%) with up to 35x fewer rollouts, and beats MIPROv2 by more than 10%., https://arxiv.org/abs/2507.19457
  - OpenThoughts (OpenThoughts-114k / OpenThoughts3-1.2M) (dataset), https://github.com/open-thoughts/open-thoughts
- products:
  - RL Environments & Infrastructure: Company-scale simulated environments for agent post-training, generated through automation plus input from human experts, with a sandboxing layer. Sold through direct engagements., https://www.bespokelabs.ai/
  - Agent Evaluation & Optimization (GEPA): Prompt and agent optimization based on GEPA. The homepage claims 200+ teams use GEPA in production (self-claimed)., https://www.bespokelabs.ai/
  - Bespoke Curator: Open-source (Apache-2.0) library for synthetic data curation and structured extraction. About 1.74k GitHub stars., https://github.com/bespokelabsai/curator
  - Bespoke Nimble / Bespoke-Nimble-9B: Local typed-decision (classifier/judge) model and tool. Qwen3.5-9B LoRA adapter, Apache-2.0 per the Hugging Face card., https://github.com/bespokelabsai/nimble
  - AutoResearchExam: Public long-horizon research-agent benchmark and leaderboard., https://benchmarks.bespokelabs.ai/autoresearchexam/
  - ConjectureBench: Open set of 14,865 open math problems., https://github.com/bespokelabsai/conjecture-bench
  - OpenThoughts: Open reasoning datasets and recipes (with the DataComp community)., https://github.com/open-thoughts/open-thoughts
- notes_for_researchers:
  - Recent open releases: AutoResearchExam (29 tasks, AUARC metric, Apache-2.0 Terminus harness), ConjectureBench (14,865 open math problems), Bespoke-Nimble-9B (contrastive data curation) and a joint Intuit/Bespoke technical report on Dynamic Semantic Tags.
  - Published RL post-training recipe: per-repo SFT + GRPO on the Inkling open coding model. fontTools held-out tasks went from 0% to 57% and the unseen SQLGlot repo from 0% to 54% (vendor-reported).
  - ICLR 2026 papers: OpenThoughts, Terminal-Bench (85-author multi-institution paper; Dimakis co-author) and GEPA (oral; Dimakis co-author).
- notes_for_program_managers:
  - $40M announced 2026-07-06: Series A led by Wing VC (SiliconANGLE reports $31.75M) plus an $8.25M seed led by 8VC. Other backers: Mayfield, The House Fund, Jeff Dean, Tristan Handy, and unnamed angels from Anthropic, OpenAI and Meta. Headcount is about 40 per TNW.
  - Named enterprise engagement: Intuit Credit Karma (SeeWhy, 2026-09-19), backed by a technical report with Intuit co-authors. Frontier-lab customers remain unnamed; 'trusted by Fortune 500 enterprises and frontier labs' is self-claimed.
  - Possible Snowflake tie: Teradata's press release says Snowflake Labs and Bespoke Labs developed data-eng-bench. Not confirmed by either company.
  - Research-led team: CEO is ex-Google DeepMind; Chief Science Officer is a UC Berkeley professor. Advisors listed on the vendor site: Joseph Gonzalez (UC Berkeley), Greg Durrett (NYU), Tasso Argyros (Databricks).
- notes_for_data_ops:
  - Tooling is built around the Terminal-Bench/Harbor ecosystem: the GitHub org hosts a Harbor fork (created 2026-09), a SkyRL fork and AutoResearchExam-Terminus.
  - A bespokelabs-python SDK repo (Apache-2.0) exists. It dates from 2024 and was last updated 2026-09-30. No public API docs found.
  - Bespoke-Nimble-9B runs locally (Apple Silicon or NVIDIA BF16) as a typed judge or classifier, which may suit labeling QA. Its accuracy figures are vendor-reported.
  - No SOC 2, other certifications or trust/security page found on bespokelabs.ai (checked 2026-10-05).
- sources:
  - https://bespokelabs.ai/blog/bespoke-labs-raises-40m-to-build-environments-that-enable-reliable-agents (accessed 2026-10-05)
  - https://siliconangle.com/2026/07/06/ai-post-training-startup-bespoke-labs-raises-40m-funding/ (accessed 2026-10-05)
  - https://thenextweb.com/news/bespoke-labs-40m-ai-agent-training-environments (accessed 2026-10-05)
  - https://www.bespokelabs.ai/blog (accessed 2026-10-05)
  - https://www.bespokelabs.ai/blog/conjecturebench-15k-open-math-problems-for-you-to-solve (accessed 2026-10-05)
  - https://www.bespokelabs.ai/blog/personalizing-inkling-for-your-code-repository-with-post-training (accessed 2026-10-05)
  - https://www.bespokelabs.ai/blog/introducing-autoresearchexam (accessed 2026-10-05)
  - https://benchmarks.bespokelabs.ai/autoresearchexam/ (accessed 2026-10-05)
  - https://www.bespokelabs.ai/blog/post-training-seewhy-with-intuit-credit-karma (accessed 2026-10-05)
  - https://files.bespokelabs.ai/ck-bespoke-tr.pdf (accessed 2026-10-05)
  - https://github.com/bespokelabsai (accessed 2026-10-05)
  - https://github.com/bespokelabsai/nimble (accessed 2026-10-05)
  - https://huggingface.co/bespokelabs/Bespoke-Nimble-9B (accessed 2026-10-05)
  - https://www.teradata.com/press-releases/2026/tera-an-agentic-coworker (accessed 2026-10-05)
  - https://github.com/Snowflake-Labs/data-eng-bench (accessed 2026-10-05)
  - https://www.bespokelabs.ai/ (accessed 2026-10-05)
  - https://bespokelabs.ai/about (accessed 2026-10-05)
  - https://smahesh.com/ (accessed 2026-10-05)
  - https://people.eecs.berkeley.edu/~alexdimakis/ (accessed 2026-10-05)
  - https://arxiv.org/abs/2601.11868 (accessed 2026-10-05)
  - https://arxiv.org/abs/2507.19457 (accessed 2026-10-05)
  - https://www.bespokelabs.ai/about-us (accessed 2026-06-07)
  - https://www.linkedin.com/company/bespokelabsai (accessed 2026-06-07)
  - https://tracxn.com/d/companies/bespokelabs/__TgeW4_XxZv-sKUrOh6M6QeTLr6e9xHzW26BbTJzHYbQ (accessed 2026-06-07)
  - https://www.formds.com/issuers/bespokelabs-ai-inc (accessed 2026-06-07)
  - https://luma.com/36sbgxgo (accessed 2026-06-07)
  - https://jobs.ashbyhq.com/bespokelabs (accessed 2026-06-07)
  - https://newsletter.semianalysis.com/p/rl-environments-and-rl-for-science (accessed 2026-06-07)

## Huzzle Labs, rank #3
- slug: huzzle-labs
- segment: Commercial vendors
- website: https://labs.huzzle.com/
- focus_areas: Coding, Computer Use, Enterprise Workflows, Long-Horizon, Private Codebases
- positioning: Huzzle Labs is the applied-AI division of the London talent platform Huzzle (co-founders Ingmar Klein, CEO, and Amit Choudhary, CTO). It builds long-horizon RL environments (coding, computer use, enterprise workflows) and expert trajectory data for frontier labs, drawing on Huzzle's talent network, which it says numbers 300k+. In 2026 it expanded into its own small computer-use models (HuzzleWorld, 1B-8B, self-reported results), a domain benchmark (InsureBench, scores pending), and enterprise custom evals and models deployed on client infrastructure.
- best_fit: Frontier labs and regulated enterprises needing custom RL environments plus expert human trajectory data and evals for code, computer-use, and long-horizon professional workflows.
- overall_confidence: medium
- founded_year: 2020 [reported], source: Search consensus (theorg.com, Tracxn, press) gives founding year 2020 by Ingmar Klein, Parham Rakhshanfar, and Amit Choudhary; refers to parent Huzzle. Conflicting aggregator dates (2021) exist. Labs is a later AI pivot (~2025). (accessed 2026-06-07)
- status: active [confirmed], source: https://labs.huzzle.com/ (accessed 2026-06-07); https://webflow.huzzle.com/aix (accessed 2026-06-07)
- hq_location: London, United Kingdom [reported], source: https://github.com/huzzle-app (accessed 2026-06-07); Tracxn registered address 85 Great Portland Street, London (accessed 2026-06-07)
- other_locations: San Francisco, USA; Berlin, Germany [reported], source: https://webflow.huzzle.com/aix (accessed 2026-10-05)
- distributed_remote: yes [reported], source: https://webflow.huzzle.com/aix (accessed 2026-06-07), operations across US and Europe running 'around the clock'; bebee.com role open to remote (accessed 2026-06-07)
- current_headcount: ~30 (15 engineers + 15 operators) for the Labs/AI division per vendor; parent Huzzle.com aggregators report ~81-93 [reported], source: https://webflow.huzzle.com/aix (accessed 2026-06-07), vendor self-description of Labs team size; parent figures from Tracxn/CBInsights aggregators
- headcount_band: 11-50 [reported], source: https://webflow.huzzle.com/aix (accessed 2026-06-07), vendor states 15 engineers + 15 operators for Labs. Parent-company aggregator figure of ~81-93 reflects the larger talent-platform entity, not the Labs division.
- headcount_growth: Parent Huzzle reportedly ranked among Sifted 100 UK & Ireland 2026 fastest-growing startups; Labs-specific growth not separately disclosed [reported], source: https://sifted.eu/leaderboards/sifted-100-uk-ireland-2026 (accessed 2026-06-07), applies to parent Huzzle, ranking via secondary sources; downgraded as the #7 figure is from a job-posting claim, not primary confirmation
- open_roles_count: ~44 open positions (parent Huzzle, 2026) [reported], source: https://apply.workable.com/huzzle/ (accessed 2026-06-07); Sifted/Glassdoor snippets, parent company, not Labs-specific
- has_researchers: yes [confirmed], source: Confirmed by Huzzle Labs (company-provided), accessed 2026-06-07
- researcher_count: ~15 engineers plus a vetted external PhD/expert network (size of dedicated research staff not separately disclosed) [reported], source: https://webflow.huzzle.com/aix (accessed 2026-06-07)
- researcher_backgrounds: RL engineer, ex-Turing; Researchers from IIT Kharagpur and IIT Bombay (incl. PhD); Ex post-training lead on BharatGen’s 2.9B-parameter PARAM-1 LLM [confirmed], source: Confirmed by Huzzle Labs (company-provided), accessed 2026-06-07
- published_papers_or_benchmarks: InsureBench: insurance underwriting and claims benchmark (2026, scores pending); HuzzleWorld computer-use models: self-reported 72.4% on ScreenSpot-Pro (3B) and 58.6% on OSWorld-Verified (8B), leaderboard snapshot 2026-09-17; OpenEnv Visual Memory and Spreadsheet environments on Hugging Face (MIT; created March 2026, updated April 2026) [confirmed], source: https://labs.huzzle.com/huzzle-world (accessed 2026-10-05)
- total_raised: $6M [confirmed], source: Confirmed by Huzzle Labs (company-provided), accessed 2026-06-07
- last_round: $6M [confirmed], source: Confirmed by Huzzle Labs (company-provided), accessed 2026-06-07
- notable_investors: 10x Founders; Angel Invest; Emerge; a16z Scout Fund; Thomas Wolf (Hugging Face co-founder); Verena Pausder (German Startup Association); Yaser Khalighi (Stanford); Bernd Heinemann (Allianz); Former CTO of Hugging Face (angel); Researchers at Meta (angels); Researchers at Applied Intuition (angels) [confirmed], source: https://labs.huzzle.com/enterprise (accessed 2026-10-05)
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: Type II [confirmed], source: https://huzzle.secureframetrustuk.com/#compliance-d852f4b3-d3ac-4c7e-9e48-bb30d52d89c9, Huzzle Labs trust center (Secureframe), accessed 2026-06-07
- other_certifications: unknown [unknown]
- security_page: https://huzzle.secureframetrustuk.com/#compliance-d852f4b3-d3ac-4c7e-9e48-bb30d52d89c9 [confirmed], source: https://huzzle.secureframetrustuk.com/#compliance-d852f4b3-d3ac-4c7e-9e48-bb30d52d89c9, Huzzle Labs trust center (Secureframe), accessed 2026-06-07
- what_they_sell: RL environments (code, computer use, enterprise workflows) and expert trajectory data for frontier labs; custom evals and custom-trained models for enterprises; plus its own HuzzleWorld computer-use models [confirmed], source: https://labs.huzzle.com/ (accessed 2026-10-05)
- open_source: partial [confirmed], source: https://huggingface.co/api/spaces?author=huzzle-labs&full=true (accessed 2026-10-05)
- license: Proprietary (core products, HuzzleWorld); OpenEnv environments MIT-licensed on Hugging Face [confirmed], source: https://huggingface.co/api/spaces?author=huzzle-labs&full=true (accessed 2026-10-05)
- deployment_model: Enterprise: trained on client data and deployed on the client's own machines, with data and models staying with the client. Frontier labs: environments and data delivered to the lab. [confirmed], source: https://labs.huzzle.com/enterprise (accessed 2026-10-05)
- maturity: GA (in production) [confirmed], source: Confirmed by Huzzle Labs (company-provided), accessed 2026-07-15, in production with paying customers / frontier-lab engagements
- notable_customers: Apple (self-claimed); Lazard (self-claimed); Financial Times (self-claimed) [reported], source: https://webflow.huzzle.com/aix (accessed 2026-06-07), listed on vendor's own site; no third-party confirmation found. Vendor also references unnamed 'frontier AI labs' as partners.
- founding_team:
  - Ingmar Klein, Co-founder & CEO, Studied at a Swiss university and moved to London to launch Huzzle (tech.eu, 2024). Leads Huzzle Labs and is its listed contact.
  - Amit Choudhary, Co-founder & CTO, Co-founder and CTO of Huzzle (TheOrg). Member of the huzzle-labs Hugging Face org that publishes the OpenEnv environments. No other background was found in public sources.
  - Parham Rakhshanfar, Co-founder, Named as Huzzle co-founder alongside Ingmar Klein in 2024 press (tech.eu spells it 'Rakshanfar'). He is not on the current TheOrg leadership chart, so his current involvement is unverified.
- recent_developments (last 6 months):
  - 2026-09 [product] Huzzle Labs publishes HuzzleWorld, a family of small computer-use models (1B, 2B, 3B, 4B, 8B), source: https://labs.huzzle.com/huzzle-world
  - 2026-04-08 [open-source] OpenEnv Visual Memory and Spreadsheet environments on Hugging Face updated with eval runs against Claude Opus 4.6 and GPT-5.4, source: https://huggingface.co/huzzle-labs
- benchmarks_and_research:
  - HuzzleWorld results on ScreenSpot-Pro, ScreenSpot-v2 and OSWorld-Verified (leaderboard), 2026-09-17, HuzzleWorld-3B: 72.4% on ScreenSpot-Pro (Zoom In; 66.4% single-pass), claimed #1 at 3B or smaller. HuzzleWorld-8B: 58.6% on OSWorld-Verified, claimed #1 at 8B or smaller. Self-reported., https://labs.huzzle.com/huzzle-world
  - OpenEnv Spreadsheet environment (environment), 2026-03-19, https://huggingface.co/spaces/huzzle-labs/spreadsheet
  - OpenEnv Visual Memory environment (environment), 2026-03-18, https://huggingface.co/spaces/huzzle-labs/visual_memory
  - InsureBench (benchmark), 2026, No results yet: 'Scores pending · Leaderboard opening 2026', https://www.insurebench.com/
- products:
  - HuzzleWorld: Family of small computer-use models (1B, 2B, 3B, 4B, 8B) that the vendor says it trained with its own RL environments and video. Results are self-reported; no public weights were found., https://labs.huzzle.com/huzzle-world
  - RL Environments: RL environments for frontier labs in three areas: coding (multi-file repos with build and test loops), computer use (desktop and browser) and enterprise workflows (CRMs, spreadsheets, ticketing, finance)., https://labs.huzzle.com/rl-environments
  - Enterprise (custom evals and models): Evaluates frontier models on client workflows, trains custom agentic models on client data deployed on client machines, and retrains from production data, https://labs.huzzle.com/enterprise
  - InsureBench: Public benchmark for insurance underwriting and claims workflows; the leaderboard is pending, https://www.insurebench.com/
  - Expert human data / trajectory data: Expert trajectory and human data drawn from the Huzzle talent platform, which the vendor says has 300,000+ talents, https://webflow.huzzle.com/aix
  - OpenEnv environments on Hugging Face: Open, MIT-licensed Visual Memory and Spreadsheet OpenEnv environments, https://huggingface.co/huzzle-labs
- notes_for_researchers:
  - Open, inspectable work: two MIT-licensed OpenEnv (v0.2.3) Docker environments on Hugging Face (Visual Memory, Spreadsheet). The Visual Memory repo includes eval trajectories dated 2026-04-07 for Claude Opus 4.6 and GPT-5.4, in ATIF and raw JSON.
  - HuzzleWorld computer-use models (1B-8B) have self-reported results: 72.4% on ScreenSpot-Pro (3B, Zoom In) and 58.6% on OSWorld-Verified (8B). No weights, model card, paper or license were found, and the vendor's pages disagree on the 8B OSWorld rank (11th of 41 vs #13).
  - InsureBench is a GDPval-style benchmark for insurance underwriting and claims, with a public methodology page. No scores have been published yet.
- notes_for_program_managers:
  - Engagement is by email or a 15-minute Calendly call with the CEO (labs.huzzle.com).
  - Frontier-lab ties: the homepage carries a testimonial attributed to Emre Guven of Meta Superintelligence Labs, and the open environments use Meta-PyTorch's OpenEnv spec. No frontier lab is publicly named or confirmed as a customer.
  - Named customers (Apple, Lazard, Financial Times) appear only on the vendor's own AIX page and are self-claimed.
  - Scale signals, all self-claimed: a Labs team of 15 engineers and 15 operators, a talent network of 300k+, and parent-company rankings of #7 on Sifted UK & Ireland 2026 and #14 in Europe (J.P. Morgan & Nebius 2026).
- notes_for_data_ops:
  - The vendor states AICPA SOC 2 on its site. The baseline recorded Type II from the Secureframe trust center; that was not rechecked in this run.
  - The published environments are Docker-based OpenEnv Spaces serving on port 8000, each with a client.py, a models.py and a server/ directory.
  - The enterprise offering says models are trained on client data and deployed on the client's own machines, and that data and models stay with the client.
  - No public API docs or MCP integrations were found in this run.
- sources:
  - https://labs.huzzle.com/ (accessed 2026-10-05)
  - https://labs.huzzle.com/huzzle-world (accessed 2026-10-05)
  - https://labs.huzzle.com/rl-environments (accessed 2026-10-05)
  - https://labs.huzzle.com/enterprise (accessed 2026-10-05)
  - https://www.insurebench.com/ (accessed 2026-10-05)
  - https://www.insurebench.com/methodology (accessed 2026-10-05)
  - https://huggingface.co/huzzle-labs (accessed 2026-10-05)
  - https://huggingface.co/api/spaces?author=huzzle-labs&full=true (accessed 2026-10-05)
  - https://huggingface.co/api/spaces/huzzle-labs/visual_memory/tree/main/outputs?recursive=true (accessed 2026-10-05)
  - https://webflow.huzzle.com/aix (accessed 2026-10-05)
  - https://theorg.com/org/huzzle (accessed 2026-10-05)
  - https://tech.eu/2024/04/18/meet-the-gen-z-founders-rewiring-the-graduate-job-market/ (accessed 2026-10-05)
  - https://huggingface.co/amitchoudhary (accessed 2026-10-05)
  - https://labs.huzzle.com/blog/openenv-visual-memory-spreadsheet (accessed 2026-10-05)
  - https://huzzle.secureframetrustuk.com/#compliance-d852f4b3-d3ac-4c7e-9e48-bb30d52d89c9 (accessed 2026-06-07)
  - https://x.com/himanshustwts/status/2041877003251695733 (accessed 2026-06-07)
  - https://bebee.com/us/jobs/founding-engagement-manager-frontier-labs-huzzlecom--theirstack-649700226 (accessed 2026-06-07)
  - https://github.com/huzzle-app (accessed 2026-06-07)
  - https://www.eu-startups.com/2024/04/london-based-huzzle-secures-e1-67-million-pre-seed-to-help-students-land-their-dream-graduate-job/ (accessed 2026-06-07)
  - https://tracxn.com/d/companies/huzzle/__gSmaXUdJfRlNTOIE3SEajswKR4IZuSyaJRpXk91s88g (accessed 2026-06-07)
  - https://sifted.eu/leaderboards/sifted-100-uk-ireland-2026 (accessed 2026-06-07)
  - https://www.linkedin.com/company/huzzle-com (accessed 2026-06-07)
  - https://www.hud.ai/resources/platforms-agent-evals-rl-training-data (accessed 2026-06-07)
  - https://alignlist.com/guides/top-40-rl-environments-startups-and-companies (accessed 2026-06-07)

## Fleet AI, rank #4
- slug: fleet-ai
- segment: Commercial vendors
- website: https://www.fleetai.com/
- focus_areas: Computer Use, Enterprise Workflows
- positioning: Fleet AI builds high-fidelity reinforcement-learning training environments ('gyms') that replicate enterprise software such as Salesforce and Excel, plus browser/desktop workflows, so frontier AI labs and large enterprises can train and evaluate computer-use agents. It ships a Python SDK, a platform API, and the open-source 'Harbor' agent-evaluation/RL-environment tooling, pairing simulated environments with human supervision.
- best_fit: A frontier lab or large enterprise that needs bespoke, high-fidelity RL environments simulating real enterprise software (CRM, spreadsheets, browser/desktop) to train and evaluate computer-use agents.
- overall_confidence: medium
- founded_year: 2024 [reported], source: https://www.kucoin.com/news/flash/ai-training-firm-fleet-eyes-750m-valuation-amid-60x-revenue-surge (accessed 2026-06-07); https://www.theinformation.com/newsletters/ai-agenda/reinforcement-learning-gym-startup-buoyed-labs-appetite-training-data-reaches-750-million-valuation (accessed 2026-06-07)
- status: active [confirmed], source: https://www.fleetai.com/ (accessed 2026-06-07); https://github.com/fleet-ai (accessed 2026-06-07), active site and recent OSS activity
- hq_location: New York, NY, USA [reported], source: https://www.linkedin.com/company/fleet-so (accessed 2026-06-07); https://sacra.com/c/fleet/ (accessed 2026-06-07), both list New York, NY; careers page conflictingly lists a San Francisco (SOMA) HQ
- other_locations: San Francisco, CA (SOMA); New York, NY (Chelsea) [confirmed], source: https://www.fleetai.com/careers (accessed 2026-10-05)
- distributed_remote: yes (in person at SF/NYC offices, or remote, per the careers page) [confirmed], source: https://www.fleetai.com/careers (accessed 2026-10-05)
- current_headcount: 11-50 per LinkedIn public company-size band as of 2026-06-07 (RocketReach ~41 is consistent); an earlier draft figure of ~133 could not be reproduced and appears erroneous [reported], source: https://www.linkedin.com/company/fleet-so (accessed 2026-06-07); https://rocketreach.co/fleet-ai-management_b6f95050c628d05e (accessed 2026-06-07)
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/fleet-so (accessed 2026-06-07), LinkedIn shows the '11-50 employees' size band
- headcount_growth: unknown [unknown]
- open_roles_count: unknown [unknown], source: https://www.fleetai.com/careers (accessed 2026-06-07), careers page does not show a numeric open-role count
- has_researchers: yes [reported], source: https://www.fleetai.com/about (accessed 2026-06-07), team described as including people with prior research roles at Anthropic, xAI, Meta Superintelligence, Essential AI, Contextual AI, etc.
- researcher_count: unknown [unknown]
- researcher_backgrounds: Team self-describes prior experience at Anthropic, xAI, Meta Superintelligence, Essential AI, Contextual AI, Mercor, Docker, Citadel, Jane Street and Cruise; Founder/CEO Nicolai (Nic) Ouporov: ex-founding engineer at Respell (acquired by Salesforce, Jan 2024); research at the Stanford Robotics & Embodied AI Lab and the Columbia Creative Machines Lab; Co-founder Andrew (X @andrewthezhou; per Fleet's Notes post and his X bio); Fred Havemeyer (Founding Member of Technical Staff, forward-deployed engineering): ex-Head of AI at Macquarie, joined October 2024 [reported], source: https://www.fleetai.com/notes/fred-havemeyer-at-fleet-ai (accessed 2026-10-05)
- published_papers_or_benchmarks: Printing Machines (2026-01-28): a benchmark of iterative visual reasoning and drawing, from Printing Machines Lab, whose contributors include Fleet CEO Nicolai Ouporov and Fleet's Deniz Zorlu. Fleet-affiliated, not a Fleet-branded release.; No peer-reviewed paper authored by Fleet found. Harbor, OpenEnv, SkyRL, Miles, zeroboot, mcp-bench, cadgenbench and EnterpriseOps-Gym in Fleet's GitHub org are forks of third-party projects. [reported], source: https://www.printingmachines.ai/team (accessed 2026-10-05)
- total_raised: ~$15M seed (reported). A further ~$45M Series A at ~$725M was claimed closed by an anonymous investor account in April 2026, but it is unannounced and conflicts with The Information's 2026-04-13 report of the round as 'in talks' at $50M/$750M. [reported], source: https://x.com/ArfurRock/status/2042048795253395563 (accessed 2026-10-05)
- last_round: Series A, reported but unconfirmed: $45M at $725M, 'led by insiders Sequoia, Bain, Menlo, SVA' per an anonymous investor account (2026-04-09). The Information (2026-04-13) reported it as in talks at $50M/$750M, with Bain Capital Ventures as lead per Sacra. Not announced by Fleet. [reported], source: https://x.com/ArfurRock/status/2042048795253395563 (accessed 2026-10-05)
- notable_investors: Sequoia Capital; Menlo Ventures; SV Angel; Bain Capital Ventures (reported Series A lead/co-lead, April 2026; not listed on Fleet's site) [reported], source: https://www.fleetai.com/about (accessed 2026-10-05)
- valuation: ~$725M-$750M (reported, April 2026; $725M per the closed-round claim, $750M per The Information's in-talks report; unconfirmed) [reported], source: https://www.theinformation.com/newsletters/ai-agenda/reinforcement-learning-gym-startup-buoyed-labs-appetite-training-data-reaches-750-million-valuation (accessed 2026-10-05)
- revenue_signals: Reported ~$60M annualized (April 2026, Sacra) to ~$63M run-rate (anonymous investor account), up from ~$1M in 2025; ~$160M claimed for the next quarter (unverified) [reported], source: https://sacra.com/c/fleet/ (accessed 2026-10-05)
- soc2: unknown [unknown], source: No security/trust page found on fleetai.com (https://www.fleetai.com/security returned 404, accessed 2026-06-07); SOC 2 search results referred to an unrelated company (fleetdm.com)
- other_certifications: unknown [unknown]
- security_page: unknown [unknown], source: https://www.fleetai.com/security (accessed 2026-06-07), returned 404; a status page exists but no trust/security page found
- what_they_sell: environments [confirmed], source: https://www.fleetai.com/about (accessed 2026-06-07); https://sacra.com/c/fleet/ (accessed 2026-06-07)
- open_source: partial: the public GitHub org is mostly forks of third-party OSS (Harbor, OpenEnv, SkyRL, Miles, zeroboot, benchmark repos), plus an archived Apache-2.0 fleet-sdk, harbor-train (a SkyRL-based Harbor training layer) and a small researcher-scrape tool. The core product is closed. [confirmed], source: https://github.com/fleet-ai (accessed 2026-10-05)
- license: Apache-2.0 for the archived fleet-sdk and harbor-train; forked repos carry their upstream licenses (e.g. Apache-2.0, BSD-3-Clause for OpenEnv); the core hosted product is proprietary [confirmed], source: https://github.com/fleet-ai (accessed 2026-10-05)
- deployment_model: managed-hosted platform with login-gated docs, plus custom forward-deployed environment builds; the public Python SDK is archived [estimated], source: https://www.fleetai.com/notes/fred-havemeyer-at-fleet-ai (accessed 2026-10-05)
- maturity: GA [estimated], source: https://sacra.com/c/fleet/ (accessed 2026-06-07), reported paying labs/enterprises and a published SDK/API imply commercial availability; vendor does not use a GA label
- notable_customers: unknown [unknown], source: No specific Fleet AI customer is named or third-party-verified. Press/profiles describe frontier labs (OpenAI, Anthropic, Meta, Google) and financial-services/insurance enterprises only as the buyer CATEGORY for RL-environment vendors, not as confirmed Fleet customers. https://sacra.com/c/fleet/ (accessed 2026-06-07); https://sapphireventures.com/blog/reinforcement-learning-environments-ai-agents/ (accessed 2026-06-07)
- founding_team:
  - Nicolai (Nic) Ouporov, Founder & CEO, Was the founding engineer and first hire at Respell (acquired by Salesforce, January 2024). Did research at the Stanford Robotics and Embodied AI Lab and the Columbia Creative Machines Lab. Trained in ballet at a professional level (Boston Ballet, SF Ballet, Ballet West) and is a 3x YoungArts winner, per his personal site.
- funding_history:
  - 2026-04-09 Series A $45M led by Sequoia Capital, Bain Capital Ventures, Menlo Ventures, SV Angel, valuation $725M, source: https://x.com/ArfurRock/status/2042048795253395563
- recent_developments (last 6 months):
  - 2026-09 [open-source] Fleet's GitHub shows active RL post-training work: forks of SkyRL, Miles, OpenEnv and benchmark repos, plus an original arXiv research scraper; fleet-sdk archived, source: https://github.com/fleet-ai
  - 2026-04-13 [funding] The Information: Fleet in talks to raise $50M at a $750M valuation as AI labs' appetite for training data grows, source: https://www.theinformation.com/newsletters/ai-agenda/reinforcement-learning-gym-startup-buoyed-labs-appetite-training-data-reaches-750-million-valuation
  - 2026-04-09 [funding] Anonymous investor account Arfur Rock says Fleet closed an unannounced $45M Series A at a $725M valuation, led by insiders Sequoia, Bain, Menlo and SV Angel, source: https://x.com/ArfurRock/status/2042048795253395563
  - 2026-04-09 [other] Reported revenue run-rate of ~$60-63M, up from ~$1M about six months earlier, with ~$160M claimed for the next quarter, source: https://sacra.com/c/fleet/
- benchmarks_and_research:
  - Printing Machines (benchmark), 2026-01-28, Qualitative only, no scores published on the page. The lab says frontier models (Claude 4.6 Opus, Gemini 3 Flash, Gemini 3.1 Pro, GPT 5.2, Grok 4.1 Fast, Kimi 2.5) "failed miserably and in surprising ways". Failure modes include repeated spiraling, misidentified objects and poor geometric precision, with errors compounding over steps., https://www.printingmachines.ai/blog/printing-machines
- products:
  - Fleet RL environments ('training gyms'): Simulated environments for training and evaluating agents. The about page calls them 'training gyms for agents'. Third parties (Sacra, press) describe replicas of enterprise SaaS, browser and IDE workflows, but Fleet's own site names no specific replicated apps., https://www.fleetai.com/about
  - Custom environment builds / forward-deployed engineering: Bespoke environments and agents built by forward-deployed engineers. The official post says these were built for 'some of the largest financial services and insurance firms in the world', without naming any of them., https://www.fleetai.com/notes/fred-havemeyer-at-fleet-ai
  - Fleet platform docs (login-gated): docs.fleetai.com returns a 307 redirect to a docs login page, so the documentation is not publicly readable. Sacra reports platform access with a 60-day free trial and recurring billing., https://docs.fleetai.com
  - fleet-sdk (Python SDK, archived): Apache-2.0 Python SDK. The repo is now archived on GitHub., https://github.com/fleet-ai/fleet-sdk
  - harbor-train: Not a GitHub fork, but its README describes it as 'Harbor GRPO training, clean upstream SkyRL + SkyPilot layer'. It is mostly upstream SkyRL code, adapted to train against Harbor environments. Apache-2.0., https://github.com/fleet-ai/harbor-train
- notes_for_researchers:
  - Fleet has no paper of its own. Its CEO and another Fleet staffer co-built the 'Printing Machines' benchmark (2026-01-28), which tests iterative drawing and visual reasoning and reports qualitative failures for frontier models. Code was promised but not yet linked.
  - Harbor is not a Fleet original: fleet-ai/harbor is a fork of harbor-framework/harbor. fleet-ai/harbor-train is a SkyRL + SkyPilot GRPO training layer for Harbor environments.
  - The public GitHub shows Fleet working with open RL stacks: forks of SkyRL, Miles, OpenEnv, zeroboot (Firecracker VM sandboxes), and of the mcp-bench, cadgenbench and ServiceNow EnterpriseOps-Gym benchmarks, plus a nightly arXiv scraper for RL and environment-simulation papers.
  - The OpenEnv README on Hugging Face lists Fleet AI as a current technical committee member, alongside Meta-PyTorch, Reflection, Unsloth, Modal, Prime Intellect, Nvidia, Mercor and Hugging Face. The date Fleet joined is not given.
- notes_for_program_managers:
  - Funding: Fleet has confirmed only its investors Sequoia, Menlo and SV Angel (about page and X bio); the ~$15M seed size is reported by Sacra. An anonymous investor account claimed on 2026-04-09 that a $45M Series A at $725M had closed. The Information reported it four days later as 'in talks' at $50M/$750M, and Synaptic (Aug 2026) still lists only the $15M seed. Treat the Series A as unconfirmed.
  - Scale signal: a reported ~$60-63M run-rate in April 2026, up from ~$1M in 2025. It comes from third parties and is attributed to frontier-lab demand; no lab customer is named or verified.
  - Engagement model: custom builds with forward-deployed engineers, plus platform access (Sacra reports a 60-day free trial). Fleet's own blog names financial-services and insurance firms as a customer segment, without naming any of them (self-claim).
  - Team: LinkedIn band 11-50. Fleet's about page self-reports alumni of Anthropic, xAI, Meta Superintelligence, Essential AI, Contextual AI, Mercor, Docker, Citadel, Jane Street and Cruise. Offices are in SF (SOMA) and NYC (Chelsea), with remote work offered.
- notes_for_data_ops:
  - No public security or trust page (fleetai.com/security returns 404) and no stated SOC 2.
  - Docs at docs.fleetai.com are login-gated and the public fleet-sdk repo is archived, so integration details cannot be verified publicly. Ask the vendor for current SDK/API documentation.
  - Format alignment: Fleet sits on the OpenEnv technical committee and maintains forks of Harbor and OpenEnv, plus a Harbor training layer. That suggests compatibility with the Harbor and OpenEnv formats, but Fleet does not state format support on an official page.
  - The homepage links a public status page that reported 'All Systems Operational' on 2026-10-05.
- sources:
  - https://www.fleetai.com/about (accessed 2026-10-05)
  - https://www.fleetai.com/ (accessed 2026-10-05)
  - https://www.fleetai.com/careers (accessed 2026-10-05)
  - https://www.fleetai.com/notes (accessed 2026-10-05)
  - https://www.fleetai.com/notes/fred-havemeyer-at-fleet-ai (accessed 2026-10-05)
  - https://www.fleetai.com/security (accessed 2026-10-05)
  - https://docs.fleetai.com (accessed 2026-10-05)
  - https://github.com/fleet-ai (accessed 2026-10-05)
  - https://github.com/fleet-ai/harbor (accessed 2026-10-05)
  - https://github.com/fleet-ai/harbor-train (accessed 2026-10-05)
  - https://github.com/fleet-ai/zeroboot (accessed 2026-10-05)
  - https://github.com/fleet-ai/gym-anything (accessed 2026-10-05)
  - https://huggingface.co/spaces/openenv/README/blob/main/README.md (accessed 2026-10-05)
  - https://x.com/ArfurRock/status/2042048795253395563 (accessed 2026-10-05)
  - https://x.com/theinformation/status/2043804171472638323 (accessed 2026-10-05)
  - https://www.theinformation.com/newsletters/ai-agenda/reinforcement-learning-gym-startup-buoyed-labs-appetite-training-data-reaches-750-million-valuation (accessed 2026-10-05)
  - https://runtimewire.com/article/fleet-raises-45m-series-a-725m (accessed 2026-10-05)
  - https://sacra.com/c/fleet/ (accessed 2026-10-05)
  - https://synaptic.com/resources/tip-offs/rl-evironments-startups-and-founders (accessed 2026-10-05)
  - https://www.nicolas.info/ (accessed 2026-10-05)
  - https://www.linkedin.com/in/nicolas-ouporov/ (accessed 2026-10-05)
  - https://api.fxtwitter.com/andrewthezhou (accessed 2026-10-05)
  - https://www.printingmachines.ai/blog/printing-machines (accessed 2026-10-05)
  - https://www.printingmachines.ai/team (accessed 2026-10-05)
  - https://www.kucoin.com/news/flash/ai-training-firm-fleet-eyes-750m-valuation-amid-60x-revenue-surge (accessed 2026-06-07)
  - https://www.linkedin.com/company/fleet-so (accessed 2026-06-07)
  - https://rocketreach.co/fleet-ai-management_b6f95050c628d05e (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/fleet-6338 (accessed 2026-06-07)
  - https://sapphireventures.com/blog/reinforcement-learning-environments-ai-agents/ (accessed 2026-06-07)
  - https://newsletter.semianalysis.com/p/rl-environments-and-rl-for-science (accessed 2026-06-07)

## Datacurve, rank #5
- slug: datacurve
- segment: Commercial vendors
- website: https://datacurve.ai/
- focus_areas: Coding, Private Codebases
- positioning: Datacurve is a YC W24 data vendor that says it supplies frontier labs with expert-sourced RL environments, long-horizon tasks, agent trajectories, SFT demonstrations and off-the-shelf datasets. It is coding-first but now also covers data science, cyber security, ML and research. Data comes from its Shipd bounty platform of vetted engineers, which has SWE, ML and data-science tracks. It publishes DeepSWE, an open (Apache-2.0) long-horizon coding benchmark used in the Artificial Analysis Coding Agent Index.
- best_fit: Frontier/foundation model labs needing expert-sourced coding SFT/RLHF data and code-execution RL environments with verifiable rewards (code execution, tight loops).
- overall_confidence: medium
- founded_year: 2024 [confirmed], source: https://www.ycombinator.com/companies/datacurve (accessed 2026-06-07), Winter 2024 batch, founded 2024
- status: active [confirmed], source: https://datacurve.ai/ (accessed 2026-06-07); https://techcrunch.com/2025/10/09/datacurve-raises-15-million-to-take-on-scaleai/ (accessed 2026-06-07)
- hq_location: San Francisco, USA [confirmed], source: https://www.ycombinator.com/companies/datacurve (accessed 2026-06-07); https://www.linkedin.com/company/datacurveai (accessed 2026-06-07)
- other_locations: unknown [unknown]
- distributed_remote: no (San Francisco office-based; careers page lists in-office meals and commuter benefits) [estimated], source: https://datacurve.ai/careers (accessed 2026-06-07)
- current_headcount: approx 29 (LinkedIn, size band 11-50) as of 2026-10-05 [reported], source: https://www.linkedin.com/company/datacurveai (accessed 2026-10-05)
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/datacurveai (accessed 2026-06-07)
- headcount_growth: down slightly (LinkedIn approx 36 in 2026-06 to approx 29 in 2026-10) [estimated], source: https://www.linkedin.com/company/datacurveai (accessed 2026-10-05)
- open_roles_count: 3 [confirmed], source: https://datacurve.ai/careers (accessed 2026-06-07), Software Engineer, Research Engineer, Growth/Operations
- has_researchers: yes [reported], source: https://datacurve.ai/careers (accessed 2026-06-07), Research Engineer role; co-founders have ML/AI research backgrounds
- researcher_count: unknown [unknown]
- researcher_backgrounds: Serena Ge (co-founder/CEO): worked on LLM reasoning during a co-op at Cohere; University of Waterloo CS; Forbes 30 Under 30; Charley Lee (co-founder): University of Waterloo CS; AI research background [reported], source: https://uwaterloo.ca/computer-science/news/cs-led-startup-secures-177m-transform-ai-training-data (accessed 2026-06-07)
- published_papers_or_benchmarks: DeepSWE: a long-horizon, contamination-resistant agentic coding benchmark with 113 original tasks across 91 repos in 5 languages (TypeScript/JavaScript/Python/Go/Rust) and hand-written behavioral verifiers. Launched 2026-05-26; v1.1 on 2026-06-14 added isolated verification and CTRF test reports. Apache-2.0 repo at github.com/datacurve-ai/deep-swe. Used in the Artificial Analysis Coding Agent Index since v1.1 (June 2026). Not the same as the Together AI/Agentica 'DeepSWE' agent. [confirmed], source: https://deepswe.datacurve.ai/blog/deepswe (accessed 2026-10-05)
- total_raised: $17.7M ($15M Series A + $2.7M seed) [confirmed], source: https://techcrunch.com/2025/10/09/datacurve-raises-15-million-to-take-on-scaleai/ (accessed 2026-06-07); https://uwaterloo.ca/computer-science/news/cs-led-startup-secures-177m-transform-ai-training-data (accessed 2026-06-07)
- last_round: Series A, $15M, October 2025 [confirmed], source: https://techcrunch.com/2025/10/09/datacurve-raises-15-million-to-take-on-scaleai/ (accessed 2026-06-07)
- notable_investors: Chemistry (Mark Goldberg, lead Series A); Y Combinator; Balaji Srinivasan (seed); angel investors who are employees of DeepMind, Vercel, Anthropic and OpenAI (individuals, not the companies) [reported], source: https://techcrunch.com/2025/10/09/datacurve-raises-15-million-to-take-on-scaleai/ (accessed 2026-06-07)
- valuation: unknown [unknown]
- revenue_signals: unknown (over $1M paid out in bounties to contributors, a payout figure, not revenue) [unknown], source: https://techcrunch.com/2025/10/09/datacurve-raises-15-million-to-take-on-scaleai/ (accessed 2026-06-07)
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: unknown [unknown]
- what_they_sell: mixed [confirmed], source: https://datacurve.ai/products (accessed 2026-10-05)
- open_source: no (company data products are commercial/proprietary; the DeepSWE benchmark and harness are open-source under Apache-2.0) [confirmed], source: https://github.com/datacurve-ai/deep-swe (accessed 2026-10-05)
- license: Apache-2.0 (DeepSWE benchmark repo); commercial data products proprietary [confirmed], source: https://api.github.com/repos/datacurve-ai/deep-swe (accessed 2026-10-05)
- deployment_model: managed-hosted (data delivered as a service; private model endpoints spun up for RLHF traces) [reported], source: https://sacra.com/c/datacurve/ (accessed 2026-06-07)
- maturity: GA [estimated], source: https://datacurve.ai/ (accessed 2026-06-07), commercially operating with paying labs and active Shipd bounty platform
- notable_customers: unknown [unknown], source: Vendor and press describe 'frontier AI labs / foundation model labs' and 'multimillion-dollar contracts with leading AI labs' but name no specific customer; not verifiable
- founding_team:
  - Serena Ge, Co-founder & CEO, Studied CS at the University of Waterloo before dropping out to found Datacurve. Worked with Cohere's CTO on LLM reasoning and coding through synthetic data generation. Forbes 30 Under 30 (per the University of Waterloo).
  - Charley Lee, Co-founder, University of Waterloo CS. Former Google intern. AI research on multi-modal RL and browser-use agents (per his YC profile). Co-author of the DeepSWE launch post.
- recent_developments (last 6 months):
  - 2026-10-05 [product] Product line now spans multiple RL-environment domains; Shipd lists SWE, ML and data-science tracks, source: https://datacurve.ai/products
  - 2026-09-03 [benchmark] DeepSWE leaderboard updated with new frontier models (latest: GPT-6 Astra, Gemini 3.8 Flash), source: https://deepswe.datacurve.ai/changelog
  - 2026-08-26 [open-source] DeepSWE repo is Apache-2.0 licensed and Harbor-format, about 1.8k stars, source: https://api.github.com/repos/datacurve-ai/deep-swe
  - 2026-06-29 [research] Datacurve founding engineer James Shi presents DeepSWE at AI Engineer World's Fair 2026, source: https://ai.engineer/orgs/datacurve
  - 2026-06-19 [benchmark] DeepLearning.AI's The Batch covers DeepSWE as a harder successor to SWE-bench, source: https://deeplearning.ai/the-batch/agentic-tests-beyond-the-bug-hunt
  - 2026-06-14 [benchmark] DeepSWE v1.1 released with isolated verification and structured test reports, source: https://deepswe.datacurve.ai/blog/deepswe-v1-1
  - 2026-06-12 [benchmark] Artificial Analysis replaces SWE-Bench-Pro-Hard with DeepSWE in its Coding Agent Index, source: https://artificialanalysis.ai/methodology/coding-agents-benchmarking
  - 2026-05-26 [benchmark] Datacurve releases DeepSWE, a long-horizon agentic coding benchmark, source: https://deepswe.datacurve.ai/blog/deepswe
- benchmarks_and_research:
  - DeepSWE leaderboard and trials browser (leaderboard), 2026-06-15, Top score 74% pass@1 (three-way tie, as of 2026-09-22), https://deepswe.datacurve.ai/changelog
  - DeepSWE (v1.1) (benchmark), 2026-05-26, As of 2026-09-22, GPT-6 Astra, Gemini 3.8 Flash and Claude Opus 5 tie at 74% pass@1 on the leaderboard. At launch GPT-5.5 led with 70%., https://deepswe.datacurve.ai/
  - DeepSWE GitHub repository (dataset), 2026-05-15, https://github.com/datacurve-ai/deep-swe
- products:
  - Reinforcement Learning environments: Environments for measuring and training agent capability in software engineering, data science, cyber security, machine learning and research., https://datacurve.ai/products
  - Long-horizon Tasks: Tasks that run over long periods and involve ambiguity, partial progress and recovery., https://datacurve.ai/products
  - OTS Datasets: Pre-built, off-the-shelf training datasets., https://datacurve.ai/products
  - Benchmarks and Evals: Custom evaluations. DeepSWE is the public example., https://datacurve.ai/products
  - Agent Trajectories: Full agent execution traces., https://datacurve.ai/products
  - Supervised Fine-tuning data: Expert demonstrations for SFT., https://datacurve.ai/products
  - Shipd: Datacurve-operated bounty platform where vetted engineers solve paid problems that become training data. It has three tracks: Mars (software engineering), Eris (ML competitions) and Monte (data science). This is Datacurve's contributor supply., https://shipd.ai/
  - DeepSWE: Public long-horizon coding benchmark, leaderboard and trials browser., https://deepswe.datacurve.ai/
- notes_for_researchers:
  - DeepSWE tasks are written from scratch rather than mined from merged PRs, to resist contamination. Datacurve reports verifier false positives of 0.3% against 8.5% for SWE-Bench Pro, and false negatives of 1.1% against 24.0%, measured on an audited sample. These are self-reported figures.
  - Graded per-rollout trials can be inspected at deepswe.datacurve.ai/data/v1.1/trials.
  - DeepSWE v1.1 grades the committed git patch in a fresh isolated container and emits per-test CTRF reports, so partial-progress analysis is possible.
  - Artificial Analysis uses DeepSWE v1.1 as one of three equally weighted components of its Coding Agent Index (v1.5, September 2026).
- notes_for_program_managers:
  - Engagement is project-based custom data and environments, through contact@datacurve.ai. No named customers are published, and frontier-lab customer ties remain self-described and unverified.
  - Funding is unchanged since the October 2025 Series A ($15M led by Chemistry), about $17.7M in total. No new round was found as of 2026-10-05. Tracxn's $18.2M total is an unreconciled aggregator figure.
  - Small team: LinkedIn shows 29 employees (2026-10-05), against about 36 in the 2026-06 baseline. Tracxn reports 28 as of 2026-07-31. The careers page lists 3 open roles in San Francisco.
  - The offering has widened from coding data to multi-domain RL environments, long-horizon tasks, agent trajectories and OTS datasets, and Shipd now has ML and data-science tracks.
- notes_for_data_ops:
  - DeepSWE tasks use the Harbor format (task.toml, instruction.md, Docker environment/, tests/ verifier, solution/) and run through Pier, a Harbor-compatible sandbox with per-agent network allowlists. Pier also supports claude-code, codex, gemini-cli and opencode agents.
  - The DeepSWE repo is Apache-2.0 licensed.
  - No public security or trust page and no SOC 2 attestation were found.
- sources:
  - https://datacurve.ai/ (accessed 2026-10-05)
  - https://datacurve.ai/products (accessed 2026-10-05)
  - https://datacurve.ai/research (accessed 2026-10-05)
  - https://datacurve.ai/careers (accessed 2026-10-05)
  - https://deepswe.datacurve.ai/ (accessed 2026-10-05)
  - https://deepswe.datacurve.ai/blog/deepswe (accessed 2026-10-05)
  - https://deepswe.datacurve.ai/blog/deepswe-v1-1 (accessed 2026-10-05)
  - https://deepswe.datacurve.ai/changelog (accessed 2026-10-05)
  - https://deepswe.datacurve.ai/data/v1.1/trials (accessed 2026-10-05)
  - https://github.com/datacurve-ai/deep-swe (accessed 2026-10-05)
  - https://api.github.com/repos/datacurve-ai/deep-swe (accessed 2026-10-05)
  - https://artificialanalysis.ai/methodology/coding-agents-benchmarking (accessed 2026-10-05)
  - https://alphasignal.ai/news/datacurve-s-deepswe-catches-claude-cheating-and-reshuffles-coding-agen (accessed 2026-10-05)
  - https://deeplearning.ai/the-batch/agentic-tests-beyond-the-bug-hunt (accessed 2026-10-05)
  - https://mer.vin/2026/05/deepswe-benchmark-how-datacurve-separates-real-agentic-coding-ability/ (accessed 2026-10-05)
  - https://ai.engineer/orgs/datacurve (accessed 2026-10-05)
  - https://www.ai.engineer/worldsfair (accessed 2026-10-05)
  - https://www.startuphub.ai/ai-news/ai-research/2026/data-curve-launches-deepswe-coding-benchmark (accessed 2026-10-05)
  - https://shipd.ai/ (accessed 2026-10-05)
  - https://www.ycombinator.com/companies/datacurve (accessed 2026-10-05)
  - https://www.linkedin.com/company/datacurveai (accessed 2026-10-05)
  - https://techcrunch.com/2025/10/09/datacurve-raises-15-million-to-take-on-scaleai/ (accessed 2026-10-05)
  - https://www.chemistry.vc/post/staying-ahead-of-the-curve (accessed 2026-10-05)
  - https://tracxn.com/d/companies/datacurve/__2w9YlTVQ2FbPLzVtvcqLEI1v_XXMwKgf4xoxLtbqb-0 (accessed 2026-10-05)
  - https://sacra.com/c/datacurve/ (accessed 2026-06-07)
  - https://uwaterloo.ca/computer-science/news/cs-led-startup-secures-177m-transform-ai-training-data (accessed 2026-06-07)
  - https://www.menlotimes.com/post/datacurve-is-taking-on-scale-ai-building-frontier-coding-data-for-foundation-model-labs (accessed 2026-06-07)

## Proximal, rank #6
- slug: proximal
- segment: Commercial vendors
- website: https://www.proximal.ai
- focus_areas: Coding, Long-Horizon, Private Codebases
- positioning: Proximal is a research lab in San Francisco and Bangalore that treats post-training data as a core research problem. It builds long-horizon coding RL environments and data for frontier labs and enterprises, and it publishes the FrontierSWE benchmark and verifier audits such as CyberGym-Verified. In September 2026 it raised a $15M seed led by General Catalyst at a $300M valuation and claimed $200M in annualized revenue, which is self-reported. It also said it will expand into drug discovery, chip design and legacy software modernization.
- best_fit: Frontier labs or AI startups needing long-horizon, real-codebase RL environments and quality-aware (fuzzy) verifiers to post-train coding agents.
- overall_confidence: medium
- founded_year: 2026 [reported], source: https://www.proximal.ai/blog/proximal (announced 2026-02-18; accessed 2026-06-07)
- status: active [confirmed], source: https://www.proximal.ai/ (accessed 2026-06-07)
- hq_location: San Francisco, CA, USA [confirmed], source: https://www.proximal.ai/blog/proximal (San Francisco); https://www.proximal.ai/careers (accessed 2026-06-07)
- other_locations: Bangalore, India [confirmed], source: https://www.proximal.ai/careers (accessed 2026-10-05)
- distributed_remote: no [reported], source: https://www.proximal.ai/careers (emphasizes in-person collaboration in SF and Bangalore; accessed 2026-06-07)
- current_headcount: 11-50 band (LinkedIn); '~25' is an unverified estimate [estimated], source: https://www.linkedin.com/company/proximalhq (public snippet shows 11-50; accessed 2026-06-07)
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/proximalhq (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: 5 [confirmed], source: https://www.proximal.ai/careers (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: https://www.proximal.ai/blog/proximal (team of engineers and researchers; members published at leading conferences; accessed 2026-06-07)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Justus Mattern (co-founder): built open RL infrastructure at Prime Intellect, co-founded Revideo (YC S23), early engineer at Dynamo AI, research on ML and differential privacy (confirmed via justusmattern.com); Calvin Chen (co-founder): prior-exit specifics not corroborated; Navid Pour (co-founder): prior Cursor experience per founder posts, no specific title corroborated; Team from Cursor, Google DeepMind, Meta Superintelligence, Prime Intellect, Citadel and Jane Street; former founders are more than half the team (company-stated, 2026-09-29) [reported], source: https://www.proximal.ai/blog/new-frontiers/ (accessed 2026-10-05)
- published_papers_or_benchmarks: FrontierSWE v2: 34 ultra-long-horizon coding tasks, Proximus harness (2026-09). https://www.frontierswe.com/blog/v2 ; https://github.com/Proximal-Labs/frontier-swe-v2; FrontierSWE (v1): 17 tasks, 20-hour budgets. https://www.proximal.ai/blog/frontierswe/ ; https://github.com/Proximal-Labs/frontier-swe; Robustly Evaluating Cyber Capabilities: CyberGym audit and CyberGym-Verified (2026-09-30). https://www.proximal.ai/blog/cybergym-cyberrl/ [confirmed], source: https://www.proximal.ai/blog (accessed 2026-10-05)
- total_raised: At least $15M disclosed (2026-09 seed). The amount of the earlier Scribble Ventures-led round, if it was a separate round, is undisclosed. [confirmed], source: https://www.proximal.ai/blog/new-frontiers/ (accessed 2026-10-05)
- last_round: Seed: $15M led by General Catalyst at a $300M valuation (announced 2026-09-29) [confirmed], source: https://www.proximal.ai/blog/new-frontiers/ (accessed 2026-10-05)
- notable_investors: General Catalyst (lead, 2026-09 seed); Scribble Ventures (led an earlier investment, per its announcement); SV Angel; Go Global Ventures (Diede van Lamoen); Chemistry; Liam Fedus (CEO, Periodic Labs); Kevin Weil (former CPO, OpenAI); Erik Bernhardsson (CEO, Modal); Angels from OpenAI, Anthropic, Thinking Machines, Renaissance Technologies, Google DeepMind, xAI, Meta Superintelligence, Cursor and Cognition (per founder Calvin Chen's site; self-reported, not independently verified) [confirmed], source: https://www.proximal.ai/blog/new-frontiers/ (accessed 2026-10-05)
- valuation: $300M (company-stated, 2026-09-29) [confirmed], source: https://www.proximal.ai/blog/new-frontiers/ (accessed 2026-10-05)
- revenue_signals: Company claims it was able 'to scale to $200M in annualized revenue' (2026-09-29). This is self-reported. The Information's headline refers to a 'revenue milestone' but the figure is paywalled and not independently verified. [reported], source: https://www.proximal.ai/blog/new-frontiers/ (accessed 2026-10-05)
- soc2: unknown [unknown], source: No security/trust page found; https://www.proximal.ai/security returned 404 (accessed 2026-06-07)
- other_certifications: unknown [unknown]
- security_page: unknown [unknown], source: https://www.proximal.ai/security returned 404 (accessed 2026-06-07)
- what_they_sell: environments [confirmed], source: https://www.proximal.ai/blog/proximal ; https://www.proximal.ai/blog/our-problems (high-fidelity, long-horizon RL environments / coding data engine; accessed 2026-06-07)
- open_source: yes [confirmed], source: https://github.com/Proximal-Labs (accessed 2026-10-05)
- license: No licence on frontier-swe, frontier-swe-v2 or px-eval. v2 task content is 'provided for evaluation purposes'. [confirmed], source: https://github.com/Proximal-Labs/frontier-swe-v2 (accessed 2026-10-05)
- deployment_model: unknown [unknown]
- maturity: unknown [unknown]
- notable_customers: unknown [unknown], source: Company states it works with frontier labs / AI startups but names no specific customers publicly; no third-party customer confirmation found (accessed 2026-06-07)
- founding_team:
  - Justus Mattern, Co-founder, Built open RL infrastructure at Prime Intellect, co-founded Revideo (YC S23, an open-source animation library) and was one of the first engineers at Dynamo AI (federated learning, on-device ML). Studied physics at RWTH Aachen and left in 2023. Did research on ML and differential privacy.
  - Calvin Chen, Co-founder, Named as a co-founder in Scribble Ventures' announcement and listed as an author on the FrontierSWE v2 core team. His site says he 'invests, writes' and that the founding team includes 'second-time exited founders', but it names no specific prior company or exit.
  - Navid Pour, Co-founder, Named as a co-founder in Scribble Ventures' announcement and listed as an author on the FrontierSWE v2 core team. Prior Cursor experience comes only from founder posts saying the team includes 'early engineers from Cursor'; no specific title is corroborated.
- funding_history:
  - 2026-09-29 Seed $15M led by General Catalyst, valuation $300M, source: https://www.proximal.ai/blog/new-frontiers/
- recent_developments (last 6 months):
  - 2026-10-05 [hire] Careers page lists 5 open roles across San Francisco and Bangalore, source: https://www.proximal.ai/careers
  - 2026-09-30 [research] 'Robustly Evaluating Cyber Capabilities': Proximal audits CyberGym and releases a filtered CyberGym-Verified subset, source: https://www.proximal.ai/blog/cybergym-cyberrl/
  - 2026-09-29 [funding] Proximal announces a $15M seed round led by General Catalyst at a $300M valuation and claims $200M in annualized revenue, source: https://www.proximal.ai/blog/new-frontiers/
  - 2026-09-19 [open-source] px-eval open-sourced: a thin runner for Harbor tasks, source: https://github.com/Proximal-Labs/px-eval
  - 2026-09 [funding] The Information: 'General Catalyst Backs Coding Data Startup Proximal As It Reaches Revenue Milestone', source: https://www.theinformation.com/newsletters/ai-agenda/general-catalyst-backs-coding-data-startup-proximal-reaches-revenue-milestone
  - 2026-09 [benchmark] FrontierSWE v2 released: 34 ultra-long-horizon tasks, 20-hour budgets and a new Proximus harness, source: https://www.frontierswe.com/blog/v2
  - 2026-07-06 [legal] Public Master Development Services Agreement posted (last updated 2026-07-06), source: https://www.proximal.ai/msa/
- benchmarks_and_research:
  - CyberGym-Verified (CyberGym audit) (paper), 2026-09-30, Kimi K3 scores 95.6% pass@1 on CyberGym-Verified, against 81.9% on the original. On Verified, GLM 5.3 scores 93.5%, GPT-5.6 Sol 91.1% and Claude Fable 5.1/Opus 5 87.8%., https://www.proximal.ai/blog/cybergym-cyberrl/
  - FrontierSWE v2 (benchmark), 2026-09, Release post: Claude Fable 5.1 56.29% (Opus 5 used as fallback on content-filter-blocked tasks), GPT-5.6 Sol 32.2%, GLM-5.3 30.2%. The live leaderboard, accessed 2026-10-05, now includes later models: GPT-6 Astra 65.5% ± 8.9, Claude Opus 5.5 62.3% ± 9.8, Gemini 4 Argon 55.0% ± 9.9. It shows Fable 5.1 at 56.5%, slightly different from the post., https://www.frontierswe.com/
  - FrontierSWE (v1) (benchmark), 2026-07-09, Claude Fable 5 ranked first on mean@5 with an average rank of 2.88. The write-up says 'the only models consistently producing at least partial solutions are GPT-5.4 in Codex and Claude Opus 4.6 in Claude Code.', https://github.com/Proximal-Labs/frontier-swe
- products:
  - Post-training data and long-horizon coding RL environments (commercial): Custom data and environments for frontier labs and enterprises training coding agents, sold under a Master Development Services Agreement. There is no public product SKU, pricing or self-serve offering, and no customers are named., https://www.proximal.ai/
  - FrontierSWE (benchmark and leaderboard): Public ultra-long-horizon coding benchmark and leaderboard, now at v2 (34 tasks)., https://www.frontierswe.com/
  - Proximus harness: Minimal coding-agent harness for multi-hour tasks, described as 'essentially mini-swe-agent with small additions'. It adds context compaction, image viewing and Git-backed submission checkpointing. The post does not say whether its code is public., https://www.frontierswe.com/blog/v2
  - px-eval: Open-source thin runner for Harbor tasks that runs rollouts on Modal., https://github.com/Proximal-Labs/px-eval
- notes_for_researchers:
  - FrontierSWE v2 has 34 tasks, a 20-hour budget and 5 trials per model. Its anti-reward-hacking measures are separate-container verification, a non-root agent user with root-owned 0700 scored paths, 1:1 structural mutation at verification time and deterministic instruction-count metrics for performance tasks (https://www.frontierswe.com/blog/v2).
  - The live v2 leaderboard has been extended beyond the models in the release post, and Fable 5.1 shows 56.5% there against 56.29% in the post. Cite the source and the access date when quoting scores.
  - The CyberGym audit shows that differential-execution scoring produces both false negatives and false positives. On the filtered Verified subset, several models score above 87%, which suggests the original benchmark is close to saturated.
  - Stated research themes are fuzzy verifiers, reward-hacking detection and long-horizon environments built from real codebases (https://www.proximal.ai/blog/our-problems/).
- notes_for_program_managers:
  - Seed round of $15M led by General Catalyst at a company-stated $300M valuation, announced 2026-09-29. The Information covered it, but its article is paywalled.
  - The company claims $200M in annualized revenue. That figure is self-reported and unverified; The Information's headline mentions only a 'revenue milestone'.
  - The company says it works 'with frontier labs and enterprises' but names no customers. Kevin Weil (ex-OpenAI CPO), Liam Fedus (Periodic Labs) and Erik Bernhardsson (Modal) are angels; an angel investment does not make their companies customers.
  - It plans to expand beyond coding into drug discovery, chip design and legacy software modernization.
  - Offices are in San Francisco and Bangalore, with 5 open roles as of 2026-10-05.
- notes_for_data_ops:
  - Under the public template MSA (last updated 2026-07-06), Proximal owns the deliverables and the customer gets a non-exclusive internal-use licence. Proximal may keep learnings and model training derived from the work. Negotiate exclusivity or assignment in the statement of work if you need it.
  - Public tasks use the Harbor format (instruction.md, task.toml, environment/Dockerfile, solution/solve.sh, preflight/). Images are pinned by digest in task.toml and can be rebuilt from each task's Dockerfile.
  - px-eval runs Harbor tasks on Modal and needs MODAL_TOKEN_ID and MODAL_TOKEN_SECRET plus model-provider API keys.
  - The frontier-swe, frontier-swe-v2 and px-eval repos show no OSS licence. v2 task content is 'provided for evaluation purposes', with upstream attributions per task.
  - There is no public security or trust page (/security/ and /trust/ return 404), and neither the site nor the MSA mentions SOC 2.
- sources:
  - https://www.proximal.ai/ (accessed 2026-10-05)
  - https://www.proximal.ai/blog (accessed 2026-10-05)
  - https://www.proximal.ai/blog/new-frontiers/ (accessed 2026-10-05)
  - https://www.theinformation.com/newsletters/ai-agenda/general-catalyst-backs-coding-data-startup-proximal-reaches-revenue-milestone (accessed 2026-10-05)
  - https://www.proximal.ai/blog/cybergym-cyberrl/ (accessed 2026-10-05)
  - https://www.frontierswe.com/blog/v2 (accessed 2026-10-05)
  - https://www.frontierswe.com/ (accessed 2026-10-05)
  - https://www.proximal.ai/blog/frontierswe/ (accessed 2026-10-05)
  - https://www.proximal.ai/blog/proximal/ (accessed 2026-10-05)
  - https://www.proximal.ai/msa/ (accessed 2026-10-05)
  - https://github.com/Proximal-Labs (accessed 2026-10-05)
  - https://github.com/Proximal-Labs/frontier-swe-v2 (accessed 2026-10-05)
  - https://github.com/Proximal-Labs/px-eval (accessed 2026-10-05)
  - https://www.proximal.ai/careers (accessed 2026-10-05)
  - https://www.proximal.ai/security/ (accessed 2026-10-05)
  - https://www.justusmattern.com/ (accessed 2026-10-05)
  - https://www.calvinjaychen.com/ (accessed 2026-10-05)
  - https://www.linkedin.com/posts/imstusmith_excited-to-finally-share-that-scribble-ventures-activity-7430007099004432384-cqbR (accessed 2026-10-05)
  - https://www.proximal.ai/blog/proximal (accessed 2026-06-07)
  - https://www.proximal.ai/blog/our-problems (accessed 2026-06-07)
  - https://www.linkedin.com/company/proximalhq (accessed 2026-06-07)
  - https://www.linkedin.com/posts/justus-mattern-a04230184_incredibly-excited-to-introduce-proximal-activity-7429982692999438336-Bsdr (accessed 2026-06-07)
  - https://www.frontierswe.com (accessed 2026-06-07)
  - https://github.com/Proximal-Labs/frontier-swe (accessed 2026-06-07)
  - https://www.proximal.ai/security (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/proximal (accessed 2026-06-07)

## Gray Swan AI, rank #7
- slug: gray-swan-ai
- segment: Commercial vendors
- website: https://www.grayswan.ai/
- focus_areas: unknown
- positioning: Gray Swan AI is a Pittsburgh AI security company spun out of Carnegie Mellon (Fredrikson, Kolter). It sells adversarial red teaming and runtime protection for AI models and agents through three products. Shade does automated red teaming. Cygnal enforces policies and detects indirect prompt injection at runtime, and is now also available through the TrueFoundry and Bifrost gateways. Arena is a crowdsourced red-teaming network of more than 15,000 researchers. Gray Swan is a pre-release evaluation partner to frontier labs: Anthropic, OpenAI, Google DeepMind and Meta all cite its Shade, IPI Arena or ART evaluations in their 2026 system cards and safety reports. It is not a general RL-environment vendor.
- best_fit: Buyers needing adversarial evaluation, red-teaming arenas, and runtime guardrails for frontier or enterprise LLM/agent deployments.
- overall_confidence: high
- founded_year: 2023 or 2024 (conflicting: LinkedIn lists 2023; Tracxn lists 2024; public product launch July 16, 2024) [reported], source: https://www.linkedin.com/company/grayswanai (public snippet, accessed 2026-06-07); https://tracxn.com/d/companies/gray-swan-ai/__nZCNR23H086-Z-r6gED-nlgrD6otsq465-kAbWKigfY (accessed 2026-06-07); https://www.grayswan.ai/blog/gray-swan-launch (accessed 2026-06-07)
- status: active [confirmed], source: https://www.grayswan.ai/news/gray-swan-announces-series-a (accessed 2026-06-07)
- hq_location: Pittsburgh, Pennsylvania, USA [reported], source: https://www.linkedin.com/company/grayswanai (public snippet, accessed 2026-06-07); https://www.finsmes.com/2026/06/gray-swan-raises-40m-in-series-a-funding.html (accessed 2026-06-07)
- other_locations: unknown [unknown]
- distributed_remote: yes (careers page states flexible work arrangements) [reported], source: https://www.grayswan.ai/careers (accessed 2026-06-07)
- current_headcount: ~29-75 (LinkedIn shows ~75 employees listed; Tracxn reports 59 as of 2026-04-30; PitchBook reports 29) - figure uncertain, sources diverge [estimated], source: https://www.linkedin.com/company/grayswanai (accessed 2026-06-07); https://tracxn.com/d/companies/gray-swan-ai/__nZCNR23H086-Z-r6gED-nlgrD6otsq465-kAbWKigfY (accessed 2026-06-07); https://pitchbook.com/profiles/company/633269-80 (accessed 2026-06-07)
- headcount_band: 11-50 [estimated], source: https://pitchbook.com/profiles/company/633269-80 (accessed 2026-06-07); https://tracxn.com/d/companies/gray-swan-ai/__nZCNR23H086-Z-r6gED-nlgrD6otsq465-kAbWKigfY (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: unknown (careers page advertises ML Engineer roles; exact count not enumerated) [unknown], source: https://www.grayswan.ai/careers (accessed 2026-06-07)
- has_researchers: yes [confirmed], source: https://www.grayswan.ai/about (accessed 2026-06-07)
- researcher_count: unknown (research team referenced; exact count not published) [unknown], source: https://www.grayswan.ai/about (accessed 2026-06-07)
- researcher_backgrounds: Zico Kolter (Co-founder, Chief Scientist): Professor and Department Head of the CMU Machine Learning Department; OpenAI board member and chair of its Safety and Security Committee; Matt Fredrikson (Co-founder, CEO): CMU faculty, adversarial-ML researcher, GCG co-author; Andy Zou: lead or co-author on Gray Swan papers (ART, GCG, IPI). Tracxn names him a co-founder, but the company's leadership page does not list him (unconfirmed as co-founder); Frequent co-authors on Gray Swan papers include Mateusz Dziemian, Maxwell Lin, Eliot Jones, Micha Nowak and Nick Winter (from arXiv author lists; arXiv shows no affiliations, so employment is not confirmed) [reported], source: https://zicokolter.com/bio (accessed 2026-10-05)
- published_papers_or_benchmarks: Evaluations cited in 23 frontier model system cards and reports listed on the vendor homepage (Claude family through Opus/Sonnet 5.5, GPT-5, GPT-6 Astra, o1, o3-mini, Gemini 3.7 Flash, 3.8 Flash and 4 Argon, Meta Muse Spark and Muse Spark 1.1). The count is vendor-stated; the Opus 5.5, GPT-6 Astra, Gemini 3.7 Flash, Gemini 3.8 Flash, Gemini 4 Argon and Muse Spark 1.1 documents were checked and do cite Gray Swan.; IPI Arena / How Vulnerable Are AI Agents to Indirect Prompt Injections? (arXiv 2603.15714, March 2026); Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing / ARTEMIS (arXiv 2512.09882, December 2025, with Stanford); D-REX deceptive reasoning benchmark (arXiv 2509.17938, September 2025); Agent Red Teaming (ART) benchmark, from Security Challenges in AI Agent Deployment, with UK AISI (arXiv 2507.20526, July 2025); Team research listed on the research page: GCG, HarmBench, AgentHarm, WMDP, Cybench, Circuit Breakers, RepE, VLA attacks, Safety Pretraining; NIST CAISI blog on the UK AISI x Gray Swan agent red-teaming competition [confirmed], source: https://www.grayswan.ai/research (accessed 2026-10-05)
- total_raised: $40M disclosed (Series A $40M; any prior seed amount not publicly disclosed) [reported], source: https://www.grayswan.ai/news/gray-swan-announces-series-a (accessed 2026-06-07); https://www.finsmes.com/2026/06/gray-swan-raises-40m-in-series-a-funding.html (accessed 2026-06-07)
- last_round: Series A, $40M, May 28, 2026 [confirmed], source: https://www.grayswan.ai/news/gray-swan-announces-series-a (accessed 2026-06-07); https://www.finsmes.com/2026/06/gray-swan-raises-40m-in-series-a-funding.html (accessed 2026-06-07)
- notable_investors: Wing Venture Capital (co-lead); Madrona (co-lead); Obvious Ventures; Snowflake Ventures; Hudson River Trading; Samsung Next; Magarac Venture Partners (existing) [confirmed], source: https://www.grayswan.ai/news/gray-swan-announces-series-a (accessed 2026-06-07); https://www.finsmes.com/2026/06/gray-swan-raises-40m-in-series-a-funding.html (accessed 2026-06-07)
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: Type II [reported], source: https://trust.grayswan.ai (accessed 2026-06-07)
- other_certifications: Cyber Essentials; TSM Awardable badge shown in site footer (presumably the DoD CDAO Tradewinds Solutions Marketplace; self-displayed, not independently verified) [reported], source: https://www.grayswan.ai/ (accessed 2026-10-05)
- security_page: https://trust.grayswan.ai [confirmed], source: https://www.grayswan.ai/ (accessed 2026-06-07)
- what_they_sell: mixed (adversarial red-teaming/evals + runtime security infra; Arena crowdsourced red-teaming generates attack-trajectory data) [confirmed], source: https://www.grayswan.ai/about (accessed 2026-06-07); https://www.grayswan.ai/ (accessed 2026-06-07)
- open_source: no [estimated], source: https://www.grayswan.ai/ (accessed 2026-06-07)
- license: unknown [unknown]
- deployment_model: Managed-hosted / API. The SaaS platform at platform.grayswan.ai offers Cygnal runtime protection (also through the TrueFoundry and Bifrost AI gateways), Shade automated red teaming, hosted Arena competitions and managed adversarial evaluations. [confirmed], source: https://www.grayswan.ai/news/gray-swan-integrates-with-bifrost (accessed 2026-10-05)
- maturity: GA [reported], source: https://www.grayswan.ai/news/gray-swan-announces-series-a (accessed 2026-06-07)
- notable_customers: Anthropic (verified, frontier-lab tie); OpenAI (verified, frontier-lab tie); Meta (verified, frontier-lab tie); Google DeepMind (verified, frontier-lab tie); xAI (self-claimed, frontier-lab tie); Amazon (self-claimed); Snowflake (self-claimed); ByteDance (self-claimed); ElevenLabs (self-claimed); Intercom (self-claimed); Deloitte (self-claimed); UK AI Security Institute (AISI) (verified); OpenHands (self-claimed); AIUC (self-claimed) [confirmed], source: https://storage.googleapis.com/deepmind-media/gemini/gemini_3-7_flash_fsf_report.pdf (accessed 2026-10-05)
- founding_team:
  - Matt Fredrikson, Co-founder, CEO, Carnegie Mellon University computer science faculty and adversarial-ML/AI-security researcher. Co-author of GCG (Universal and Transferable Adversarial Attacks on Aligned Language Models).
  - Zico Kolter, Co-founder, Chief Scientist, Professor and Department Head of the Machine Learning Department at CMU. Serves on OpenAI's Board and chairs its Safety and Security Committee.
  - Rob Jenks, Chief Strategy Officer (not a founder; appointed 2026-01-08), Former SVP Strategy & Innovation at Tanium. Led product and go-to-market functions at C3.ai. PhD in Physics, Harvard.
  - Spencer Whitman, Chief Product Officer (not a founder), Listed as CPO on the company's leadership page, which gives no prior background. He is a co-author of the 2026 IPI competition paper (arXiv 2603.15714).
- funding_history:
  - 2026-05-28 Series A $40M led by Wing Venture Capital, Madrona, source: https://www.grayswan.ai/news/gray-swan-announces-series-a
- recent_developments (last 6 months):
  - 2026-10-01 [partnership] Gray Swan integrates Cygnal with Bifrost, Maxim's AI gateway, source: https://www.grayswan.ai/news/gray-swan-integrates-with-bifrost
  - 2026-09-30 [partnership] Google's Gemini 4 Argon launch cites Gray Swan's Indirect Prompt Injection benchmark, source: https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-4-argon/
  - 2026-09-22 [customer] Claude Opus 5.5 system card uses Gray Swan's Shade and IPI benchmark for external red teaming, source: https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf
  - 2026-09-03 [customer] OpenAI's GPT-6 Astra system card reports external IPI evaluations run with Gray Swan, source: https://deploymentsafety.openai.com/gpt-6-astra/gpt-6-astra.pdf
  - 2026-08 [partnership] Google DeepMind's Gemini 3.7 Flash safety report says it 'partnered with Gray Swan' for cyber and bio red teaming, source: https://storage.googleapis.com/deepmind-media/gemini/gemini_3-7_flash_fsf_report.pdf
  - 2026-07-31 [partnership] Gray Swan integrates Cygnal with TrueFoundry AI Gateway, source: https://www.grayswan.ai/news/gray-swan-integrates-with-truefoundry
  - 2026-07-09 [customer] Meta's Muse Spark 1.1 evaluation report includes third-party red teaming by Gray Swan on its ART benchmark, source: https://ai.meta.com/static-resource/muse-spark-1-1-evaluation-report
  - 2026-05-28 [funding] Gray Swan raises $40M Series A co-led by Wing and Madrona, source: https://www.grayswan.ai/news/gray-swan-announces-series-a
- benchmarks_and_research:
  - Gray Swan Indirect Prompt Injection (IPI) Arena / benchmark (benchmark), 2026-03-16, 464 participants submitted 272,000 attacks against 13 frontier models, producing 8,648 successful attacks across 41 scenarios. Attack success rates ranged from 0.5% (Claude Opus 4.5) to 8.5% (Gemini 2.5 Pro), and universal attacks transferred across 21 of the 41 behaviors., https://arxiv.org/abs/2603.15714
  - Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing (ARTEMIS) (paper), 2025-12-10, ARTEMIS placed second overall with 9 valid vulnerabilities and an 82% valid-submission rate, outperforming 9 of the 10 humans. Some ARTEMIS variants cost about $18/hr, against about $60/hr for professional penetration testers., https://arxiv.org/abs/2512.09882
  - D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models (benchmark), 2025-09-22, https://arxiv.org/abs/2509.17938
  - Agent Red Teaming (ART) benchmark: Security Challenges in AI Agent Deployment (benchmark), 2025-07-28, Participants submitted 1.8M prompt-injection attacks, and more than 60,000 caused policy violations. Nearly all agents violated policy for most behaviors within 10-100 queries., https://arxiv.org/abs/2507.20526
  - Gray Swan Arena / Proving Ground (leaderboard), 2025-06-24, https://app.grayswan.ai/arena
  - Adversarial Attacks on Robotic Vision Language Action Models (paper), 2025-06-03, https://arxiv.org/abs/2506.03350
  - Safety Pretraining: Toward the Next Generation of Safe AI (paper), 2025-04-23, Attack success rate fell from 38.8% to 8.4% on standard safety benchmarks with no loss of general performance., https://arxiv.org/abs/2504.16980
  - AgentHarm (benchmark), 2024-10-11, https://arxiv.org/abs/2410.09024
  - Cybench (benchmark), 2024-08-15, https://arxiv.org/abs/2408.08926
  - Circuit Breakers (Improving Alignment and Robustness with Circuit Breakers) (paper), 2024-06-06, https://arxiv.org/abs/2406.04313
  - WMDP (benchmark), 2024-03-05, https://arxiv.org/abs/2403.03218
  - HarmBench (benchmark), 2024-02-06, https://arxiv.org/abs/2402.04249
  - Representation Engineering (RepE) (paper), 2023-10-02, https://arxiv.org/abs/2310.01405
  - GCG / llm-attacks (Universal and Transferable Adversarial Attacks on Aligned Language Models) (paper), 2023-07-27, https://llm-attacks.org
  - MMLU (benchmark), 2020-09-07, https://arxiv.org/abs/2009.03300
- products:
  - Shade: Automated AI red-teaming agent for adversarial testing of models and agents before deployment and in CI/CD. Google DeepMind (Gemini 3.7 Flash) and Anthropic (Claude Opus 5.5) name it in their 2026 pre-release red teaming., https://www.grayswan.ai/
  - Cygnal: Runtime enforcement that screens requests, responses and tool outputs against custom policies, including indirect prompt injection detection. It is available via API and through the TrueFoundry (2026-07) and Bifrost (2026-10) AI gateways., https://www.grayswan.ai/
  - Arena: Crowdsourced adversarial red-teaming network (15,000+ researchers, self-claimed) running public competitions, quarterly IPI Arenas and weekly Proving Ground challenges., https://app.grayswan.ai/arena
  - Adversarial Evaluation: Pre-release adversarial safety evaluations of frontier models, including prompt injection, agent red teaming and cyber/bio red teaming. These are cited in system cards and safety reports from Anthropic, OpenAI, Google DeepMind and Meta., https://www.grayswan.ai/
  - Gray Swan Platform: Customer platform and API console (login page)., https://platform.grayswan.ai
- notes_for_researchers:
  - The IPI competition paper (arXiv 2603.15714, March 2026) reports 272K attacks against 13 models with ASR from 0.5% to 8.5%. The competition environment and 95 Qwen-specific attacks are open-sourced, and the full dataset is shared with UK AISI and US CAISI.
  - OpenAI's GPT-6 Astra card (2026-09-03) evaluates on 1,810 curated attacks from Gray Swan's Q1 and Q2 2026 IPI Arenas. Meta's Muse Spark 1.1 report (2026-07-09) uses Gray Swan's private ART benchmark with pass@k ASR.
  - The ART benchmark (arXiv 2507.20526, with UK AISI) is built from 1.8M attacks, and agents violate policy within 10-100 queries.
  - D-REX (2025) targets deceptive chain-of-thought. The ARTEMIS paper (December 2025, with Stanford) compares agent penetration testers with human professionals.
  - The research page also lists GCG, HarmBench, AgentHarm, WMDP, Cybench, Circuit Breakers and RepE, which are co-authored by team members.
- notes_for_program_managers:
  - Gray Swan closed a $40M Series A in May 2026, co-led by Wing and Madrona. It claims more than 20 customers across frontier labs and enterprises (self-claimed).
  - Four frontier labs have confirmed the relationship in their own documents during the window: Anthropic (Claude Opus 5.5 card), OpenAI (GPT-6 Astra card), Google DeepMind (Gemini 3.7 Flash safety report and Gemini 3.8 Flash and 4 Argon posts) and Meta (Muse Spark 1.1 report). xAI is still only a logo.
  - The homepage lists 23 system cards that cite Gray Swan, up from the 11 cited at the Series A. The count is vendor-stated, though the cards checked in this run do mention Gray Swan.
  - The UK AISI partnership on agent red teaming is corroborated by NIST CAISI and co-authored papers. Anthropic's card describes the IPI benchmark as built with UK AISI, US CAISI and model developers.
  - Gray Swan sells a SaaS platform (Shade, Cygnal) plus managed adversarial evaluations and sponsored Arena challenges.
  - The site footer shows a 'TSM Awardable' badge, presumably the DoD CDAO Tradewinds Solutions Marketplace. It is self-displayed and not independently verified.
- notes_for_data_ops:
  - Cygnal is available through the TrueFoundry AI Gateway (2026-07-31) and the Bifrost gateway (2026-10-01) as a config change with no code changes. On TrueFoundry it can target users, teams, models or MCP servers and can start in observe-only mode. The vendor states about 100-200ms of screening latency on TrueFoundry.
  - The trust center at trust.grayswan.ai lists SOC 2 Type 2 and Cyber Essentials.
  - No Harbor, Terminal-Bench or MCP environment packaging is published. This is a security and evaluation vendor, not an RL-environment packager.
- sources:
  - https://www.grayswan.ai/news (accessed 2026-10-05)
  - https://www.grayswan.ai/news/gray-swan-integrates-with-bifrost (accessed 2026-10-05)
  - https://www.grayswan.ai/news/gray-swan-integrates-with-truefoundry (accessed 2026-10-05)
  - https://www.grayswan.ai/news/gray-swan-announces-series-a (accessed 2026-10-05)
  - https://www.grayswan.ai/news/gray-swan-appoints-rob-jenks-as-chief-strategy-officer-to-lead-global-ai-security-market-expansion (accessed 2026-10-05)
  - https://www.grayswan.ai/about (accessed 2026-10-05)
  - https://www.grayswan.ai/ (accessed 2026-10-05)
  - https://www.grayswan.ai/research (accessed 2026-10-05)
  - https://www.grayswan.ai/careers (accessed 2026-10-05)
  - https://www.grayswan.ai/blog (accessed 2026-10-05)
  - https://trust.grayswan.ai (accessed 2026-10-05)
  - https://www.grayswan.ai/news/introducing-the-gray-swan-ai-proving-ground (accessed 2026-10-05)
  - https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-4-argon/ (accessed 2026-10-05)
  - https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/ (accessed 2026-10-05)
  - https://storage.googleapis.com/deepmind-media/gemini/gemini_3-7_flash_fsf_report.pdf (accessed 2026-10-05)
  - https://deploymentsafety.openai.com/gpt-6-astra/gpt-6-astra.pdf (accessed 2026-10-05)
  - https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf (accessed 2026-10-05)
  - https://ai.meta.com/static-resource/muse-spark-1-1-evaluation-report (accessed 2026-10-05)
  - https://arxiv.org/abs/2603.15714 (accessed 2026-10-05)
  - https://arxiv.org/abs/2512.09882 (accessed 2026-10-05)
  - https://arxiv.org/abs/2509.17938 (accessed 2026-10-05)
  - https://arxiv.org/abs/2507.20526 (accessed 2026-10-05)
  - https://arxiv.org/abs/2406.04313 (accessed 2026-10-05)
  - https://arxiv.org/abs/2310.01405 (accessed 2026-10-05)
  - https://llm-attacks.org (accessed 2026-10-05)
  - https://zicokolter.com/bio (accessed 2026-10-05)
  - https://platform.grayswan.ai (accessed 2026-10-05)
  - https://www.linkedin.com/company/grayswanai (accessed 2026-06-07)
  - https://tracxn.com/d/companies/gray-swan-ai/__nZCNR23H086-Z-r6gED-nlgrD6otsq465-kAbWKigfY (accessed 2026-06-07)
  - https://www.finsmes.com/2026/06/gray-swan-raises-40m-in-series-a-funding.html (accessed 2026-06-07)
  - https://technical.ly/entrepreneurship/gray-swan-ai-security-40m-series-a/ (accessed 2026-06-07)
  - https://natlawreview.com/press-releases/gray-swan-ai-security-company-trusted-every-major-frontier-lab-raises-40m (accessed 2026-06-07)
  - https://www.nist.gov/blogs/caisi-research-blog/insights-ai-agent-security-large-scale-red-teaming-competition (accessed 2026-06-07)
  - https://www.grayswan.ai/news/uk-aisi-x-gray-swan-agent-red-teaming-challenge-results-snapshot (accessed 2026-06-07)

## Vals AI, rank #8
- slug: vals-ai
- segment: Commercial vendors
- website: https://www.vals.ai
- focus_areas: unknown
- positioning: Vals AI is an independent, a16z-backed third-party evaluator. It benchmarks frontier models and AI applications on economically valuable professional work (finance, legal, tax, coding, healthcare) and, since 2026, on frontier risks (recursive self-improvement, cyber, child safety). It publishes the Vals Index and more than 20 proprietary leaderboards, open-sources its Valkyrie eval infrastructure, offers the self-serve Vals Smith for custom coding benchmarks, and runs a public-sector practice from Washington, DC.
- best_fit: Buyers who need neutral, domain-specific (legal/finance/healthcare) benchmarking and ongoing evaluation of LLM applications on their own data and tasks.
- overall_confidence: high
- founded_year: 2023 [reported], source: Tracxn, Grokipedia, QA-Financial founder coverage (accessed 2026-06-07); note Crunchbase/CB Insights list 2024 and first press coverage is April 2024
- status: active [confirmed], source: https://www.vals.ai/home (accessed 2026-06-07)
- hq_location: San Francisco, USA [reported], source: https://www.linkedin.com/company/vals-ai (public snippet, accessed 2026-06-07)
- other_locations: Washington, DC [confirmed], source: https://www.vals.ai/blogs/vals-public-sector-memo (accessed 2026-10-05)
- distributed_remote: unknown [unknown]
- current_headcount: ~27 employees (Tracxn, as of July 2026; aggregator). The company says the team tripled in the six months to Aug 2026. [reported], source: https://tracxn.com/d/companies/vals-ai/__Aeq7C2n56rLfgsWjTrbrPIwiOteKWlKhIWYPbapLuow (accessed 2026-10-05)
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/vals-ai (accessed 2026-06-07)
- headcount_growth: Team tripled in the six months before Aug 2026 (company statement); actively hiring [confirmed], source: https://www.vals.ai/blogs/series-a (accessed 2026-10-05)
- open_roles_count: unknown [unknown]
- has_researchers: yes [reported], source: arXiv:2508.00828 authored by Vals AI team; founders ex-Stanford AI master's; Stanford collaborations (accessed 2026-06-07)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Co-founder/CEO Rayan Krishnan - left Stanford's AI master's program to start Vals; Co-founder/CTO Langston Nashold - Stanford AI master's program; Glenn Parham - Head of Public Sector (previously AI for national security at the Pentagon and in big tech); Rez Havaei - founding engineer per TechTimes (2024); QA-Financial (2024) calls him a co-founder; current status unverified; Research staff publishing on the blog: Daniel Fein, Andrea Mock, Nikil Ravi, Geby Jaff, Hung Tran, Olivia Fowler, Omar Almatov; Collaborators: Stanford SALT Lab (Diyi Yang, Valerie Chen), Stanford Psychiatry (Eric Lin, MD), Caroline Figueroa (teen-safety co-author) [confirmed], source: https://www.vals.ai/blogs (accessed 2026-10-05)
- published_papers_or_benchmarks: Finance Agent Benchmark - arXiv:2508.00828; Finance Agent v2 (927 questions); Vals Index v2.1 and Vals Multimodal Index; Vals RSI Index (with CoreWeave, 2026-08); SRE Bench (2026-08) and CyberBench v1.1; Web Search Index (2026-07; expanded 2026-10); Excel Modeling Benchmark (2026-07); Public Benefits Bench (with Code for America / Center for Civic Futures, 2026-06); Legal Research Bench, Tax Agent Bench, MedCode, MedScribe, ProofBench, MysteryMechanism, SAGE, Vibe Code Bench, Code Migration, CUA-bench, Time Horizon Index: KSP; Teen AI safety study (with Stanford SALT Lab, 2026-09); AI cheating audit, AI detection benchmark, Environmental Impacts Report (2026-09); Vals Legal AI Report (VLAIR), Feb 2025 [confirmed], source: https://www.vals.ai/benchmarks (accessed 2026-10-05)
- total_raised: $40M Series A (Aug 2026) disclosed. Earlier pre-seed/seed money from 8VC, Pear VC and Bloomberg Beta is confirmed but the amounts are not; aggregators' earlier '$5M seed' figure remains unverified. [confirmed], source: https://www.vals.ai/blogs/series-a (accessed 2026-10-05)
- last_round: Series A, $40M, 2026-08-13, led by a16z [confirmed], source: https://www.vals.ai/blogs/series-a (accessed 2026-10-05)
- notable_investors: Andreessen Horowitz (a16z); 8VC; Pear VC; Bloomberg Beta; HRT Ventures; Next Ladder Ventures [confirmed], source: https://www.vals.ai/blogs/series-a (accessed 2026-10-05)
- valuation: $400M (Series A, Aug 2026; company-stated) [confirmed], source: https://www.vals.ai/blogs/series-a (accessed 2026-10-05)
- revenue_signals: Company says 2026 revenue is up 8x compared with all of 2025 and its customer base has doubled (self-reported, Aug 2026). No absolute figure is disclosed. [confirmed], source: https://www.vals.ai/blogs/series-a (accessed 2026-10-05)
- soc2: unknown (the SOC 2 badge previously on /product is gone after the redesign; no audit report or trust center published) [unknown], source: https://www.vals.ai/product (accessed 2026-10-05)
- other_certifications: unknown [unknown], source: https://www.vals.ai/product (accessed 2026-10-05)
- security_page: https://www.vals.ai/vulnerability-disclosure-program (Vulnerability Disclosure Program, effective 2026-08-31; no trust center or certifications listed) [confirmed], source: https://www.vals.ai/vulnerability-disclosure-program (accessed 2026-10-05)
- what_they_sell: evals [confirmed], source: https://www.vals.ai/product (accessed 2026-06-07); https://www.vals.ai/home (accessed 2026-06-07)
- open_source: yes: Valkyrie eval infrastructure (AGPL-3.0); model-library, finance-agent, finance-agent-v2, legal-research-bench and web-search-agent-public (MIT); plus proof-bench, ioi-agent and benchmark-service repos. Core benchmark test sets are largely private. [confirmed], source: https://github.com/vals-ai (accessed 2026-10-05)
- license: AGPL-3.0 (Valkyrie); MIT (finance-agent, finance-agent-v2, model-library, legal-research-bench) [confirmed], source: https://github.com/vals-ai/Valkyrie (accessed 2026-10-05)
- deployment_model: Public leaderboards at vals.ai; self-serve Vals Smith (connects through GitHub); evaluation infrastructure through Valkyrie (Vals-hosted or self-hosted on AWS, CLI/SDK); public-sector evaluations [confirmed], source: https://github.com/vals-ai/Valkyrie (accessed 2026-10-05)
- maturity: GA; Series A company with a broad public leaderboard suite and a generally available self-serve product (Vals Smith) [confirmed], source: https://www.vals.ai/blogs/series-a (accessed 2026-10-05)
- notable_customers: OpenAI (Vals says its results were cited in OpenAI model cards) (self-claimed, frontier-lab tie); Anthropic (Vals says its results were cited in Anthropic model cards) (self-claimed, frontier-lab tie); Google (Vals says its results were cited in Google model cards) (self-claimed, frontier-lab tie); Meta (Vals says its results were cited in Meta model cards) (self-claimed, frontier-lab tie); xAI (Vals says its results were cited in xAI model cards) (self-claimed, frontier-lab tie); Harvey (industry-partner benchmark run by Vals; partner, not a stated customer) (self-claimed); Exa, Keenable, Parallel, Tavily (Web Search Index partners) (self-claimed); CoreWeave (RSI Index compute collaborator) (self-claimed); Code for America and Center for Civic Futures (Public Benefits Bench partners) (self-claimed); Fisher Phillips, McDermott Will & Emery, Reed Smith, Legal Technology Hub (Legal Research Bench evaluation partners) (self-claimed); US Department of Commerce (Vals says it supported Commerce's AI policy work) (self-claimed) [reported], source: https://www.vals.ai/blogs/series-a (accessed 2026-10-05)
- founding_team:
  - Rayan Krishnan, Co-founder & CEO, Left Stanford's master's program in AI to start Vals AI (TechTimes, 2024). Authored the Series A and Public Sector posts and co-authored the Finance Agent Benchmark paper (arXiv:2508.00828). The CEO title comes from the prior record; the sources opened here call him co-founder without a title.
  - Langston Nashold, Co-founder & CTO, Stanford master's program in AI; QA-Financial (Aug 2024) names him CTO and founder. Co-author of the Finance Agent Benchmark paper (arXiv:2508.00828). a16z's Series A post names him as a founder.
  - Glenn Parham, Head of Public Sector, Joined to lead Vals Public Sector (July 2026). Previously worked on AI for national security at the Pentagon and in big tech, per the Vals memo. Co-authored the Child AI Safety post.
- funding_history:
  - 2026-08-13 Series A $40M led by Andreessen Horowitz (a16z), valuation $400M, source: https://www.vals.ai/blogs/series-a
- recent_developments (last 6 months):
  - 2026-10-02 [benchmark] Vals Web Search Index blog post: search tools tested on expert legal and finance tasks, with partners Exa, Keenable, Parallel and Tavily, source: https://www.vals.ai/blogs/web-search-index
  - 2026-10-01 [partnership] Harvey's Legal Agent Benchmark run by Vals as an 'Industry Partner' benchmark (leaderboard as of 2026-10-01), source: https://www.vals.ai/benchmarks/hlab
  - 2026-10-01 [benchmark] Finance Agent v2 leaderboard update: Gemini 4 Argon leads at 65.40%, source: https://www.vals.ai/benchmarks/fabv2
  - 2026-09-30 [research] 'Has the Bitter Lesson Come for AI Detectors?' AI-text detection benchmark, source: https://www.vals.ai/blogs/ai-detection-benchmark
  - 2026-09-30 [other] Vals press page features NYT coverage of Google's Gemini 4 Argon launch, source: https://www.vals.ai/news
  - 2026-09-28 [research] Ten Claude Sonnet 5.5 agents produce a Lean proof of the Thomson problem for N = 7, source: https://www.vals.ai/blogs/thomson-n7-lean-proof
  - 2026-09-23 [research] Teen-conversation AI safety study with Stanford, covered by the Washington Post and Politico, source: https://www.vals.ai/blogs/evaluating-ai-safety-in-teen-conversations
  - 2026-09-20 [research] Claude Opus 5.5 agents develop a shortest-path algorithm (C-HD) verified in Lean, source: https://www.vals.ai/blogs/faster-shortest-path-algorithm
  - 2026-09-15 [research] 'AI Cheating is on the Rise': audit of benchmark shortcut-seeking, source: https://www.vals.ai/blogs/cheating-on-the-rise
  - 2026-09-14 [other] Bloomberg Tech segment: 'Can Independent Testing Make AI Safer?', source: https://www.youtube.com/watch?v=MpYAoufS588
  - 2026-09-03 [research] Vals Environmental Impacts Report, source: https://www.vals.ai/blogs/vals-environmental-impacts-report
  - 2026-08-31 [other] Vulnerability Disclosure Program published, source: https://www.vals.ai/vulnerability-disclosure-program
  - 2026-08-31 [research] Claude Fable 5.1 decodes the roughly 370-year-old Cyphral Distich cipher, source: https://www.vals.ai/blogs/fable-solves-cyphral-distich
  - 2026-08-13 [funding] Vals raises $40M Series A at a $400M valuation led by a16z, source: https://www.vals.ai/blogs/series-a
  - 2026-08-13 [product] Vals Smith generally available: custom coding benchmarks from any GitHub repo, source: https://www.vals.ai/blogs/series-a
  - 2026-08-13 [benchmark] Frontier Risk Benchmarks released: RSI Index with CoreWeave and SRE Bench cyber benchmark, source: https://www.vals.ai/blogs/series-a
  - 2026-08-12 [research] Child AI Safety initiative: youth-specific evaluations, source: https://www.vals.ai/blogs/child-ai-safety
  - 2026-07-02 [hire] Vals Public Sector launched; Glenn Parham is Head of Public Sector; Washington, DC office opened, source: https://www.vals.ai/blogs/vals-public-sector-memo
  - 2026-07-01 [benchmark] Excel Modeling Benchmark (EMB) for finance agents launched, source: https://letsdatascience.com/news/vals-ai-launches-excel-modeling-benchmark-for-finance-agents-6a2d5a85
  - 2026-06-09 [benchmark] Public Benefits Bench study published with Center for Civic Futures and Code for America, source: https://www.vals.ai/benchmarks/public-benefits-bench
- benchmarks_and_research:
  - Vals Index (v2.1) (leaderboard), 2026-10-02, Gemini 4 Argon 68.90%, Claude Sonnet 5.5 67.04%, Claude Opus 5.5 66.97% (as of 2026-10-02), https://www.vals.ai/benchmarks/vals_index
  - Web Search Index (benchmark), 2026-10-02, No search: 2.9% legal / 7.4% finance; with search: roughly 30-50%, https://www.vals.ai/blogs/web-search-index
  - Finance Agent Benchmark (v1) / Finance Agent v2 (benchmark), 2026-10-01, Gemini 4 Argon 65.40% on v2 (as of 2026-10-01), https://www.vals.ai/benchmarks/fabv2
  - Harvey's Legal Agent Benchmark (industry partner benchmark, run by Vals) (leaderboard), 2026-10-01, Muse Spark 1.2 25.42% task pass rate (as of 2026-10-01), https://www.vals.ai/benchmarks/hlab
  - Legal Research Bench (benchmark), 2026-10-01, Three-way tie at 55.29% (Muse Spark 1.3 Max, Claude Opus 5, Claude Fable 5.1) as of 2026-10-01, https://www.vals.ai/benchmarks/legal_research
  - AI detection benchmark (benchmark), 2026-09-30, Claude Opus 5.5 98.38% balanced accuracy, https://www.vals.ai/blogs/ai-detection-benchmark
  - Teen conversation AI safety evaluation (paper), 2026-09-23, 27.5% of conversations had at least one serious safety failure, https://www.vals.ai/blogs/evaluating-ai-safety-in-teen-conversations
  - Vals RSI Index (benchmark), 2026-08-13, Claude Opus 5.5 leads at 37.31% (as of 2026-10-01), https://www.vals.ai/benchmarks/rsi_index
  - SRE Bench (benchmark), 2026-08-13, GPT-6 Astra 56.87% (as of 2026-09-29), https://www.vals.ai/benchmarks/srebench
  - Excel Modeling Benchmark (EMB) (benchmark), 2026-07-01, Claude Fable 5.1 76.67% (as of 2026-10-01); Claude Opus 4.8 69.4% at launch (reported), https://www.vals.ai/benchmarks/emb
  - Public Benefits Bench (v1.1) (benchmark), 2026-06-09, Claude Opus 5 76.93%; Vals concludes no general-purpose model is reliable enough for SNAP guidance, https://www.vals.ai/benchmarks/public-benefits-bench
  - Vals Legal AI Report (VLAIR) (benchmark), 2025-02, https://www.lawnext.com/2025/05/vals-ai-issues-open-call-for-vendors-to-participate-in-its-legal-research-and-other-legal-ai-benchmarking-studies.html
  - Other Vals proprietary benchmarks: Vals Multimodal Index, Tax Agent Bench, MedCode, MedScribe, ProofBench v1.1, MysteryMechanism, SAGE, Vibe Code Bench (v1.1 and 1-100), Code Migration, CyberBench v1.1, CUA-bench, Time Horizon Index: KSP, Poker Agent (leaderboard), https://www.vals.ai/benchmarks
- products:
  - Vals Index and public benchmark leaderboards: Independent public evaluations of frontier models across finance, legal, coding, tax, healthcare, math, science and frontier risk, including industry-partner benchmarks., https://www.vals.ai/benchmarks
  - Vals Smith: Self-serve tool that turns a GitHub repo's merged PRs into a custom coding benchmark with hidden tests. Users sign in with GitHub and can pick public or private repos. Launched with 120 free credits., https://www.vals.ai/vals-smith
  - Valkyrie (open source): Cloud-native infrastructure for running AI agent evaluations on any benchmark (AGPL-3.0). Runs Vals-hosted or self-hosted on AWS, with a CLI and Python SDK; agents are defined by contract.yaml. Docs at docs.valkyrie.vals.ai., https://github.com/vals-ai/Valkyrie
  - model-library (open source): Simple, provider-agnostic LLM gateway (MIT)., https://github.com/vals-ai/model-library
  - Vals Public Sector: Independent AI evaluation for government agencies; includes Public Benefits Bench., https://www.vals.ai/gov
  - Application Reports: Evaluations of specific AI applications (e.g., Alexi)., https://www.vals.ai/application-reports/alexi
- notes_for_researchers:
  - Publishes many open methodology posts. Recent ones cover benchmark cheating and shortcut-seeking (SWE-bench Verified, Terminal-Bench 2.1), AI-detector robustness, multi-turn teen-safety failures, and agent-produced Lean proofs (Thomson N=7, C-HD shortest path).
  - Open source: Valkyrie eval infrastructure (AGPL-3.0), plus MIT-licensed model-library, finance-agent, finance-agent-v2, legal-research-bench and web-search-agent-public. Also proof-bench, ioi-agent and benchmark-service skeletons for SWE-bench and Terminal-Bench.
  - Finance Agent Benchmark paper: arXiv:2508.00828. The dataset is on Hugging Face (vals-ai/finance_agent_benchmark).
  - Collaborators named on official pages: Stanford SALT Lab (Diyi Yang, Valerie Chen) and Stanford Psychiatry, CoreWeave and Marimo (RSI Index), Code for America and Center for Civic Futures, Harvey (partner benchmark), and Exa, Keenable, Parallel and Tavily (Web Search Index).
- notes_for_program_managers:
  - Raised a $40M Series A at a $400M valuation led by a16z (2026-08-13), confirmed by both the company and a16z. Self-reported: 2026 revenue is 8x all of 2025, the customer base doubled, and the team tripled in six months.
  - Self-claimed frontier-lab ties: Vals says its results have been cited in model cards from OpenAI, Anthropic, Google, Meta and xAI. These are citations, not stated paying contracts.
  - Positioned as an independent third-party evaluator, now including frontier risk (RSI Index, SRE Bench and CyberBench, child and teen safety). Has a public-sector arm led by Glenn Parham out of a Washington, DC office. No agency customers are named.
  - Says it has supported the Department of Commerce and members of Congress on AI policy (self-claimed).
- notes_for_data_ops:
  - Valkyrie can be Vals-hosted or self-hosted on AWS. It is driven by a CLI and Python SDK, with agents defined in contract.yaml.
  - Vals Smith connects to public or private repos through GitHub sign-in. Vals says code 'never leaves your control' and other services cannot access your data (vendor claim).
  - Security: a Vulnerability Disclosure Program has been in effect since 2026-08-31 (security@vals.ai). The redesigned site shows no SOC 2 or GDPR claims, and no trust center was found.
- sources:
  - https://www.vals.ai/blogs/series-a (accessed 2026-10-05)
  - https://a16z.com/announcement/investing-in-vals/ (accessed 2026-10-05)
  - https://pear.vc/companies/ (accessed 2026-10-05)
  - https://www.finsmes.com/2026/08/vals-ai-raises-40m-in-series-a-funding-at-400m-valuation.html (accessed 2026-10-05)
  - https://tracxn.com/d/companies/vals-ai/__Aeq7C2n56rLfgsWjTrbrPIwiOteKWlKhIWYPbapLuow (accessed 2026-10-05)
  - https://www.vals.ai/blogs/rss.xml (accessed 2026-10-05)
  - https://www.vals.ai/blogs (accessed 2026-10-05)
  - https://www.vals.ai/news (accessed 2026-10-05)
  - https://www.youtube.com/watch?v=MpYAoufS588 (accessed 2026-10-05)
  - https://www.vals.ai/home (accessed 2026-10-05)
  - https://www.vals.ai/about (accessed 2026-10-05)
  - https://www.vals.ai/benchmarks (accessed 2026-10-05)
  - https://www.vals.ai/benchmarks/vals_index (accessed 2026-10-05)
  - https://www.vals.ai/benchmarks/rsi_index (accessed 2026-10-05)
  - https://www.vals.ai/benchmarks/srebench (accessed 2026-10-05)
  - https://www.vals.ai/benchmarks/fabv2 (accessed 2026-10-05)
  - https://www.vals.ai/benchmarks/hlab (accessed 2026-10-05)
  - https://www.vals.ai/benchmarks/emb (accessed 2026-10-05)
  - https://www.vals.ai/benchmarks/public-benefits-bench (accessed 2026-10-05)
  - https://www.vals.ai/benchmarks/legal_research (accessed 2026-10-05)
  - https://www.vals.ai/benchmarks/web_search (accessed 2026-10-05)
  - https://www.vals.ai/benchmarks/cyber (accessed 2026-10-05)
  - https://www.vals.ai/blogs/web-search-index (accessed 2026-10-05)
  - https://www.vals.ai/blogs/vals-public-sector-memo (accessed 2026-10-05)
  - https://www.vals.ai/blogs/child-ai-safety (accessed 2026-10-05)
  - https://www.vals.ai/blogs/evaluating-ai-safety-in-teen-conversations (accessed 2026-10-05)
  - https://www.vals.ai/blogs/cheating-on-the-rise (accessed 2026-10-05)
  - https://www.vals.ai/blogs/ai-detection-benchmark (accessed 2026-10-05)
  - https://www.vals.ai/blogs/vals-environmental-impacts-report (accessed 2026-10-05)
  - https://www.vals.ai/blogs/thomson-n7-lean-proof (accessed 2026-10-05)
  - https://www.vals.ai/blogs/faster-shortest-path-algorithm (accessed 2026-10-05)
  - https://www.vals.ai/blogs/fable-solves-cyphral-distich (accessed 2026-10-05)
  - https://www.vals.ai/vals-smith (accessed 2026-10-05)
  - https://www.vals.ai/gov (accessed 2026-10-05)
  - https://www.vals.ai/vulnerability-disclosure-program (accessed 2026-10-05)
  - https://www.vals.ai/product (accessed 2026-10-05)
  - https://github.com/vals-ai (accessed 2026-10-05)
  - https://github.com/vals-ai/Valkyrie (accessed 2026-10-05)
  - https://letsdatascience.com/news/vals-ai-launches-excel-modeling-benchmark-for-finance-agents-6a2d5a85 (accessed 2026-10-05)
  - https://qa-financial.com/industry-on-a-mission-the-long-road-to-uniform-ai-testing/ (accessed 2026-10-05)
  - https://www.techtimes.com/articles/303524/20240412/standardized-ai-performance-test-tested-out-new-startup.htm (accessed 2026-10-05)
  - https://arxiv.org/abs/2508.00828 (accessed 2026-10-05)
  - https://www.vals.ai/authors/rayan.png (accessed 2026-10-05)
  - https://www.vals.ai/authors/glenn.png (accessed 2026-10-05)
  - https://www.linkedin.com/company/vals-ai (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/vals-ai (accessed 2026-06-07)
  - https://getlatka.com/companies/vals.ai/funding (accessed 2026-06-07)
  - https://www.bloomberg.com/news/newsletters/2024-04-11/this-startup-is-trying-to-test-how-well-ai-models-actually-work (accessed 2026-06-07)
  - https://www.deeplearning.ai/the-batch/vals-ai-evaluates-large-language-models-on-industry-specific-tasks/ (accessed 2026-06-07)
  - https://www.lawnext.com/2025/05/vals-ai-issues-open-call-for-vendors-to-participate-in-its-legal-research-and-other-legal-ai-benchmarking-studies.html (accessed 2026-06-07)
  - https://finance.yahoo.com/news/vals-legal-ai-report-establishes-160000601.html (accessed 2026-06-07)
  - https://huggingface.co/datasets/vals-ai/finance_agent_benchmark (accessed 2026-06-07)
  - https://github.com/vals-ai/finance-agent (accessed 2026-06-07)
  - https://www.legaltechnologyhub.com/vendors/vals-ai/ (accessed 2026-06-07)

## HUD, rank #9
- slug: hud
- segment: Commercial vendors
- website: https://www.hud.ai
- focus_areas: Computer Use, Enterprise Workflows
- positioning: HUD (Human Union Data, YC W25, formerly hud.so) calls itself the platform for building high-quality post-training datasets. It offers an MIT-licensed SDK for defining RL environments, tasks and graders across coding, browser, computer-use and robotics; a hosted training and eval platform; and DataVendor, a marketplace where independent builders sell RL environments and data to AI labs. It announced a $16M Series A led by Standard Capital in June 2026.
- best_fit: Teams that want to build, evaluate and RL-train agents on reproducible environments (computer-use, browser, coding, robotics), or that want to buy or sell RL environments through a marketplace.
- overall_confidence: high
- founded_year: 2025 [confirmed], source: https://www.ycombinator.com/companies/hud (accessed 2026-06-07)
- status: active [confirmed], source: https://www.ycombinator.com/companies/hud (accessed 2026-06-07)
- hq_location: San Francisco, USA [confirmed], source: https://www.ycombinator.com/companies/hud (accessed 2026-06-07)
- other_locations: Singapore [confirmed], source: https://www.hud.ai/careers (accessed 2026-10-05)
- distributed_remote: partial (SF and Singapore offices; most roles open to remote in Asia or North America) [confirmed], source: https://www.ycombinator.com/companies/hud/jobs (accessed 2026-10-05)
- current_headcount: ~24 (per YC profile, accessed 2026-10-05) [reported], source: https://www.ycombinator.com/companies/hud (accessed 2026-10-05)
- headcount_band: 11-50 [reported], source: https://www.ycombinator.com/companies/hud (accessed 2026-10-05)
- headcount_growth: growing (YC team size 15 to 24 between 2026-06-07 and 2026-10-05; ~30 open roles) [reported], source: https://www.ycombinator.com/companies/hud (accessed 2026-10-05)
- open_roles_count: 31 [reported], source: https://www.ycombinator.com/companies/hud/jobs (accessed 2026-10-05)
- has_researchers: yes [reported], source: https://www.ycombinator.com/companies/hud ; https://www.workatastartup.com/companies/hud (accessed 2026-06-07) - team described as including IOI/IPhO Olympiad medalists and researchers with ICLR/NeurIPS publications; 'Research Engineer' roles open
- researcher_count: unknown [unknown]
- researcher_backgrounds: Jay Ram (CEO) - consumer apps, ML/quant research (YC bio); Lorenss Martinsons (Co-CEO) - Cognitive Science, Yale (FounderTrace); Parth Patel (CTO) - evals and RL environments; Named research authors: Lukass Kellijs (robotics), Ilya Kuhta and Reinis Salaks (evals/task design), Shriya Lolabattu (browser evals); Team described as including Olympiad medalists and ICLR/NeurIPS-published researchers (carried over from the 2026-06 baseline; not re-verified) [reported], source: https://www.hud.ai/research (accessed 2026-10-05)
- published_papers_or_benchmarks: Assemble Bench (robotics, NIST assembly board; CG-DAgger) - 2026-07-31; LiveWeb samples (with Browserbase) - 2026-09-22/28; Production-debugging RL environment (hud-ops-diagnostics) - 2026-01-20; SheetBench-50 (with Sepal AI) - 2025-10-01; OSWorld-Verified (XLANG, 2025-07-28; HUD credited as 'Human Data' for feedback/fixes, hosts taskset) [confirmed], source: https://www.hud.ai/research (accessed 2026-10-05)
- total_raised: $16M+ (Series A $16M plus an undisclosed seed) [confirmed], source: https://www.hud.ai/blog/announcing-16m-series-a (accessed 2026-10-05)
- last_round: Series A, $16M (announced 2026-06-18), led by Standard Capital [confirmed], source: https://www.hud.ai/blog/announcing-16m-series-a (accessed 2026-10-05)
- notable_investors: Standard Capital (Series A lead); Y Combinator (W25); Exceptional Capital; Liquid 2 Ventures; Twenty Two Ventures; Dylan Patel (SemiAnalysis, angel); Roon (OpenAI, angel); Ivan Burazin (Daytona, angel); Theo Browne (angel) [confirmed], source: https://www.hud.ai/blog/announcing-16m-series-a (accessed 2026-10-05)
- valuation: unknown [unknown]
- revenue_signals: Self-reported: 50+ businesses on platform; vendor marketplace moves 'millions' monthly; 1.3M+ task runs, 15M+ inference calls [reported], source: https://www.hud.ai/blog/announcing-16m-series-a (accessed 2026-10-05)
- soc2: claimed (homepage: 'SOC 2 compliant infrastructure' on Enterprise plan); no public report or trust center found [reported], source: https://www.hud.ai/ (accessed 2026-10-05)
- other_certifications: unknown [unknown]
- security_page: https://www.hud.ai/dpa (Data Processing Addendum) [reported], source: https://www.hud.ai/dpa surfaced via search (accessed 2026-06-07); no dedicated trust/security page or SOC 2/ISO certification found
- what_they_sell: environments + platform (SDK, training and eval platform, DataVendor environment and data marketplace) [confirmed], source: https://www.hud.ai/ (accessed 2026-10-05)
- open_source: yes [confirmed], source: https://github.com/hud-evals/hud-python (accessed 2026-06-07)
- license: MIT [confirmed], source: https://github.com/hud-evals/hud-python (accessed 2026-06-07) - MIT confirmed via repo license field on re-fetch
- deployment_model: managed-hosted + self-hosted (open-source SDK run locally or on owned infrastructure; hosted platform with REST API, TrainingClient and model gateway; Docker, Modal and Daytona runtimes; Fireworks RL integration documented) [confirmed], source: https://docs.hud.ai/llms.txt (accessed 2026-10-05)
- maturity: GA [estimated], source: https://pypi.org/project/hud-python/ ; https://docs.hud.ai/ (accessed 2026-06-07) - public OSS SDK with regular releases and self-serve docs/login; no formal GA label found
- notable_customers: UiPath (self-claimed); Sharpe (self-claimed); OpenAI (self-claimed, frontier-lab tie); Anthropic (self-claimed, frontier-lab tie) [reported], source: https://www.hud.ai/ (accessed 2026-10-05)
- founding_team:
  - Jay Ram, Co-founder & CEO, YC bio: 'prev: consumer apps, ml research, quant research'.
  - Lorenss Martinsons, Co-founder & Co-CEO (previously listed as CPO), Cognitive Science background at Yale (per FounderTrace). Wrote HUD's 2025 'HUD Autonomy' post on agent evaluation.
  - Parth Patel, Co-founder & CTO, Focuses on evals and RL environments (per FounderTrace). Co-authored the 2026-09 Browserbase x HUD browser-task post.
- funding_history:
  - 2026-06-18 Series A $16M led by Standard Capital, source: https://www.hud.ai/blog/announcing-16m-series-a
- recent_developments (last 6 months):
  - 2026-10-05 [hire] Hiring surge: about 30 open roles across SF, Singapore and remote (Asia and North America), source: https://www.ycombinator.com/companies/hud/jobs
  - 2026-09-25 [product] SDK pre-release adds managed Claude Code and Codex CLI agents, source: https://github.com/hud-evals/hud-python/releases
  - 2026-09-22 [partnership] HUD and Browserbase publish joint research on browser-eval reliability and open-source four LiveWeb sample tasks, source: https://www.browserbase.com/blog/browserbase-hud-frontier-evals
  - 2026-09-22 [research] HUD publishes 'What is a good task?' guidance on task realism, fairness and difficulty, source: https://www.hud.ai/research/what-is-a-good-task
  - 2026-08 [product] SDK adds Docker Compose environments, Harbor interoperability, a Fireworks RL cookbook and a unified robot stack; Daytona RL cookbook follows in September, source: https://github.com/hud-evals/hud-python/releases?page=2
  - 2026-07-31 [benchmark] HUD releases Assemble Bench, an open robotics benchmark for contact-rich assembly, plus the CG-DAgger method, source: https://www.hud.ai/research/assemble-benchmark
  - 2026-06-18 [funding] HUD announces a $16M Series A led by Standard Capital, source: https://www.hud.ai/blog/announcing-16m-series-a
  - 2026-06 [product] HUD repositions as a post-training data platform: SDK, Training + Eval Platform and the DataVendor marketplace, source: https://www.hud.ai/
- benchmarks_and_research:
  - LiveWeb samples (with Browserbase) (dataset), 2026-09-22, Self-reported: on Ladybird, GPT-6 Astra averaged 93.3%, Claude Fable 56%, and Claude Sonnet 5 and Qwen 3.8 Max 0%. On AirNow, 20 of 21 runs got full marks. On Costco login, all four models scored 75%., https://github.com/hud-evals/liveweb-samples
  - Assemble Bench (benchmark), 2026-07-31, Self-reported: after CG-DAgger rounds trained only on the 8mm round-peg task, pi0.5 improved dense reward on the 12mm and 16mm round pegs, with little transfer to 4mm. Sparse task success stayed limited at all sizes., https://github.com/hud-evals/assemble-bench
  - Production debugging RL environment (hud-ops-diagnostics) (environment), 2026-01-20, Self-reported: a Sentry subagent fine-tuned from o4-mini with OpenAI RFT improved from 6.3% to 13% success, about 2x., https://github.com/hud-evals/hud-ops-diagnostics
  - SheetBench-50 (benchmark), 2025-10-01, https://huggingface.co/datasets/hud-evals/SheetBench-50
  - OSWorld-Verified (HUD-hosted taskset; HUD credited as a feedback contributor) (benchmark), 2025-07-28, https://huggingface.co/datasets/hud-evals/OSWorld-Verified
- products:
  - HUD SDK (`hud` package, formerly hud-python): MIT-licensed Python framework for defining environments, tasks and graders. It connects over SSH, MCP, CDP (browser), RFB/VNC (computer use) and a beta robot WebSocket, with native Claude, OpenAI and Gemini harnesses. TrainingClient turns graded rollouts into RL training data. Install with pip install hud (renamed on PyPI 2026-07-15)., https://github.com/hud-evals/hud-python
  - Training + Eval Platform: Hosted platform for running evals at scale, viewing traces, managing tasksets and environments, and RL training from graded rollouts. It has a REST API and an OpenAI-compatible model gateway., https://www.hud.ai/
  - DataVendor: Marketplace connecting independent RL-environment and data builders with model labs. HUD says it moves 'millions' a month (self-reported)., https://www.hud.ai/
  - HUD docs and cookbooks: Documentation including Harbor interoperability, Compose environments, and Daytona RL, Fireworks Serverless RL, robot-benchmark and ops-diagnostics cookbooks. An LLM-readable index is at docs.hud.ai/llms.txt., https://docs.hud.ai/
- notes_for_researchers:
  - Open research posts: Assemble Bench and CG-DAgger for VLA robotics (2026-07-31), LiveWeb browser-eval reliability with Browserbase (Browserbase 2026-09-22, HUD 2026-09-28), and 'What is a good task?' on task and grader design (2026-09-22).
  - Open releases under hud-evals on Hugging Face: AssembleBench dataset, two pi0.5 AssembleBench checkpoints, SheetBench-50, OSWorld-Verified, SpreadSheetBench, Online-Mind2Web-Tiny. On GitHub: assemble-bench, liveweb-samples, hud-ops-diagnostics (all MIT where checked).
  - The SDK's TrainingClient turns graded eval runs straight into training data (every rollout returns a trace_id and a reward), so you can bring your own GRPO or PPO. Documented RL paths exist for Daytona sandboxes and Fireworks Serverless RL.
- notes_for_program_managers:
  - $16M Series A announced 2026-06-18, led by Standard Capital. Angels include Roon (OpenAI), Dylan Patel (SemiAnalysis) and Ivan Burazin (Daytona). Valuation is undisclosed.
  - Self-reported scale: 50+ businesses on the platform, 1.3M+ task runs, 2,500+ environments created, 15M+ inference calls, and a marketplace moving 'millions' a month. These are unaudited company claims.
  - Engagement is now a platform plus a marketplace (DataVendor), where third-party builders sell environments to labs. Check the provenance and QA of any environment sourced through the marketplace.
  - Published plan tiers on the homepage: SDK + Platform access free, Cloud at $0.10 per environment-hour, Enterprise custom, Academic $100 in free credits for .edu emails.
  - Team of 24 on the YC profile with about 30 open roles. Offices are in SF and Singapore. Lorenss Martinsons is now listed as Co-CEO next to CEO Jay Ram.
  - No frontier-lab customer is publicly confirmed by name. Public lab ties are the OSWorld-Verified feedback co-acknowledgement and an OpenAI employee (Roon) investing as an angel. Current homepage logos show only Sharpe and UiPath (UI-CUBE), both self-claimed.
- notes_for_data_ops:
  - The SDK connects over MCP, CDP browser control, RFB/VNC computer use, SSH shell and files, and robot control over WebSocket (beta). Environments can be declared as Docker Compose projects (since v0.6.13, 2026-08-07).
  - Harbor interoperability is documented: Harbor tasks can be imported as HUD tasksets and Compose artifacts, and HUD tasks exported to Harbor folders.
  - Robotics data is published in LeRobot format. The HF org also shows a 'datavendor-test-lerobot' dataset.
  - PyPI package renamed from hud-python to `hud` in v0.6.10 (2026-07-15). Update pinned dependencies.
  - Security: the homepage claims 'SOC 2 compliant infrastructure' for the Enterprise plan, but there is no public SOC 2 report or trust center. A current YC posting for a Senior Security Engineer says the hire will 'Own SOC 2 and customer trust, including ... audits'. Ask for the attestation letter or report. The only legal document found is the DPA at hud.ai/dpa.
  - REST API reference and an LLM-readable docs index at docs.hud.ai/llms.txt.
- sources:
  - https://www.hud.ai/blog/announcing-16m-series-a (accessed 2026-10-05)
  - https://api.fxtwitter.com/hud_evals/status/2067646134332567770 (accessed 2026-10-05)
  - https://www.hud.ai/blog (accessed 2026-10-05)
  - https://www.hud.ai/ (accessed 2026-10-05)
  - https://www.ycombinator.com/companies/hud (accessed 2026-10-05)
  - https://www.ycombinator.com/companies/hud/jobs (accessed 2026-10-05)
  - https://www.ycombinator.com/companies/hud/jobs/fi9ZRlo-senior-security-engineer (accessed 2026-10-05)
  - https://www.hud.ai/careers (accessed 2026-10-05)
  - https://foundertrace.com/companies/hud_yc_w25/ (accessed 2026-10-05)
  - https://github.com/hud-evals/hud-python (accessed 2026-10-05)
  - https://github.com/hud-evals/hud-python/releases (accessed 2026-10-05)
  - https://github.com/hud-evals/hud-python/releases?page=2 (accessed 2026-10-05)
  - https://docs.hud.ai/llms.txt (accessed 2026-10-05)
  - https://docs.hud.ai/v6/cookbooks/daytona-rl.md (accessed 2026-10-05)
  - https://www.hud.ai/research (accessed 2026-10-05)
  - https://www.hud.ai/research/assemble-benchmark (accessed 2026-10-05)
  - https://github.com/hud-evals/assemble-bench (accessed 2026-10-05)
  - https://www.hud.ai/research/can-you-trust-your-browser-eval (accessed 2026-10-05)
  - https://www.browserbase.com/blog/browserbase-hud-frontier-evals (accessed 2026-10-05)
  - https://github.com/hud-evals/liveweb-samples (accessed 2026-10-05)
  - https://www.hud.ai/research/what-is-a-good-task (accessed 2026-10-05)
  - https://www.hud.ai/research/debugging-rl-environment (accessed 2026-10-05)
  - https://www.hud.ai/research/sheetbench-50 (accessed 2026-10-05)
  - https://huggingface.co/hud-evals (accessed 2026-10-05)
  - https://xlang.ai/blog/osworld-verified (accessed 2026-10-05)
  - https://www.cua.ai/blog/hud-agent-evals (accessed 2026-10-05)
  - https://arxiv.org/abs/2607.28367 (accessed 2026-10-05)
  - https://pypi.org/project/hud-python/ (accessed 2026-06-07)
  - https://docs.hud.ai/ (accessed 2026-06-07)
  - https://www.hud.ai/case-studies/sheetbench-50 (accessed 2026-06-07)
  - https://www.hud.ai/dpa (accessed 2026-06-07)
  - https://www.workatastartup.com/companies/hud (accessed 2026-06-07)
  - https://newsletter.semianalysis.com/p/rl-environments-and-rl-for-science (accessed 2026-06-07)
  - https://www.linkedin.com/company/hud-evals (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/hud (accessed 2026-06-07)
  - https://x.com/hud_evals/status/1919262852088570225 (accessed 2026-06-07)

## Halluminate, rank #10
- slug: halluminate
- segment: Commercial vendors
- website: https://www.halluminate.ai/
- focus_areas: Computer Use, Enterprise Workflows
- positioning: Halluminate (YC S25, founded 2024, San Francisco) calls itself a data research lab building benchmarks and RL environments for knowledge work, starting with financial services (PE, IB, due diligence) and consulting. It raised a $30M Series A led by Oak HC/FT in October 2026 ($38.5M total). With about nine people, it says it works with four of the five leading closed-source US AI labs and has a mid-eight-figure revenue run rate (self-reported). Its flagship benchmark is Westworld Finance Diligence Bench.
- best_fit: Frontier labs that need long-horizon, expert-authored finance and knowledge-work RL environments and benchmarks (due diligence, PE deal analysis, modeling) in realistic desktop sandboxes.
- overall_confidence: high
- founded_year: 2024 [reported], source: https://www.ycombinator.com/companies/halluminate (accessed 2026-06-07)
- status: active [confirmed], source: https://www.halluminate.ai/ (accessed 2026-06-07)
- hq_location: San Francisco, CA, USA [confirmed], source: https://www.ycombinator.com/companies/halluminate ; https://www.linkedin.com/company/halluminate (accessed 2026-06-07)
- other_locations: unknown [unknown]
- distributed_remote: unknown [unknown]
- current_headcount: 9 (as of 2026-10-01) [reported], source: https://fortune.com/2026/10/01/halluminate-raises-30-million-series-a-oakhc-ft/ (accessed 2026-10-05)
- headcount_band: 1-10 [reported], source: https://fortune.com/2026/10/01/halluminate-raises-30-million-series-a-oakhc-ft/ (accessed 2026-10-05)
- headcount_growth: unknown [unknown]
- open_roles_count: 9 listed on the YC jobs page as of 2026-10-05 (incl. MTS Research/Post-Training, MTS Platform Engineering, MTS Finance Researcher, Strategic Project Lead, Chief of Staff) [confirmed], source: https://www.ycombinator.com/companies/halluminate (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: https://www.halluminate.ai/blog/due-diligence-bench (accessed 2026-10-05)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Jerry Wu (co-founder/CEO): ex-Capital One Labs (led product and research; 3 patents); Cornell CS & Economics; Wyatt Marshall (co-founder/CTO): Cornell Milstein Scholar; data engineering at two early-stage NYC startups; Team includes former founders, researchers, particle physicists, experts and engineers from Meta, Scale AI, Capital One Labs, McKinsey and Goldman Sachs (company statement) [reported], source: https://www.halluminate.ai/blog/series-a (accessed 2026-10-05)
- published_papers_or_benchmarks: Westworld Finance Diligence Bench (2026-08-11): https://www.halluminate.ai/blog/due-diligence-bench; DealTrace Bench (2026-05/06): https://www.halluminate.ai/blog/dealtrace; Simulated-to-Real Transfer for Web Agents (2026-07-27): https://www.halluminate.ai/blog/sim-to-real-webagents; Westworld v1 simulated-web environments (2025-11-23): https://github.com/Halluminate/westworld; BrowserBench (2025-11-22): https://github.com/Halluminate/browserbench; Web Bench (2025-05-30; ~5,750 tasks across 500+ sites in full, ~2,454 tasks across 452 sites open-sourced): https://github.com/Halluminate/WebBench [confirmed], source: https://www.halluminate.ai/research (accessed 2026-10-05)
- total_raised: $38.5M (including $30M Series A, Oct 2026) [confirmed], source: https://www.halluminate.ai/blog/series-a (accessed 2026-10-05)
- last_round: Series A, $30M, 2026-10-01, led by Oak HC/FT [confirmed], source: https://www.halluminate.ai/blog/series-a (accessed 2026-10-05)
- notable_investors: Oak HC/FT (Series A lead); Y Combinator (S25); Orange Collective; FT Partners; Heavybit; Unnamed angel ML researchers from Anthropic, OpenAI and Meta (press-reported); Antigravity Capital (baseline, portfolio-listed); Batch Ventures (baseline, aggregator-reported); Team Ignite Ventures (baseline, aggregator-reported); Transpose Platform Management (baseline, aggregator-reported) [reported], source: https://www.halluminate.ai/blog/series-a (accessed 2026-10-05)
- valuation: not disclosed [unknown], source: https://fortune.com/2026/10/01/halluminate-raises-30-million-series-a-oakhc-ft/ (accessed 2026-10-05)
- revenue_signals: Self-reported: grew from $0 to a mid-eight-figure annualized revenue run rate in the 10 months before Oct 2026, 'strongly profitable' [reported], source: https://www.halluminate.ai/blog/series-a (accessed 2026-10-05)
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: unknown [unknown]
- what_they_sell: mixed [confirmed], source: https://www.halluminate.ai/ ; https://www.ycombinator.com/companies/halluminate ; https://news.ycombinator.com/item?id=44865290 (accessed 2026-06-07), RL/sandbox environments for computer-use agents plus human/annotation data and evaluation benchmarks
- open_source: yes [confirmed], source: https://github.com/Halluminate (accessed 2026-06-07)
- license: Mixed: WebBench (MIT), westworld (Apache-2.0), noodle-flights (MIT), browserbench (no license shown); the 2026 finance benchmarks are not open-sourced (DealTrace repo empty) [confirmed], source: https://github.com/Halluminate (accessed 2026-10-05)
- deployment_model: managed-hosted (fully managed, parallelizable sandbox environments); also bespoke/custom environments for enterprise clients [confirmed], source: https://www.ycombinator.com/companies/halluminate ; https://news.ycombinator.com/item?id=44865290 (accessed 2026-06-07)
- maturity: GA [estimated], source: https://www.ycombinator.com/companies/halluminate (accessed 2026-06-07), states paying customers, indicating commercial availability
- notable_customers: Four of the five leading closed-source US AI labs (unnamed) (self-claimed, frontier-lab tie) [reported], source: https://www.halluminate.ai/blog/series-a (accessed 2026-10-05)
- founding_team:
  - Jerry Wu, Co-founder & CEO, Previously led product and research at Capital One Labs, where he launched one of the first AI agents in banking and co-authored three patents. Studied Computer Science and Economics at Cornell.
  - Wyatt Marshall, Co-founder & CTO, Cornell Milstein Scholar. Twice an early data/software engineer, doing large-scale data engineering at two early-stage NYC startups. Co-author of Westworld Finance Diligence Bench.
- funding_history:
  - 2026-10-01 Series A $30M led by Oak HC/FT, source: https://www.halluminate.ai/blog/series-a
- recent_developments (last 6 months):
  - 2026-10-01 [funding] Halluminate raises a $30M Series A led by Oak HC/FT; total funding now $38.5M, source: https://fortune.com/2026/10/01/halluminate-raises-30-million-series-a-oakhc-ft/
  - 2026-10-01 [customer] Company says four of the five leading closed-source US AI labs are customers (self-claimed, none named), source: https://www.halluminate.ai/blog/series-a
  - 2026-10-01 [other] CEO describes a 'Moore's law of environments', source: https://fortune.com/2026/10/01/halluminate-raises-30-million-series-a-oakhc-ft/
  - 2026-08-11 [benchmark] Westworld Finance Diligence Bench released, source: https://www.halluminate.ai/blog/due-diligence-bench
  - 2026-07-27 [research] Sim-to-real transfer study for web agents (Goodbuy clone vs Best Buy), source: https://www.halluminate.ai/blog/sim-to-real-webagents
  - 2026-06-17 [benchmark] DealTrace Bench: staged evaluation of a real private-equity deal, source: https://www.halluminate.ai/blog/dealtrace
- benchmarks_and_research:
  - Westworld Finance Diligence Bench (benchmark), 2026-08-11, Current blog table: top mean score 0.59 (Opus 5.5, native CLI and CU+TU). Fable 5.1 scored 0.56/0.58 and GPT-6.1 Sol 0.54/0.55. Series A press cited a 51% top score, likely an earlier model set., https://www.halluminate.ai/blog/due-diligence-bench
  - Simulated-to-Real Transfer for Web Agents (Goodbuy/Best Buy study) (paper), 2026-07-27, Mixed sim+real training beat either alone (39.1% sim / 23.4% real). Real-to-sim transfer was much stronger than sim-to-real., https://www.halluminate.ai/blog/sim-to-real-webagents
  - DealTrace Bench (benchmark), 2026-06-17, GPT-5.5 was strongest of Claude Opus 4.7, GPT-5.5 and Gemini 3.1 Pro. Forecast quality dominates recommendation quality (standardized coefficient 0.75)., https://www.halluminate.ai/blog/dealtrace
  - Westworld v1 (environment), 2025-11-23, https://www.halluminate.ai/blog/westworld
  - BrowserBench (benchmark), 2025-11-22, https://www.halluminate.ai/blog/browserbench
  - Web Bench (benchmark), 2025-05-30, Anthropic Computer Use was best fully automated at 66.0% of tasks; the best agent reached 46.6% on write-heavy tasks., https://www.halluminate.ai/blog/benchmark
- products:
  - RL environments and benchmarks for knowledge work / financial services (custom, for frontier labs): Custom simulated desktop and web environments with expert-written tasks and verifiers for training agents on finance and consulting workflows. Sold mainly to frontier labs (self-claimed: 4 of the 5 leading closed-source US labs)., https://www.halluminate.ai/
  - Westworld Finance Diligence Bench: Flagship long-horizon finance benchmark (88 due-diligence problems)., https://www.halluminate.ai/blog/due-diligence-bench
  - Westworld (simulated websites): Open-source simulated-web sandbox environments for computer-use agents (Apache-2.0)., https://github.com/Halluminate/westworld
  - Expert Network: Paid network of finance professionals (PE, IB, consulting, accounting, FP&A) who work as 1099 contractors. They write problems and rubrics at $100-$250/hour, with at least 20 hours a week for 6+ weeks, fully remote. Launch date not stated., https://www.halluminate.ai/experts
  - Data Partnerships: Program inviting companies to 'turn your company's data and workflows into a new revenue opportunity' by partnering with Halluminate. Launch date not stated., https://www.halluminate.ai/data-partnerships
  - noodle-flights: Open-source flight-search app for benchmarking and training agents (MIT). Last updated 2026-10-02., https://github.com/Halluminate/noodle-flights
- notes_for_researchers:
  - The finance benchmarks (Westworld Finance Diligence Bench, DealTrace) are published only as write-ups with per-model scores. No public task set or leaderboard is linked, and the DealTrace GitHub repo was empty on 2026-10-05.
  - Westworld Finance Diligence Bench scores differ between the current blog table (top 0.59, Opus 5.5) and the Series A press (top 51%). Cite the blog table with its date, because the model set appears to have been updated.
  - The sim-to-real web-agent study (2026-07-27) is reproducible: the HF dataset victoria-kp/goodbuy-bestbuy-mirrored-tasks, HF models and a W&B report are public. Method: SFT then GRPO on Qwen2.5-VL-7B-Instruct.
  - Older open releases: WebBench (MIT; ~2,454 of ~5,750 tasks open-sourced), westworld (Apache-2.0, plus an HF dataset), browserbench (no license shown), noodle-flights (MIT).
- notes_for_program_managers:
  - $30M Series A led by Oak HC/FT on 2026-10-01, bringing the total raised to $38.5M. Fortune reported nine employees at the announcement, while the YC profile still shows a team size of 5.
  - Self-claimed and unaudited: a mid-eight-figure annualized run rate reached in 10 months, 'strongly profitable', and work with 4 of the 5 leading closed-source US AI labs. No lab is named.
  - Stated near-term focus (TechTimes): go deeper with frontier-lab customers rather than sell to enterprises. The homepage says it is expanding from finance into consulting and adjacent verticals.
  - The YC jobs page lists 9 open roles, including MTS Research/Post-Training, MTS Platform Engineering, MTS Finance Researcher, Strategic Project Lead and Chief of Staff.
- notes_for_data_ops:
  - Experts are supplied as 1099 finance contractors (PE, IB, consulting, accounting, FP&A) paid $100-$250/hour with a minimum of 20 hours a week for 6 weeks, per the company's Expert Network page.
  - The finance environments run as desktop sandboxes graded by deterministic checks plus rubric-based verifiers, per the company's benchmark write-up.
  - The GitHub org keeps forks of Harbor and harbor-cookbook (updated Aug 2026), which suggests familiarity with the Harbor eval/RL format. Official support is not stated.
  - No SOC 2, security page or trust center was found on halluminate.ai as of 2026-10-05.
- sources:
  - https://www.halluminate.ai/ (accessed 2026-10-05)
  - https://www.halluminate.ai/blog/series-a (accessed 2026-10-05)
  - https://fortune.com/2026/10/01/halluminate-raises-30-million-series-a-oakhc-ft/ (accessed 2026-10-05)
  - https://www.techtimes.com/articles/328407/20261001/finance-ai-tops-out-51-due-diligence-startup-raises-30m-fix-it.htm (accessed 2026-10-05)
  - https://www.finsmes.com/2026/10/halluminate-raises-usd30m-in-series-a-funding.html (accessed 2026-10-05)
  - https://www.halluminate.ai/blog/due-diligence-bench (accessed 2026-10-05)
  - https://www.halluminate.ai/blog/dealtrace (accessed 2026-10-05)
  - https://www.halluminate.ai/blog/sim-to-real-webagents (accessed 2026-10-05)
  - https://www.halluminate.ai/research (accessed 2026-10-05)
  - https://www.halluminate.ai/blog/benchmark (accessed 2026-10-05)
  - https://www.halluminate.ai/blog/westworld (accessed 2026-10-05)
  - https://www.halluminate.ai/experts (accessed 2026-10-05)
  - https://www.halluminate.ai/data-partnerships (accessed 2026-10-05)
  - https://www.ycombinator.com/companies/halluminate (accessed 2026-10-05)
  - https://github.com/Halluminate (accessed 2026-10-05)
  - https://github.com/Halluminate/DealTrace (accessed 2026-10-05)
  - https://huggingface.co/datasets/victoria-kp/goodbuy-bestbuy-mirrored-tasks (accessed 2026-10-05)
  - https://news.ycombinator.com/item?id=44865290 (accessed 2026-06-07)
  - https://getlatka.com/companies/halluminate.ai (accessed 2026-06-07)
  - https://www.orangecollective.vc/portfolio/halluminate (accessed 2026-06-07)
  - https://antigravity.capital/portfolio/halluminate (accessed 2026-06-07)
  - https://www.halluminate.ai/blog/browserbench (accessed 2026-06-07)
  - https://www.paraform.com/company/halluminate/cmo0vioko000e0djsvyvu9n6m (accessed 2026-06-07)
  - https://www.linkedin.com/in/jerry-wu-7814b0100/ (accessed 2026-06-07)

## Veris AI, rank #11
- slug: veris-ai
- segment: Commercial vendors
- website: https://veris.ai
- focus_areas: Enterprise Workflows
- positioning: Veris AI sells simulation sandboxes that recreate an agent's production environment (mocked SaaS tools, seeded databases, simulated users) so enterprises can benchmark, regression-test, and RL/SFT-train agents before deployment. In 2026 it added a self-serve tier (Veris Plus), a coding-agent verification product (Agentic SDLC), and public voice-agent benchmarks (VAmoS, VAmoS Pro) backed by arXiv papers.
- best_fit: Enterprise teams shipping customer-facing chat, voice or tool-calling agents (especially in finance, telecom and utilities) who need policy-specific benchmarks, CI regression gating, and simulated environments for RFT before production.
- overall_confidence: high
- founded_year: 2025 [reported], source: https://www.linkedin.com/company/veris-ai public snippet (founded 2025); https://www.businesswire.com/news/home/20250603868539/en/ (emerged from stealth June 2025) (accessed 2026-06-07)
- status: active [confirmed], source: https://veris.ai/about; https://veris.ai/ (accessed 2026-06-07)
- hq_location: San Francisco, CA, USA [reported], source: https://www.linkedin.com/company/veris-ai public snippet (HQ San Francisco) (accessed 2026-06-07)
- other_locations: unknown [unknown]
- distributed_remote: unknown [unknown]
- current_headcount: 1-10 (approx 9) as of 2026-06-07 [reported], source: https://www.linkedin.com/company/veris-ai public snippet (9 employees / 1-10 band); https://veris.ai/about (named team members) (accessed 2026-06-07)
- headcount_band: 1-10 [reported], source: https://www.linkedin.com/company/veris-ai public snippet (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: unknown [unknown], source: Careers links to external jobs board; at least one role (Member of Technical Staff, SF) seen on LinkedIn Jobs but no reliable total count (accessed 2026-06-07)
- has_researchers: yes [confirmed], source: https://veris.ai/about (Applied Research Engineer / Senior Applied Research Engineer roles); founders hold PhDs (accessed 2026-06-07)
- researcher_count: ~2 applied research engineers (Sahar Shayegan, Ritiz Tambi) plus 2 PhD co-founders; 8 of the 9 people on the team page co-author the arXiv papers [estimated], source: https://www.veris.ai/company (accessed 2026-10-05)
- researcher_backgrounds: CEO Mehdi Jamei: PhD EECS UC Berkeley; previously led agentic AI at System and Workmate; CTO Andi Partovi: PhD (brain-computer interfaces) University of Melbourne; ex-Solutions Architect at Google; ex-founder/CTO KeyLead Health [reported], source: https://www.businesswire.com/news/home/20250603868539/en/; https://pulse2.com/veris-ai-8-5-million-raised-for-training-ai-agents/; https://www.linkedin.com/in/mehdijamei/; https://www.linkedin.com/in/andi-partovi/ (accessed 2026-06-07)
- published_papers_or_benchmarks: VAmoS Pro Bench / 'VAmoS Part Deux' (arXiv 2609.38512, Sep 2026) - https://arxiv.org/abs/2609.38512; VAmoS Bench: Voice Agent Simulation Bench (arXiv 2607.27453, Jul 2026) - https://arxiv.org/abs/2607.27453; Public leaderboards VAmoS and VAmoS Pro - https://www.veris.ai/leaderboard; User-simulation realism: Veris vs τ²-bench (Aug 2026, vendor blog) - https://veris.ai/blog/user-simulation-realism-veris-vs-tau2-bench; RFT case study with Savi Security, Qwen3 4B scam detection (Mar 2026, vendor blog) - https://veris.ai/blog/rft-can-make-agents-with-small-models-outperform-big-models-on-enterprise-tasks; BankingBench (Feb 2026, vendor blog) - https://veris.ai/blog/bankingbench-measuring-model-performance; SantaBench (Dec 2025, vendor blog) - https://veris.ai/blog/santabench; Technical Report: How Reinforcement Fine-tuning Trains Enterprise-Grade Domain-specific Agents (Dec 2025) - https://veris.ai/blog/technical-report-reinforcement-learning-fine-tuning-for-enterprise-ai-agents [confirmed], source: https://veris.ai/blog (accessed 2026-10-05)
- total_raised: $8.5M [confirmed], source: https://www.businesswire.com/news/home/20250603868539/en/; https://www.finsmes.com/2025/06/veris-ai-raises-8-5m-in-seed-funding.html; https://pulse2.com/veris-ai-8-5-million-raised-for-training-ai-agents/; https://www.citybiz.co/article/701730/ (accessed 2026-06-07)
- last_round: Seed, $8.5M, June 2025 [confirmed], source: https://www.businesswire.com/news/home/20250603868539/en/ (stealth emergence June 2025); https://pulse2.com/veris-ai-8-5-million-raised-for-training-ai-agents/ (round stage: Seed) (accessed 2026-06-07)
- notable_investors: Decibel Ventures (lead); Acrew Capital (lead); The House Fund; Ian Livingstone; Idris Mokhtarzada (Rocket Money); Dorothy Chang [confirmed], source: https://www.businesswire.com/news/home/20250603868539/en/; https://pulse2.com/veris-ai-8-5-million-raised-for-training-ai-agents/; https://www.citybiz.co/article/701762/; https://www.citybiz.co/article/701797/ (accessed 2026-06-07)
- valuation: unknown [unknown]
- revenue_signals: Usage-based pricing (by simulation volume and environment complexity); self-serve tier launched Apr 2026; Google Cloud Marketplace listing (Mar 2026). No revenue figures disclosed. [confirmed], source: https://veris.ai/faqs (accessed 2026-10-05)
- soc2: claimed-unverified [reported], source: https://veris.ai/faqs (accessed 2026-10-05)
- other_certifications: HIPAA (self-claimed); GDPR (self-claimed) [reported], source: https://veris.ai/faqs (accessed 2026-10-05)
- security_page: unknown [unknown], source: No dedicated /security or /trust page found; SOC 2 and security controls described on FAQ page only (accessed 2026-06-07)
- what_they_sell: environments [confirmed], source: https://veris.ai/ (accessed 2026-10-05)
- open_source: partial [confirmed], source: https://github.com/veris-ai (accessed 2026-10-05)
- license: Proprietary platform; open-source CLI, sandbox integrations and benchmark agents under MIT / Apache-2.0 [confirmed], source: https://github.com/veris-ai (accessed 2026-10-05)
- deployment_model: managed-hosted (Veris Cloud), self-hosted (customer VPC on AWS/GCP/Azure), and on-prem [confirmed], source: https://veris.ai/faqs ('supports deployment on your own cloud (AWS, GCP, Azure) or on-prem'; Veris Cloud managed option) (accessed 2026-06-07)
- maturity: GA [reported], source: https://veris.ai/faqs (no beta designation; standard onboarding/deployment described; waitlist opened at launch per BusinessWire) (accessed 2026-06-07)
- notable_customers: Savi Security (RFT research collaborator / co-author) (self-claimed); Telecom operator (unnamed) - production customer-service agent (self-claimed); Consumer fintech company (unnamed) - compliant chatbots (self-claimed); HR tech / executive-assistant agent company (unnamed) (self-claimed); Manufacturer - supply chain agent (unnamed) (self-claimed) [reported], source: https://veris.ai/blog/rft-can-make-agents-with-small-models-outperform-big-models-on-enterprise-tasks (accessed 2026-10-05)
- founding_team:
  - Mehdi Jamei, Co-founder & CEO, PhD EECS, UC Berkeley; previously led agentic AI at System and Workmate.
  - Andi Partovi, Co-founder & CTO, PhD (brain-computer interfaces), University of Melbourne; ex-Google Solutions Architect; ex-founder/CTO of KeyLead Health.
- recent_developments (last 6 months):
  - 2026-10-01 [benchmark] Veris releases VAmoS Pro Bench, a voice-agent benchmark on utility billing calls, with a public leaderboard of 14 stacks, source: https://veris.ai/blog/vamos-pro-bench-a-harder-exam-for-voice-agents
  - 2026-09-29 [research] arXiv paper 'VAmoS Part Deux: Harder, More Realistic Voice-Agent Simulation' introduces the VAmoS Energy benchmark, source: https://arxiv.org/abs/2609.38512
  - 2026-09-24 [other] Business Insider quotes Veris CEO Mehdi Jamei on a personal AI agent misreporting its own actions, source: https://www.businessinsider.com/ai-agents-gone-wrong-horror-stories-instinct-muse-privacy-security-2026-9
  - 2026-09-13 [research] 'Don't upgrade the model, simulate first': prompt scaffolding vs model upgrades on an NVIDIA NemoClaw sales-outreach agent, source: https://veris.ai/blog/dont-upgrade-the-model-simulate-first
  - 2026-08-31 [research] Veris publishes a user-simulation realism comparison against τ²-bench ('Real People Don't Talk Like That'), source: https://veris.ai/blog/user-simulation-realism-veris-vs-tau2-bench
  - 2026-08-11 [research] 'The Right Model for the Right Task': six models compared on a production telecom customer-service agent, source: https://veris.ai/blog/the-right-model-for-the-right-task
  - 2026-08-06 [benchmark] VAmoS Bench leaderboard (credit-card support voice agents, 17 stacks) updated, source: https://www.veris.ai/leaderboard/vamos
  - 2026-07-29 [open-source] arXiv paper 'VAmoS Bench: Voice Agent Simulation Bench' released with the open-source riley-agent repo (MIT), source: https://arxiv.org/abs/2607.27453
  - 2026-07-06 [product] Case study: continuous-improvement loop on a NemoClaw product-analytics agent, source: https://veris.ai/blog/the-loop-is-the-easy-part
  - 2026-06-08 [product] Veris demonstrates simulation testing of an OpenClaw agent through a config-only .veris/ folder, source: https://veris.ai/blog/let-openclaw-run-wild-in-simulation
  - 2026-05-18 [partnership] Veris publishes an integration walkthrough for AWS Bedrock AgentCore, source: https://veris.ai/blog/battle-testing-enterprise-ai-agents-veris-ai-aws-bedrock-agentcore
  - 2026-04-27 [product] Veris Plus launches as a self-serve simulation sandbox, source: https://veris.ai/blog/veris-plus-is-live
- benchmarks_and_research:
  - VAmoS Pro Bench (VAmoS Energy) (benchmark), 2026-10-01, Best of 14 stacks: Grok Voice at 44.7% task completion, ahead of Gemini 3.8 Live at 40.1% and GPT-Live 1 at 39.2%; the range is 17.3-44.7%., https://www.veris.ai/leaderboard/vamos-pro
  - User-simulation realism: Veris vs τ²-bench (paper), 2026-08-31, Veris 84.8% vs τ²-bench-retail 68.6% naturalness (vendor-run)., https://veris.ai/blog/user-simulation-realism-veris-vs-tau2-bench
  - VAmoS Bench (Voice Agent Simulation Bench) (benchmark), 2026-07-29, Best of 17 stacks: pipecat at 71.0% task completion (leaderboard updated 2026-08-06)., https://www.veris.ai/leaderboard/vamos
  - RFT case study with Savi Security: Qwen3 4B scam detection (paper), 2026-03-30, Qwen3 4B rose from about 80% to 94.4% accuracy after RFT and ran roughly 2-13x faster per sample than the comparison models (vendor-run)., https://veris.ai/blog/rft-can-make-agents-with-small-models-outperform-big-models-on-enterprise-tasks
  - BankingBench (benchmark), 2026-02, Overall pass rates were 88.2-93.5%, but category spreads reached 66 pp (error handling: DeepSeek 80.6% vs Grok-3-fast 14.7%)., https://veris.ai/blog/bankingbench-measuring-model-performance
  - SantaBench (benchmark), 2025-12-18, DeepSeek V3.2 had the highest tool reliability at 95%, ahead of Grok 4 at 75% and GPT-5.2 at 65%., https://veris.ai/blog/santabench
  - Technical Report: How Reinforcement Fine-tuning Trains Enterprise-Grade Domain-specific Agents (paper), 2025-12-01, Qwen 14B improved from 0.401 to 0.604, approaching GPT-5 at 0.6815 at roughly 200x lower inference cost (vendor-run)., https://veris.ai/blog/technical-report-reinforcement-learning-fine-tuning-for-enterprise-ai-agents
  - Dynamic Benchmarking: Evaluate AI Agents through Environments, not Datasets (paper), 2025-10-28, https://veris.ai/blog/dynamic-benchmarking
- products:
  - Veris for Benchmarking: Lets an organization build custom benchmarks around its own policies and systems to evaluate vendor and internal agents end-to-end against production twins., https://veris.ai/
  - Veris for Agentic SDLC: A stateful sandbox where coding agents verify their work before pull requests open. The homepage names Claude Code, Codex, Cursor and GitHub Copilot., https://veris.ai/
  - Veris Plus: Self-serve simulation sandbox with 30+ built-in SaaS simulations, simulated users, auto-generated success criteria, parallel rollouts and audit reports., https://veris.ai/blog/veris-plus-is-live
  - Veris CLI: Open-source (MIT) Go CLI for the Veris dependency sandbox. Commands are veris login, env create, up, run and down; configuration lives in .veris/twin.yaml, and CI uses VERIS_API_KEY. The repo was created in Aug 2026., https://github.com/veris-ai/veris-cli
  - Public leaderboards (VAmoS, VAmoS Pro): Public voice-agent benchmark leaderboards: VAmoS covers 17 stacks (updated 2026-08-06) and VAmoS Pro covers 14 stacks (2026-10-01)., https://www.veris.ai/leaderboard
  - Google Cloud Marketplace listing: The platform is available through Google Cloud Marketplace. The announcement includes a quote from Google Cloud's Dai Vu., https://veris.ai/blog/veris-ai-announces-availability-on-google-cloud-marketplace-to-deliver-reliable-production-ready-ai-agents
- notes_for_researchers:
  - Veris has two arXiv papers whose author lists are the Veris team: VAmoS Bench (2607.27453) and VAmoS Part Deux / VAmoS Energy (2609.38512). Both score voice agents on backend state changes, not just transcripts.
  - Benchmark agent code is open at github.com/veris-ai/riley-agent and github.com/veris-ai/rory-agent (MIT).
  - RFT results: Qwen3 4B reached 94.4% on scam detection (with Savi Security, Mar 2026), and Qwen 14B rose from 0.401 to 0.604 on Sigma-rule generation (Dec 2025). Both are vendor-run and not peer-reviewed.
  - A vendor study claims Veris's simulated users are more realistic than τ²-bench's (84.8% vs 68.6% naturalness, with the same LLM judge for both). It has not been independently replicated.
- notes_for_program_managers:
  - Funding is unchanged: an $8.5M seed in June 2025, led by Decibel and Acrew. No new round was found for Apr-Oct 2026.
  - The team is small: 9 people on the team page, including a Head of Sales.
  - Sales channels: self-serve Veris Plus (Apr 2026) and Google Cloud Marketplace (Mar 2026). The site also shows AWS Partner Network, NVIDIA Inception, Accenture FinTech Innovation Lab and Plug and Play Tech Center logos; these are self-displayed program memberships, not verified partnerships.
  - No frontier-lab customer ties were found. The public benchmarks evaluate frontier models from xAI, Google and OpenAI, but these are not partnerships. Most customers are anonymized; the only named collaborator is Savi Security.
- notes_for_data_ops:
  - Integration is based on a CLI and containers. A .veris/ folder holds veris.yaml and Dockerfile.sandbox; declared external services such as Slack are intercepted at the DNS layer and routed to LLM-powered mocks, with no agent code changes. The open-source veris-cli uses .veris/twin.yaml and a VERIS_API_KEY environment variable for CI.
  - The docs at docs.veris.ai cover services (PostgreSQL, Kubernetes, Elasticsearch, Splunk), voice frameworks (ElevenLabs, LiveKit, Vapi), CI/CD regression gating, and RL/SFT training. The E2B and Daytona sandbox integrations are open-sourced (Apache-2.0).
  - Deployment options are Veris Cloud, the customer's own cloud (AWS, GCP or Azure), or on-prem.
  - Security claims are self-reported: SOC 2 Type II (AICPA logo shown), HIPAA and GDPR. There is no trust portal and no named auditor. Simulations run on synthetic data.
- sources:
  - https://veris.ai/blog (accessed 2026-10-05)
  - https://veris.ai/blog/vamos-pro-bench-a-harder-exam-for-voice-agents (accessed 2026-10-05)
  - https://arxiv.org/abs/2609.38512 (accessed 2026-10-05)
  - https://arxiv.org/abs/2607.27453 (accessed 2026-10-05)
  - https://www.veris.ai/leaderboard (accessed 2026-10-05)
  - https://www.veris.ai/leaderboard/vamos-pro (accessed 2026-10-05)
  - https://www.veris.ai/leaderboard/vamos (accessed 2026-10-05)
  - https://github.com/veris-ai (accessed 2026-10-05)
  - https://github.com/veris-ai/veris-cli (accessed 2026-10-05)
  - https://www.businessinsider.com/ai-agents-gone-wrong-horror-stories-instinct-muse-privacy-security-2026-9 (accessed 2026-10-05)
  - https://veris.ai/blog/dont-upgrade-the-model-simulate-first (accessed 2026-10-05)
  - https://veris.ai/blog/user-simulation-realism-veris-vs-tau2-bench (accessed 2026-10-05)
  - https://veris.ai/blog/the-right-model-for-the-right-task (accessed 2026-10-05)
  - https://veris.ai/blog/the-loop-is-the-easy-part (accessed 2026-10-05)
  - https://veris.ai/blog/let-openclaw-run-wild-in-simulation (accessed 2026-10-05)
  - https://veris.ai/blog/battle-testing-enterprise-ai-agents-veris-ai-aws-bedrock-agentcore (accessed 2026-10-05)
  - https://veris.ai/blog/veris-plus-is-live (accessed 2026-10-05)
  - https://veris.ai/blog/rft-can-make-agents-with-small-models-outperform-big-models-on-enterprise-tasks (accessed 2026-10-05)
  - https://veris.ai/blog/veris-ai-announces-availability-on-google-cloud-marketplace-to-deliver-reliable-production-ready-ai-agents (accessed 2026-10-05)
  - https://veris.ai/blog/bankingbench-measuring-model-performance (accessed 2026-10-05)
  - https://veris.ai/blog/santabench (accessed 2026-10-05)
  - https://veris.ai/blog/technical-report-reinforcement-learning-fine-tuning-for-enterprise-ai-agents (accessed 2026-10-05)
  - https://veris.ai/blog/dynamic-benchmarking (accessed 2026-10-05)
  - https://www.veris.ai/company (accessed 2026-10-05)
  - https://veris.ai/ (accessed 2026-10-05)
  - https://veris.ai/faqs (accessed 2026-10-05)
  - https://docs.veris.ai/ (accessed 2026-10-05)
  - https://news.google.com/rss/search?q=%22Veris+AI%22&hl=en-US&gl=US&ceid=US:en (accessed 2026-10-05)
  - https://veris.ai/about (accessed 2026-06-07)
  - https://www.alleywatch.com/2025/06/veris-ai-enterprise-ai-agents-agentic-simulation-based-training-platform-mehdi-jamei/ (accessed 2026-06-07)
  - https://www.businesswire.com/news/home/20250603868539/en/ (accessed 2026-06-07)
  - https://www.citybiz.co/article/701797/ (accessed 2026-06-07)
  - https://www.linkedin.com/company/veris-ai (accessed 2026-06-07)
  - https://mjamei.github.io/ (accessed 2026-06-07)
  - https://www.linkedin.com/in/andi-partovi/ (accessed 2026-06-07)
  - https://qconsf.com/speakers/andipartovi (accessed 2026-06-07)
  - https://www.decibel.vc/articles/veris-ai-why-the-environment-is-everything-for-autonomous-ai-agents (accessed 2026-06-07)

## Collinear, rank #12
- slug: collinear
- segment: Commercial vendors
- website: https://www.collinear.ai/
- focus_areas: Coding, Computer Use, Enterprise Workflows, Long-Horizon
- positioning: Collinear AI runs a Simulation Lab of sandboxed, stateful environments with simulated users (NPCs), tools, tasks and verifiers for agent RL training and evaluation. It now positions itself mainly as a supplier of training data, environments and evals to frontier labs across cybersecurity, software engineering, computer use and complex agent behaviour. Its CWE-bench defensive-cyber benchmark is part of the Artificial Analysis Cyber Index.
- best_fit: Labs and agent teams that need held-out evals and verifier-graded RL tasks or training data for defensive cybersecurity, SWE, computer use, or long-horizon workflows with simulated users.
- overall_confidence: medium
- founded_year: 2023 [reported], source: https://www.technologyreview.com/innovator/nazneen-rajani/ and https://www.innovatorsunder35.com/the-list/nazneen-rajani/ (founded after leaving Hugging Face); https://www.crunchbase.com/person/nazneen-rajani-9727 (CEO since Jan 2023); Tracxn (2023). LinkedIn/prospeo list 2024, conflict; 2023 better supported (accessed 2026-06-07)
- status: active [confirmed], source: https://www.collinear.ai/ (accessed 2026-06-07)
- hq_location: Sunnyvale, California, USA [confirmed], source: https://www.collinear.ai/about-us (accessed 2026-10-05)
- other_locations: San Francisco, CA (office opening); Bengaluru, India [confirmed], source: https://www.collinear.ai/careers (accessed 2026-10-05)
- distributed_remote: unknown [unknown], source: https://www.collinear.ai/careers states no remote/distributed policy; physical CA office implied (accessed 2026-06-07)
- current_headcount: ~16-23 (About page lists 16 team members; Tracxn ~23 as of 2026-04-30; Crunchbase 1-10; LinkedIn band 11-50; prospeo 11-20) [reported], source: https://www.collinear.ai/about-us (16 named); https://tracxn.com/d/companies/collinearai (~23, 2026-04-30); https://www.crunchbase.com/organization/collinear-ai (1-10); https://www.linkedin.com/company/collinearai (11-50) (accessed 2026-06-07)
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/collinearai (11-50); corroborated by About page (16 members) and Tracxn (~23); note Crunchbase lists 1-10 (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: unknown [unknown], source: https://www.collinear.ai/careers (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: https://www.collinear.ai/careers (Research Scientist/Engineer, Head of Research roles); https://www.collinear.ai/about-us (team from Hugging Face, Salesforce, Google, Amazon, Stanford) (accessed 2026-06-07)
- researcher_count: unknown [unknown], source: About page lists 16 team members but does not split research vs engineering (accessed 2026-06-07)
- researcher_backgrounds: Founder/CEO Nazneen Rajani: ex-Robustness Research Lead at Hugging Face, ex-Research Scientist at Salesforce, PhD University of Texas at Austin (MIT TR Innovators Under 35); Team described as researchers/engineers from Hugging Face, Salesforce, Google, Amazon, Stanford (per company About page) [reported], source: https://www.collinear.ai/about-us ; https://www.technologyreview.com/innovator/nazneen-rajani/ ; https://www.linkedin.com/in/nazneenrajani (accessed 2026-06-07)
- published_papers_or_benchmarks: CWE-bench v1 (2026-09, cwe-bench.com; part of the Artificial Analysis Cyber Index); YC-Bench (arXiv 2604.01212, 2026-04); TraitBasis / Impatient Users Confuse AI Agents (arXiv 2510.04491); The Valley of Code Reasoning (arXiv 2510.06101); Cats Confuse Reasoning LLM (arXiv 2503.01781); VERITAS (arXiv 2411.03300); Self-rationalization improves LLM as a fine-grained judge (arXiv 2410.05495) [confirmed], source: https://www.collinear.ai/research (accessed 2026-10-05)
- total_raised: ~$10M (reported by Caplight, aggregator); no round publicly announced [reported], source: https://www.caplight.com/company/collinear-ai (accessed 2026-10-05)
- last_round: unknown [unknown], source: No round stage/amount/date found in public sources (accessed 2026-06-07)
- notable_investors: Engineering Capital; 112 Capital (11.2 Capital); B Capital; Storm Ventures; Hitachi Ventures [confirmed], source: https://www.collinear.ai/about-us (accessed 2026-10-05)
- valuation: unknown [unknown], source: prospeo lists ~$5.3M but explicitly labels it an industry-average estimate, not a reported figure (accessed 2026-06-07)
- revenue_signals: AWS case study: co-founder/CPO says Collinear works with 'three of the world's top four AI labs' and with household-name customers including Amazon (self-claimed, labs unnamed). [reported], source: https://aws.amazon.com/solutions/case-studies/collinear-sagemaker/ (accessed 2026-10-05)
- soc2: SOC 2 listed as 'Compliant' on the company Trust Centre (type I/II not stated publicly) [confirmed], source: https://trustcentre.collinear.ai/ (accessed 2026-10-05)
- other_certifications: ISO/IEC 42001:2023 (listed 'Compliant' on Trust Centre); VAPT completed (Trust Centre) [confirmed], source: https://trustcentre.collinear.ai/ (accessed 2026-10-05)
- security_page: https://trustcentre.collinear.ai/ [confirmed], source: https://www.collinear.ai/ (accessed 2026-10-05)
- what_they_sell: environments, evals/benchmarks and verifier-graded training data (incl. cybersecurity task corpus) [confirmed], source: https://www.prweb.com/releases/collinear-ai-launches-cwe-bench-to-test-frontier-coding-agents-on-defensive-cybersecurity-capabilities-302868204.html (accessed 2026-10-05)
- open_source: partial (YC-Bench, spider, tau-trait and CatAttack are on GitHub with MIT licences; Simulation Lab and the CWE-bench eval set are not open) [confirmed], source: https://github.com/collinear-ai (accessed 2026-10-05)
- license: MIT (open-source repos: yc-bench, spider, tau-trait, CatAttack); core platform proprietary [confirmed], source: https://github.com/collinear-ai (accessed 2026-10-05)
- deployment_model: managed-hosted / API (environment endpoints) [reported], source: https://blog.collinear.ai/p/rl-env-as-a-service - 'point your trainer and sampler at Collinear's environment endpoints' (accessed 2026-06-07)
- maturity: unknown [unknown], source: Product pages and blog do not state beta/GA status (accessed 2026-06-07)
- notable_customers: Amazon (self-claimed); Three of the world's top four AI labs (unnamed) (self-claimed, frontier-lab tie) [reported], source: https://aws.amazon.com/solutions/case-studies/collinear-sagemaker/ (accessed 2026-10-05)
- founding_team:
  - Nazneen Rajani, Co-founder & CEO, Previously led post-training/robustness research at Hugging Face and was a senior research scientist at Salesforce Research. PhD in CS from UT Austin. MIT Technology Review Innovators Under 35 (2024).
  - Soumyadeep Bakshi, Co-founder & Chief Product Officer, Per the company About page, he previously led Deloitte's AI practice, advising frontier labs on data for model development. He works on data strategy, verification and evaluation infrastructure and co-authored YC-Bench. The AWS case study gives his title as Cofounder and Chief Product Officer.
- recent_developments (last 6 months):
  - 2026-09-28 [benchmark] CWE-bench v1 released: 120 tasks, 73 CWEs, 8 languages, source: https://blog.collinear.ai/p/cwe-bench-v1
  - 2026-09-02 [benchmark] Collinear launches CWE-bench, a defensive cybersecurity benchmark for coding agents, source: https://www.prweb.com/releases/collinear-ai-launches-cwe-bench-to-test-frontier-coding-agents-on-defensive-cybersecurity-capabilities-302868204.html
  - 2026-09 [partnership] CWE-bench included in the Artificial Analysis Cyber Index as CWE-Bench-AA, source: https://artificialanalysis.ai/evaluations/cwe-bench-aa
  - 2026-04-30 [research] Newsletter #11: exec dinner series, Sunnyvale research social, push toward agentic NPCs, source: https://blog.collinear.ai/p/collinear-newsletter-11-notes-on
  - 2026-04-02 [product] Collinear launches self-serve SimLab CLI for agent staging and testing, source: https://blog.collinear.ai/p/simlab-the-self-serve-staging-playground
  - 2026-04-01 [research] YC-Bench paper posted to arXiv; benchmark open-sourced under MIT, source: https://arxiv.org/abs/2604.01212
- benchmarks_and_research:
  - CWE-bench (v1) (benchmark), 2026-09-28, Blog (2026-09-28): Grok 4.7 and GPT-6 Astra lead at 68% pass@1, with Claude Opus 5.5 at 67% for about a quarter of the cost. The live leaderboard on cwe-bench.com (as of 2026-10-05) shows a three-way tie at 68% (Grok 4.7, Gemini 4 Argon, GPT-6 Astra), with 5 of 120 tasks unsolved by every model., https://cwe-bench.com/
  - CWE-bench (v0, launch) (benchmark), 2026-09-02, Fable 5 led with 47% pass@1 (max reasoning); 18 of 100 tasks unsolved by every model., https://blog.collinear.ai/p/cwe-bench
  - CWE-Bench-AA (Artificial Analysis Cyber Index component) (leaderboard), 2026-09, Grok 4.7 (Xhigh) and DeepSeek V4.1 Flash (Max) at 68.3% pass@1 on the AA leaderboard (as of 2026-10-05)., https://artificialanalysis.ai/evaluations/cwe-bench-aa
  - YC-Bench: Benchmarking AI Agents for Long-Term Planning and Consistent Execution (benchmark), 2026-04-01, Only 3 of 12 models consistently ended above the $200K starting capital. Claude Opus 4.6 led with $1.27M average final funds and GLM-5 was second at $1.21M. Adversarial-client detection failures caused 47% of bankruptcies., https://github.com/collinear-ai/yc-bench
  - The Valley of Code Reasoning: Scaling Knowledge Distillation of Large Language Models (paper), 2025-10-07, https://arxiv.org/abs/2510.06101
  - Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents (TraitBasis) (paper), 2025-10-06, https://arxiv.org/abs/2510.04491
  - Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Models (CatAttack) (paper), 2025-03-03, https://arxiv.org/abs/2503.01781
  - VERITAS: A Unified Approach to Reliability Evaluation (paper), 2024-11-05, https://arxiv.org/abs/2411.03300
  - Self-rationalization improves LLM as a fine-grained judge (paper), 2024-10-07, https://arxiv.org/abs/2410.05495
- products:
  - Simulation Lab (SimLab): Isolated, reproducible environments with simulated applications, users (NPCs), tools and workflows for agent RL training and evaluation, plus automated verifiers and reward functions. A self-serve SimLab CLI was announced on 2026-04-02; its GitHub repo returned 404 on 2026-10-05., https://docs.collinear.ai/introduction
  - Off-the-shelf task libraries: Pre-built task libraries across cybersecurity, software engineering, computer use and knowledge work. The homepage claims '1,000+ tasks off the shelf' (vendor claim)., https://www.collinear.ai/
  - CWE-bench and cyber training corpus: Held-out cyber-defence eval (v1: 120 tasks, 73 CWEs) plus a separately offered training corpus of 4,000+ tasks across 215 CWEs (vendor-stated). Eval tasks never appear in training deliveries., https://cwe-bench.com/
  - YC-Bench: Open-source (MIT) long-horizon agent benchmark., https://github.com/collinear-ai/yc-bench
- notes_for_researchers:
  - Publishes on arXiv regularly: YC-Bench (Apr 2026), TraitBasis (Oct 2025), Valley of Code Reasoning (Oct 2025, company lists it as NeurIPS), CatAttack (Mar 2025), VERITAS (Nov 2024) and self-rationalizing judges (Oct 2024).
  - Open-source code on github.com/collinear-ai: yc-bench, spider (on/off-policy distillation, MIT), tau-trait (MIT) and CatAttack (MIT).
  - The CWE-bench eval set is private to Collinear. Public scores are on cwe-bench.com, and the Artificial Analysis Cyber Index runs an independent implementation (CWE-Bench-AA).
  - Current research direction is NPCs with traits, agency, secrets and limited context inside Simulation Lab, plus 'hard but fair' verifier design (blog posts from May to September 2026).
- notes_for_program_managers:
  - The media-kit boilerplate now reads: 'Collinear AI provides data to frontier labs for faster model improvement', covering cybersecurity, SWE, computer use and complex agent behaviour.
  - An AWS case study (undated) quotes co-founder/CPO Soumyadeep Bakshi saying Collinear works with 'three of the world's top four AI labs' and with Amazon. These are self-claimed, and the labs are not named.
  - The About page now lists B Capital, Storm Ventures and Hitachi Ventures alongside Engineering Capital and 112 Capital; Firestreak is no longer shown. No round has been announced. Caplight (an aggregator) reports about $10M raised, seed-stage.
  - CWE-bench is part of the Artificial Analysis Cyber Index, which counts as third-party validation of the benchmark, not of customer work.
  - Locations: Sunnyvale HQ, San Francisco office opening, and a Bengaluru team per the careers page.
- notes_for_data_ops:
  - The Trust Centre (Sprinto-hosted, trustcentre.collinear.ai) lists SOC 2 'Compliant', ISO 42001 v2023 'Compliant' and VAPT complete. The SOC 2 type (I or II) is not stated publicly; reports are presumably available on request.
  - Docs are on Mintlify at docs.collinear.ai, with an llms.txt index. Integration formats are not documented in the intro pages.
  - CWE-bench v1 runs agents in a sandbox with no internet access during tasks, graded by programmatic verifiers plus a three-judge panel across model families. Eval tasks are kept out of training deliveries.
  - AWS case study: Collinear's RL training workflow runs on Amazon SageMaker AI (vendor/AWS-stated).
- sources:
  - https://www.collinear.ai/ (accessed 2026-10-05)
  - https://www.collinear.ai/about-us (accessed 2026-10-05)
  - https://trustcentre.collinear.ai/ (accessed 2026-10-05)
  - https://www.collinear.ai/research (accessed 2026-10-05)
  - https://www.collinear.ai/careers (accessed 2026-10-05)
  - https://www.collinear.ai/media-kit (accessed 2026-10-05)
  - https://cwe-bench.com/ (accessed 2026-10-05)
  - https://blog.collinear.ai/p/cwe-bench (accessed 2026-10-05)
  - https://blog.collinear.ai/p/cwe-bench-v1 (accessed 2026-10-05)
  - https://www.prweb.com/releases/collinear-ai-launches-cwe-bench-to-test-frontier-coding-agents-on-defensive-cybersecurity-capabilities-302868204.html (accessed 2026-10-05)
  - https://artificialanalysis.ai/evaluations/cwe-bench-aa (accessed 2026-10-05)
  - https://arxiv.org/abs/2604.01212 (accessed 2026-10-05)
  - https://github.com/collinear-ai/yc-bench (accessed 2026-10-05)
  - https://github.com/collinear-ai (accessed 2026-10-05)
  - https://blog.collinear.ai/archive (accessed 2026-10-05)
  - https://blog.collinear.ai/p/simlab-the-self-serve-staging-playground (accessed 2026-10-05)
  - https://blog.collinear.ai/p/collinear-newsletter-10-notes-on (accessed 2026-10-05)
  - https://blog.collinear.ai/p/collinear-newsletter-11-notes-on (accessed 2026-10-05)
  - https://aws.amazon.com/solutions/case-studies/collinear-sagemaker/ (accessed 2026-10-05)
  - https://www.caplight.com/company/collinear-ai (accessed 2026-10-05)
  - https://docs.collinear.ai/introduction (accessed 2026-10-05)
  - https://blog.collinear.ai/p/rl-env-as-a-service (accessed 2026-06-07)
  - https://www.linkedin.com/company/collinearai (accessed 2026-06-07)
  - https://tracxn.com/d/companies/collinearai/__afhJh0xW8gZpWYbw5rCoeqG-KD4fTP3wJr3zxaPZ6Y0 (accessed 2026-06-07)
  - https://pitchbook.com/profiles/company/550057-87 (accessed 2026-06-07)
  - https://www.linkedin.com/in/nazneenrajani (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/collinear-ai (accessed 2026-06-07)

## Chakra Labs, rank #13
- slug: chakra-labs
- segment: Commercial vendors
- website: https://www.chakra.dev/
- focus_areas: Computer Use
- positioning: Chakra Labs ('Frontier Data Laboratory') builds deterministic clones of enterprise and productivity software that expose both GUI and MCP surfaces, along with verifiable task sets and crowdsourced trajectory data for training and evaluating computer-use and tool-use agents. It delivers them through its Dojo hub with Harbor, Verifiers and Verl support. In 2026 its messaging centred on task quality (difficult, realistic, reliably verifiable, hard to hack), and it published MAGI, a 1,000-task mixed GUI and MCP benchmark on which it reports frontier models below 30% pass@1.
- best_fit: Teams training or evaluating computer-use and tool-use (MCP) agents on long-horizon, multi-application enterprise workflows that need deterministic software clones, rubric-verified tasks and a Harbor-compatible harness.
- overall_confidence: medium
- founded_year: 2024 [confirmed], source: https://www.sec.gov/Archives/edgar/data/2106724/000123191926000079/primary_doc.xml (accessed 2026-10-05)
- status: active [confirmed], source: https://www.chakra.dev/ (accessed 2026-06-07); GitHub org repos with pushes through 2026-06-05 https://github.com/chakra-network (accessed 2026-06-07)
- hq_location: Brooklyn, New York, USA [confirmed], source: https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0002106724&type=D&dateb=&owner=include&count=40 (accessed 2026-10-05)
- other_locations: unknown [unknown]
- distributed_remote: unknown [unknown]
- current_headcount: ~14 [reported], source: https://www.linkedin.com/company/chakra-labs (public snippet, accessed 2026-06-07), single source; LinkedIn size band still listed as 2-10
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/chakra-labs (public snippet, accessed 2026-06-07), LinkedIn shows a '2-10' size band but ~14 associated employees; band placed at 11-50 on the higher employee count. Conflicting signals; treat as approximate.
- headcount_growth: unknown [unknown]
- open_roles_count: 4 (Simplify listing; includes Member of Technical Staff, New York) [reported], source: https://simplify.jobs/c/Chakra-Labs (accessed 2026-10-05)
- has_researchers: yes [reported], source: https://www.chakra.dev/ (accessed 2026-06-07), self-described 'applied research team pushing the boundaries of agents'; publishes research/CUA market analyses
- researcher_count: unknown [unknown]
- researcher_backgrounds: Nirmal Krishnan (co-founder and CEO per the SEC Form D signature): Johns Hopkins BS/MS (CS/ML, computational genomics); prior data, markets and early-stage startups; Alexander Fung (co-founder; Director per SEC Form D): University of Waterloo; reported ex-Palantir, Snap and Fin (snippets only); David Finkelstein (Creative Director, not a researcher): leads creative and editorial strategy [reported], source: https://www.sec.gov/Archives/edgar/data/2106724/000123191926000079/primary_doc.xml (accessed 2026-10-05)
- published_papers_or_benchmarks: MAGI: MCP and GUI Interaction benchmark (2026-07-10): 1,000 tasks across 8 cloned enterprise apps; self-reported frontier pass@1 under 30% (Opus 4.8 28%, GPT-5.5 15%); arXiv preprint forthcoming; Dojo-Bench-Mini public task set on Hugging Face (chakra-labs/dojo-bench-mini, 219 tasks); Pango crowdsourced computer-use trajectory datasets on Hugging Face (public sample; full set gated); GLADOS-1: CUA model post-trained on crowd-sourced trajectories (GitHub repo, not a peer-reviewed paper); Editorial and infrastructure posts on chakra.dev/blog (e.g. 'RL Environments are Worthless', Jan 2026; 'Quality Control as an Infrastructure Problem', Sep 2026) [confirmed], source: https://www.chakra.dev/magi-bench (accessed 2026-10-05)
- total_raised: $10.06M (SEC Form D, Chakra Digital Labs, Inc., filed 2026-01-23; includes about $3.05M of convertible securities) [confirmed], source: https://www.sec.gov/Archives/edgar/data/2106724/000123191926000079/primary_doc.xml (accessed 2026-10-05)
- last_round: $10.06M private placement under Reg D Rule 506(b) (Form D filed 2026-01-23; first sale 2025-12-22). Equity plus token warrants; includes $3,047,867 of convertible securities and $8,500 of token warrants. 50 accredited investors; no lead disclosed. Stage not stated in the filing. [confirmed], source: https://www.sec.gov/Archives/edgar/data/2106724/000123191926000079/primary_doc.xml (accessed 2026-10-05)
- notable_investors: unknown [unknown], source: AlleyWatch cites ~50 investors per SEC filing but names none (accessed 2026-06-07). NOTE: do not confuse with same-named crypto/Bitcoin-restaking 'Chakra' (StarkWare/ABCDE), which is a different company.
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: unknown [unknown], source: chakra.dev/security returned 404 (accessed 2026-06-07)
- what_they_sell: environments [confirmed], source: https://trydojo.ai/, 'The RL Environment Hub for Computer Use Agents'; https://www.chakra.dev/, deterministic RL environments plus human computer-use trajectory datasets (2,500+ hrs trajectories, 10M screenshot/action pairs) (accessed 2026-06-07). Note: AlleyWatch frames the company as turning public web data into structured datasets, older/adjacent framing.
- open_source: yes [confirmed], source: https://github.com/chakra-network (accessed 2026-06-07), public repos GLADOS-1, harbor, marina, dojo-spas, software-agent-sdk
- license: Mixed. Original Chakra repos: GLADOS-1 is Apache-2.0; dojo-spas and verl-tools have no declared license. Forks: marina and harbor (of harbor-framework/harbor, Apache-2.0), software-agent-sdk (of OpenHands, MIT), Office-Word-MCP-Server and cli-mcp-server (third-party, MIT). The dojo-bench-mini HF dataset declares no license. [confirmed], source: https://github.com/chakra-network (accessed 2026-10-05)
- deployment_model: Managed-hosted with access on request (Dojo), plus on-premises installation through Harbor per the homepage. Environments are delivered as containers, with native Harbor, Verifiers and Verl support. Internal eval and QC runs use Daytona sandboxes via Harbor. [confirmed], source: https://www.chakra.dev/ (accessed 2026-10-05)
- maturity: private beta (request access) [estimated], source: https://trydojo.ai/, 'Request Access' gating; Dojo launched 2025-10-31 per https://www.chakra.dev/publications/product-launch-dojo (accessed 2026-06-07)
- notable_customers: unknown [unknown], source: Site references collaboration with unnamed 'leading research teams' (https://www.chakra.dev/, https://trydojo.ai/, accessed 2026-06-07), self-claimed, no named or verified customers
- founding_team:
  - Nirmal Krishnan, Co-founder and CEO (signed SEC Form D as Chief Executive Officer; Executive Officer and Director), Bachelor's and master's at Johns Hopkins (CS/ML, computational genomics, with published papers on prostate cancer and iPSCs). Earlier work in data, markets and early-stage startups (per his chakra.dev author bio).
  - Alexander Fung, Co-founder; Director, University of Waterloo (per LinkedIn). Prior roles at Palantir, Snap and Fin are reported in search snippets only and not confirmed on a primary page.
- recent_developments (last 6 months):
  - 2026-10 [hire] Hiring Member of Technical Staff (New York) for agent orchestration at scale, source: https://simplify.jobs/p/6a8b3f39-c088-40a1-874e-d0a0f909b9ef/Member-of-Technical-Staff
  - 2026-09-11 [partnership] Chakra moves QC trials from Kubernetes to Daytona sandboxes (partnership post), source: https://www.chakra.dev/blog/quality-control-as-an-infrastructure-problem
  - 2026-07-10 [benchmark] Chakra Labs publishes MAGI (MCP and GUI Interaction) benchmark, source: https://www.chakra.dev/magi-bench
- benchmarks_and_research:
  - MAGI: MCP and GUI Interaction benchmark (benchmark), 2026-07-10, Self-reported: frontier models complete fewer than 30% of tasks at pass@1. Opus 4.8 scores 28%, GPT-5.5 15%, and Kimi K2.7-code is lowest (exact value shown only in a chart). Only 21/200 test tasks are solved by both Opus 4.8 and GPT-5.5., https://www.chakra.dev/magi-bench
  - Dojo-SPAs (environment), 2026-02-19, https://github.com/chakra-network/dojo-spas
  - Dojo-Bench-Mini (dataset), 2025-10-13, Self-reported chart on the chakra.dev homepage: Claude Opus 4.6 41.39%, 'ChatGPT 5.2-High' 29.51%, Gemini 2.5 Pro 9.43%, Llama 4 Maverick 3.69%., https://huggingface.co/datasets/chakra-labs/dojo-bench-mini
  - Pango Sample (crowdsourced computer-use trajectories) (dataset), 2025-07-21, https://huggingface.co/datasets/chakra-labs/pango-sample
- products:
  - Dojo: Collaborative RL environment hub for computer-use research: interactive clones of production software with native support for Harbor, Verifiers and Verl. Access is by request only., https://trydojo.ai/
  - Chakra environment clones: Deterministic, pixel-perfect environment clones exposing GUI and MCP surfaces. The site lists Notion, Figma, Canva, Gmail, LinkedIn, Salesforce, Amazon, Slack, GitHub, Grafana and Google Calendar. Delivered by containerization, either hosted or on-premises through Harbor (per the homepage)., https://www.chakra.dev/
  - MAGI benchmark and train split: 1,000-task benchmark mixing MCP and GUI. Teams can ask to run their model on it, and the 800-task train split is available on request., https://www.chakra.dev/magi-bench
  - Pango trajectory datasets: Crowdsourced human computer-use trajectories from productivity applications. A public sample is available; the full dataset is gated., https://huggingface.co/datasets/chakra-labs/pango
- notes_for_researchers:
  - MAGI (2026-07-10) has 1,000 mixed GUI and MCP tasks across 8 cloned enterprise apps, all sharing one seeded company state. Self-reported pass@1 is Opus 4.8 28% and GPT-5.5 15%, with Kimi K2.7-code lowest. Runs use one adapted OpenHands harness on Harbor rather than native harnesses, and the arXiv preprint has not been posted.
  - MAGI's failure analysis finds GUI-interaction errors (about 51% of failed trajectories) and premature termination (about 40%) dominate, while MCP tool errors are near zero. Reward drops sharply on tasks spanning 6 or more apps.
  - Public data on Hugging Face: dojo-bench-mini (219 tasks, no license) and pango-sample (crowdsourced trajectories). The full Pango dataset and other sets are gated.
  - The GitHub org contains original repos (GLADOS-1, a computer-use model post-trained on crowd-sourced trajectories, Apache-2.0; and dojo-spas) plus forks. 'marina' and 'harbor' are forks of harbor-framework/harbor, and software-agent-sdk, Office-Word-MCP-Server and cli-mcp-server are forks of third-party projects.
  - Editorial stance (Nirmal Krishnan, Jan 2026): 'Labs are not buying environments, they are buying tasks.' Good tasks are difficult, realistic, reliably verifiable and hard to hack, and Chakra cites pass@5 below 30% as the difficulty range labs expect.
- notes_for_program_managers:
  - Published onboarding model (methodology page, undated): Discovery 3–5 days, Provision 2–3 days, Benchmark 1–2 days, Train (highly variable), New Benchmark 1–2 days. That is about 7–12 days plus training time. The page also claims custom task specification to evaluation-ready deployment in under 24 hours (self-claimed).
  - No named customers. The site cites unnamed 'leading research teams', which is self-claimed. The Hugging Face org has gated datasets with codename labels (e.g. pango-customer-*, dojo-bench-customer-colossus), but no customer identities are disclosed.
  - Funding: one SEC Form D (filed 2026-01-23, first sale 2025-12-22) shows $10,056,320 sold to 50 accredited investors. The total includes $3,047,867 of convertible securities and $8,500 of token warrants. No lead investor or valuation is disclosed, and no new round was found in Apr–Oct 2026.
  - Named infrastructure partner: Daytona (sandboxes). Its CEO is quoted in Chakra's Sep 2026 post.
  - Team: Simplify reports 11–50 employees and 4 open roles, including an in-person MTS role in New York (aggregator data, reported).
- notes_for_data_ops:
  - Integration: Dojo supports Harbor, Verifiers and Verl natively. Environments ship as containers, hosted or on-prem via Harbor (per the homepage). MAGI runs through an OpenHands harness on Harbor.
  - Execution backend: Daytona on-demand sandboxes via Harbor. Self-reported P99 start time fell from 16.7 hours to 46 seconds, and cost per trial fell about 85% at list price.
  - QC process (self-described): each task is run at least 10 times per frontier model. RLVR verifiers are audited against a judge for false positives and negatives, LLM-judge tasks are checked for agreement with human reviewers, and expert operators produce golden traces before a task ships.
  - No SOC 2 or other certification and no security page found. Contacts are info@, partner@ and pr@chakra-labs.com.
- sources:
  - https://www.chakra.dev/ (accessed 2026-10-05)
  - https://www.chakra.dev/blog (accessed 2026-10-05)
  - https://www.chakra.dev/research (accessed 2026-10-05)
  - https://www.chakra.dev/magi-bench (accessed 2026-10-05)
  - https://www.chakra.dev/blog/quality-control-as-an-infrastructure-problem (accessed 2026-10-05)
  - https://www.chakra.dev/methodology (accessed 2026-10-05)
  - https://www.chakra.dev/blog/rl-environments-are-worthless (accessed 2026-10-05)
  - https://trydojo.ai/ (accessed 2026-10-05)
  - https://huggingface.co/datasets/chakra-labs/dojo-bench-mini (accessed 2026-10-05)
  - https://huggingface.co/datasets/chakra-labs/pango-sample (accessed 2026-10-05)
  - https://github.com/chakra-network (accessed 2026-10-05)
  - https://www.sec.gov/Archives/edgar/data/2106724/000123191926000079/primary_doc.xml (accessed 2026-10-05)
  - https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0002106724&type=D&dateb=&owner=include&count=40 (accessed 2026-10-05)
  - https://alleywatch.com/2026/02/nyctech-week-in-review-1-25-26-1-31-26/ (accessed 2026-10-05)
  - https://simplify.jobs/c/Chakra-Labs (accessed 2026-10-05)
  - https://simplify.jobs/p/6a8b3f39-c088-40a1-874e-d0a0f909b9ef/Member-of-Technical-Staff (accessed 2026-10-05)
  - https://www.linkedin.com/in/nirmal-krishnan (accessed 2026-10-05)
  - https://www.linkedin.com/in/alexfung (accessed 2026-10-05)
  - https://arxiv.org/abs/2609.04298 (accessed 2026-10-05)
  - https://www.chakra.dev/publications/product-launch-dojo (accessed 2026-06-07)
  - https://www.linkedin.com/company/chakra-labs (accessed 2026-06-07)
  - https://www.alleywatch.com/2026/01/the-alleywatch-startup-daily-funding-report-1-26-2026/ (accessed 2026-06-07)
  - https://x.com/chakra_ai (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/chakra-labs (accessed 2026-06-07)
  - https://jobs.ashbyhq.com/chakra-labs (accessed 2026-06-07)

## Andon Labs, rank #14
- slug: andon-labs
- segment: Commercial vendors
- website: https://andonlabs.com
- focus_areas: Computer Use, Long-Horizon
- positioning: Andon Labs (YC W24, formerly Vectorview) builds long-horizon agent benchmarks (Vending-Bench, Blueprint-Bench, Drone-Bench, Butter-Bench) and runs real AI-operated businesses (Andon Market in SF, Andon Café in Stockholm, Andon FM, vending machines) as live safety testbeds. In September 2026 it launched Pion, a research-preview platform for agents that run whole businesses. It is a repeat Anthropic research partner (Project Vend, and Project Pilot with the Frontier Red Team).
- best_fit: Frontier labs and safety teams that want held-out evals of long-horizon behaviour, agentic misalignment and dual-use physical capability (Vending-Bench, Drone-Bench), plus data from real autonomous deployments. Pion suits businesses that want to hand operations to AI agents on a revenue-share basis.
- overall_confidence: high
- founded_year: 2023 [reported], source: https://www.ycombinator.com/companies/andon-labs (accessed 2026-06-07); https://tracxn.com/d/companies/andonlabs (accessed 2026-06-07). Note: company was formerly named Vectorview and rebranded to Andon Labs around late 2024; predecessor Vectorview activity may predate 2023.
- status: active [confirmed], source: https://andonlabs.com/ (accessed 2026-06-07); https://www.anthropic.com/research/project-vend-2 (accessed 2026-06-07); https://fortune.com/2026/06/02/anthropic-office-vending-machine-ai-agents-vendo-andon-lukas-petersson/ (accessed 2026-06-07)
- hq_location: San Francisco, USA [reported], source: https://www.ycombinator.com/companies/andon-labs (accessed 2026-06-07); https://tracxn.com/d/companies/andonlabs (accessed 2026-06-07). Note: Swedish origins (formerly Vectorview, Bromma/Stockholm, Sweden); operates in both SF and Stockholm. Some databases list HQ as Bromma, Sweden.
- other_locations: Stockholm, Sweden (Andon Café, Norrbackagatan 48; AI manager 'Mona'); San Francisco, USA (Andon Market, 2102 Union St, Cow Hollow; AI manager 'Luna') [confirmed], source: https://andonlabs.com/blog/andon-market-launch (accessed 2026-10-05)
- distributed_remote: unknown [unknown]
- current_headcount: 11 (YC company page, accessed 2026-10-05); earlier third-party estimates ~16-18 [reported], source: https://www.ycombinator.com/companies/andon-labs (accessed 2026-10-05)
- headcount_band: 11-50 [estimated], source: Inferred from Tracxn ~16 employees and LinkedIn ~18 profiles (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: 3 (Research/Engineering: AI Safety and Evals; Software/Entrepreneurial: Scaling SAOs; Design Engineer for Pion). In person in SF, $120k-$180k salary plus $300k-$900k equity. [confirmed], source: https://andonlabs.com/join (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: https://arxiv.org/abs/2510.21860 (Butter-Bench paper, accessed 2026-06-07), authored by Andon Labs team; company publishes benchmark research.
- researcher_count: ~7 (counted as authors on the Butter-Bench paper; not a full team census) [estimated], source: https://arxiv.org/abs/2510.21860 (accessed 2026-06-07): authors include Callum Sharrock, Lukas Petersson, Hanna Petersson, Axel Backlund, Axel Wennstrom, Kristoffer Nordstrom, Elias Aronsson
- researcher_backgrounds: Lukas Petersson (co-founder; CEO per Fortune 2026-06, 'Founder/CTO' on YC). Co-founded Vectorview.; Axel Backlund (co-founder; first author of Vending-Bench); Emil Froberg (co-founder and former CEO of Vectorview/Andon Labs; stepped down around Dec 2024 and remains a shareholder) [reported], source: https://www.linkedin.com/posts/emil-froberg_i-co-founded-vectorview-now-andon-labs-activity-7269617615793238016-BRZ9 (accessed 2026-10-05)
- published_papers_or_benchmarks: Vending-Bench (arXiv:2502.15840); Vending-Bench 2 / Vending-Bench Arena (https://andonlabs.com/evals/vending-bench-2); Blueprint-Bench (arXiv:2509.25229); Blueprint-Bench 2 (https://andonlabs.com/evals/blueprint-bench-2), 2026-05; Butter-Bench (arXiv:2510.21860); Drone-Bench (https://andonlabs.com/docs/Drone_Bench.pdf), with Anthropic Project Pilot, 2026-07; Andon FM (AI-run radio stations); Project Vend (with Anthropic); Cheating in Drone-Bench (2026-08-03); AI bosses studies, parts 1 and 2 (2026-08) [confirmed], source: https://andonlabs.com/ (accessed 2026-10-05)
- total_raised: unknown (PitchBook reports ~$500K; Tracxn lists unfunded, sources directly conflict) [unknown], source: https://pitchbook.com/profiles/company/541549-09 (accessed 2026-06-07, ~$500K via search snippet, fetcher 403); https://tracxn.com/d/companies/andonlabs (accessed 2026-06-07, lists unfunded). YC W24 participation confirmed; specific raise amount not corroborated by a primary/announcement source.
- last_round: Y Combinator W24 (pre-seed); raise amount unconfirmed [reported], source: https://www.ycombinator.com/companies/andon-labs (accessed 2026-06-07). YC W24 batch participation confirmed; PitchBook ~$500K is single-source and conflicts with Tracxn's 'unfunded'.
- notable_investors: Y Combinator (W24) [reported], source: https://www.ycombinator.com/companies/andon-labs (accessed 2026-06-07). YC W24 confirmed. Additional names (Breakpoint Capital, Juniper Ventures, Phosphor Capital, Superangel, Seldon Lab) appear only in PitchBook/Tracxn search snippets, not on a primary source, and Tracxn lists the company as unfunded, so they are not independently confirmed and are excluded pending verification.
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown], source: Anecdotal only: Stockholm AI cafe 'Mona' reportedly earned ~44,000 SEK (~$4,659) in first two weeks (https://www.businesstoday.in/, accessed 2026-06-07). Not a company-level revenue figure.
- soc2: unknown [unknown], source: No SOC 2 attestation or trust/security page found as of 2026-06-07
- other_certifications: unknown [unknown]
- security_page: unknown [unknown], source: No dedicated trust/security page found as of 2026-06-07
- what_they_sell: evals + platform for agents that run businesses (Pion, research preview) + its own AI-run real-world deployments [confirmed], source: https://andonlabs.com/blog/why-we-built-pion (accessed 2026-10-05)
- open_source: unknown [unknown], source: No clear public OSS product/license found as of 2026-06-07; benchmarks published with papers but licensing not confirmed.
- license: unknown [unknown]
- deployment_model: Pion: hosted platform in research preview (waitlist). Andon funds selected ideas with seed tokens and plans to take a small share of the revenue the agent creates. Evals are run in-house on held-out benchmarks. [confirmed], source: https://andonlabs.com/pion (accessed 2026-10-05)
- maturity: GA for benchmarks and live deployments; Pion platform in research preview (waitlist) [confirmed], source: https://andonlabs.com/pion (accessed 2026-10-05)
- notable_customers: Anthropic (verified, frontier-lab tie); xAI (verified, frontier-lab tie); OpenAI (self-claimed, frontier-lab tie); Google DeepMind (self-claimed, frontier-lab tie) [reported], source: https://andonlabs.com/ (accessed 2026-10-05)
- founding_team:
  - Lukas Petersson, Co-founder and CEO (Fortune 2026-06-02 says CEO; the YC profile lists 'Founder/CTO'), Co-founded Vectorview, which became Andon Labs (YC W24). Co-author of Vending-Bench (arXiv:2502.15840), Blueprint-Bench (arXiv:2509.25229) and Butter-Bench (arXiv:2510.21860). His own site describes him as an 'aspiring astronaut and ML enthusiast'.
  - Axel Backlund, Co-founder, Co-founder of Andon Labs (confirmed on his own site and on the Cognitive Revolution podcast). First author of Vending-Bench (arXiv:2502.15840) and co-author of Blueprint-Bench and Butter-Bench. Based in Stockholm. No verified prior employer or education found. A CTO title is reported elsewhere but not confirmed on a primary source.
  - Emil Froberg, Co-founder and former CEO (stepped down around Dec 2024), In his own LinkedIn post, he says he co-founded Vectorview (now Andon Labs) with Lukas and stepped down as CEO. He remains a shareholder and has moved on to a new venture on AI agent adoption.
- recent_developments (last 6 months):
  - 2026-09-24 [benchmark] Opus 5.5, GPT-6 Sol and Grok 4.7 evaluated on Vending-Bench, source: https://andonlabs.com/blog/opus-5-5-gpt-6-sol-grok-4-7-vending-bench
  - 2026-09-14 [product] Andon Labs launches Pion, a platform for AI agents that run businesses (research preview), source: https://andonlabs.com/blog/why-we-built-pion
  - 2026-09-07 [benchmark] Astra vs Fable on Vending-Bench: More Money, More Aligned, source: https://andonlabs.com/blog/gpt-6-astra-vending-bench
  - 2026-08-14 [research] 'AI bosses are slow to fire and quick to hire' (AI-employer study, part 2), source: https://andonlabs.com/blog/ai-bosses-2
  - 2026-08-04 [research] 'AI bosses are kind, but sometimes dumb, which hurts employees' (AI-employer study, part 1), source: https://andonlabs.com/blog/ai-bosses-1
  - 2026-08-03 [research] Cheating in Drone-Bench: 50.6% of Claude Opus 5 runs flagged, source: https://andonlabs.com/blog/cheating-in-drone-bench
  - 2026-07-28 [benchmark] Opus 5 on Vending-Bench: 'Once Again the Best Capitalist, Once Again Misaligned', source: https://andonlabs.com/blog/opus-5-vending-bench
  - 2026-07-24 [partnership] Project Pilot with Anthropic's Frontier Red Team; Drone-Bench released, source: https://anthropic.com/research/project-pilot
  - 2026-07-07 [research] Andon FM, Six Weeks Later, source: https://andonlabs.com/blog/andon-fm-2
  - 2026-06-30 [research] Why Gemini 3.1 Pro lost money running Andon Café, source: https://andonlabs.com/blog/why-gemini-lost-money-andon-cafe
  - 2026-06-09 [benchmark] Fable 5 on Vending-Bench: Misbehaving, with Plausible Deniability, source: https://andonlabs.com/blog/fable5-vending-bench
  - 2026-06-02 [other] Fortune: Anthropic's office AI vending machine became AI-run stores and cafés within a year, source: https://fortune.com/2026/06/02/anthropic-office-vending-machine-ai-agents-vendo-andon-lukas-petersson/
  - 2026-05-28 [benchmark] Opus 4.8 on Vending-Bench: Better Alignment, Worse Performance, source: https://andonlabs.com/blog/opus-4-8-vending-bench
  - 2026-05-13 [research] Andon FM: four AIs run radio stations as businesses, source: https://andonlabs.com/blog/andon-fm
  - 2026-05-04 [product] Our AI started a café in Stockholm (Andon Café, AI manager 'Mona'), source: https://andonlabs.com/blog/ai-cafe-stockholm
  - 2026-05 [benchmark] Blueprint-Bench 2 released, source: https://andonlabs.com/evals/blueprint-bench-2
  - 2026-04-22 [benchmark] GPT-5.5 on Vending-Bench: Bad behavior is not necessary, source: https://andonlabs.com/blog/openai-gpt-5-5-vending-bench
  - 2026-04-21 [other] New York Times covers Andon Market, the AI-managed store in San Francisco, source: https://www.nytimes.com/2026/04/21/us/san-francisco-store-managed-ai-agent.html
  - 2026-04-10 [product] Andon Market launches: AI agent 'Luna' given a 3-year retail lease in SF, source: https://andonlabs.com/blog/andon-market-launch
- benchmarks_and_research:
  - Project Pilot (with Anthropic Frontier Red Team) (paper), 2026-07-24, Per Anthropic, Claude Fable 5 was the top performer, beating the baseline on 4 of 5 subtasks (consistently on only 3). Reconstruction reached about 47% of baseline., https://anthropic.com/research/project-pilot
  - Drone-Bench (benchmark), 2026-07, Leaderboard (accessed 2026-10-05): Claude Opus 5.5 is highest at about 97% average progress toward baseline. No model has beaten the Reconstruct baseline, so end-to-end success is 0%., https://andonlabs.com/evals/drone-bench
  - Andon FM (environment), 2026-05-13, Only Gemini secured a sponsorship (one $45 deal). Grok made up sponsor relationships., https://andonlabs.com/blog/andon-fm
  - Blueprint-Bench 2 (benchmark), 2026-05, Gemini 4 Argon 0.544, Claude Opus 5.5 0.512, GPT-6 Astra 0.497. The human baseline is 0.586, measured on 12 apartments., https://andonlabs.com/evals/blueprint-bench-2
  - Project Vend (with Anthropic) (environment), 2025-12-18, https://www.anthropic.com/research/project-vend-2
  - Butter-Bench (paper), 2025-10-23, The best LLMs score 40%; the human mean is 95%., https://andonlabs.com/evals/butter-bench
  - Blueprint-Bench (benchmark), 2025-09-24, Humans score 0.547 and GPT-5 0.431. Most models score at or below the random baseline of 0.279., https://andonlabs.com/evals/blueprint-bench
  - Vending-Bench (benchmark), 2025-02-20, Every model tested had some runs that failed unpredictably, including 'meltdown' loops. The failures did not correlate with the context window filling up., https://andonlabs.com/evals/vending-bench
  - Vending-Bench 2 (and Vending-Bench Arena) (leaderboard), 2025, Leaderboard (accessed 2026-10-05): #1 GPT-6 Astra $15,514.70, #2 GPT-6 Sol $14,427.85, #3 Gemini 4 Argon $13,718.16, #4 Claude Opus 5 $11,181.87, #5 Claude Opus 4.7 $10,936.76., https://andonlabs.com/evals/vending-bench-2
- products:
  - Pion: A platform for AI agents that run businesses autonomously, with tools for email, phone, banking, browsing and secure compute. In research preview with a waitlist. During the preview, Andon funds selected ideas with 'seed tokens' and plans to take a small share of the revenue the agent creates., https://andonlabs.com/pion
  - Vending-Bench / Vending-Bench 2 / Arena: A long-horizon benchmark where agents run a simulated business, with a public leaderboard., https://andonlabs.com/evals/vending-bench-2
  - Blueprint-Bench (1 and 2): Spatial-reasoning benchmark (turning photos into floor plans)., https://andonlabs.com/evals/blueprint-bench-2
  - Drone-Bench: A held-out benchmark of AI-written code for drone control., https://andonlabs.com/evals/drone-bench
  - Andon Market: An AI-run retail store at 2102 Union St, San Francisco. The AI manager is Luna; the homepage lists the current model as Claude Fable 5.1., https://andonlabs.com/blog/andon-market-launch
  - Andon Café: An AI-run café at Norrbackagatan 48, Stockholm. The AI manager is Mona; the homepage lists the current model as GPT-6 Astra., https://andonlabs.com/blog/ai-cafe-stockholm
  - Andon FM: Four AI-run radio stations., https://andonlabs.com/blog/andon-fm
- notes_for_researchers:
  - Papers: Vending-Bench (arXiv:2502.15840), Blueprint-Bench (arXiv:2509.25229), Butter-Bench (arXiv:2510.21860), and the Drone-Bench PDF (andonlabs.com/docs/Drone_Bench.pdf).
  - Andon publishes Vending-Bench results for major frontier releases within days of launch, with notes on supplier deception, collusion and refund behaviour.
  - The Drone-Bench cheating report (2026-08-03) documents reward hacking, such as exfiltrating test data to filebin.net and x0.at and encoding scores in feedback channels, in 50.6% of Claude Opus 5 runs. A useful case study for eval integrity.
  - Real-world deployments (Andon Market, Andon Café, Andon FM, and vending machines in SF, NY and London) produce incident data on AI agents handling money and managing human employees. The 'AI bosses' posts replay real incidents across models.
- notes_for_program_managers:
  - Frontier-lab ties: Anthropic credits Andon Labs on its own site for Project Vend (2025-12-18) and Project Pilot with the Frontier Red Team (2026-07-24). xAI's Grok account confirmed the Grokbox. The homepage says it is 'trusted by' Anthropic, OpenAI and Google DeepMind; OpenAI and DeepMind are self-claimed only. These are confirmed partnerships, not confirmed paying contracts.
  - A small team: YC lists 11 employees. The only publicly confirmed backer is YC (W24). A former co-founder says the company 'secured significant funding' (self-claim, no amount given). No priced round has been announced.
  - Benchmarks are run in-house as held-out evals; the Drone-Bench page says labs cannot train on it. Labs get evaluated by Andon Labs rather than licensing data.
  - Pion (2026-09-14) is in research preview, with a model of seed tokens plus a share of revenue and no named external customers.
- notes_for_data_ops:
  - No public API docs, SOC 2 report or trust page found. Pion is waitlist-only.
  - No published integration formats for the benchmarks were found.
  - The benchmarks are held out and not open-sourced. Contact: founders (at) andonlabs.com.
- sources:
  - https://andonlabs.com/ (accessed 2026-10-05)
  - https://andonlabs.com/blog (accessed 2026-10-05)
  - https://andonlabs.com/blog/why-we-built-pion (accessed 2026-10-05)
  - https://andonlabs.com/pion (accessed 2026-10-05)
  - https://andonlabs.com/blog/opus-5-5-gpt-6-sol-grok-4-7-vending-bench (accessed 2026-10-05)
  - https://andonlabs.com/blog/gpt-6-astra-vending-bench (accessed 2026-10-05)
  - https://andonlabs.com/blog/ai-bosses-2 (accessed 2026-10-05)
  - https://andonlabs.com/blog/cheating-in-drone-bench (accessed 2026-10-05)
  - https://andonlabs.com/blog/ai-bosses-1 (accessed 2026-10-05)
  - https://andonlabs.com/blog/opus-5-vending-bench (accessed 2026-10-05)
  - https://andonlabs.com/blog/andon-fm-2 (accessed 2026-10-05)
  - https://andonlabs.com/blog/why-gemini-lost-money-andon-cafe (accessed 2026-10-05)
  - https://andonlabs.com/blog/fable5-vending-bench (accessed 2026-10-05)
  - https://andonlabs.com/blog/opus-4-8-vending-bench (accessed 2026-10-05)
  - https://andonlabs.com/blog/andon-fm (accessed 2026-10-05)
  - https://andonlabs.com/blog/ai-cafe-stockholm (accessed 2026-10-05)
  - https://andonlabs.com/blog/openai-gpt-5-5-vending-bench (accessed 2026-10-05)
  - https://andonlabs.com/blog/andon-market-launch (accessed 2026-10-05)
  - https://andonlabs.com/evals/drone-bench (accessed 2026-10-05)
  - https://andonlabs.com/docs/Drone_Bench.pdf (accessed 2026-10-05)
  - https://andonlabs.com/evals/blueprint-bench (accessed 2026-10-05)
  - https://andonlabs.com/evals/blueprint-bench-2 (accessed 2026-10-05)
  - https://andonlabs.com/evals/vending-bench-2 (accessed 2026-10-05)
  - https://andonlabs.com/join (accessed 2026-10-05)
  - https://anthropic.com/research/project-pilot (accessed 2026-10-05)
  - https://www.anthropic.com/research/project-vend-2 (accessed 2026-10-05)
  - https://fortune.com/2026/06/02/anthropic-office-vending-machine-ai-agents-vendo-andon-lukas-petersson/ (accessed 2026-10-05)
  - https://www.ycombinator.com/companies/andon-labs (accessed 2026-10-05)
  - https://arxiv.org/abs/2502.15840 (accessed 2026-10-05)
  - https://arxiv.org/abs/2509.25229 (accessed 2026-10-05)
  - https://arxiv.org/abs/2510.21860 (accessed 2026-10-05)
  - https://lukaspetersson.com/ (accessed 2026-10-05)
  - https://axelbacklund.se/ (accessed 2026-10-05)
  - https://www.cognitiverevolution.ai/autonomous-organizations-vending-bench-beyond-w-lukas-petersson-axel-backlund-of-andon-labs/ (accessed 2026-10-05)
  - https://www.linkedin.com/posts/emil-froberg_i-co-founded-vectorview-now-andon-labs-activity-7269617615793238016-BRZ9 (accessed 2026-10-05)
  - https://andonlabs.com/evals/vending-bench (accessed 2026-06-07)
  - https://andonlabs.com/evals/butter-bench (accessed 2026-06-07)
  - https://tracxn.com/d/companies/andonlabs/__kbDcsnD7GXkDkPhJe7NzhyYdc1aYNmLVZaGPytuX8Ck (accessed 2026-06-07)
  - https://pitchbook.com/profiles/company/541549-09 (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/andon-labs (accessed 2026-06-07)
  - https://www.linkedin.com/company/andonlabs (accessed 2026-06-07)
  - https://www.linkedin.com/posts/andonlabs_vectorview-is-now-andon-labs-we-help-frontier-activity-7270578670040170497-GIyX (accessed 2026-06-07)
  - https://www.pymnts.com/artificial-intelligence-2/2026/andon-labs-handed-an-ai-a-cafe-and-business-boomed/ (accessed 2026-06-07)
  - https://x.com/andonlabs/status/1943182987371098151 (accessed 2026-06-07)
  - https://x.com/grok/status/1947365557248135298 (accessed 2026-06-07)
  - https://www.businesstoday.in/amp/technology/story/meet-mona-the-ai-running-a-real-cafe-in-stockholm-527972-2026-04-29 (accessed 2026-06-07)

## BenchFlow, rank #15
- slug: benchflow
- segment: Commercial vendors
- website: https://www.benchflow.ai/
- focus_areas: Coding, Computer Use, Enterprise Workflows
- positioning: BenchFlow is a small, open-source-first 'frontier environment lab' in the Bay Area. It publishes agent benchmarks and environments: SkillsBench and ClawsBench, the env0 simulated workspaces, Robo Use and GenesisBench for embodied agents, and FrontierPhysics. It runs the PostTrain Arena for community-contributed post-training environments and maintains an Apache-2.0 runtime for RL environments, post-training and evals. It has event-level ties to Google DeepMind, Kaggle, Anthropic, Hugging Face and Prime Intellect.
- best_fit: Teams needing open-source, reproducible agent evaluation environments and a runtime to benchmark coding/computer-use/workplace agents at low setup cost.
- overall_confidence: medium
- founded_year: 2024 [reported], source: https://startupintros.com/orgs/benchflow (accessed 2026-06-07); https://www.linkedin.com/company/benchflow-ai (accessed 2026-06-07)
- status: active [confirmed], source: https://www.benchflow.ai/ (accessed 2026-06-07); https://github.com/benchflow-ai/benchflow (accessed 2026-06-07), release 0.5.2 dated 2026-06-05
- hq_location: San Francisco Bay Area, CA, USA [confirmed], source: https://www.benchflow.ai/about (accessed 2026-10-05)
- other_locations: unknown [unknown]
- distributed_remote: unknown [unknown]
- current_headcount: 3 named team members (founder + 2 Members of Technical Staff) on the company About page [confirmed], source: https://www.benchflow.ai/about (accessed 2026-10-05)
- headcount_band: 1-10 [reported], source: https://www.linkedin.com/company/benchflow-ai (accessed 2026-06-07); https://startupintros.com/orgs/benchflow (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: unknown [unknown]
- has_researchers: yes [confirmed], source: https://www.benchflow.ai/research (accessed 2026-10-05)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Xiangyi Li (founder/CEO): previously coding-model inference at Tesla; first author of SkillsBench and ClawsBench; Bingran You (MTS): PhD in quantum physics, UC Berkeley; author of the env0 study; Carrie Chen (MTS): CS, Cornell; host of the Cornell RL research seminar [confirmed], source: https://www.benchflow.ai/about (accessed 2026-10-05)
- published_papers_or_benchmarks: SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (arXiv:2602.12670, Feb 2026; v4 Jun 2026; leaderboard v1.1 at skillsbench.ai); ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces (arXiv:2604.05172, Apr 2026); env0 SFT study on SkillsBench/ClawsBench (Jul 2026); GenesisBench (LLM agents improving robot policies, 2026); Robo Use (embodied-agent environments, Sep 2026); FrontierPhysics v0.1 (physics-research agent benchmark); PostTrain Arena (community post-training environment arena) [confirmed], source: https://www.benchflow.ai/research (accessed 2026-10-05)
- total_raised: unknown (aggregator reports $1.0M; founder stated in Jun 2026 that no seed round was raised) [unknown], source: https://x.com/xdotli/status/2070256643930255519 (accessed 2026-10-05)
- last_round: Unconfirmed. An aggregator reports a $1M seed (Jan 2025), but the founder stated on X on 2026-06-25 'We didn't even raise a seed'. No priced round has been announced. [reported], source: https://x.com/xdotli/status/2070256643930255519 (accessed 2026-10-05)
- notable_investors: Founders, Inc. (confirmed via its portfolio page); Y Combinator (reported by aggregator only; no YC directory page found); Pear VC (reported); Construct Capital (reported); FAST by GETTYLAB (reported); Ankit Jain (angel, reported) [reported], source: https://f.inc/portfolio/benchflow/ (accessed 2026-10-05)
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: unknown [unknown], source: No SOC2 mention or trust page found on official site (accessed 2026-06-07)
- other_certifications: unknown [unknown]
- security_page: unknown [unknown], source: No dedicated trust/security page found (https://www.benchflow.ai/ accessed 2026-06-07)
- what_they_sell: environments [confirmed], source: https://www.benchflow.ai/ (accessed 2026-06-07), 'A frontier environment lab for AI agents'; environments + evaluation infrastructure (SkillsBench, ClawsBench, runtime)
- open_source: yes [confirmed], source: https://github.com/benchflow-ai/benchflow (accessed 2026-06-07)
- license: Apache-2.0 [confirmed], source: https://github.com/benchflow-ai/benchflow (accessed 2026-06-07)
- deployment_model: Self-hosted open-source runtime (CLI + Python SDK; sandboxes via Docker, Daytona, Modal, AWS AgentCore, Vertex); hosted public benchmark sites and leaderboards [confirmed], source: https://github.com/benchflow-ai/benchflow (accessed 2026-10-05)
- maturity: research preview / early-stage (multiple open benchmarks and an OSS runtime released frequently at v0.7.x; PostTrain Arena runs paused; very small team; funding status unclear) [estimated], source: https://github.com/benchflow-ai/benchflow/releases (accessed 2026-10-05)
- notable_customers: Google DeepMind (self-claimed (internal SkillsBench use); separately, GDM presented the SkillsBench launch event (verified on Luma), frontier-lab tie); Meta (self-claimed, frontier-lab tie); Microsoft (self-claimed, frontier-lab tie); Tencent (self-claimed); Zhipu (GLM) (self-claimed); MiniMax (self-claimed) [reported], source: https://www.benchflow.ai/news/skillsbench-paper (accessed 2026-10-05)
- founding_team:
  - Xiangyi Li, Founder & CEO, Previously worked on coding-model inference and inference at Tesla. First author of SkillsBench and ClawsBench; co-organizer of Agent Skills '26.
  - Kirk Xu, Co-founder (reported; listed by Founders, Inc.), Named as a founder only on the Founders, Inc. portfolio page. Not shown on BenchFlow's own About page. A GitHub account 'kirk-xuhj' has 17 commits to the runtime. Background unknown.
  - Moritz Wallawitsch, Former early co-founder (reported departure Feb 2025), Listed as an early co-founder who left in February 2025, per the startupintros aggregator only. No longer with the company.
- recent_developments (last 6 months):
  - 2026-09-21 [product] Robo Use released on PyPI: 'agent as a policy for embodied agents', source: https://pypi.org/project/robouse/
  - 2026-09-15 [open-source] BenchFlow runtime v0.7.x: sandbox egress hardening, rubric reviews, hf:// dataset sources, AgentCore VPC and Vertex sandboxes, source: https://github.com/benchflow-ai/benchflow/releases
  - 2026-08-15 [partnership] BenchFlow partners on re:AGENT, an agentic-science build weekend presented with Anthropic and Arc Institute, source: https://luma.com/g6org075
  - 2026-07-02 [research] env0: BenchFlow SFT on simulated workspaces lifts Qwen3.5-9B on SkillsBench v1.1 by 69.4% (relative), source: https://www.benchflow.ai/news/improving-skillsbench-with-env0
  - 2026-07 [benchmark] GenesisBench: BenchFlow benchmark on LLM agents improving robot policies, source: https://genesisbench.benchflow.ai
  - 2026-06 [product] PostTrain Arena: BenchFlow's open arena that scores contributed environments by the post-training lift they produce, source: https://posttrain.com
  - 2026-05-30 [partnership] BenchFlow lists a Google DeepMind Enterprise Build Day at AGI House among its events (self-reported), source: https://www.benchflow.ai/
  - 2026-05-27 [partnership] SkillsBench 1.x launch party in SF, presented by Google DeepMind with Kaggle, source: https://luma.com/deepmind-634c
  - 2026-05-26 [research] BenchFlow co-organizes Agent Skills '26, the first agent-skills workshop, at ACM CAIS (San Jose), source: https://www.agentskills-workshop.org/
  - 2026-04-07 [research] ClawsBench paper released on arXiv (2604.05172), source: https://arxiv.org/abs/2604.05172
- benchmarks_and_research:
  - Robo Use (environment), 2026-09, https://robouse.ai
  - env0 SkillsBench/ClawsBench SFT study (paper), 2026-07-02, SkillsBench v1.1 rose from 7.01% to 11.88% (+69.4% relative). ClawsBench rose from 10.56% to 13.89% (+31.6% relative)., https://www.benchflow.ai/news/improving-skillsbench-with-env0
  - GenesisBench (benchmark), 2026-07, https://genesisbench.benchflow.ai
  - awesome-evals (dataset), 2026-06-24, https://github.com/benchflow-ai/awesome-evals
  - PostTrain Arena (leaderboard), 2026-06, https://posttrain.com
  - ClawsBench (benchmark), 2026-04-06, With full scaffolding, agents reach 39-64% task success but take unsafe actions 7-33% of the time (6 models, 4 harnesses). Without scaffolding, all six models score 0-8%., https://clawsbench.benchflow.ai
  - SkillsBench (benchmark), 2026-02-13, Paper v4: curated skills raise the average pass rate from 33.9% to 50.5% (+16.6 pp). v1.1 leaderboard: GPT-5.5 + OpenHands leads at 67.3% with skills vs 51.5% without (skillsbench.ai)., https://skillsbench.ai
  - FrontierPhysics (benchmark), 2026, Claude Opus 5.5 leads at 22.0% overall pass rate, ahead of GPT-6 Astra at 14.9% (BenchFlow-run leaderboard)., https://www.benchflow.ai/frontierphysics
- products:
  - BenchFlow Runtime (open source): Apache-2.0 'Research infra for creating RL environments, post-training, and evals'. Supports single-agent, multi-agent (for example coder plus reviewer, or a simulated user) and multi-round runs. Sandboxes run on Docker, Daytona, Modal, AWS AgentCore and Vertex. Tasks use the native task.md package format. Includes a CLI and Python SDK, sandbox hardening to protect verifiers and hidden task data, and LLM rubric scoring. Latest release v0.7.8 (2026-09-15); about 353 stars., https://github.com/benchflow-ai/benchflow
  - SkillsBench: Agent-skills benchmark and leaderboard (v1.1)., https://skillsbench.ai
  - ClawsBench: Simulated-workspace capability and safety benchmark., https://clawsbench.benchflow.ai
  - env0: High-fidelity simulated Google Workspace and Slack services, used to generate post-training trajectories., https://www.benchflow.ai/news/improving-skillsbench-with-env0
  - PostTrain Arena: Community environment-contribution and post-training arena, supported by Hugging Face OpenEnv. Runs are currently paused., https://posttrain.com
  - Robo Use: Embodied-agent environments ('agent as a policy for embodied agents'); pip install robouse., https://robouse.ai
  - GenesisBench: Benchmark of language agents improving robot policies across simulators., https://genesisbench.benchflow.ai
  - FrontierPhysics: End-to-end physics-research agent benchmark and leaderboard (v0.1)., https://www.benchflow.ai/frontierphysics
- notes_for_researchers:
  - Two arXiv papers, both first-authored by founder Xiangyi Li: SkillsBench (2602.12670, Feb 2026, v4 Jun 2026; +16.6 pp from curated skills) and ClawsBench (2604.05172, Apr 2026).
  - SkillsBench and ClawsBench datasets are on Hugging Face (benchflow/SkillsBench, benchflow/ClawsBench). The SkillsBench repo has about 1.8k stars.
  - BenchFlow says SkillsBench is cited in the Qwen 3.6 Max model card. skillsbench.ai also points to citations in Tencent Hunyuan and Huawei openPangu model cards. These are self-reported; we did not check the model cards.
  - July 2026 env0 study: teacher-student SFT on simulated-workspace trajectories gave +69.4% relative on SkillsBench for Qwen3.5-9B, for $10.67 of H100 compute. The model is public at huggingface.co/benchflow/benchflow-qwen35-9b.
  - Moving into embodied and science evals: Robo Use (PyPI, Sep 2026), GenesisBench (Jul 2026), FrontierPhysics v0.1, and partner of the re:AGENT agentic-science build weekend (Aug 2026).
- notes_for_program_managers:
  - Very small team. The About page lists 3 people: the founder plus two Members of Technical Staff (a Berkeley quantum-physics PhD and a Cornell CS student). Founders, Inc. also lists a team size of 3. Paper author lists and GitHub contributors show a wider academic collaborator network.
  - Frontier-lab ties are event-level only. Google DeepMind presented the SkillsBench launch party (with Kaggle), and Anthropic was a presenter of the re:AGENT event where BenchFlow was a partner. BenchFlow self-claims internal SkillsBench use at Google DeepMind, Meta, Microsoft, Tencent, GLM and MiniMax; this is unverified, and no commercial contracts are disclosed.
  - BenchFlow names Prime Intellect, Fireworks and Hugging Face (OpenEnv) as compute and ecosystem partners.
  - Funding is unclear. Aggregators report a $1M seed in Jan 2025 (YC, Pear, Construct, FAST/GETTYLAB, Ankit Jain), but the founder posted on X on 2026-06-25: 'We didn't even raise a seed.' Founders, Inc. is the only investor confirmed on the investor's own page. BenchFlow has no YC directory page. No priced round has been announced.
  - Mainly community and open-source driven: hackathons with 600+ participants and more than $100K in prize pools (Mar 2026), plus workshops and build weekends. The commercial engagement model is not publicly described.
- notes_for_data_ops:
  - The runtime is Apache-2.0 with a CLI and Python SDK. Sandboxes run on Docker, Daytona, Modal, AWS AgentCore (with VPC support) and Vertex. The README lists Claude Code, Codex, Gemini, Goose, Qwen-Code and Prime-Agent as agents; release notes also mention OpenCode.
  - Native task package is task.md with environment and verifier sidecars. PostTrain Arena tasks are Docker env + verifier + oracle packages, submitted as public Hugging Face datasets.
  - Recent releases (Aug-Sep 2026) add URL-denylist egress enforcement, weighted rubric reviews, hf:// dataset sources, and trajectory upload with redaction and rate limiting.
  - Note the mixed licenses: the runtime is Apache-2.0, but the posttrainarena, env0 and artifacts repos are AGPL-3.0 and GenesisBench is GPL-3.0.
  - No SOC 2, certifications or trust page found.
- sources:
  - https://www.benchflow.ai/ (accessed 2026-10-05)
  - https://www.benchflow.ai/about (accessed 2026-10-05)
  - https://www.benchflow.ai/assets/team/xiangyi-li.webp (accessed 2026-10-05)
  - https://www.benchflow.ai/news (accessed 2026-10-05)
  - https://www.benchflow.ai/news/improving-skillsbench-with-env0 (accessed 2026-10-05)
  - https://www.benchflow.ai/news/skillsbench-paper (accessed 2026-10-05)
  - https://www.benchflow.ai/news/clawsbench-paper (accessed 2026-10-05)
  - https://www.benchflow.ai/news/agent-skills-26-workshop (accessed 2026-10-05)
  - https://www.benchflow.ai/news/agent-skills-hackathon-ii (accessed 2026-10-05)
  - https://www.benchflow.ai/research (accessed 2026-10-05)
  - https://www.benchflow.ai/frontierphysics (accessed 2026-10-05)
  - https://www.benchflow.ai/docs/benchflow (accessed 2026-10-05)
  - https://arxiv.org/abs/2604.05172 (accessed 2026-10-05)
  - https://arxiv.org/abs/2602.12670 (accessed 2026-10-05)
  - https://skillsbench.ai/ (accessed 2026-10-05)
  - https://luma.com/deepmind-634c (accessed 2026-10-05)
  - https://luma.com/g6org075 (accessed 2026-10-05)
  - https://www.agentskills-workshop.org/ (accessed 2026-10-05)
  - https://posttrain.com (accessed 2026-10-05)
  - https://github.com/benchflow-ai/posttrainarena (accessed 2026-10-05)
  - https://robouse.ai (accessed 2026-10-05)
  - https://pypi.org/project/robouse/ (accessed 2026-10-05)
  - https://genesisbench.benchflow.ai (accessed 2026-10-05)
  - https://github.com/benchflow-ai/GenesisBench (accessed 2026-10-05)
  - https://github.com/benchflow-ai (accessed 2026-10-05)
  - https://github.com/benchflow-ai/benchflow (accessed 2026-10-05)
  - https://github.com/benchflow-ai/benchflow/releases (accessed 2026-10-05)
  - https://f.inc/portfolio/benchflow/ (accessed 2026-10-05)
  - https://x.com/xdotli/status/2070256643930255519 (accessed 2026-10-05)
  - https://www.ycombinator.com/companies/benchflow (accessed 2026-10-05)
  - https://startupintros.com/orgs/benchflow (accessed 2026-10-05)
  - https://tracxn.com/d/companies/benchflow/__n6eEhuEioOF6iyeia5-JulQQPuB9g_ddBiUaBZ1DuxQ/funding-and-investors (accessed 2026-10-05)
  - https://docs.benchflow.ai/introduction (accessed 2026-06-07)
  - https://github.com/benchflow-ai/skillsbench (accessed 2026-06-07)
  - https://www.linkedin.com/company/benchflow-ai (accessed 2026-06-07)
  - https://www.linkedin.com/in/l1xiangyi/ (accessed 2026-06-07)
  - https://www.inverse.com/tech/building-ais-testing-ground-benchflows-mission-as-explained-by-xiangyi-li (accessed 2026-06-07)
  - https://news.ycombinator.com/item?id=43440893 (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/benchflow (accessed 2026-06-07)
  - https://pitchbook.com/profiles/company/711737-02 (accessed 2026-06-07)

## Refresh, rank #16
- slug: refresh
- segment: Commercial vendors
- website: https://www.refresh.dev
- focus_areas: Coding, Computer Use, Private Codebases
- positioning: Refresh (YC X25; appears to have grown out of Operative.sh) builds RL environments with verifiable rewards for coding, computer use (high-fidelity clones of software suites such as EHRs and enterprise apps) and 3D animation and simulation in Blender. Data ships in Harbor format. Public releases include BlenderBench, the Gauntlet 4K RLVR dataset (access by request), the open-source computer-1 harness in Harbor, and the trajectories.sh trajectory-sharing platform.
- best_fit: Frontier labs needing custom RL training environments and datasets for software-engineering and computer-use agent capabilities.
- overall_confidence: medium
- founded_year: 2025 [confirmed], source: https://www.ycombinator.com/companies/refresh accessed 2026-06-07 (YC Spring 2025 / X25, founded 2025); https://www.linkedin.com/company/refresh-dot-dev accessed 2026-06-07
- status: active [confirmed], source: https://www.refresh.dev/ accessed 2026-06-07; https://www.ycombinator.com/companies/refresh accessed 2026-06-07
- hq_location: San Francisco, CA, USA [confirmed], source: https://www.ycombinator.com/companies/refresh accessed 2026-06-07; https://www.linkedin.com/company/refresh-dot-dev accessed 2026-06-07
- other_locations: unknown [unknown]
- distributed_remote: no [estimated], source: https://www.refresh.dev/careers accessed 2026-06-07 (roles listed San Francisco, in-person)
- current_headcount: ~8 employees (as of 2026-06-07) [reported], source: https://www.ycombinator.com/companies/refresh accessed 2026-06-07 (lists 8 employees); https://www.linkedin.com/company/refresh-dot-dev accessed 2026-06-07 (public snippet ~7)
- headcount_band: 1-10 [confirmed], source: https://www.ycombinator.com/companies/refresh accessed 2026-06-07 (8 employees); https://www.linkedin.com/company/refresh-dot-dev accessed 2026-06-07
- headcount_growth: unknown [unknown]
- open_roles_count: 4 [reported], source: https://www.refresh.dev/careers (accessed 2026-10-05)
- has_researchers: yes [reported], source: https://www.ycombinator.com/launches/Ncy-refresh-turning-real-work-into-rl-training-grounds accessed 2026-06-07 (mechanistic interpretability work); https://www.refresh.dev/careers accessed 2026-06-07 (Research Engineer, Benchmarking role)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Christopher Settles (CEO): led evaluation for Uber's generative-AI platform and shipped its first computer-use agents; ex-ML at Uber; CS at UIUC; Erik Quintanilla (CTO): Amazon production-ops automation and web-scraping patents, Capital One platform modernization, ML at LynkAI; designs verifiers and post-training harnesses; Sid Santbakshsing (Head of Operations): former founder; leads delivery, domain-expert recruiting, gold-trajectory recording and verifier QA [confirmed], source: https://www.refresh.dev/about (accessed 2026-10-05)
- published_papers_or_benchmarks: BlenderBench (2026-09-22) https://www.refresh.dev/blog/blenderbench; Gauntlet 4K RLVR dataset, access by request (2026-02-17) https://www.refresh.dev/blog/gauntlet-4k-rlvr; computer-1 computer-use harness in Harbor (2026-07-30) https://www.refresh.dev/blog/trajectories-and-computer-1 [confirmed], source: https://www.refresh.dev/blog (accessed 2026-10-05)
- total_raised: unknown [unknown], source: YC-backed (X25). No total disclosed in any primary source; an unverified ~$500K pre-seed figure surfaced in a LinkedIn snippet only, not confirmed.
- last_round: unknown [unknown], source: YC investment implied by batch participation; specific round stage/amount/date not in any primary source as of 2026-06-07.
- notable_investors: Y Combinator [reported], source: https://www.ycombinator.com/companies/refresh accessed 2026-06-07 (YC X25 batch participation). Reported angel/Weekend Fund participation could not be confirmed against any primary source.
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: unknown [unknown], source: No security/trust page found; https://www.refresh.dev/security returns 404 (accessed 2026-06-07)
- other_certifications: unknown [unknown]
- security_page: unknown [unknown], source: https://www.refresh.dev/security returns 404 (accessed 2026-06-07)
- what_they_sell: environments [confirmed], source: https://www.refresh.dev/ accessed 2026-06-07 (simulation engines for coding and computer use); https://www.ycombinator.com/companies/refresh accessed 2026-06-07
- open_source: partial [confirmed], source: https://github.com/refreshdotdev (accessed 2026-10-05)
- license: Open-source repos: Apache-2.0 (web-eval-agent; computer-1 within Apache-2.0 Harbor) and MIT (proactive-engineer); commercial datasets and environments proprietary [confirmed], source: https://github.com/refreshdotdev (accessed 2026-10-05)
- deployment_model: Datasets and environments delivered to labs in Harbor format (Docker); hosted SaaS for trajectories.sh [reported], source: https://www.refresh.dev/blog/gauntlet-4k-rlvr (accessed 2026-10-05)
- maturity: early commercial (bespoke lab engagements; public trajectories.sh platform launched 2026-07) [estimated], source: https://www.refresh.dev/blog/trajectories-and-computer-1 (accessed 2026-10-05)
- notable_customers: Frontier AI labs (unnamed) (self-claimed, frontier-lab tie) [reported], source: https://www.refresh.dev/ accessed 2026-06-07 (self-described partnerships with 'frontier labs and enterprises'); https://www.ycombinator.com/launches/Ncy-refresh-turning-real-work-into-rl-training-grounds accessed 2026-06-07 (self-claimed). No specific customer named or third-party verified.
- founding_team:
  - Christopher Settles, Co-founder & CEO, Led evaluation for Uber's generative-AI platform and shipped its first computer-use agents (about page). ex-ML at Uber, CS at UIUC (YC profile). At Refresh he directs world generation and fidelity assurance for the computer-use environments.
  - Erik Quintanilla, Co-founder & CTO, Built automation behind Amazon's production operations and filed web-scraping patents there, then led platform modernization at Capital One and ML at LynkAI. At Refresh he designs verifiers and post-training harnesses and coordinates Blender reference creation.
  - Sid Santbakshsing, Head of Operations (not a founder), Previously founded a company. At Refresh he leads delivery across engagements: recruits domain experts, coordinates human gold-trajectory recordings, and manages verifier QA and graded model runs.
- recent_developments (last 6 months):
  - 2026-09-22 [benchmark] Refresh publishes BlenderBench, a 20-task benchmark of Blender 3D workflows, source: https://www.refresh.dev/blog/blenderbench
  - 2026-07-30 [product] Refresh launches trajectories.sh and the open-source computer-1 harness in Harbor, source: https://www.refresh.dev/blog/trajectories-and-computer-1
  - 2026-06-27 [open-source] Operative.sh-era web-eval-agent MCP server archived, source: https://github.com/refreshdotdev/web-eval-agent
- benchmarks_and_research:
  - BlenderBench (benchmark), 2026-09-22, GPT-6 Astra mean reward 0.686 (7/20 tasks at 0.90 or above); Claude Opus 5.5 0.642 (3/20). Vendor-run., https://www.refresh.dev/blog/blenderbench
  - computer-1 (Harbor computer-use agent harness) (environment), 2026-07-30, https://github.com/harbor-framework/harbor/tree/main/src/harbor/agents/computer_1
  - Gauntlet 4K RLVR (dataset), 2026-02-17, Per Refresh's post: rnj-1-instruct 8B went from 3.4% to 10.1% pass@1 on Terminal-Bench 2.0 (3 to 9 problems solved)., https://www.refresh.dev/blog/gauntlet-4k-rlvr
- products:
  - Computer-use software worlds: High-fidelity clones of full software suites that connect multiple applications, 'from EHRs to enterprise apps', for training and evaluating computer-use agents with verifiable rewards., https://www.refresh.dev/
  - 3D animation & simulation environments (Blender): Blender environments for scene building, materials, character rigging, animation keyframing and physics simulation. BlenderBench is built on them., https://www.refresh.dev/
  - Coding task environments / datasets: Software-engineering tasks aimed at GDPval-style economically valuable work, with verifiers, delivered in Harbor format (e.g. Gauntlet 4K RLVR)., https://www.refresh.dev/
  - trajectories.sh: Hosted platform and CLI (trajectories-sh) for uploading, replaying and sharing Harbor agent trajectories, with private, unlisted (default) and public visibility., https://www.trajectories.sh
  - computer-1: Open-source computer-use agent harness in the Harbor framework., https://github.com/harbor-framework/harbor/tree/main/src/harbor/agents/computer_1
- notes_for_researchers:
  - BlenderBench (2026-09-22): 20 Blender tasks. Vendor-run results: GPT-6 Astra 0.686 vs Claude Opus 5.5 0.642 mean reward.
  - Gauntlet 4K RLVR post (2026-02-17) gives a full training recipe: Harbor + SkyRL rollouts, DAPO (GRPO was unstable), LoRA rank 64, 8x A100 for 80 hours. Refresh reports rnj-1-instruct 8B went from 3.4% to 10.1% pass@1 on Terminal-Bench 2.0.
  - computer-1 is a model-agnostic computer-use harness in the open-source Harbor repo. Trajectories can be shared via trajectories.sh.
- notes_for_program_managers:
  - Small team: YC lists 8 people. The about page names 3 leaders (CEO, CTO, Head of Operations).
  - No funding disclosed beyond YC (X25). Frontier-lab and enterprise customers are self-claimed and unnamed.
  - In 2026 the scope grew from coding and computer use to enterprise software clones (EHRs) and 3D/Blender environments.
  - Appears to have grown out of Operative.sh (inferred from the archived web-eval-agent README; not formally stated).
  - Careers page lists 3 SF engineering roles (Ex-Founder, GTM Engineer for frontier-lab pilots, Research Engineer for benchmarking) plus an open-ended 'create your own role'.
- notes_for_data_ops:
  - Data ships in Harbor format with Docker environments and pytest binary-reward verifiers (stated for Gauntlet 4K RLVR).
  - trajectories.sh uploads use Harbor's ATIF job format, via a CLI (npx trajectories-sh upload ...).
  - No security or trust page (refresh.dev/security returns 404). No SOC 2 stated.
  - Public GitHub: github.com/refreshdotdev. Most active repos are forks: harbor, harbor-mm and harbor-terminus fork harbor-framework/harbor, and long-horizon forks abundant-ai/swe-marathon. Original repos include web-eval-agent (archived) and proactive-engineer (MIT).
- sources:
  - https://www.refresh.dev/ (accessed 2026-10-05)
  - https://www.refresh.dev/about (accessed 2026-10-05)
  - https://www.refresh.dev/images/chris_headshot.png (accessed 2026-10-05)
  - https://www.refresh.dev/images/erik-headshot.png (accessed 2026-10-05)
  - https://www.refresh.dev/images/sid.png (accessed 2026-10-05)
  - https://www.refresh.dev/blog (accessed 2026-10-05)
  - https://www.refresh.dev/blog/blenderbench (accessed 2026-10-05)
  - https://www.refresh.dev/blog/trajectories-and-computer-1 (accessed 2026-10-05)
  - https://www.refresh.dev/blog/gauntlet-4k-rlvr (accessed 2026-10-05)
  - https://www.trajectories.sh (accessed 2026-10-05)
  - https://github.com/refreshdotdev (accessed 2026-10-05)
  - https://github.com/refreshdotdev/web-eval-agent (accessed 2026-10-05)
  - https://github.com/refreshdotdev/long-horizon (accessed 2026-10-05)
  - https://github.com/refreshdotdev/harbor-mm (accessed 2026-10-05)
  - https://github.com/harbor-framework/harbor/tree/main/src/harbor/agents/computer_1 (accessed 2026-10-05)
  - https://www.refresh.dev/careers (accessed 2026-10-05)
  - https://www.ycombinator.com/companies/refresh (accessed 2026-10-05)
  - https://www.ycombinator.com/launches/Ncy-refresh-rl-training-gyms (accessed 2026-10-05)
  - https://www.refresh.dev/security (accessed 2026-10-05)
  - https://www.withrefresh.com/ (accessed 2026-06-07)
  - https://www.ycombinator.com/launches/Ncy-refresh-turning-real-work-into-rl-training-grounds (accessed 2026-06-07)
  - https://www.linkedin.com/company/refresh-dot-dev (accessed 2026-06-07)
  - https://www.linkedin.com/in/erikquintanilla/ (accessed 2026-06-07)
  - https://www.linkedin.com/in/christopher-settles (accessed 2026-06-07)

## Matrices, rank #17
- slug: matrices
- segment: Commercial vendors
- website: https://matrices.ai
- focus_areas: Computer Use
- positioning: Matrices builds reinforcement-learning training environments for frontier AI labs to train agents that use computers and browsers like humans, described as a 'gamified replica of the internet' where thousands of agents learn via RL. The company frames its mission as 'towards self-driving computers' and says it helps labs train computer-use agents (Operator-class systems). Note: this is the correct browser-native entity (matrices.ai / LinkedIn 'matricesapp'), distinct from the similarly named 'Matrice.ai' computer-vision company and 'Matrix AI Network' blockchain project.
- best_fit: A frontier lab needing large-scale, realistic browser/computer-use RL environments to train and evaluate web-navigating agents.
- overall_confidence: medium
- founded_year: 2023 [reported], source: https://www.linkedin.com/company/matricesapp ; https://tracxn.com/d/companies/matrices/ (accessed 2026-06-07)
- status: active [reported], source: https://www.linkedin.com/company/matricesapp (accessed 2026-06-07); https://theinflectionpoint.ai/p/building-a-fake-internet-for-ai-agents (accessed 2026-06-07)
- hq_location: San Francisco, California, USA [reported], source: https://www.linkedin.com/company/matricesapp (accessed 2026-06-07); https://tracxn.com/d/companies/matrices/ (accessed 2026-06-07)
- other_locations: unknown [unknown]
- distributed_remote: unknown [unknown]
- current_headcount: ~12-20 employees (sources conflict: LinkedIn shows 14 associated employees on 2026-10-05; Tracxn 19 as of 2026-05-31; Caplight 12 current, with earlier 2026 signals of 27-32) [reported], source: https://www.linkedin.com/company/matricesapp (accessed 2026-10-05)
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/matricesapp (size band 11-50); https://tracxn.com/d/companies/matrices/ (accessed 2026-06-07)
- headcount_growth: unknown [unknown], source: Note: theinflectionpoint.ai article described team as 'still just 3 people' at an earlier date; current LinkedIn/Tracxn show ~19-21, implying growth, but no quantified period available
- open_roles_count: unknown [unknown], source: https://matrices.ai/careers (accessed 2026-06-07), actively hiring noted but exact count not retrievable (JS-rendered page)
- has_researchers: unknown [unknown]
- researcher_count: unknown [unknown]
- researcher_backgrounds: Co-founder John Qian: UIUC; Co-founder Leonardo Axel Setyanto: UT Austin, ex-Loku. His LinkedIn now lists OpenAI as current company (undated); his ongoing role at Matrices is unverified; Early co-founder Dina Y.: background unknown [reported], source: https://www.linkedin.com/in/axelsetyanto/ (accessed 2026-10-05)
- published_papers_or_benchmarks: unknown [unknown]
- total_raised: $5M (Seed, 2024-03; reported by Caplight and theinflectionpoint.ai 2025-07-22; no new round found through 2026-10-05) [reported], source: https://www.caplight.com/company/matrices (accessed 2026-10-05)
- last_round: Seed, $5M, 2024-03 (reported; date and amount per Caplight; amount also in a 2025-07-22 third-party article) [reported], source: https://www.caplight.com/company/matrices (accessed 2026-10-05)
- notable_investors: AI Grant (Batch 3; confirmed on aigrant.com); Index Ventures (reported); Naval Ravikant (reported, single source) [reported], source: https://aigrant.com/ (accessed 2026-10-05)
- valuation: unknown [unknown]
- revenue_signals: Self-claimed '7-figure contracts with multiple/top AI labs'; no verified figure [reported], source: https://theinflectionpoint.ai/p/building-a-fake-internet-for-ai-agents ; matrices.ai/careers snippet (accessed 2026-06-07), vendor self-claim, labs unnamed
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: unknown [unknown]
- what_they_sell: environments [confirmed], source: https://matrices.ai/ (title: 'Matrices - Training Environments for LLM Agents'); https://www.linkedin.com/company/matricesapp (tagline 'Towards self-driving computers'); https://theinflectionpoint.ai/p/building-a-fake-internet-for-ai-agents (accessed 2026-06-07)
- open_source: no [estimated], source: No public GitHub org or OSS product found as of 2026-06-07
- license: unknown [unknown]
- deployment_model: managed-hosted (inferred); commercial access is contact-only with no self-serve plans or published pricing [estimated], source: https://www.therundown.ai/tools/matrices (accessed 2026-10-05)
- maturity: unknown [unknown]
- notable_customers: Unnamed frontier AI labs (described as signing 7-figure contracts; agents like OpenAI 'Operator' referenced as the type they help train) (self-claimed) [reported], source: https://matrices.ai/careers (snippet) ; https://theinflectionpoint.ai/p/building-a-fake-internet-for-ai-agents (accessed 2026-06-07), labs not named; no specific frontier lab confirmed as a customer
- founding_team:
  - John Qian, Co-Founder, University of Illinois Urbana-Champaign. Former Weights & Biases colleague Ramit Goolry later joined as founding engineer. Both the LinkedIn company page and Tracxn list Qian as a founder.
  - Leonardo Axel Setyanto, Co-Founder (listed as CTO by Caplight; current status at Matrices unverified), Attended UT Austin (2010-2012) and was previously an engineer at the startup Loku. His LinkedIn top card now lists OpenAI as his current company, with no date shown, while his GitHub still lists @matrices-app. He may have moved to OpenAI; this is reported but not confirmed.
- recent_developments (last 6 months):
  - 2026-04-29 [other] Matrices listed in Specter Monitor #225 'Investor Signals of the Week' (featured 2x in 90 days), source: https://insights.tryspecter.com/specter-monitor-225/
  - 2026-04-05 [other] Aggregator signal: headcount up 42% quarter over quarter, 19 to 27 (Caplight), source: https://www.caplight.com/company/matrices
- products:
  - Matrices simulated computer-use training environments: Simulated computer and website environments for training and evaluating multimodal LLM agents on realistic computer-use tasks. They offer repeatable starting states and task-level feedback for RL. The team says its broader goal is to recreate the parts of the internet and the workflows that agents need to practice on, and it runs many agent attempts at high throughput. Access for AI labs and agent teams is arranged directly with the company: there are no self-serve plans and no published pricing (third-party listing, last reviewed 2026-08-30). A 2025 article described the product as a 'gamified replica of the internet'. All capability statements here are self-described., https://www.therundown.ai/tools/matrices
  - Level editor (roadmap item): A third-party listing describes this as a roadmap item that would let outside contributors build simulated tasks. It has no launch date and no published terms., https://www.therundown.ai/tools/matrices
- notes_for_researchers:
  - No papers, benchmarks, datasets or original open-source repos by Matrices were found. The matrices-app GitHub org contains only forks and test repos. The approach is described only in marketing terms: a simulated or recreated internet with repeatable start states and task-level RL feedback for multimodal computer-use agents.
  - A 'level editor' for contributor-built tasks is described as a roadmap item. It has not been released.
- notes_for_program_managers:
  - Seed stage: $5M reported, dated 2024-03 by Caplight. AI Grant is confirmed as an investor; Index Ventures is reported; Naval Ravikant comes from a single source. No new round was found through 2026-10-05.
  - Frontier-lab ties are self-claimed only ('7-figure contracts with leading AI labs', July 2025), and no lab is named.
  - Key-person risk: co-founder/CTO Leonardo Axel Setyanto's LinkedIn now lists OpenAI as his current company. Ask the vendor to confirm who leads the team today.
  - Access is by direct contact only, with no self-serve tier and no public pricing.
- notes_for_data_ops:
  - No public API docs, integration formats, security page or SOC 2 statement were found.
  - matrices.ai, its careers page and matrices.app all returned HTTP 503 to automated fetches on 2026-10-05, so first-party site content could not be checked.
- sources:
  - https://www.caplight.com/company/matrices (accessed 2026-10-05)
  - https://aigrant.com/ (accessed 2026-10-05)
  - https://www.therundown.ai/tools/matrices (accessed 2026-10-05)
  - https://www.linkedin.com/company/matricesapp (accessed 2026-10-05)
  - https://www.linkedin.com/in/axelsetyanto/ (accessed 2026-10-05)
  - https://www.linkedin.com/in/qianjohn/ (accessed 2026-10-05)
  - https://github.com/laxels (accessed 2026-10-05)
  - https://tracxn.com/d/companies/matrices/__ZJZYPaHQVI36aOdqiIFAQCzYo2o_oy7cIBgidpUfYPM (accessed 2026-10-05)
  - https://insights.tryspecter.com/specter-monitor-225/ (accessed 2026-10-05)
  - https://theinflectionpoint.ai/p/building-a-fake-internet-for-ai-agents (accessed 2026-10-05)
  - https://matrices.ai/ (accessed 2026-10-05)
  - https://matrices.ai/careers (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/matrices-f9d0 (accessed 2026-06-07)
  - https://tracxn.com/d/companies/matrices/ (accessed 2026-06-07)
  - https://rocketreach.co/matrices-management_b6cf945ec78254fc (accessed 2026-06-07)
  - https://techcrunch.com/2025/09/21/silicon-valley-bets-big-on-environments-to-train-ai-agents/ (accessed 2026-06-07)

## Andromede, rank #18
- slug: andromede
- segment: Commercial vendors
- website: https://andromede.ai/
- focus_areas: Long-Horizon
- positioning: Andromede is an early-stage RL data lab (reported as founded 2025 in Lausanne, backed by Unusual Ventures) that programmatically generates RL environments, tasks and verifiers from real-world data for post-training and evaluating long-horizon frontier agents. Its core product is still in private beta with unnamed partners. Since mid-2026 the EPFL-linked team, co-founded by Alexandre Sallinen and Guillaume Allègre, has published open research: the Messier cross-benchmark agent-evaluation corpus (arXiv 2607.25891; self-reported EMNLP 2026 Main), a paper on predicting task difficulty without rollouts, and the Game of Agents multi-agent testbed (ICML 2026 workshop).
- best_fit: Buyers needing custom RL environments and verifiers derived from real-world data for post-training/evaluating long-horizon agentic models.
- overall_confidence: medium
- founded_year: 2025 [reported], source: https://pitchbook.com/profiles/company/1158131-35 (search snippet; PitchBook page returns HTTP 403, accessed 2026-06-07); corroborated by https://www.unusual.vc/portfolio/ (accessed 2026-06-07)
- status: active [confirmed], source: https://andromede.ai/ (accessed 2026-06-07)
- hq_location: Lausanne, Switzerland [reported], source: https://pitchbook.com/profiles/company/1158131-35 (search snippet; page returns HTTP 403, accessed 2026-06-07)
- other_locations: New York, USA (founder/president Guillaume Allegre is LinkedIn-listed as New York-based; no confirmed office) [estimated], source: https://www.linkedin.com/in/guillaume-allegre/ (search snippet, accessed 2026-06-07)
- distributed_remote: yes [estimated], source: https://www.linkedin.com/in/guillaume-allegre/ (accessed 2026-10-05)
- current_headcount: ~6 (6 members on the Andromede-AI Hugging Face org as of 2026-10-05; LinkedIn band 2-10) [estimated], source: https://huggingface.co/Andromede-AI (accessed 2026-10-05)
- headcount_band: 1-10 [reported], source: https://huggingface.co/Andromede-AI (accessed 2026-10-05)
- headcount_growth: unknown [unknown]
- open_roles_count: unknown [unknown], source: https://andromede.ai/careers returns HTTP 404; no LinkedIn jobs visible (accessed 2026-06-07)
- has_researchers: yes [confirmed], source: https://arxiv.org/html/2608.05797 (accessed 2026-10-05)
- researcher_count: ~4-6 publishing researchers (Sallinen, Krsteski, Meyer, O'Halloran, plus Allegre and Zhang as co-authors and HF org members) [estimated], source: https://huggingface.co/Andromede-AI (accessed 2026-10-05)
- researcher_backgrounds: Alexandre Sallinen (co-founder) - EPFL; Meditron medical-LLM and MMORE co-author; Guillaume Allegre (co-founder; Founder & President per LinkedIn) - ex-BCG X; MIT (ML & operations research) per LinkedIn snippet; Stefan Krsteski - Andromede AI / EPFL; lead author of Messier and the task-difficulty paper; Charlotte Meyer - Andromede AI / EPFL; co-author of Messier and the task-difficulty paper; Tony O'Halloran - lead author of House Rules / Game of Agents; co-author of Messier [reported], source: https://arxiv.org/html/2608.05797 (accessed 2026-10-05)
- published_papers_or_benchmarks: Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation (arXiv:2607.25891, 2026; self-reported EMNLP 2026 Main) - https://arxiv.org/abs/2607.25891; Predicting Task Difficulty Without Rollouts (arXiv:2608.05797, 2026; self-reported COLM 2026 WAB workshop) - https://arxiv.org/abs/2608.05797; House Rules: Institutional Design in Multi-Agent LLM Code Markets / Game of Agents (ICML 2026 Agents in the Wild workshop) - https://icml.cc/virtual/2026/67795 [confirmed], source: https://andromede-ai.github.io/messier/ (accessed 2026-10-05)
- total_raised: unknown [unknown], source: No disclosed funding figure found; PitchBook funding data paywalled (accessed 2026-06-07)
- last_round: unknown [unknown], source: No round stage/amount/date disclosed; PitchBook funding data paywalled (HTTP 403), accessed 2026-06-07
- notable_investors: Unusual Ventures [reported], source: https://www.unusual.vc/portfolio/ (Andromede listed in Unusual Ventures' own portfolio page, accessed 2026-06-07)
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: unknown [unknown], source: Site lists a security@andromede.ai contact but no SOC 2 / trust page found (accessed 2026-06-07)
- other_certifications: unknown [unknown]
- security_page: unknown [unknown], source: No /security or /trust page found; only security@andromede.ai contact listed (accessed 2026-06-07)
- what_they_sell: environments [confirmed], source: https://andromede.ai/ (accessed 2026-06-07)
- open_source: partial (research artifacts: Messier code/corpus under MIT, Game of Agents under Apache-2.0, OpenCode fork under MIT; the core environment-generation product is closed / private beta) [confirmed], source: https://github.com/Andromede-AI (accessed 2026-10-05)
- license: MIT (Messier code/corpus, with third-party benchmark material under its original terms; OpenCode fork inherits upstream MIT); Apache-2.0 (Game of Agents); core product proprietary [confirmed], source: https://huggingface.co/datasets/Andromede-AI/messier (accessed 2026-10-05)
- deployment_model: unknown [unknown]
- maturity: private beta [confirmed], source: https://andromede.ai/ (accessed 2026-06-07)
- notable_customers: unknown [unknown], source: Site states it works with 'a small set of partners' but names none (accessed 2026-06-07)
- founding_team:
  - Alexandre Sallinen, Co-founder, EPFL researcher whose Google Scholar affiliation is 'Andromede AI / EPFL'. Co-author of the Meditron medical LLMs (Meditron-70B, Llama-3-Meditron) and MMORE, and a member of the EPFL LLM and OpenMeditron HF orgs. Co-author of Messier and House Rules.
  - Guillaume Allègre, Co-founder, Ex-BCG X; MIT (machine learning and operations research), per the baseline's LinkedIn snippet. LinkedIn returned HTTP 999 on 2026-10-05 so this could not be re-checked. Co-author of the Messier and House Rules papers; HF user 'gandromede'. Not the Sciences Po economist of the same name.
- recent_developments (last 6 months):
  - 2026-10 [research] Messier to be presented at EMNLP 2026 Main Conference; difficulty paper at COLM 2026 Workshop on Agent Behavior (self-reported), source: https://github.com/Andromede-AI/messier
  - 2026-09-26 [open-source] Messier dataset publicly released on Hugging Face with 612 new evaluations, source: https://huggingface.co/datasets/Andromede-AI/messier
  - 2026-09-18 [open-source] Public OpenCode fork with configurable session retries 'for Andromede training', source: https://github.com/Andromede-AI/opencode
  - 2026-08-06 [research] 'Predicting Task Difficulty Without Rollouts' (arXiv:2608.05797), source: https://arxiv.org/abs/2608.05797
  - 2026-08-01 [research] Messier presented as a poster at the Berkeley RDI Summit (self-reported), source: https://huggingface.co/datasets/Andromede-AI/messier
  - 2026-07-28 [benchmark] Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation (arXiv:2607.25891), source: https://arxiv.org/abs/2607.25891
  - 2026-07-09 [open-source] Open-sourced the game-of-agents repo (Apache-2.0), source: https://github.com/Andromede-AI/game-of-agents
  - 2026-07 [research] ICML 2026 workshop poster: 'House Rules: Institutional Design in Multi-Agent LLM Code Markets' (Game of Agents), source: https://icml.cc/virtual/2026/67795
- benchmarks_and_research:
  - Predicting Task Difficulty Without Rollouts (paper), 2026-08-06, Token-level entropy is a useful difficulty signal, AUC can hide poor difficulty estimates, and prediction gaps flag contaminated or infeasible tasks., https://arxiv.org/abs/2608.05797
  - Messier (cross-benchmark agent evaluation corpus) (dataset), 2026-07-28, IRT capability scores correlate with Epoch's Evaluation Capability Index at Spearman rho = 0.84. Scoring rules can change measured performance and agent rankings., https://huggingface.co/datasets/Andromede-AI/messier
  - Game of Agents (House Rules) (environment), 2026-07, Removing public reviews cut 3-hour purchases about 5x (to ~2.5). Additive settlement produced about 2.6x more trades than net settlement (32.0 vs 12.2). Placement-based scoring raised extreme laddering about 25x over chance., https://github.com/Andromede-AI/game-of-agents
- products:
  - Andromede RL environment generation (private beta): Programmatic generation of RL environments, tasks and verifiers from real-world data for post-training and evaluating long-horizon agents. Access is by request (hello@andromede.ai)., https://andromede.ai/
  - Messier dataset (open): Open cross-benchmark agent evaluation corpus with Python loaders. Code is MIT; third-party material stays under its original terms., https://huggingface.co/datasets/Andromede-AI/messier
  - Game of Agents (open): Open, Apache-2.0 multi-agent LLM economy testbed., https://github.com/Andromede-AI/game-of-agents
- notes_for_researchers:
  - Three public research outputs since July 2026: Messier (arXiv:2607.25891, self-reported EMNLP 2026 Main), Predicting Task Difficulty Without Rollouts (arXiv:2608.05797, self-reported COLM 2026 WAB workshop poster) and House Rules / Game of Agents (ICML 2026 Agents in the Wild workshop poster).
  - Messier links tasks, verifiers, scoring rules, records and trajectories across benchmarks, which makes it usable for IRT-style capability estimation and verifier-sensitivity studies. Contributions of new benchmarks are invited through CONTRIBUTING.md.
  - The team is closely tied to EPFL: the difficulty-paper authors list joint Andromede AI / EPFL affiliations, and co-founder Sallinen comes from the Meditron medical-LLM work.
- notes_for_program_managers:
  - The product is still in private beta (homepage, 2026-10-05). There are no named customers or partners; the site only mentions 'a small set of partners'.
  - The only confirmed investor is Unusual Ventures (its portfolio page). No round amount, stage or date has been disclosed.
  - Ignore aggregator claims (e.g. Caplight) of a $1.5B valuation or a Paradigm-led Series A on 2026-03-18. Those belong to the unrelated GPU company Andromeda AI.
  - Team signal: the HF org lists 6 members (Sallinen, Allegre, Krsteski, Meyer, O'Halloran, Zhang), up from about 2 named people at baseline. Some may be EPFL collaborators rather than employees.
- notes_for_data_ops:
  - No published API, docs for the commercial product, integration format or SOC 2 / trust page. Security contact is security@andromede.ai.
  - Messier ships as five JSONL tables plus task files on Hugging Face (~12.2 GB), with loaders installed via uv. Some task files are left out where their licences do not allow redistribution.
  - A public OpenCode fork with configurable retry and backoff (headless Linux builds) is labelled 'for Andromede training'. This suggests an OpenCode-based agent harness internally, but no supported integration format is published.
- sources:
  - https://andromede.ai/ (accessed 2026-10-05)
  - https://www.unusual.vc/portfolio/ (accessed 2026-10-05)
  - https://github.com/Andromede-AI (accessed 2026-10-05)
  - https://github.com/Andromede-AI/messier (accessed 2026-10-05)
  - https://github.com/Andromede-AI/game-of-agents (accessed 2026-10-05)
  - https://github.com/Andromede-AI/opencode/blob/andromede/ANDROMEDE.md (accessed 2026-10-05)
  - https://arxiv.org/abs/2607.25891 (accessed 2026-10-05)
  - https://arxiv.org/abs/2608.05797 (accessed 2026-10-05)
  - https://arxiv.org/html/2608.05797 (accessed 2026-10-05)
  - https://andromede-ai.github.io/messier/ (accessed 2026-10-05)
  - https://huggingface.co/datasets/Andromede-AI/messier (accessed 2026-10-05)
  - https://huggingface.co/Andromede-AI (accessed 2026-10-05)
  - https://huggingface.co/alexsallinen (accessed 2026-10-05)
  - https://icml.cc/virtual/2026/67795 (accessed 2026-10-05)
  - https://scholar.google.com/citations?user=-ajWSEcAAAAJ (accessed 2026-10-05)
  - https://www.startuphub.ai/startups/andromede-ai (accessed 2026-10-05)
  - https://www.caplight.com/company/andromede-ai (accessed 2026-10-05)
  - https://raising.fi/news/andromeda-ai-undisclosed-march-2026 (accessed 2026-10-05)
  - https://www.linkedin.com/company/andromedeai (accessed 2026-06-07)
  - https://pitchbook.com/profiles/company/1158131-35 (accessed 2026-06-07)
  - https://alignlist.com/guides/top-40-rl-environments-startups-and-companies (accessed 2026-06-07)
  - https://aimultiple.com/rl-environments (accessed 2026-06-07)
  - https://andromede.ai/careers (accessed 2026-06-07)

## Vmax, rank #19
- slug: vmax
- segment: Commercial vendors
- website: https://vmax.ai/
- focus_areas: Coding, Long-Horizon
- positioning: Vmax is a San Francisco RL company and applied research lab, founded in 2025 by RL PhDs Augustine Mavor-Parker and Matthew Sargent and backed by South Park Commons and Race Capital. It aims to automate reinforcement learning by turning proprietary data and evals into new environments, with long-horizon agents as the target. Its 2026 research is about generating RL tasks and environments automatically: PopuLoRA (self-play curricula), unix-ctf (procedural shell CTF environments) and PROPEL (probe-guided task generation, with a Goodfire AI co-author). Earlier, with Martian, it released 1k Harbor-format JavaScript tasks.
- best_fit: Teams needing custom, research-grade RL environments to train coding and long-horizon shell/terminal agents from proprietary data.
- overall_confidence: medium
- founded_year: 2025 [reported], source: https://www.southparkcommons.com/companies/vmax/ (accessed 2026-06-07); corroborated by PitchBook (https://pitchbook.com/profiles/company/907262-38) and search aggregators, no primary incorporation record reviewed
- status: active [confirmed], source: https://vmax.ai/ (accessed 2026-06-07); https://job-boards.greenhouse.io/vmax actively hiring (accessed 2026-06-07)
- hq_location: San Francisco, USA [reported], source: https://www.linkedin.com/company/vmax-ai (public snippet: Brannan St, San Francisco, CA 94107; accessed 2026-06-07); https://www.southparkcommons.com/companies/vmax/
- other_locations: unknown [unknown]
- distributed_remote: unknown [unknown], source: All 8 Greenhouse roles are listed San Francisco (https://job-boards.greenhouse.io/vmax), suggests onsite/SF-based, but remote policy not stated
- current_headcount: ~11 (LinkedIn 'all 11 employees' link, 2026-06-07); official LinkedIn size band 2-10 [reported], source: https://www.linkedin.com/company/vmax-ai (accessed 2026-06-07). Note: one search source cited ~5 employees, so exact count is uncertain
- headcount_band: 1-10 [estimated], source: https://www.linkedin.com/company/vmax-ai (accessed 2026-06-07), official size band '2-10 employees'; ~11 surfaced via profile but within small-team range
- headcount_growth: unknown [unknown]
- open_roles_count: 8 [confirmed], source: https://api.ashbyhq.com/posting-api/job-board/vmax (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: https://vmax.ai/ (accessed 2026-06-07), founders are RL PhDs; unix-ctf paper (arXiv:2605.29115) authored by Vmax-affiliated researchers; multiple research/MTS roles open
- researcher_count: Small research-led team. Six Vmax-affiliated researchers are named as authors on vmax.ai: Augustine N. Mavor-Parker, Matthew Sargent (Daborn-Sargent), Geoffrey Bradway, Lorenz Wolf, Maxwill Lin, Roger Creus Castanyer. PROPEL also credits Connor Watts, affiliated with Goodfire AI, not Vmax. [confirmed], source: https://vmax.ai/team/propel (accessed 2026-10-05)
- researcher_backgrounds: Augustine Mavor-Parker: RL PhD, UCL; co-founder (CTO per LinkedIn); previously Redwood Research, CSHL, Illumina; Matthew Sargent (also published as Matthew James Sargent / Matthew Daborn-Sargent): RL PhD (UCL per LinkedIn); co-founder; Vmax-affiliated research authors: Geoffrey Bradway, Lorenz Wolf, Maxwill Lin, Roger Creus Castanyer [reported], source: https://www.southparkcommons.com/companies/vmax/ (accessed 2026-10-05)
- published_papers_or_benchmarks: PROPEL: Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier (arXiv:2606.18284, 2026-06-10; one Goodfire AI co-author). Share of frontier-difficulty generated tasks roughly doubles across math, code and SWE.; unix-ctf: Procedural Environments for Unix-Competence Reinforcement Learning (arXiv:2605.29115, posted 2026-05-25, arXiv v1 2026-05-27). Qwen3-8B held-out solve rate 11.6% to 43.6%.; PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Play (arXiv:2605.16727, 2026-05-16); ~1k JavaScript coding tasks in Harbor format, released with Martian/ARES (2026-01-30) [confirmed], source: https://vmax.ai/research (accessed 2026-10-05)
- total_raised: unknown [unknown], source: No disclosed amount on PitchBook (https://pitchbook.com/profiles/company/907262-38), Crunchbase, or press as of 2026-06-07, funding unannounced/undisclosed
- last_round: unknown [unknown], source: No round stage/date disclosed publicly as of 2026-06-07
- notable_investors: Race Capital; South Park Commons [reported], source: https://www.southparkcommons.com/companies/vmax/ (SPC self-lists as backer); PitchBook lists Race Capital and South Park Commons (accessed 2026-06-07), backers named, round/amount undisclosed
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: unknown [unknown], source: No trust/security page found as of 2026-06-07
- other_certifications: unknown [unknown]
- security_page: unknown [unknown]
- what_they_sell: environments [confirmed], source: https://vmax.ai/ (accessed 2026-06-07), reinforcement learning company converting proprietary data and evals into RL environments for long-horizon LLM-agent tasks
- open_source: no [estimated], source: https://vmax.ai/ (accessed 2026-06-07), no public OSS product repo identified; unix-ctf released as a research paper (arXiv:2605.29115), not a maintained product repo
- license: unknown [unknown]
- deployment_model: unknown [unknown]
- maturity: unknown [unknown], source: Early-stage (founded 2025); no GA product page or pricing found as of 2026-06-07
- notable_customers: Martian / ARES team (withmartian), partnership: jointly releasing ~1k JavaScript coding tasks in the Harbor format (Harbor = Terminal-Bench task format) (self-claimed) [reported], source: https://withmartian.com/post/ares-open-source-infrastructure-for-online-rl-on-coding-agents (accessed 2026-06-07), described as a partnership/collaboration, not a confirmed paying customer. (Draft incorrectly attributed this to 'Harbor / Laude Institute'; the releasing partner is Martian/ARES.)
- founding_team:
  - Augustine Mavor-Parker, Co-founder, RL PhD at UCL. Previously AI safety at Redwood Research, NeuroAI at Cold Spring Harbor Laboratory and AI for genomics at Illumina (per South Park Commons).
  - Matthew Sargent, Co-founder, PhD in RL (UCL per his LinkedIn; SPC says only 'PhD in RL'); building new ways of using RL with LLMs. He is credited as 'Matthew Daborn-Sargent' on PROPEL (June 2026).
- recent_developments (last 6 months):
  - 2026-06-10 [research] Vmax publishes PROPEL: training task generators at the learnable frontier, co-authored with a Goodfire AI researcher, source: https://vmax.ai/team/propel
  - 2026-05-25 [research] unix-ctf: procedural CTF environments for training Unix-competent shell agents, source: https://vmax.ai/team/unix-ctf-procedural-environments-for-unix-competence-reinforcement-learning
  - 2026-05-20 [research] PopuLoRA: population-based asymmetric self-play for RLVR post-training, source: https://vmax.ai/team/populora-co-evolving-llm-populations-for-reasoning-self-play
  - 2026-05-20 [hire] Vmax posts five new roles, including mechanistic interpretability; MTS pay listed at $300k-$500k, source: https://api.ashbyhq.com/posting-api/job-board/vmax
- benchmarks_and_research:
  - PROPEL (Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier) (paper), 2026-06-10, Share of generated tasks at the learnable frontier roughly doubles. Code: 10.1% to 20.0% (3B solver) and 5.3% to 12.6% (7B solver). SWE: 9.8% to 19.6% (27B agent, 11 unseen repos)., https://vmax.ai/team/propel
  - unix-ctf (environment), 2026-05-25, Fine-tuning Qwen3-8B raised the held-out solve rate from 11.6% to 43.6% (n=225), with 32/100 on InterCode-CTF (paper abstract)., https://vmax.ai/team/unix-ctf-procedural-environments-for-unix-competence-reinforcement-learning
  - PopuLoRA (paper), 2026-05-16, Reports gains over single-agent self-play on HumanEval+, MBPP+ and LiveCodeBench, plus transfer to math benchmarks (AIME, AMC and others), which the authors call 'suggestive'. 8 adapters at 1.31x wall-clock overhead., https://vmax.ai/team/populora-co-evolving-llm-populations-for-reasoning-self-play
  - ~1k JavaScript tasks in Harbor format (with Martian ARES) (dataset), 2026-01-30, https://withmartian.com/post/ares-open-source-infrastructure-for-online-rl-on-coding-agents
- products:
  - RL environments / automated RL from proprietary data (custom engagements): Homepage meta description: 'Vmax is automating reinforcement learning. We transform proprietary data and evals into new sets of environments. We refine agents on new examples of the tasks they are intended to perform.' The careers board calls it an applied research lab for open-ended learning. The site lists no named commercial product, pricing or public API; it has only a 'Sign in' link., https://vmax.ai/
  - unix-ctf (research environment generator): Procedural CTF environment generator for shell agents, released as research. No code repository is linked from the page., https://vmax.ai/team/unix-ctf-procedural-environments-for-unix-competence-reinforcement-learning
- notes_for_researchers:
  - Three 2026 arXiv papers, all on generating tasks and curricula automatically: PopuLoRA (2605.16727, self-play), unix-ctf (2605.29115, procedural shell CTF environments) and PROPEL (2606.18284, activation-probe proxy for task difficulty).
  - PROPEL has one Goodfire AI co-author (Connor Watts), which links Vmax's RL work to mechanistic interpretability. Vmax is also hiring MTS and fellows in mech interp.
  - None of the three paper pages on vmax.ai links a code repo. GitHub user 'VMax-ai' shows no link to the company.
- notes_for_program_managers:
  - Small, research-led team. Six Vmax-affiliated researchers are named as authors on vmax.ai: Mavor-Parker, Sargent/Daborn-Sargent, Bradway, Wolf, Lin, Creus Castanyer. A seventh name, Connor Watts, is a Goodfire AI co-author.
  - No round, amount or valuation disclosed. Race Capital and South Park Commons are reported backers (SPC portfolio page; PitchBook per the baseline).
  - No frontier-lab customers are named publicly. The only public partner is Martian (ARES), which released 1k JS tasks with Vmax on 2026-01-30. That is a collaboration, not a confirmed paying customer.
  - 8 open roles on Ashby, all in SF; MTS pay $300k-$500k. A research-lab hiring profile.
- notes_for_data_ops:
  - Tasks released jointly with Martian use the Harbor format (per Martian's ARES post).
  - No public docs, API, security/trust page or SOC 2 statement found.
- sources:
  - https://vmax.ai/ (accessed 2026-10-05)
  - https://vmax.ai/research (accessed 2026-10-05)
  - https://vmax.ai/team (accessed 2026-10-05)
  - https://vmax.ai/team/propel (accessed 2026-10-05)
  - https://arxiv.org/abs/2606.18284 (accessed 2026-10-05)
  - https://vmax.ai/team/populora-co-evolving-llm-populations-for-reasoning-self-play (accessed 2026-10-05)
  - https://arxiv.org/abs/2605.16727 (accessed 2026-10-05)
  - https://vmax.ai/team/unix-ctf-procedural-environments-for-unix-competence-reinforcement-learning (accessed 2026-10-05)
  - https://arxiv.org/abs/2605.29115 (accessed 2026-10-05)
  - https://api.ashbyhq.com/posting-api/job-board/vmax (accessed 2026-10-05)
  - https://www.southparkcommons.com/companies/vmax/ (accessed 2026-10-05)
  - https://withmartian.com/post/ares-open-source-infrastructure-for-online-rl-on-coding-agents (accessed 2026-10-05)
  - https://www.grasp.upenn.edu/people/heejin-jeong/ (accessed 2026-10-05)
  - https://www.goodfire.com/research (accessed 2026-10-05)
  - https://github.com/vmax-ai (accessed 2026-10-05)
  - https://job-boards.greenhouse.io/vmax (accessed 2026-06-07)
  - https://www.linkedin.com/company/vmax-ai (accessed 2026-06-07)
  - https://www.linkedin.com/in/augustine-mavor-parker/ (accessed 2026-06-07)
  - https://www.linkedin.com/in/matthewjsargent/ (accessed 2026-06-07)
  - https://newsletter.semianalysis.com/p/rl-environments-and-rl-for-science (accessed 2026-06-07)
  - https://pitchbook.com/profiles/company/907262-38 (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/vmax (accessed 2026-06-07)
  - https://x.com/MavorParker/status/1868009967518880183 (accessed 2026-06-07)

## Plato, rank #20
- slug: plato
- segment: Commercial vendors
- website: https://plato.so
- focus_areas: Computer Use, Enterprise Workflows
- positioning: Plato (plato.so, Plato Technologies, Inc.) builds simulated worlds for training and evaluating browser and computer-use agents, recreating real websites/software (e.g. Amazon/Airbnb/Gmail-style replicas) as reinforcement-learning environments with structured APIs for interaction, state tracking and scoring. It also offers a 'Computer Use' capability driving a full Linux desktop, positioning at the intersection of browser interaction and enterprise workflow simulation.
- best_fit: AI labs/teams needing high-fidelity replica web/enterprise environments to train and evaluate browser and computer-use agents via RL.
- overall_confidence: medium
- founded_year: 2025 [reported], source: PitchBook 'Plato (United States)' and Crunchbase (plato-379d / plato-d5c7) snippets, accessed 2026-06-07
- status: active [confirmed], source: https://plato.so 2026-06-07; NYT/Cade Metz coverage Dec 2025 via https://theoutpost.ai/news-story/silicon-valley-startups-clone-amazon-and-gmail-to-train-ai-agents-on-complex-tasks-22091/ 2026-06-07
- hq_location: San Francisco, CA, USA [reported], source: PitchBook/Crunchbase snippets accessed 2026-06-07; founders' LinkedIn list San Francisco
- other_locations: unknown [unknown]
- distributed_remote: unknown [unknown]
- current_headcount: ~10 employees [estimated], source: PitchBook/Wellfound search snippets (US Plato) accessed 2026-06-07; not directly confirmed via LinkedIn public snippet
- headcount_band: 1-10 [estimated], source: PitchBook snippets cite ~10 total employees; consistent with seed/pre-seed-stage startup, accessed 2026-06-07
- headcount_growth: unknown [unknown]
- open_roles_count: unknown [unknown]
- has_researchers: yes [estimated], source: Co-founder/CTO Pranav Putta has AI research background (MultiOn, Georgia Tech); LinkedIn snippets accessed 2026-06-07
- researcher_count: unknown [unknown]
- researcher_backgrounds: Pranav Putta (Co-founder/CTO), prior MultiOn, Georgia Institute of Technology, Tonic.ai; Robert Farlow (Co-founder/CEO) [reported], source: LinkedIn public snippets for Pranav Putta (in/pranav-putta-3512b47a) and Robert Farlow (in/robfarlow), accessed 2026-06-07
- published_papers_or_benchmarks: unknown [unknown]
- total_raised: unknown [unknown], source: No plato.so-specific funding amount disclosed. Widely circulated $21M/$14.5M/$6.5M figures belong to an unrelated Berlin wholesale-AI Plato; £260k belongs to an unrelated UK edtech Plato. Crunchbase lists US browser-agents Plato as Pre-Seed only with no disclosed amount, accessed 2026-06-07
- last_round: Pre-Seed (amount and date not disclosed) [reported], source: Crunchbase (plato-379d / plato-d5c7) lists the US browser-agents Plato at Pre-Seed stage; no amount/date found, accessed 2026-06-07
- notable_investors: unknown [unknown], source: No verified investors for plato.so. Do not confuse with Berlin Plato (Atomico/Cherry Ventures) or UK edtech Plato (SFC Capital), accessed 2026-06-07
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: unknown [unknown]
- what_they_sell: environments [confirmed], source: https://docs.plato.so 2026-06-07; Crunchbase 'Evals & datasets for web agents' snippet 2026-06-07
- open_source: partial (client SDK plato-sdk-v2 is MIT-licensed on PyPI; hosted sims/platform proprietary) [confirmed], source: https://pypi.org/pypi/plato-sdk-v2/2.161.0/json (accessed 2026-10-05)
- license: MIT (Python SDK only) [confirmed], source: https://pypi.org/pypi/plato-sdk-v2/2.161.0/json (accessed 2026-10-05)
- deployment_model: managed-hosted VM-based sims at plato.so (dedicated tenant nodes at {tenant}.plato.so), accessed via API key and MIT-licensed Python SDK (plato-sdk-v2) [confirmed], source: https://docs.plato.so/concepts/overview.md (accessed 2026-10-05)
- maturity: early-stage; Python SDK self-labelled Beta on PyPI and under active release (v2.161.0, 2026-10-01) [estimated], source: https://pypi.org/pypi/plato-sdk-v2/2.161.0/json (accessed 2026-10-05)
- notable_customers: unknown [unknown], source: NYT (Cade Metz, Dec 2025) names OpenAI/Google/Amazon/Anthropic as users of the replica-website RL technique generally, but does not name them as Plato's confirmed customers; no named, verified Plato customer found, accessed 2026-06-07
- founding_team:
  - Robert Farlow, Co-founder & CEO, Co-founder and CEO of Plato. The Hard Part with Evan McCann (2025-12-11) introduces him as Co-Founder and CEO of a company building 'Training Environments for Browser Agents'. He is quoted in NYT coverage (Dec 2025): 'We want to build training environments that capture entire jobs that people do.' No prior employers were verified this run.
  - Pranav Putta, Co-founder & CTO, Previously at MultiOn (The AGI Company); Georgia Tech and Tonic.ai per his LinkedIn snippet (reported). First author of 'Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents' (arXiv 2408.07199, Aug 2024), written with MultiOn colleagues and Stanford's Chelsea Finn and Rafael Rafailov.
- recent_developments (last 6 months):
  - 2026-10-01 [product] Plato ships plato-sdk-v2 2.161.0 on PyPI (MIT-licensed Python SDK, labelled Beta), source: https://pypi.org/pypi/plato-sdk-v2/2.161.0/json
- products:
  - Plato simulated environments (Sims): A sim is a pre-built VM image of an application, such as a CRM, Git server, wiki or Linux desktop; espocrm and ubuntu-vm are the examples given. Each sim has versioned artifacts. A session groups one or more envs so they can be reset, evaluated and torn down together, and connects them over a private WireGuard mesh at {alias}.plato.internal. The platform tracks mutations (database writes and file changes). Testcases are scored in MUTATION mode (database/file changes) or OUTPUT mode (returned JSON checked against a schema)., https://docs.plato.so/concepts/overview.md
  - Plato Computer Use: An Ubuntu desktop VM that agents control through four tool surfaces: status, computer (display, mouse, keyboard), bash and edit (structured file operations). Each call is one HTTP round-trip to the VM., https://docs.plato.so/computer-use/tools.md
  - plato-sdk-v2 (Python SDK): MIT-licensed Python SDK for the Plato API, labelled Beta on PyPI. The latest release requires Python >=3.11,<3.14; the docs quickstart still says 3.10+. It connects to https://plato.so by default, or to a dedicated tenant node at https://{tenant}.plato.so through PLATO_BASE_URL., https://pypi.org/project/plato-sdk-v2/
- notes_for_researchers:
  - Plato has published no papers or benchmarks. The Research section of plato.so says 'Nothing published yet'.
  - CTO Pranav Putta is first author of Agent Q (arXiv 2408.07199), which uses guided MCTS, self-critique and off-policy DPO for web agents. The paper comes from MultiOn and Stanford, not from Plato.
  - Scoring runs on testcases: MUTATION mode checks database/file state changes and OUTPUT mode checks returned JSON against a schema. The docs say sessions created directly from envs= are for free-form work and cannot be evaluated.
- notes_for_program_managers:
  - Early-stage company. The baseline Crunchbase snippet lists a pre-seed round with no disclosed amount; this was not re-verified. No institutional funding was found for plato.so. Funding stories that mention 'Plato' usually refer to the unrelated Berlin wholesale-AI company.
  - No named customers or frontier-lab relationships are verified. The NYT (Dec 2025) places Plato in the replica-website RL environment category but names no Plato customers.
  - The SDK is labelled Beta (PyPI classifier 'Development Status :: 4 - Beta'). Dedicated tenant nodes ({tenant}.plato.so) are documented, which suggests single-tenant deployments for larger customers.
- notes_for_data_ops:
  - Integration goes through the Python SDK plato-sdk-v2 (MIT) using Personal or Organization API keys from plato.so/settings. The docs warn that a key 'has full access to your environments' and that legacy keys do not work with the Core SDK.
  - Check the Python version first: the docs say 3.10+, but current PyPI releases require >=3.11,<3.14.
  - The docs describe environments as VM images and do not mention Docker. Envs in a session share a private WireGuard mesh at {alias}.plato.internal. The docs do not mention Harbor, Terminal-Bench or MCP formats.
  - No public SOC 2, security page or trust page was found.
- sources:
  - https://docs.plato.so (accessed 2026-10-05)
  - https://docs.plato.so/llms.txt (accessed 2026-10-05)
  - https://docs.plato.so/concepts/overview.md (accessed 2026-10-05)
  - https://docs.plato.so/computer-use/tools.md (accessed 2026-10-05)
  - https://plato.so/blog (accessed 2026-10-05)
  - https://pypi.org/pypi/plato-sdk-v2/json (accessed 2026-10-05)
  - https://pypi.org/pypi/plato-sdk-v2/2.161.0/json (accessed 2026-10-05)
  - https://theoutpost.ai/news-story/silicon-valley-startups-clone-amazon-and-gmail-to-train-ai-agents-on-complex-tasks-22091/ (accessed 2026-10-05)
  - https://podcasts.apple.com/us/podcast/robert-farlow-from-plato/id1634787423?i=1000740840151 (accessed 2026-10-05)
  - https://arxiv.org/abs/2408.07199 (accessed 2026-10-05)
  - https://arxiv.org/html/2408.07199 (accessed 2026-10-05)
  - https://www.ycombinator.com/companies/plato (accessed 2026-10-05)
  - https://github.com/plato-app (accessed 2026-10-05)
  - https://plato.so/ (accessed 2026-06-07)
  - https://www.linkedin.com/in/robfarlow/ (accessed 2026-06-07)
  - https://www.linkedin.com/in/pranav-putta-3512b47a (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/plato-379d (accessed 2026-06-07)
  - https://pitchbook.com/profiles/company/752634-64 (accessed 2026-06-07)
  - https://tracxn.com/d/companies/plato/__f-KUHjIqU9_tRivtPsI5QjHBWL3AFdLMphYurLed-uw/funding-and-investors (accessed 2026-06-07)
  - https://wellfound.com/company/platoteam/people (accessed 2026-06-07)
  - https://www.techmeme.com/251203/p10 (accessed 2026-06-07)
  - https://alignlist.com/guides/top-40-rl-environments-startups-and-companies (accessed 2026-06-07)
  - https://www.eu-startups.com/2026/02/from-germany-for-the-world-plato-secures-e12-2-million-to-automate-sales-and-erp-workflows-in-distribution/ (accessed 2026-06-07)

## AIChamp, rank #21
- slug: aichamp
- segment: Commercial vendors
- website: https://aichamp.com/
- focus_areas: Enterprise Workflows, Long-Horizon
- positioning: AIChamp, formerly a vendor of RL environments ('Virtual Gyms') and expert data for enterprise AI agents, repositioned in 2026 as a vendor-neutral factory digital-twin service. It runs a free 14-day loss check on a manufacturing line, builds a simulation of the costliest station, and invites robot-AI software teams to train and be tested in that twin against a plant-set target before a floor pilot. Under its 'Shared' option it also licenses de-identified factory scenes to robotics and AI companies. It no longer markets RL environments for LLM or software agents.
- best_fit: Manufacturing plants (food, beverage, chemicals, leather/textiles) that want losses priced and robot/vision fixes tested in simulation before capex. Possibly robotics/physical-AI teams seeking factory twins or de-identified real-world factory scenes for training and evaluation. Not a current fit for buyers of LLM/agent RL environments.
- overall_confidence: medium
- founded_year: unknown [unknown], source: Conflicting: LinkedIn (https://www.linkedin.com/company/aichamp-finds-a-players) shows founded 2025; Tracxn (https://tracxn.com/d/companies/aichamp) lists 'Founded Year 2022', not reliably confirmed, accessed 2026-06-07
- status: active (pivoted) [confirmed], source: https://aichamp.com/terms.html (accessed 2026-10-05)
- hq_location: San Francisco (Mission Bay), USA [confirmed], source: https://aichamp.com/terms.html (accessed 2026-10-05)
- other_locations: unknown [unknown]
- distributed_remote: yes [reported], source: https://torre.ai/teams/AIChamp (accessed 2026-06-07) lists roles as Remote (anywhere); company sources global experts; CEO SF / Tracxn Bali suggests distributed footprint
- current_headcount: ~2-10 (as of 2026-06-07) [estimated], source: https://www.linkedin.com/company/aichamp-finds-a-players public snippet shows '2-10 employees'; https://theorg.com/org/aichamp lists 5 named members in '1-10' band; https://torre.ai/teams/AIChamp shows 11 members, accessed 2026-06-07
- headcount_band: 1-10 [estimated], source: LinkedIn '2-10 employees' and TheOrg '1-10' band (5 named members), accessed 2026-06-07. Note: Torre shows 11 (likely incl. non-employee/expert contributors)
- headcount_growth: unknown [unknown]
- open_roles_count: unknown [unknown], source: Torre shows 2 jobs posted, both closed (accessed 2026-06-07); app.aichamp.com lists recruiting expert-trainer roles but count not reliably enumerable
- has_researchers: unknown [unknown], source: Team described as 'alumni of OpenAI and xAI team' on own site (self-claimed); no named, verifiable research staff confirmed, accessed 2026-06-07
- researcher_count: unknown [unknown]
- researcher_backgrounds: Self-claimed: 'Built by engineers from Microsoft, Amazon Robotics, Google, Mercor, OpenAI and NVIDIA' (vendor site, unverified; the earlier site claimed 'alumni of OpenAI and xAI'); CEO Vol Goloshuk: previously founder-CEO of Brightest Minds (outsourced assistants/SDRs), co-founder of HyperC, ex-Mastercard analyst, earlier network/security engineer (self-reported on Torre) [reported], source: https://aichamp.com/ (accessed 2026-10-05)
- published_papers_or_benchmarks: unknown [unknown], source: No papers or benchmarks published by AIChamp found; only company blog posts, accessed 2026-06-07
- total_raised: $0 / no funding raised [reported], source: https://tracxn.com/d/companies/aichamp ('aiChamp has not raised any funding rounds yet') and search snippet describing it as 'an unfunded company', accessed 2026-06-07
- last_round: none [reported], source: https://tracxn.com/d/companies/aichamp, no funding rounds, accessed 2026-06-07
- notable_investors: unknown [reported], source: https://tracxn.com/d/companies/aichamp, no investors listed (unfunded), accessed 2026-06-07
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: unknown [unknown], source: No security/trust page found (aichamp.com/security returns 404), accessed 2026-06-07
- other_certifications: unknown [unknown]
- security_page: No security/trust page. The homepage's Privacy Policy link (aichamp.com/privacy-policy/) returns HTTP 404. Data-handling commitments are in the Loss Check Terms (aichamp.com/terms.html, effective 2026-10-01). [confirmed], source: https://aichamp.com/terms.html (accessed 2026-10-05)
- what_they_sell: services + data (pivoted): factory loss assessment and digital twins of manufacturing stations in which third-party robot-AI teams train and are tested, pilot coordination, and (Shared option) licensing of de-identified factory scenes to robotics/AI companies for training and testing. RL environments / Virtual Gyms for software agents and expert SFT/RLHF data are no longer marketed. [confirmed], source: https://aichamp.com/terms.html (accessed 2026-10-05)
- open_source: no [estimated], source: No public repos or OSS offering found as of 2026-06-07
- license: unknown [unknown]
- deployment_model: managed service (on-site camera capture at the customer's plant, simulation run by AIChamp, third-party robot-AI trials, separately contracted floor pilot), with Shared (free + scene licensing/revenue share) or Private (fixed fee) options [confirmed], source: https://aichamp.com/terms.html (accessed 2026-10-05)
- maturity: early-stage / newly pivoted (new offering launched around 2026-09; terms effective 2026-10-01; no named customers, and sample report figures are illustrations per the terms) [estimated], source: https://aichamp.com/terms.html (accessed 2026-10-05)
- notable_customers: unknown [confirmed], source: https://aichamp.com/terms.html (accessed 2026-10-05)
- status_change: Pivoted away from its earlier offering (2026-09) Pivot (still active; not acquired or renamed). The 2026-04-22 and 2026-05-26 Wayback snapshots of aichamp.com sell custom RL environments ('Virtual Gyms') for enterprise software agents plus expert data (SFT/RLHF, 'Sniper Sourcing'). Wayback also captured the old WordPress/Elementor site's assets on 2026-06-19 and 2026-06-22. On 2026-10-05 the live site is a factory loss-report and digital-twin service for manufacturing plants. It offers a free 14-day loss check, builds a simulation of the costliest station, invites third-party robot-AI software companies to train and test in that twin against a target the plant sets, then coordinates a floor pilot. Its new Loss Check Terms (effective 2026-10-01) add a 'Shared' option: the check, report, twin and simulation trials are free, and AIChamp may license de-identified scenes from the plant's footage to robotics and AI companies for training and testing, paying the plant a revenue share. The 'Private' option is a fixed fee with no sharing. The homepage cites items dated as late as SEP 2026. The pivot therefore fell between late June and October 2026; the month (Sep 2026) is estimated. AIChamp has left RL environments for LLM/software agents and moved to physical-AI / robotics simulation and to licensing factory scene data., source: https://aichamp.com/terms.html
- founding_team:
  - Vol Goloshuk, CEO, CEO of AIChamp since Feb 2024. Founder-CEO of Brightest Minds (assistants, researchers and SDRs as a service, 2018-2024) and co-founder of HyperC (neural networks for e-commerce trading, 2019-2022). Earlier he was a Senior Analyst in Global Operations at Mastercard (2013-2015) and a network/security engineer at Baker Security Networks (2007-2013). BA (Hons) in Management & Information Systems, Liverpool John Moores University. San Francisco. (Self-reported career history on Torre.)
- recent_developments (last 6 months):
  - 2026-10-01 [product] New Loss Check Terms add a 'Shared' option that licenses de-identified factory scenes to robotics and AI companies, source: https://aichamp.com/terms.html
  - 2026-09 [product] AIChamp repositions as a factory loss-report and digital-twin service for robot AI, source: https://aichamp.com/
- products:
  - Free Loss Check (factory loss report): A 14-day camera capture of a production line with no shutdown and an NDA signed first. Losses (scrap, rework, bad batches, mis-grades, yield loss) are priced in dollars, and fixes are ranked by payback. The check is free. The site's sample report and calculator are illustrations, not client results (per the terms)., https://aichamp.com/
  - Factory digital twin + robot-AI trials and pilot: A simulation of one or more stations in which independent robot-AI software companies train and are tested against a target the plant sets, followed by a floor pilot coordinated with a robot maker and installer under a separate agreement. The homepage claims 'pay only if it hits the target'. The terms say the Private option carries a fixed fee for the report and twin, and pilot economics are agreed separately in writing. Vendor-neutral; does not sell robots. Launch month estimated., https://aichamp.com/
  - Shared option: de-identified factory scene licensing: Under the Shared option, AIChamp licenses de-identified scenes from participating plants to robotics and AI companies for training and testing, and pays the plant a revenue share. Terms effective 2026-10-01., https://aichamp.com/terms.html
  - Plant referral program: Pays 3% of the fees a referred, new-to-AIChamp company pays in its first 12 months as a client, within 30 days of each payment clearing. The page is undated; the launch date is unknown and probably came with the redesign., https://aichamp.com/referrals.html
  - Expert application portal: Legacy talent/expert application portal, still linked from the homepage as 'Join us'., https://app.aichamp.com/apply/
- notes_for_researchers:
  - No papers, benchmarks, datasets or open-source releases from AIChamp were found at any date.
  - New offer: sim-to-real for industrial robotics. Stations are simulated and robot-AI policies from independent teams are trained and tested against a plant-set target. Under the 'Shared' option, de-identified real-factory scenes are licensed to robotics/AI companies for training and testing (terms effective 2026-10-01). No technical details of the simulation stack or scene format are published.
- notes_for_program_managers:
  - Pivoted between late June and October 2026 (Sep 2026 estimated) from RL environments and expert data for AI labs to factory loss assessment, robot-AI evaluation in twins, and factory scene licensing. Do not treat it as an active LLM/agent RL-environment supplier without confirming directly.
  - No disclosed funding (baseline: unfunded, reported), no named customers and no frontier-lab ties. The terms say the sample report figures are illustrations. The team claim ('engineers from Microsoft, Amazon Robotics, Google, Mercor, OpenAI and NVIDIA') is self-claimed.
  - Engagement: free loss check, a written target, twin-based trials, then a separately contracted pilot. The homepage says 'pay only if it hits the target'. Under the terms, the Private option has a fixed fee for the report and twin, and pilot economics are agreed in writing.
- notes_for_data_ops:
  - No security/trust page and no SOC 2 or other certifications published. The homepage's Privacy Policy link (/privacy-policy/) returns 404, so data terms live in aichamp.com/terms.html (effective 2026-10-01; Delaware law, San Francisco courts).
  - Stated data handling: NDA before cameras go up, cameras on their own network not connected to plant systems, faces blurred, no biometric identifiers, no individual worker rating, data deleted within 30 days of project end (except scenes already licensed under Shared). Under Shared, scenes are stripped of company name, brands, customers, location, recipes and prices, and the plant can review and block clips.
  - No APIs, docs or integration formats (Harbor/MCP/Docker etc.) published.
- sources:
  - https://aichamp.com/ (accessed 2026-10-05)
  - https://aichamp.com/terms.html (accessed 2026-10-05)
  - https://aichamp.com/referrals.html (accessed 2026-10-05)
  - http://web.archive.org/web/20260526200648/https://aichamp.com/ (accessed 2026-10-05)
  - http://web.archive.org/cdx/search/cdx?url=aichamp.com/&from=2026 (accessed 2026-10-05)
  - https://aichamp.com/blog/ (accessed 2026-10-05)
  - https://aichamp.com/privacy-policy/ (accessed 2026-10-05)
  - https://app.aichamp.com/apply/ (accessed 2026-10-05)
  - https://theorg.com/org/aichamp (accessed 2026-10-05)
  - https://torre.ai/volgoloshuk (accessed 2026-10-05)
  - https://www.rl-list.com/vendors/aichamp (accessed 2026-10-05)
  - https://aichamp.com/security/ (accessed 2026-06-07)
  - https://www.linkedin.com/company/aichamp-finds-a-players (accessed 2026-06-07)
  - https://www.linkedin.com/in/goloshuk/ (accessed 2026-06-07)
  - https://www.linkedin.com/in/matiroy/ (accessed 2026-06-07)
  - https://torre.ai/teams/AIChamp (accessed 2026-06-07)
  - https://app.aichamp.com/apply/jobs (accessed 2026-06-07)

## Habitat Inc, rank #22
- slug: habitat-inc
- segment: Commercial vendors
- website: https://www.habitat.inc/
- focus_areas: Coding, Computer Use, Enterprise Workflows
- positioning: Habitat Inc is a very early-stage San Francisco company (2-10 employees). Third-party market maps (Chemistry VC, SemiAnalysis, AlignList) list it as building reinforcement-learning environments for code and computer-use agent workflows, made of programmatically verifiable problems. Until mid-2026 its own LinkedIn described it as 'RL environments for white-collar work'. By October 2026 that tagline read 'Autonomous firms.', which may signal a repositioning. It has disclosed no funding, customers, publications or security certifications, and its website is a placeholder.
- best_fit: Buyers needing RL environments that simulate enterprise/desktop and coding workflows to post-train computer-use and coding agents.
- overall_confidence: low
- founded_year: unknown [unknown]
- status: active [reported], source: https://www.linkedin.com/company/habitat-ai-inc (accessed 2026-06-07); https://www.habitat.inc/ (accessed 2026-06-07)
- hq_location: San Francisco, CA, USA [reported], source: https://www.linkedin.com/company/habitat-ai-inc (accessed 2026-10-05)
- other_locations: unknown [unknown], source: SF was inferred from a single founder LinkedIn profile location (https://www.linkedin.com/in/maximenis/, accessed 2026-06-07); insufficient to assert a company office, downgraded to unknown
- distributed_remote: unknown [unknown]
- current_headcount: 2-10 (3 associated on LinkedIn) [reported], source: https://www.linkedin.com/company/habitat-ai-inc (accessed 2026-10-05)
- headcount_band: 1-10 [reported], source: https://www.linkedin.com/company/habitat-ai-inc (accessed 2026-06-07), '2-10 employees'
- headcount_growth: unknown [unknown]
- open_roles_count: unknown [unknown]
- has_researchers: unknown [unknown]
- researcher_count: unknown [unknown]
- researcher_backgrounds: Maxim Enis (co-founder), Williams College '24; prior Ramp association per LinkedIn; co-author (with Mark Hopkins, Williams) of arXiv:2404.13813 'From LLM to NMT: Advancing Low-Resource Machine Translation with Claude' (2024, academic, predates company); Andrew Megalaa (co-founder), Williams College '24 [reported], source: https://www.linkedin.com/in/maximenis/ (accessed 2026-06-07); https://arxiv.org/abs/2404.13813 (accessed 2026-06-07); https://www.instagram.com/p/C6ra5D1PrGo/ (accessed 2026-06-07)
- published_papers_or_benchmarks: unknown [unknown], source: No company-published papers or benchmarks found. Co-founder M. Enis authored arXiv:2404.13813 academically prior to Habitat; not attributed to the company
- total_raised: unknown [unknown], source: No Crunchbase/press/announcement found for habitat.inc as of 2026-06-07; Crunchbase 'Habitat' hits are unrelated entities
- last_round: unknown [unknown]
- notable_investors: unknown [unknown]
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: unknown [unknown]
- what_they_sell: environments [reported], source: https://www.linkedin.com/company/habitat-ai-inc (accessed 2026-06-07), 'RL environments for white-collar work'; https://alignlist.com/guides/top-40-rl-environments-startups-and-companies (accessed 2026-06-07); https://www.chemistry.vc/post/rl-reigns-supreme (accessed 2026-06-07)
- open_source: no [estimated], source: No public GitHub org or OSS repos found for habitat.inc as of 2026-06-07
- license: unknown [unknown]
- deployment_model: unknown [unknown]
- maturity: unknown [unknown]
- notable_customers: unknown [unknown]
- founding_team:
  - Maxim Enis, Founding team member; reported co-founder (title not publicly stated), Williams College 2020-2024. Co-authored arXiv:2404.13813 ('From LLM to NMT: Advancing Low-Resource Machine Translation with Claude', April 2024) with Mark Hopkins; this is academic work from before Habitat. LinkedIn shows a Ramp connection. Based in San Francisco.
  - Andrew Megalaa, Reported founding team member; co-founder status and title unconfirmed, Williams College class of 2024. The 2026-06 baseline tied him to Habitat Inc through a LinkedIn search snippet. His profile could not be fetched (HTTP 999), and the company page names only Maxim Enis, so his current affiliation was not re-verified.
- recent_developments (last 6 months):
  - 2026-10 [other] LinkedIn company page now reads 'Autonomous firms.' and lists a San Francisco HQ, source: https://www.linkedin.com/company/habitat-ai-inc
- products:
  - RL environments for code / computer-use / white-collar work (unnamed): Third parties describe environments of 'hundreds of diverse, programmatically verifiable problems just out of reach of current models' (Chemistry VC, 2025-07-07) and 'code and desktop-style interaction environments for practical agent workflows' (AlignList, 2026-04-09). The official site shows only the company name, and no named product is published. LinkedIn no longer uses the RL-environments tagline., https://www.habitat.inc/
- notes_for_researchers:
  - The company has published no papers, benchmarks, datasets or open-source repos. Maxim Enis's 2024 arXiv paper on low-resource MT with Claude, which curated new MT evaluation sets, is academic work from before Habitat and is not a Habitat output.
  - The only method described anywhere is programmatically verifiable problems set just beyond frontier-model ability. That comes from a 2025 Chemistry VC quote, not from Habitat's own documentation.
- notes_for_program_managers:
  - Very early stage: 2-10 employees, 3 of them on LinkedIn. No disclosed funding, investors or customers.
  - Third-party market maps list it among RL-environment builders: Chemistry VC (2025-07), SemiAnalysis (2026-01) and AlignList (2026-04). No frontier-lab customer is publicly confirmed.
  - Since mid-2026 the LinkedIn page has said 'Autonomous firms.' and given a San Francisco address. Check what the company currently focuses on before engaging.
- notes_for_data_ops:
  - No public docs, API, delivery formats, security page or SOC 2 claim. The website is a placeholder showing only the company name.
- sources:
  - https://www.linkedin.com/company/habitat-ai-inc (accessed 2026-10-05)
  - https://www.habitat.inc/ (accessed 2026-10-05)
  - https://www.linkedin.com/in/maximenis/ (accessed 2026-10-05)
  - https://www.linkedin.com/in/andrewmegalaa/ (accessed 2026-10-05)
  - https://arxiv.org/abs/2404.13813 (accessed 2026-10-05)
  - https://www.chemistry.vc/post/rl-reigns-supreme (accessed 2026-10-05)
  - https://alignlist.com/guides/top-40-rl-environments-startups-and-companies (accessed 2026-10-05)
  - https://newsletter.semianalysis.com/p/rl-environments-and-rl-for-science (accessed 2026-10-05)
  - https://github.com/maximenis (accessed 2026-10-05)
  - https://www.vcbacked.co/company/habitat (accessed 2026-10-05)
  - https://aithority.com/ai-machine-learning-projects/habitat-ai-secures-16m-to-launch-groundbreaking-agentic-ai-infrastructure-powered-by-renewable-energy/ (accessed 2026-10-05)
  - https://www.instagram.com/p/C6ra5D1PrGo/ (accessed 2026-06-07)

## Taste Labs, rank #23
- slug: taste-labs
- segment: Commercial vendors
- website: https://tastelabs.com/
- focus_areas: unknown
- positioning: Taste Labs (Taste) is a San Francisco startup, founded in October 2025 by former Exa growth lead Thais Castello Branco, that calls itself 'the taste layer for AI' and a hybrid research lab and infrastructure company. For model labs, it builds post-training data and RL environments for subjective domains, starting with design. Amplify describes these as preference datasets, reasoning data, rubrics and evaluation environments. The data is made by its TasteMakers network of expert creatives. Taste says it works with 'the top frontier labs' but names none. In September 2026 it launched a self-serve Brand API and MCP server that lets agents extract, search and verify brand design systems. It raised an $18.5M seed co-led by CRV and Amplify Partners, announced 2026-06-16.
- best_fit: Labs that need expert preference data, rubrics or eval environments for aesthetic and design work (UI, web, visual generation), and agent builders that need their outputs to stay on-brand.
- overall_confidence: high
- founded_year: 2025 [reported], source: https://www.crv.com/companies/taste-labs (accessed 2026-10-05)
- status: active [confirmed], source: https://tastelabs.com/ (accessed 2026-10-05)
- hq_location: San Francisco, CA, USA [confirmed], source: https://tastelabs.com/ (accessed 2026-10-05)
- other_locations: unknown [unknown], source: https://tastelabs.com/careers (accessed 2026-10-05)
- distributed_remote: unknown [unknown]
- current_headcount: ~12 (team pictured on careers page) [estimated], source: https://tastelabs.com/careers (accessed 2026-10-05)
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/taste-ai-labs (accessed 2026-10-05)
- headcount_growth: unknown [unknown], source: https://tastelabs.com/careers (accessed 2026-10-05)
- open_roles_count: 9 [confirmed], source: https://tastelabs.com/ (accessed 2026-10-05)
- has_researchers: yes [reported], source: https://tastelabs.com/ (accessed 2026-10-05)
- researcher_count: unknown [unknown], source: https://tastelabs.com/careers (accessed 2026-10-05)
- researcher_backgrounds: unknown [unknown], source: https://tastelabs.com/careers (accessed 2026-10-05)
- published_papers_or_benchmarks: unknown [unknown], source: https://tastelabs.com/blog/requests-for-research (accessed 2026-10-05)
- total_raised: $18.5M (seed, 2026-06) [confirmed], source: https://www.amplifypartners.com/blog-posts/kill-the-slop-announcing-our-investment-in-taste (accessed 2026-10-05)
- last_round: Seed, $18.5M, 2026-06-16, co-led by CRV and Amplify Partners [confirmed], source: https://tastelabs.com/ (accessed 2026-10-05)
- notable_investors: CRV; Amplify Partners [confirmed], source: https://www.amplifypartners.com/blog-posts/kill-the-slop-announcing-our-investment-in-taste (accessed 2026-10-05)
- valuation: unknown [unknown], source: https://www.amplifypartners.com/blog-posts/kill-the-slop-announcing-our-investment-in-taste (accessed 2026-10-05)
- revenue_signals: Self-claims work with 'the top frontier labs' (none named). Paid Brand API plans ($99-$749/mo plus enterprise). One named API customer case study (vendor-published). No revenue figures disclosed. [confirmed], source: https://tastelabs.com/api (accessed 2026-10-05)
- soc2: unknown [unknown], source: https://tastelabs.com/ (accessed 2026-10-05)
- other_certifications: unknown [unknown], source: https://tastelabs.com/ (accessed 2026-10-05)
- security_page: none found [unknown], source: https://tastelabs.com/ (accessed 2026-10-05)
- what_they_sell: human data + environments [confirmed], source: https://www.amplifypartners.com/blog-posts/kill-the-slop-announcing-our-investment-in-taste (accessed 2026-10-05)
- open_source: no [estimated], source: https://tastelabs.com/blog (accessed 2026-10-05)
- license: proprietary [estimated], source: https://tastelabs.com/api (accessed 2026-10-05)
- deployment_model: Custom data/environment engagements with frontier labs; self-serve SaaS API + MCP server (Brand API) for agent/app builders [confirmed], source: https://tastelabs.com/api (accessed 2026-10-05)
- maturity: early-stage (seed; Brand API in beta/alpha) [confirmed], source: https://tastelabs.com/api (accessed 2026-10-05)
- notable_customers: General Intelligence Company of New York (Cofounder) (self-claimed) [confirmed], source: https://tastelabs.com/case-study/general-intelligence-company (accessed 2026-10-05)
- founding_team:
  - Thais Castello Branco, Founder & CEO, Previously led growth at Exa, the AI search company, where she was known for its billboard campaigns (per Amplify).
- funding_history:
  - 2026-06-16 Seed $18.5M led by CRV, Amplify Partners, source: https://www.amplifypartners.com/blog-posts/kill-the-slop-announcing-our-investment-in-taste
- recent_developments (last 6 months):
  - 2026-10-05 [customer] Case study: General Intelligence Company of New York uses Taste's Brand API for brand import in Cofounder, source: https://tastelabs.com/case-study/general-intelligence-company
  - 2026-09-16 [research] Founder Thais Castello Branco presents 'Training Taste' at AI Engineer, source: https://www.youtube.com/watch?v=sDMGWK4wZ_w
  - 2026-09-15 [product] Taste launches the Brand API (Extract, Search, Verify) with an MCP server, source: https://tastelabs.com/blog/helping-agents-create-things-worth-making
  - 2026-08-16 [research] Taste launches Prototype research fellowship and publishes Requests for Research, source: https://tastelabs.com/blog/introducing-prototype
  - 2026-06-16 [funding] Taste emerges with $18.5M seed co-led by CRV and Amplify Partners, source: https://www.amplifypartners.com/blog-posts/kill-the-slop-announcing-our-investment-in-taste
  - 2026-06-16 [other] Taste publishes 'A manifesto for judgement' and 'Stop debating taste', source: https://tastelabs.com/blog/a-manifesto-for-judgement
- products:
  - Taste for Training Models (post-training data and RL environments for frontier labs): Taste's own name for its lab offering. It says it works 'with the top frontier labs to give their models taste, vision and design capabilities'. Amplify describes the offering as preference datasets, reasoning data, rubrics and evaluation environments. The manifesto says it delivers this through 'post-training data and RL environments'., https://tastelabs.com/
  - Brand API (Extract, Search, Verify) + MCP server: Part of 'Taste for Agents or Apps'. It has three parts: brand-system extraction from a URL (beta), style search over a curated brand corpus (alpha), and brand-adherence verification against a reference site (alpha). Extract and Verify run as async jobs. Results come back as JSON plus captured HTML, CSS and screenshots. Access is via REST or the MCP server at mcp.tastelabs.com/mcp., https://tastelabs.com/api
  - Prototype fellowship: A research fellowship on machine creativity, design and human-model interaction. Fellows get human data, compute, collaborators and a stipend., https://tastelabs.com/blog/introducing-prototype
  - TasteMakers community: Taste's human-data network of expert creatives who run paid design projects in UI/UX, brand, web and generative work. Applicants pass a short test to join. Taste describes it as 'a human data and creative network working with leading AI labs, tech companies, and brands' (self-claim)., http://tastemakers.tastelabs.com/
- notes_for_researchers:
  - Taste describes itself as 'making the unverifiable verifiable, starting with design' (homepage) and says it works through 'post-training data and RL environments' (manifesto, 2026-06-16).
  - Its Requests for Research (2026-08-16) lists open problems: high-fidelity web environments (it uses Playwright for capture), rubrics and evals for creative work, procedural fingerprinting of agents, inferring user intent from edit sequences, and reward hacking it says it saw in agents that rebuild website sub-pages to match a brand.
  - The Requests for Research links arXiv 2606.16988, 'Agent trajectories as programs: fingerprinting and programming coding-agent behavior' (Hamidah Oderinwale, 2026-06-15), as work Taste 'explored'. The arXiv listing does not name Taste as an affiliation. Apart from that, Taste has published no papers, datasets, benchmarks or open-source repos as of 2026-10-05.
- notes_for_program_managers:
  - Taste is seed stage. It raised an $18.5M seed co-led by CRV and Amplify Partners, announced 2026-06-16. CRV lists it as founded October 2025 and partnered April 2026.
  - Taste says it works 'with the top frontier labs' but names no lab, so frontier-lab work is a self-claim.
  - Amplify says Taste sells preference datasets, reasoning data, rubrics and evaluation environments to frontier labs. Amplify names Runway, Figma, Character AI and Adobe only as context or companies it has spoken to, not as customers.
  - The team is small. The careers page pictures 12 people, and LinkedIn self-reports 11-50 employees. There are 9 open roles in San Francisco, including 'AI Engineer, RL' and 'Design Research Engineer'.
  - The only named customer is General Intelligence Company of New York (Cofounder), a Brand API user, in a case study Taste published itself.
- notes_for_data_ops:
  - Brand API Extract and Verify jobs run asynchronously: you poll the submission while it runs and sections come back as they finish. Search is synchronous. Results are cached per URL, and force: true forces a fresh run.
  - The MCP server at https://mcp.tastelabs.com/mcp accepts OAuth sign-in or an API key as a bearer token. Taste documents it for Claude Code, Cursor and Codex.
  - Published Brand API prices are Starter $99/mo, Pro $749/mo and custom Enterprise with SLA and Slack support. These cover the API only; lab data engagements have no published pricing.
  - Data for labs comes from the TasteMakers network, where creatives apply, take a short test and are placed on paid projects.
  - Taste publishes no security page, SOC 2 or other certification.
- sources:
  - https://www.amplifypartners.com/blog-posts/kill-the-slop-announcing-our-investment-in-taste (accessed 2026-10-05)
  - https://tastelabs.com/ (accessed 2026-10-05)
  - https://www.crv.com/companies/taste-labs (accessed 2026-10-05)
  - https://tastelabs.com/api (accessed 2026-10-05)
  - https://tastelabs.com/blog (accessed 2026-10-05)
  - https://tastelabs.com/blog/a-manifesto-for-judgement (accessed 2026-10-05)
  - https://tastelabs.com/blog/helping-agents-create-things-worth-making (accessed 2026-10-05)
  - https://tastelabs.com/blog/introducing-prototype (accessed 2026-10-05)
  - https://tastelabs.com/blog/requests-for-research (accessed 2026-10-05)
  - https://tastelabs.com/case-study/general-intelligence-company (accessed 2026-10-05)
  - https://tastelabs.com/careers (accessed 2026-10-05)
  - http://tastemakers.tastelabs.com/ (accessed 2026-10-05)
  - https://www.youtube.com/watch?v=sDMGWK4wZ_w (accessed 2026-10-05)
  - https://arxiv.org/abs/2606.16988 (accessed 2026-10-05)
  - https://www.linkedin.com/company/taste-ai-labs (accessed 2026-10-05)

## Idler, rank #24
- slug: idler
- segment: Commercial vendors
- website: https://idler.ai/
- focus_areas: Coding, Enterprise Workflows, Long-Horizon
- positioning: Idler (YC S25, founded 2025, San Francisco) calls itself a frontier data research lab that builds evals and RL environments from real production work. Its public benchmarks are ShelfLife, a digital twin of a live multi-brand e-commerce company; ShelfLife E-Sim, a stateful retail-operations simulator; and CorpLaw, built from a law firm's anonymized data. It also offers private, on-request collections in long-horizon SWE, cybersecurity, recursive self-improvement and terminal agents. The roughly 12-person team is led by Dark Forest/0xPARC co-founder Ivan Chub and says leading frontier labs use its work, without naming any. A $9M Paradigm-led seed was reported by an aggregator in August 2026 but is not confirmed by the company.
- best_fit: Frontier labs that want bespoke long-horizon RL environments and evals grounded in real company data (e-commerce operations, corporate law), or private long-horizon SWE and cybersecurity task sets.
- overall_confidence: medium
- founded_year: 2025 [confirmed], source: https://www.ycombinator.com/companies/idler (accessed 2026-10-05)
- status: active [confirmed], source: https://www.ycombinator.com/companies/idler (accessed 2026-10-05)
- hq_location: San Francisco, CA, USA [confirmed], source: https://idler.ai/careers (accessed 2026-10-05)
- other_locations: unknown [unknown]
- distributed_remote: unknown [unknown]
- current_headcount: 12 (YC profile team size, as of 2026-10-05) [confirmed], source: https://www.ycombinator.com/companies/idler (accessed 2026-10-05)
- headcount_band: 11-50 [confirmed], source: https://www.ycombinator.com/companies/idler (accessed 2026-10-05)
- headcount_growth: unknown [unknown]
- open_roles_count: 5 as of 2026-10-05 (Forward Deployed Engineer, Design Engineer, Research Scientist, Software Engineer, Special Projects Operator) [confirmed], source: https://idler.ai/careers (accessed 2026-10-05)
- has_researchers: unknown [unknown], source: https://idler.ai/careers (accessed 2026-10-05)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Ivan Chub (CEO): co-founder of Dark Forest, 0xPARC, Zupass and Hack Lodge; ex-Facebook, Dynasty, AppSheet; Tony Goss (CTO): previously Dark Forest DAO and 0xPARC; won $400k by (ethically) hacking Dark Forest; Nalu Concepcion (co-founder): ex-Microsoft SWE, Poshmark product, PayMongo data science [confirmed], source: https://www.ycombinator.com/companies/idler (accessed 2026-10-05)
- published_papers_or_benchmarks: ShelfLife (measured 2026-08-13): https://idler.ai/collections/shelflife; ShelfLife E-Sim (measured 2026-08-26): https://idler.ai/collections/shelflife-e-sim; CorpLaw (measured 2026-08-18; access on request): https://idler.ai/collections/corplaw [confirmed], source: https://idler.ai/collections (accessed 2026-10-05)
- total_raised: $9M+ (seed led by Paradigm, reported 2026-08; plus undisclosed YC S25 investment) [reported], source: https://thesaasnews.com/news/idler-raises-9m-seed (accessed 2026-10-05)
- last_round: Seed, $9M, 2026-08-21, led by Paradigm with Y Combinator and Long Journey [reported], source: https://thesaasnews.com/news/idler-raises-9m-seed (accessed 2026-10-05)
- notable_investors: Paradigm (seed lead, reported); Y Combinator (S25) [reported], source: https://thesaasnews.com/news/idler-raises-9m-seed (accessed 2026-10-05)
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: none found [unknown], source: https://idler.ai/ (accessed 2026-10-05)
- what_they_sell: environments + evals [confirmed], source: https://idler.ai/careers (accessed 2026-10-05)
- open_source: no [reported], source: https://idler.ai/collections (accessed 2026-10-05)
- license: unknown [unknown]
- deployment_model: Bespoke/custom environments and private datasets delivered to labs on request. Public leaderboards, docs and samples are on idler.ai. The delivery mechanism is not publicly documented. [confirmed], source: https://idler.ai/collections (accessed 2026-10-05)
- maturity: early commercial (YC S25; public benchmarks launched Aug 2026) [estimated], source: https://www.ycombinator.com/companies/idler (accessed 2026-10-05)
- notable_customers: Leading frontier AI labs (unnamed) (self-claimed, frontier-lab tie) [reported], source: https://idler.ai/careers (accessed 2026-10-05)
- founding_team:
  - Ivan Chub, Co-founder & CEO, Previously co-founded Dark Forest, 0xPARC, Zupass and Hack Lodge. Worked at Facebook, Dynasty and AppSheet.
  - Tony Goss, Co-founder & CTO, Previously at Dark Forest DAO (@d_fdao) and 0xPARC, per his X bio. His YC bio says he '(ethically) hacked Dark Forest to win $400k'.
  - Nalu Concepcion, Co-founder, Leads the scaling of RL environments. Previously a software engineer at Microsoft, in product at Poshmark and in data science at PayMongo, and organized events of 500+ people.
- funding_history:
  - 2026-08-21 Seed $9M led by Paradigm, source: https://thesaasnews.com/news/idler-raises-9m-seed
  - 2025 Accelerator undisclosed led by Y Combinator, source: https://www.ycombinator.com/companies/idler
- recent_developments (last 6 months):
  - 2026-08-26 [benchmark] ShelfLife E-Sim leaderboard published: long-horizon retail-operations simulator, source: https://idler.ai/collections/shelflife-e-sim
  - 2026-08-21 [funding] Idler raises a $9M seed from Paradigm to build RL environments (reported), source: https://thesaasnews.com/news/idler-raises-9m-seed
  - 2026-08-18 [benchmark] CorpLaw: a corporate-law benchmark built from a real law firm's anonymized data, source: https://idler.ai/collections/corplaw
  - 2026-08-13 [benchmark] ShelfLife: a digital twin of a live e-commerce company, launched as the flagship benchmark, source: https://idler.ai/collections/shelflife
- benchmarks_and_research:
  - ShelfLife E-Sim (environment), 2026-08-26, Grok 4.6 is top at 65.0 ± 1.7 mean reward points; Opus 5 scores 39.5., https://idler.ai/collections/shelflife-e-sim
  - CorpLaw (benchmark), 2026-08-18, Claude Opus 5 is top at 42.8% (per the collections index). The company says 10 of 50 tasks have never been passed by any of the six models evaluated at high reasoning effort., https://idler.ai/collections
  - ShelfLife (benchmark), 2026-08-13, Claude Opus 5 leads at 69.0% Pass@1, ahead of GPT-5.6 Sol (55.7%) and Claude Fable 5 (54.7%). Company-reported: training Nemotron 3 Nano on 41 ShelfLife tasks raised its Finance Agent Benchmark score from 30.0% to 41.8%., https://idler.ai/collections/shelflife
- products:
  - Custom RL environments and evals for frontier labs: RL environments 'built from real production work', covering frontier coding, economically valuable workflows and bespoke evaluations. Engagement is through an Airtable contact form or a booked call., https://idler.ai/
  - ShelfLife: A benchmark and environment set built as a digital twin of an e-commerce company. Public leaderboard, docs and sample; full data on request., https://idler.ai/collections/shelflife
  - ShelfLife E-Sim: A stateful, long-horizon retail-operations simulator., https://idler.ai/collections/shelflife-e-sim
  - CorpLaw: A corporate legal practice benchmark built from anonymized law-firm data (access on request)., https://idler.ai/collections/corplaw
  - Long-Horizon SWE collection: Repo tasks in Go, Python, Rust, C++, TS and Java. The company says each task has 100+ steps and a frontier-model pass rate below 50%. Access on request., https://idler.ai/collections/long-horizon-swe
  - Cybersecurity collection: CVE finding, patching and exploitation across public and private codebases. Access on request., https://idler.ai/collections/cybersecurity
  - Recursive Self-Improvement collection: Model training, data curation, inference optimization and harness engineering. Access on request., https://idler.ai/collections/recursive-self-improvement
  - Terminal Agents collection: Execution-environment setup, enterprise API and tool use, and data cleaning and anonymization. Access on request., https://idler.ai/collections/terminal-agents
- notes_for_researchers:
  - ShelfLife, ShelfLife E-Sim and CorpLaw are published on idler.ai as leaderboards, with documentation and a sample task for ShelfLife. No paper, GitHub repo or Hugging Face dataset was found, and full datasets are available only on request.
  - ShelfLife grading splits reward into quantitative (partial credit), integrity (penalties only, for specified harmful actions) and withholding (policy-violation deductions) criteria. The docs say models are not penalized for insufficient data, safety-motivated refusals, undisclosed rules or unexpected answer formatting.
  - Data is deliberately left unsanitized. The docs say: 'We do not clean or alter schemas in the data workspace. We preserve the natural defects and gaps present in the raw data.'
  - Company-reported training signal: Nemotron 3 Nano post-trained on 41 ShelfLife tasks rose from 30.0% to 41.8% on Finance Agent Benchmark (p ≤ 0.001), per the YC launch. No write-up or checkpoints were found.
- notes_for_program_managers:
  - YC S25, founded 2025, San Francisco. The YC profile listed a team size of 12 and 5 open roles (Forward Deployed Engineer, Design Engineer, Research Scientist, Software Engineer, Special Projects Operator) on 2026-10-05.
  - The SaaS News (2026-08-21) reports a $9M seed led by Paradigm, with Y Combinator, Long Journey and angel investors. There is no primary announcement from the company or Paradigm, so treat it as reported.
  - The company says 'the world's leading frontier labs' use its evals and environments but names none (self-claimed).
  - Engagement is bespoke. Collections beyond the public leaderboards are 'Access Upon Request' via a scheduled call or an Airtable interest form. The Forward Deployed Engineer role 'own[s] the entire system that turns our partners' real work into evals & RL environments'.
  - Environments are built from real partner businesses' data (a live retailer, a law firm) in collaboration with the people who did the work.
- notes_for_data_ops:
  - The ShelfLife sample shows MCP-style tools (read_file, list_files, read_policy, release_run, request_approval, whoami) and a /workspace/data directory of CSV exports (transfers, inventory snapshots, sales). Some tasks also give full terminal access.
  - Some tasks run on a 'state-machine-driven business engine' that reproduces the sequence of information and events a human would meet, including irreversible decisions. An integrity violation (e.g. acting under an account that is not the agent's own) zeros the run.
  - No public API, SDK, download or standard eval-harness format is documented. Access is arranged by request.
  - No SOC 2, security page or trust center was found on idler.ai on 2026-10-05. Data is described as anonymized law-firm data or real retailer operational data.
- sources:
  - https://idler.ai/ (accessed 2026-10-05)
  - https://idler.ai/careers (accessed 2026-10-05)
  - https://www.ycombinator.com/companies/idler (accessed 2026-10-05)
  - https://idler.ai/collections (accessed 2026-10-05)
  - https://idler.ai/collections/shelflife (accessed 2026-10-05)
  - https://idler.ai/collections/shelflife/documentation (accessed 2026-10-05)
  - https://idler.ai/collections/shelflife/sample (accessed 2026-10-05)
  - https://idler.ai/collections/shelflife-e-sim (accessed 2026-10-05)
  - https://idler.ai/collections/corplaw (accessed 2026-10-05)
  - https://idler.ai/collections/long-horizon-swe (accessed 2026-10-05)
  - https://idler.ai/collections/cybersecurity (accessed 2026-10-05)
  - https://idler.ai/collections/recursive-self-improvement (accessed 2026-10-05)
  - https://idler.ai/collections/terminal-agents (accessed 2026-10-05)
  - https://thesaasnews.com/news/idler-raises-9m-seed (accessed 2026-10-05)
  - https://www.paradigm.xyz/portfolio (accessed 2026-10-05)
  - https://api.fxtwitter.com/chubivan (accessed 2026-10-05)
  - https://api.fxtwitter.com/cha0sg0d_ (accessed 2026-10-05)
  - https://api.fxtwitter.com/idler_ai (accessed 2026-10-05)
  - https://idler.ai/blog (accessed 2026-10-05)
  - https://github.com/idler-ai (accessed 2026-10-05)

## Scale AI (not ranked, incumbent / infrastructure / open-source)
- slug: scale-ai
- segment: Incumbents also building RL environments
- website: https://scale.com
- focus_areas: Coding, Computer Use, Enterprise Workflows, Long-Horizon
- positioning: Scale AI is the data-labeling and AI-data incumbent that has extended into RL environments (simulated web apps, macOS/Windows-like desktop VMs and MCP-tool environments with expert rubrics and automated verifiers). It runs a large public evaluation program through Scale Labs (SWE-Bench Pro, MCP Atlas, Humanity's Last Exam, Remote Labor Index). After Meta's ~$14.3B deal in June 2025 (~49% non-voting stake) and Alexandr Wang's move to Meta, several frontier labs reportedly scaled back work. In 2026 Scale grew mainly in government and enterprise: the Pentagon CDAO agreement ceiling rose to $500M, it acquired ICG Solutions, and Francis deSouza (ex-Google Cloud COO) became CEO in August 2026.
- best_fit: Government, defense and enterprise buyers who want an established FedRAMP High / IL4 / SOC 2 vendor for expert-built RL environments, human data and evaluation, and who can buy through existing vehicles (CDAO OTA, GSA MAS via Carahsoft). Also researchers who use Scale Labs' public benchmarks (SWE-Bench Pro, MCP Atlas, HLE). Frontier labs should weigh Meta's ~49% stake.
- overall_confidence: high
- founded_year: 2016 [confirmed], source: https://en.wikipedia.org/wiki/Scale_AI (accessed 2026-06-07); founded 2016 by Alexandr Wang and Lucy Guo via Y Combinator
- status: active (independent; Meta holds ~49% non-voting minority stake as of June 2025) [confirmed], source: https://techcrunch.com/2025/06/13/scale-ai-confirms-significant-investment-from-meta-says-ceo-alexandr-wang-is-leaving/ (accessed 2026-06-07); https://www.cnbc.com/2025/11/04/scale-ais-life-after-meta-has-been-rocky-cfo-insists-not-a-zombie.html
- hq_location: San Francisco, California, USA [confirmed], source: https://scale.com/about (accessed 2026-06-07); https://en.wikipedia.org/wiki/Scale_AI
- other_locations: St. Louis, Missouri, USA (opened 2022) [reported], source: https://en.wikipedia.org/wiki/Scale_AI (accessed 2026-06-07)
- distributed_remote: unknown [unknown]
- current_headcount: 1,000+ (company-stated on scale.com/about, accessed 2026-10-05); Wikipedia infobox ~1,200 (2025) [reported], source: https://scale.com/about (accessed 2026-10-05)
- headcount_band: 200+ [confirmed], source: https://techcrunch.com/2025/07/16/scale-ai-lays-off-14-of-staff-largely-in-data-labeling-business/ (accessed 2026-06-07); multiple sources cite ~1,000-1,400 FTE
- headcount_growth: ~14% FTE reduction in July 2025 (~200 cut from ~1,400) plus ~500 contractors; interim CEO signaled intent to staff up enterprise and government sales units in H2 2025 [reported], source: https://techcrunch.com/2025/07/16/scale-ai-lays-off-14-of-staff-largely-in-data-labeling-business/ (accessed 2026-06-07)
- open_roles_count: unknown [unknown]
- has_researchers: yes [confirmed], source: https://scale.com/blog/scale-labs (accessed 2026-10-05)
- researcher_count: unknown [unknown]
- researcher_backgrounds: unknown [unknown]
- published_papers_or_benchmarks: Scale Labs (launched 2026-03-09, extends SEAL) publishes papers and leaderboards at labs.scale.com; SWE-Bench Pro v1 (2025-09-19) and v2 (2026-09-22; HF ScaleAI/SWE-bench_Pro); SWE Atlas: Codebase QnA, Test Writing, Refactoring (2026-05-07); MCP Atlas (2025-12-18; HF ScaleAI/MCP-Atlas; GitHub scaleapi/mcp-atlas); Humanity's Last Exam (with Center for AI Safety, 2025-01); Remote Labor Index (2025-10-28); HiL-Bench (2026-04-13); DrugDiscoveryBench with Phylo (2026-06-30); READY or Not enterprise agent deployment benchmarks (2026-09-03; arXiv 2609.02095); RSI Bench (GitHub scaleapi/rsi-benchmark); PRBench, MultiChallenge, VisualToolBench, EnigmaEval, MASK, TutorBench and other leaderboards; Rubric-based RL papers (Chasing the Tail; Agentic Rubrics; Reward Hacking in Rubric-Based RL; Not Every Rubric Teaches Equally; Rubric-Guided Self-Distillation); Agent-environment papers (ChainWorld; SWE-INTERACT; Spine-Branch multi-agent computer use) [confirmed], source: https://labs.scale.com/papers (accessed 2026-10-05)
- total_raised: ~$1.6B in disclosed equity rounds prior to Meta; Meta's June 2025 ~$14.3B was a strategic stake purchase (largely secondary/cash to existing holders), not a conventional primary funding round, so combining them into a single ~$15.9B 'raised' figure overstates capital raised [reported], source: https://techcrunch.com/2025/06/13/scale-ai-confirms-significant-investment-from-meta-says-ceo-alexandr-wang-is-leaving/ (accessed 2026-06-07); https://techcrunch.com/2024/05/21/data-labeling-startup-scale-ai-raises-1b-as-valuation-doubles-to-13-8b/
- last_round: Meta strategic investment, ~$14.3B for ~49% non-voting stake, June 2025 (valuing Scale at ~$29B); prior Series F ~$1B in May 2024 at ~$13.8B valuation [reported], source: https://techcrunch.com/2025/06/13/scale-ai-confirms-significant-investment-from-meta-says-ceo-alexandr-wang-is-leaving/ (accessed 2026-06-07); https://techcrunch.com/2024/05/21/data-labeling-startup-scale-ai-raises-1b-as-valuation-doubles-to-13-8b/
- notable_investors: Meta Platforms; Accel; Amazon; Nvidia; Founders Fund; Index Ventures; Tiger Global Management; Dragoneer Investment Group; Greenoaks; Y Combinator [reported], source: https://en.wikipedia.org/wiki/Scale_AI (accessed 2026-06-07); https://news.crunchbase.com/ai/scale-holistic-raise-big-accel-nvda-amzn/
- valuation: ~$29B (post Meta June 2025 investment) [reported], source: https://techcrunch.com/2025/06/13/scale-ai-confirms-significant-investment-from-meta-says-ceo-alexandr-wang-is-leaving/ (accessed 2026-06-07); corroborated by Bloomberg and multiple outlets
- revenue_signals: ~$870M revenue in 2024 (reported). Company guided ~$2B for 2025 (Bloomberg, Apr 2025; forward guidance, not audited). In May 2026 the company reported 700+ customers to date and $1B+ paid to contributors. Pentagon CDAO agreement ceiling raised from $100M to $500M in May 2026 (a ceiling, not booked revenue). No audited 2025 or 2026 revenue has been published. [reported], source: https://scale.com/blog/ten-years-of-scale (accessed 2026-10-05)
- soc2: Type II [confirmed], source: https://scale.com/security (accessed 2026-06-07); page states 'SOC 2 Type II'; reports available via trust.scale.com
- other_certifications: ISO/IEC 27001:2022; FedRAMP High Authorized; DoD IL4 Provisional Authorization (DISA) [confirmed], source: https://scale.com/security (accessed 2026-06-07), HIPAA REMOVED: not listed on the current security page (page omits HIPAA); only verifiable via separate older blog claim
- security_page: https://scale.com/security (Trust Center: https://trust.scale.com/) [confirmed], source: https://scale.com/security (accessed 2026-06-07)
- what_they_sell: mixed (human data / data labeling, RL environments, model evaluations and leaderboards, enterprise GenAI agent platform, defense and intelligence AI including Donovan and ICG LUX) [confirmed], source: https://scale.com/blog/scale-acquires-icg-solutions (accessed 2026-10-05)
- open_source: no [estimated], source: https://scale.com/rlenvironments (accessed 2026-06-07); offered as a commercial managed product, no OSS license indicated
- license: unknown [unknown]
- deployment_model: managed-hosted (vendor-operated environments/infrastructure to run agent training and evaluation; simulated APIs, MCP servers, GUIs) [reported], source: https://scale.com/rlenvironments (accessed 2026-06-07)
- maturity: GA (RL Environments product publicly offered) [reported], source: https://scale.com/rlenvironments (accessed 2026-06-07); https://techcrunch.com/2025/09/21/silicon-valley-bets-big-on-environments-to-train-ai-agents/
- notable_customers: U.S. Department of War / DoD (CDAO production OTA, ceiling raised to $500M May 2026) (verified); Mayo Clinic (named Mayo executive quoted) (verified); BP (named in the company's CEO announcement) (self-claimed); U.S. Department of Energy (Genesis Mission MOU and consortium membership; partner, not a disclosed contract) (self-claimed); General Motors (verified); Cisco (logo on scale.com/about) (self-claimed); Cohere (logo on scale.com/about) (self-claimed, frontier-lab tie); OpenAI (former; reportedly wound down work after the Meta deal) (verified, frontier-lab tie); Google / Alphabet (former data customer that reportedly cut ties in 2025; Google Cloud is a 2026 reference-architecture partner) (verified, frontier-lab tie); xAI (reportedly paused work after the Meta deal; not firmly confirmed) (self-claimed, frontier-lab tie) [reported], source: https://scale.com/blog/scale-ai-pentagon-cdao-500-million-agreement (accessed 2026-10-05)
- founding_team:
  - Alexandr Wang, Co-founder; former CEO (2016-2025); Founder and Chairman per the 2026-07-30 announcement, Left MIT to co-found Scale in 2016 through Y Combinator. Earlier a software engineer at Quora and Addepar. Since June 2025 he has been Meta's Chief AI Officer, leading Meta Superintelligence Labs.
  - Lucy Guo, Co-founder (left in 2018), Left Carnegie Mellon in 2014 as a Thiel Fellow. Interned at Facebook and was Snapchat's first female designer. Founded Backend Capital (2019) and Passes (2022).
  - Francis deSouza, CEO (announced 2026-07-30, effective 2026-08-10), Previously COO and President of Security Products at Google Cloud, President and CEO of Illumina, and President of Products and Services at Symantec. Founded two companies acquired by Microsoft and Symantec. BS and MS in EECS from MIT.
  - Jason Droege, Interim CEO (June 2025 - August 2026); assisting the CEO transition, Joined Scale as Chief Strategy Officer in September 2024. Earlier, as Uber VP, he built Uber Eats from its first order in 2015 to a $19B GMV run rate by 2020.
- recent_developments (last 6 months):
  - 2026-09-22 [benchmark] SWE-Bench Pro v2: 642 public tasks, a 51-task Hard split, a 272-task private set and network-locked evaluation, source: https://labs.scale.com/blog/swe-bench-pro-v2
  - 2026-09-22 [partnership] Scale and Google Cloud publish a reference architecture for deploying Scale GenAI Portfolio (SGP) agents on Google Cloud and Gemini Enterprise, source: https://scale.com/blog/deploying-enterprise-ai-agents-with-scale-and-google-cloud
  - 2026-09-03 [research] Scale Labs releases READY ('READY or Not'), enterprise agent-deployment benchmarks, source: https://labs.scale.com/blog/ready
  - 2026-09-02 [research] Scale Labs 'Who Grades the Graders?': study of verifier reliability for computer-use agent tasks, source: https://labs.scale.com/blog/verifier-design-for-cua
  - 2026-07-30 [hire] Scale AI appoints Francis deSouza as CEO (effective 2026-08-10); interim CEO Jason Droege to assist the transition, source: https://scale.com/blog/scale-appoints-new-ceo
  - 2026-07-16 [partnership] Scale joins the DOE Genesis Mission Consortium as a formal member, source: https://scale.com/blog/scale-ai-joins-genesis-mission-consortium
  - 2026-07-15 [customer] Mayo Clinic collaboration on clinical AI applications built on Scale GenAI Platform, source: https://scale.com/blog/mayo-clinic-scale
  - 2026-07-07 [partnership] Scale and Singapore's IMDA sign an MOU on joint AI evaluation research, starting with a Singapore-law legal benchmark, source: https://scale.com/blog/singapore-imda-mou
  - 2026-07-06 [research] Scale Labs publishes VeRO, a framework in which an optimizer agent edits a target agent, source: https://scale.com/blog/vero
  - 2026-06-30 [benchmark] DrugDiscoveryBench released with Phylo, source: https://scale.com/blog/drugdiscoverybench
  - 2026-06-24 [partnership] Scale and Circadence announce a collaboration on AI-driven cyber readiness, source: https://scale.com/blog/scale-circadence-ai-driven-cyber-readiness
  - 2026-06 [research] Scale Labs publishes rubric-based RL and agent-environment papers (Reward Hacking in Rubric-Based RL, 2026-05-12; Not Every Rubric Teaches Equally, 2026-05-19; Rubric-Guided Self-Distillation, 2026-06-10; ChainWorld, 2026-06-19; SWE-INTERACT, 2026-06-29), source: https://labs.scale.com/papers
  - 2026-05-20 [product] Scale's GenAI Platform, T&E and services added to the GSA Multiple Award Schedule through Carahsoft, source: https://scale.com/blog/scale-ai-gsa-schedule-available
  - 2026-05-14 [other] 'Ten Years of Scale': company reports 700+ customers, 15B human decisions and $1B+ paid to contributors, source: https://scale.com/blog/ten-years-of-scale
  - 2026-05-07 [benchmark] SWE Atlas completed: Codebase QnA, Test Writing and Refactoring benchmarks, 284 tasks in total, source: https://scale.com/blog/swe-atlas-complete
  - 2026-05-06 [customer] Pentagon CDAO raises the ceiling on Scale's production OTA from $100M to $500M, source: https://scale.com/blog/scale-ai-pentagon-cdao-500-million-agreement
  - 2026-05-01 [partnership] Scale signs an MOU with the U.S. Department of Energy for the Genesis Mission, source: https://scale.com/blog/scale-ai-doe-genesis-mission-mou
  - 2026-04-20 [acquisition] Scale acquires ICG Solutions, maker of the LUX real-time streaming analytics platform for defense and intelligence, source: https://scale.com/blog/scale-acquires-icg-solutions
  - 2026-04-13 [benchmark] Scale Labs releases HiL-Bench, which measures agents' help-seeking judgment, source: https://labs.scale.com/leaderboard/hil
- benchmarks_and_research:
  - SWE-Bench Pro (v2) (benchmark), 2026-09-22, Claude Opus 5: 638/642 public (99.4%) vs 222/272 private (81.6%); Hard split leader Opus 5 (Claude Code) 98.00, https://labs.scale.com/leaderboard/swe_bench_pro_public_v2
  - READY or Not: Reliable Enterprise Agent Deployment (benchmark), 2026-09-03, Two systems with near-identical accuracy (72.8% vs 72.5%) needed different human review rates (39.2% vs 29.6%) to reach the same 76% reliability target, https://labs.scale.com/blog/ready
  - Who Grades the Graders? Verifier design for computer-use agents (paper), 2026-09-02, None of 38 programmatic verifiers failed a deliverable with its most critical number corrupted; programmatic and VLM judges agreed on 99.5% of judge-passed items, https://labs.scale.com/blog/verifier-design-for-cua
  - DrugDiscoveryBench (benchmark), 2026-06-30, Top: GPT 6 Astra (mini-SWE-agent) 68.70 +/- 2.50, https://labs.scale.com/leaderboard/drugdiscoverybench
  - Agent-environment papers (ChainWorld; SWE-INTERACT; Spine-Branch Coordination for Multi-agent Computer Use) (paper), 2026-06-19, https://labs.scale.com/papers
  - SWE Atlas (Codebase QnA, Test Writing, Refactoring) (benchmark), 2026-05-07, Current leaders: Test Writing Fable-5.1 (Claude Code) 67.04 +/- 5.33; Codebase QnA Opus 5 (Claude Code) 63.17 +/- 5.01; Refactoring GPT 6 Astra and Fable-5.1 roughly 57-59, https://scale.com/blog/swe-atlas-complete
  - HiL-Bench (Human-in-the-Loop) (benchmark), 2026-04-13, Top: Claude Fable 5.1 61.50 +/- 6.47 Pass@3, https://labs.scale.com/leaderboard/hil
  - MCP Atlas (benchmark), 2025-12-18, Top: Muse Spark 1.1, 88.1% pass rate, https://labs.scale.com/leaderboard/mcp_atlas
  - Professional Reasoning Bench (PRBench: Legal, Finance) (benchmark), 2025-11-13, Top: Muse Spark 1.3 (Legal 61.56 +/- 1.72; Finance 59.54 +/- 1.66), https://labs.scale.com/leaderboard/prbench-legal
  - Remote Labor Index (RLI) (benchmark), 2025-10-28, Top automation rate: GPT 6 Astra 20.83%, https://labs.scale.com/leaderboard/rli
  - Rubric-based RL papers (Chasing the Tail; ResearchRubrics; Agentic Rubrics as Contextual Verifiers; Reward Hacking in Rubric-Based RL; Not Every Rubric Teaches Equally; Rubric-Guided Self-Distillation) (paper), 2025-09, https://labs.scale.com/papers
  - Reasoning Gym (environment), 2025-05, https://arxiv.org/abs/2505.24760
  - Humanity's Last Exam (HLE) leaderboards (benchmark), 2025-01-23, Main: GPT 6 Astra 54.80 +/- 1.94; Diamond: gpt-6-astra 60.60 +/- 3.00 (labs.scale.com, accessed 2026-10-05), https://labs.scale.com/leaderboard/humanitys_last_exam
  - RSI Bench (benchmark), https://labs.scale.com/rsi-benchmark
  - Other Scale Labs/SEAL leaderboards: VisualToolBench, MultiChallenge, MultiNRC, AudioMultiChallenge, EnigmaEval, MASK, Fortress, PropensityBench, TutorBench, SciPredict, VISTA (leaderboard), https://labs.scale.com/leaderboard
- products:
  - RL Environments: Managed simulated environments for agent training and evaluation: browser-based SaaS and productivity apps, macOS- and Windows-like desktop VMs, and MCP servers for enterprise tools (Slack, HubSpot, Linear and others). Includes expert-designed objectives, rubrics, automated verifiers and 'standard environment interfaces' (trajectory tracking, state resets). The page names no customers., https://scale.com/rlenvironments
  - Scale Data Engine: Data labeling and human-data platform, including computer vision and ML operations., https://scale.com/blog/scale-ai-pentagon-cdao-500-million-agreement
  - Scale GenAI Platform / GenAI Portfolio (SGP): Enterprise platform for building, evaluating, tracing and operating AI agents. Google Cloud reference architecture published 2026-09-22; listed on the GSA MAS through Carahsoft since 2026-05-20., https://scale.com/blog/deploying-enterprise-ai-agents-with-scale-and-google-cloud
  - Scale Donovan: AI decision-support platform for defense and national security, authorized on NIPR, SIPR and JWICS under the CDAO OTA., https://scale.com/blog/scale-ai-pentagon-cdao-500-million-agreement
  - ICG LUX (via the ICG Solutions subsidiary): Real-time streaming data ingestion, correlation and alerting for defense and intelligence. Product of ICG Solutions, which Scale acquired on 2026-04-20., https://scale.com/blog/scale-acquires-icg-solutions
  - Scale Labs (research hub and public leaderboards): Research hub launched 2026-03-09 that extends SEAL. Hosts public model leaderboards (HLE, SWE-Bench Pro, SWE Atlas, MCP Atlas, RLI, HiL-Bench and others) and a papers index., https://labs.scale.com/leaderboard
  - Outlier: Expert contributor platform for AI training work, described on its site as 'a platform operated by Scale AI'., https://outlier.ai
- notes_for_researchers:
  - Scale Labs (labs.scale.com, launched 2026-03-09) extends SEAL and keeps a dated papers index. Its 2026 output centers on rubric rewards for RL (reward hacking, policy-aware rubrics, rubric-guided self-distillation), long-horizon agent environments (ChainWorld from OSWorld tasks, SWE-INTERACT, Spine-Branch multi-agent computer use), agent optimization (VeRO) and deployment evaluation (READY).
  - Open releases: SWE-Bench Pro (HF ScaleAI/SWE-bench_Pro; GitHub scaleapi/SWE-bench_Pro-os), MCP Atlas (HF ScaleAI/MCP-Atlas; GitHub scaleapi/mcp-atlas) and RSI Bench (GitHub scaleapi/rsi-benchmark). SWE-Bench Pro and MCP Atlas keep held-out private splits.
  - SWE-Bench Pro v2 shows a large public vs private gap (Claude Opus 5: 99.4% vs 81.6%), which Scale attributes mainly to training-time exposure. Treat public-split scores as likely contaminated.
  - The 'Who Grades the Graders?' study (2026-09-02) found that programmatic verifiers missed critical-number corruption in all 38 tested deliverables. It recommends deterministic checks plus vision-capable judges and corruption testing, which is a useful reference for reward design in computer-use environments.
- notes_for_program_managers:
  - Leadership changed: Francis deSouza (ex-Google Cloud COO, ex-Illumina CEO) has been CEO since 2026-08-10, replacing interim CEO Jason Droege. Alexandr Wang (Meta Chief AI Officer) is still on the board and is called 'Founder and Chairman' in the announcement. Meta holds a ~49% non-voting stake from June 2025.
  - Momentum in the window is mostly government and enterprise: ICG Solutions acquisition (2026-04-20), DOE Genesis MOU (2026-05-01) and consortium membership (2026-07-16), Pentagon CDAO OTA ceiling raised to $500M (2026-05-06), GSA MAS listing (2026-05-20), Circadence cyber partnership (2026-06-24), Singapore IMDA evaluation MOU (2026-07-07), Mayo Clinic (2026-07-15) and the Google Cloud reference architecture (2026-09-22).
  - Company-stated scale: 700+ customers to date, 15B human decisions, $1B+ paid to contributors (2026-05-14 post), plus 1,000+ employees and a $29B valuation (scale.com/about). All self-reported.
  - No new primary funding round, tender or secondary was found for 2026-04 to 2026-10.
  - No public 2026 confirmation was found that OpenAI, Google DeepMind, Anthropic or xAI buy RL environments from Scale. Labs worried about Meta's stake should do their own diligence.
- notes_for_data_ops:
  - Security: SOC 2 Type II, ISO/IEC 27001:2022, FedRAMP High Authorized, DoD IL4 Provisional Authorization (DISA). Trust Center at trust.scale.com. HIPAA is not listed on the security page; the Mayo work runs inside Mayo's own HIPAA-compliant environment.
  - RL Environments are a managed offering (simulated apps, desktop VMs, MCP servers) that claims 'standard environment interfaces' with trajectory tracking and state resets. No public API docs, self-host option or published interchange format was found.
  - Scale's open benchmarks ship as Docker-image environments with verifiers (SWE-Bench Pro) or containerized MCP servers (MCP Atlas). READY is built on Inspect AI.
  - Procurement vehicles through Carahsoft: GSA MAS 47QSWA18D008F, SEWP V, ITES-SW2, NASPO ValuePoint and OMNIA Partners. DoD components can use the CDAO production OTA without a new competition.
- sources:
  - https://scale.com/blog/scale-appoints-new-ceo (accessed 2026-10-05)
  - https://scale.com/blog/scale-ai-pentagon-cdao-500-million-agreement (accessed 2026-10-05)
  - https://scale.com/blog/scale-acquires-icg-solutions (accessed 2026-10-05)
  - https://scale.com/blog/scale-labs (accessed 2026-10-05)
  - https://scale.com/blog/ten-years-of-scale (accessed 2026-10-05)
  - https://scale.com/blog/deploying-enterprise-ai-agents-with-scale-and-google-cloud (accessed 2026-10-05)
  - https://scale.com/blog/mayo-clinic-scale (accessed 2026-10-05)
  - https://scale.com/blog/scale-ai-gsa-schedule-available (accessed 2026-10-05)
  - https://scale.com/blog/scale-ai-doe-genesis-mission-mou (accessed 2026-10-05)
  - https://scale.com/blog/scale-ai-joins-genesis-mission-consortium (accessed 2026-10-05)
  - https://scale.com/blog/singapore-imda-mou (accessed 2026-10-05)
  - https://scale.com/blog/scale-circadence-ai-driven-cyber-readiness (accessed 2026-10-05)
  - https://scale.com/blog/vero (accessed 2026-10-05)
  - https://scale.com/blog/swe-atlas-complete (accessed 2026-10-05)
  - https://scale.com/blog/drugdiscoverybench (accessed 2026-10-05)
  - https://scale.com/blog/scale-ai-announces-next-phase-of-company-evolution (accessed 2026-10-05)
  - https://scale.com/blog/sitemap.xml (accessed 2026-10-05)
  - https://labs.scale.com/leaderboard (accessed 2026-10-05)
  - https://labs.scale.com/papers (accessed 2026-10-05)
  - https://labs.scale.com/blog/swe-bench-pro-v2 (accessed 2026-10-05)
  - https://labs.scale.com/leaderboard/swe_bench_pro_public_v2 (accessed 2026-10-05)
  - https://labs.scale.com/blog/ready (accessed 2026-10-05)
  - https://arxiv.org/abs/2609.02095 (accessed 2026-10-05)
  - https://labs.scale.com/blog/verifier-design-for-cua (accessed 2026-10-05)
  - https://labs.scale.com/leaderboard/humanitys_last_exam (accessed 2026-10-05)
  - https://labs.scale.com/leaderboard/hil (accessed 2026-10-05)
  - https://labs.scale.com/rsi-benchmark (accessed 2026-10-05)
  - https://labs.scale.com/leaderboard/mcp_atlas (accessed 2026-10-05)
  - https://huggingface.co/datasets/ScaleAI/MCP-Atlas (accessed 2026-10-05)
  - https://labs.scale.com/leaderboard/rli (accessed 2026-10-05)
  - https://arxiv.org/abs/2505.24760 (accessed 2026-10-05)
  - https://scale.com/about (accessed 2026-10-05)
  - https://scale.com/rlenvironments (accessed 2026-10-05)
  - https://scale.com/security (accessed 2026-10-05)
  - https://outlier.ai (accessed 2026-10-05)
  - https://en.wikipedia.org/wiki/Alexandr_Wang (accessed 2026-10-05)
  - https://en.wikipedia.org/wiki/Lucy_Guo (accessed 2026-10-05)
  - https://en.wikipedia.org/wiki/Scale_AI (accessed 2026-10-05)
  - https://news.google.com/rss/search?q=%22Scale+AI%22+when:180d&hl=en-US&gl=US&ceid=US:en (accessed 2026-10-05)
  - https://scale.com/blog/rl-environments (accessed 2026-06-07)
  - https://scale.com/blog/soc2-hipaa (accessed 2026-06-07)
  - https://techcrunch.com/2025/06/13/scale-ai-confirms-significant-investment-from-meta-says-ceo-alexandr-wang-is-leaving/ (accessed 2026-06-07)
  - https://techcrunch.com/2025/09/21/silicon-valley-bets-big-on-environments-to-train-ai-agents/ (accessed 2026-06-07)
  - https://www.cnbc.com/2025/07/16/scale-ai-cuts-14percent-of-workforce-after-meta-investment-hiring-of-wang.html (accessed 2026-06-07)
  - https://techcrunch.com/2024/05/21/data-labeling-startup-scale-ai-raises-1b-as-valuation-doubles-to-13-8b/ (accessed 2026-06-07)
  - https://news.crunchbase.com/ai/scale-holistic-raise-big-accel-nvda-amzn/ (accessed 2026-06-07)
  - https://www.linkedin.com/company/scaleai (accessed 2026-06-07)
  - https://fortune.com/2025/06/14/self-made-billionaire-college-dropout-alexandr-wang-signs-14-3-billion-deal-to-bolster-metas-ai-efforts-theres-a-huge-premium-to-naivete/ (accessed 2026-06-07)

## Snorkel AI (not ranked, incumbent / infrastructure / open-source)
- slug: snorkel-ai
- segment: Incumbents also building RL environments
- website: https://snorkel.ai/
- focus_areas: Coding, Computer Use, Enterprise Workflows, Long-Horizon
- positioning: Snorkel AI is a San Francisco company spun out of the Stanford AI Lab in 2019 and known for the Snorkel weak-supervision project. It now calls itself 'the frontier AI data lab'. In 2025 it moved from selling Snorkel Flow software to an expert Data-as-a-Service model, and it now supplies expert agentic tasks, RL environments (computer-use, terminal and simulated-enterprise), rubrics and evals to AI labs and enterprises. It combines expert contributors with synthetic generation and agent-based QC, maintains Terminal-Bench, and co-authors or hosts agent benchmarks such as OSWorld 2.0, Agents' Last Exam and Senior SWE-bench through a $3M Open Benchmarks Grants program. It raised a $350M Series E at $3.5B in September 2026 and says its ARR passed $375M.
- best_fit: Frontier labs and enterprise AI teams that want a research-led, SOC 2 Type II/HIPAA-attested managed partner for hard expert data, rubrics and RL environments in coding/terminal, computer-use and regulated domains (finance, insurance, legal, healthcare), especially teams already using Terminal-Bench.
- overall_confidence: high
- founded_year: 2019 [confirmed], source: https://snorkel.ai/company/ (accessed 2026-10-05)
- status: active [confirmed], source: https://snorkel.ai/press/snorkel-ai-raises-350m-to-scale-the-data-factory-for-frontier-ai/ (accessed 2026-10-05)
- hq_location: San Francisco, CA, USA [confirmed], source: https://snorkel.ai/press/snorkel-ai-raises-350m-to-scale-the-data-factory-for-frontier-ai/ (accessed 2026-10-05)
- other_locations: New York City, NY, USA; Redwood City, CA, USA; US remote [reported], source: https://www.linkedin.com/company/snorkel-ai (accessed 2026-10-05)
- distributed_remote: unknown [unknown]
- current_headcount: Not publicly disclosed. LinkedIn band is 201-500 employees, and about 1,461 LinkedIn members list Snorkel, which likely includes expert contributors. [reported], source: https://www.linkedin.com/company/snorkel-ai (accessed 2026-10-05)
- headcount_band: 201-500 [reported], source: https://www.linkedin.com/company/snorkel-ai (accessed 2026-10-05)
- headcount_growth: unknown [unknown]
- open_roles_count: 41 (Greenhouse, 2026-10-05) [confirmed], source: https://job-boards.greenhouse.io/snorkelai (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: https://snorkel.ai/research/ (accessed 2026-10-05)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Founders from the Stanford AI Lab (Ratner, Stanford CS PhD; Varma, Stanford EE PhD; Ré, Stanford professor and MacArthur Fellow); Chief Scientist Frederic Sala, UW-Madison assistant professor (2024 DARPA Young Faculty Award); Co-founders Hancock and Ehrenberg came from Facebook/Google applied ML; Company claims 250+ peer-reviewed papers cited more than 25,000 times [confirmed], source: https://snorkel.ai/author/fred-sala/ (accessed 2026-10-05)
- published_papers_or_benchmarks: Terminal-Bench (2.0 to 4.0; Snorkel maintains and hosts it); Terminal-Bench+ (proprietary training curriculum); OSWorld 2.0 (XLANG Lab-led; Snorkel co-authors); Agents' Last Exam (UC Berkeley RDI-led; arXiv 2606.05405); Senior SWE-bench (with Princeton and UW-Madison); LibraryDesignBench (arXiv 2609.36730); SlopCodeBench; Continual Learning Bench; FinQA / SnorkelFinance environment and benchmark; Benchmarking Agents in Insurance Underwriting Environments (arXiv 2602.00456, CAIS); Learning from Less: RLVR in Low Data Regimes (arXiv 2604.18381; MLSys 2026 per Snorkel); RIFT / Hitchhiker's Guide to Rubric Quality (arXiv 2604.01375); Shrinking the Generation-Verification Gap with Weak Verifiers (NeurIPS 2025); BigLaw Bench: Research (co-released with Harvey); Snorkel: Rapid Training Data Creation with Weak Supervision (PVLDB 2017) [confirmed], source: https://snorkel.ai/research/ (accessed 2026-10-05)
- total_raised: $585M+ (disclosed rounds: $15M seed/A 2020, $35M B 2021, $85M C 2021, $100M D 2025, $350M E 2026-09; plus an undisclosed Accenture strategic investment in 2025-08) [reported], source: https://snorkel.ai/press/snorkel-ai-raises-350m-to-scale-the-data-factory-for-frontier-ai/ (accessed 2026-10-05)
- last_round: Series E, $350M, 2026-09-22 (co-led by Insight Partners and S32) [confirmed], source: https://snorkel.ai/press/snorkel-ai-raises-350m-to-scale-the-data-factory-for-frontier-ai/ (accessed 2026-10-05)
- notable_investors: Insight Partners (co-led Series E); S32 (co-led Series E); Addition (led Series D, co-led Series C); Lightspeed Venture Partners (led Series B); BlackRock (co-led Series C); Greylock; GV; In-Q-Tel; Accenture (strategic, 2025); Prosperity7; Wells Fargo; D.E. Shaw; SV Angel [confirmed], source: https://snorkel.ai/company/ (accessed 2026-10-05)
- valuation: $3.5B (Series E, Sep 2026) [confirmed], source: https://snorkel.ai/press/snorkel-ai-raises-350m-to-scale-the-data-factory-for-frontier-ai/ (accessed 2026-10-05)
- revenue_signals: Company-stated ARR above $375M (Sep 2026), up more than 18x in under a year since the September 2025 Data-as-a-Service launch. TechCrunch reports a $375M annualized run rate, up eighteenfold over 12 months. Reuters reports a run rate above $350M, up from about $20M a year earlier. TechCrunch notes that expert payments sit in cost of goods sold, not in headline revenue. [reported], source: https://techcrunch.com/2026/09/22/snorkel-ai-triples-valuation-to-3-5b-as-demand-for-ai-training-data-booms/ (accessed 2026-10-05)
- soc2: SOC 2 Type II [confirmed], source: https://security.snorkel.ai/ (accessed 2026-10-05)
- other_certifications: HIPAA; NIST CSF; AWS Qualified Software [confirmed], source: https://security.snorkel.ai/ (accessed 2026-10-05)
- security_page: https://security.snorkel.ai/ [confirmed], source: https://snorkel.ai/ (accessed 2026-10-05)
- what_they_sell: human data + environments [confirmed], source: https://snorkel.ai/ (accessed 2026-10-05)
- open_source: partial (FinQA OpenEnv environment and finqa-data dataset, Senior SWE-bench Harbor dataset, grant-funded open benchmarks, and the original Snorkel weak-supervision library; commercial data services and Terminal-Bench+ are proprietary) [confirmed], source: https://huggingface.co/snorkelai (accessed 2026-10-05)
- license: mixed: finqa-data Apache-2.0 (evaluation only); other open releases' licenses not verified; commercial services proprietary [confirmed], source: https://huggingface.co/datasets/snorkelai/finqa-data (accessed 2026-10-05)
- deployment_model: managed service (expert Data-as-a-Service delivering datasets, environments, rubrics and evals); some open environments and benchmarks published via OpenEnv/Hugging Face and Harbor [estimated], source: https://snorkel.ai/press/snorkel-ai-raises-350m-to-scale-the-data-factory-for-frontier-ai/ (accessed 2026-10-05)
- maturity: GA [confirmed], source: https://techcrunch.com/2026/09/22/snorkel-ai-triples-valuation-to-3-5b-as-demand-for-ai-training-data-booms/ (accessed 2026-10-05)
- notable_customers: OpenAI (self-claimed, frontier-lab tie); Anthropic (self-claimed, frontier-lab tie); Google (self-claimed, frontier-lab tie); Mistral AI (self-claimed, frontier-lab tie); Microsoft (self-claimed); Amazon Web Services (self-claimed); Accenture (verified) [reported], source: https://snorkel.ai/ (accessed 2026-10-05)
- founding_team:
  - Alex Ratner, Co-founder & CEO, Stanford CS PhD (advised by Christopher Ré), where he started and led the open-source Snorkel project. Affiliate assistant professor of computer science at the University of Washington.
  - Chris Ré, Co-founder, Stanford CS professor (Stanford AI Lab) and MacArthur Fellow. Also co-founded SambaNova, Lattice and Inductiv (both acquired by Apple). Sits on the Open Benchmarks Grants steering committee.
  - Paroma Varma, Co-founder & Head of Research, Stanford PhD in electrical engineering. Her research is on machine learning for domain experts without large labeled datasets, applied to medical imaging and autonomous driving. Co-authored Snorkel's 2026 RLVR low-data paper and the RIFT rubric paper.
  - Braden Hancock, Co-founder & Head of Technology, Spent four years on programmatic data labeling and augmentation at the Stanford AI Lab, Facebook and Google. Earlier did NLP/ML research at Johns Hopkins and MIT Lincoln Laboratory. BS in mechanical engineering from BYU.
  - Henry Ehrenberg, Co-founder (technical strategy and engineering), Built the original open-source Snorkel library at the Stanford AI Lab and later led the representation-learning team at Facebook Applied AI. Stanford MS in computational and mathematical engineering, Yale BS in applied mathematics. Led the Senior SWE-bench project in 2026.
  - Frederic Sala, Chief Scientist (non-founder), Assistant professor at UW-Madison working on data-centric AI and foundation models, with a 2024 DARPA Young Faculty Award. Sits on the Open Benchmarks Grants steering committee and co-authored OSWorld 2.0, LibraryDesignBench, RIFT and the RLVR low-data paper.
- funding_history:
  - 2026-09-22 Series E $350M led by Insight Partners, S32, valuation $3.5B, source: https://snorkel.ai/press/snorkel-ai-raises-350m-to-scale-the-data-factory-for-frontier-ai/
  - 2025-08-06 Strategic investment undisclosed led by Accenture, source: https://snorkel.ai/press/accenture-invests-in-snorkel-ai-to-help-financial-services-firms-transform-data-into-ai-solutions/
  - 2025-05-29 Series D $100M led by Addition, valuation $1.3B, source: https://snorkel.ai/press/snorkel-ai-raises-100-million-to-build-better-evaluators-for-ai-models/
  - 2021-08-09 Series C $85M led by BlackRock, Addition, valuation $1B, source: https://snorkel.ai/blog/85-million-series-c-accelerating-data-centric-ai-enterprise/
  - 2021-04-07 Series B $35M led by Lightspeed Venture Partners, source: https://snorkel.ai/blog/introducing-application-studio/
  - 2020-07 Seed + Series A $15M, source: https://snorkel.ai/blog/introducing-application-studio/
- recent_developments (last 6 months):
  - 2026-10-02 [research] Snorkel grant supports MedPAIR, a dataset comparing physician and LLM relevance judgments, source: https://snorkel.ai/blog/medpair-measuring-whether-physicians-and-ai-agree-on-what-matters-in-medical-qa/
  - 2026-09-28 [research] Blog: 'RL environments for LLM agents: Design, rewards, and validation', source: https://snorkel.ai/blog/rl-environments-for-llm-agents-design-rewards-and-validation/
  - 2026-09-22 [funding] Snorkel AI raises $350M Series E at $3.5B, co-led by Insight Partners and S32, source: https://snorkel.ai/press/snorkel-ai-raises-350m-to-scale-the-data-factory-for-frontier-ai/
  - 2026-09-22 [funding] TechCrunch: Snorkel AI triples valuation to $3.5B as demand for AI training data booms, source: https://techcrunch.com/2026/09/22/snorkel-ai-triples-valuation-to-3-5b-as-demand-for-ai-training-data-booms/
  - 2026-09-22 [product] CEO post 'Data 2.0 and the research era of AI data' describes the Agentic Data Platform, source: https://snorkel.ai/blog/data-2-0-and-the-research-era-of-ai-data/
  - 2026-09-15 [product] Snorkel describes Terminal-Bench+, a training-scale coding-agent curriculum dataset, source: https://snorkel.ai/blog/curriculum-learning-coding-agents/
  - 2026-09-04 [other] Fortune quotes Snorkel's Vincent Sunn Chen on OpenAI changing Astra eval metrics, source: https://snorkel.ai/press/fortune-openai-astra-evaluation-metrics/
  - 2026-09-03 [benchmark] OSWorld 2.0 blog: long-horizon computer-use agents still fail most tasks, source: https://snorkel.ai/blog/osworld-2-0-why-computer-use-agents-fail-most-tasks/
  - 2026-08-28 [benchmark] Terminal-Bench 4.0 and continuous benchmark QA, source: https://snorkel.ai/blog/terminal-bench-4-continuous-qa/
  - 2026-08-03 [research] Snorkel details enterprise RL environments (simulated companies) and training gains, source: https://snorkel.ai/blog/enterprise-environments-ai-agents/
  - 2026-07-24 [open-source] First wave of $3M Open Benchmarks Grants projects, source: https://snorkel.ai/press/snorkel-ai-highlights-first-wave-of-open-benchmarks-grants-projects/
  - 2026-07-01 [benchmark] Snorkel AI, Princeton and UW-Madison release Senior SWE-bench, source: https://snorkel.ai/press/researchers-from-snorkel-ai-princeton-and-uw-madison-release-senior-swe-bench/
  - 2026-04-20 [research] Paper: 'Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes', source: https://arxiv.org/abs/2604.18381
- benchmarks_and_research:
  - LibraryDesignBench (Can Agents Design Libraries for Agents?) (benchmark), 2026-09-29, On 11 of 15 tasks, agent designers reproduce the abstractions of the human-written production library. Downstream agents underuse the libraries and reimplement what they already provide., https://snorkel.ai/research-paper/can-agents-design-libraries-for-agents/
  - Terminal-Bench+ (training-scale curriculum) (dataset), 2026-09-15, Across 740 GPT-5.5 and Claude Opus 4.8 trajectories, skipping verification caused 43% of failures (self-reported)., https://snorkel.ai/blog/curriculum-learning-coding-agents/
  - Terminal-Bench 4.0 (benchmark), 2026-08-28, Leaderboard top: GPT-6 Astra, 58.2% ±2.8 (accessed 2026-10-05), https://snorkel.ai/leaderboard/terminal-bench-4-0
  - Senior SWE-bench (benchmark), 2026-07-01, Leaderboard top: Claude Fable 5.1 at a 34.7% 'tasteful solve rate' (pass@1, medium effort; accessed 2026-10-05), https://senior-swe-bench.snorkel.ai/
  - Agents' Last Exam (benchmark), 2026-06, Average full pass rate of 2.6% across mainstream harness and backbone configurations, https://snorkel.ai/research-paper/agents-last-exam/
  - Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes (paper), 2026-04-20, Mixed-complexity training gives up to 5x sample efficiency over training on easy tasks, https://arxiv.org/abs/2604.18381
  - The Hitchhiker's Guide to Rubric Quality Understanding and Enrichment (RIFT: Rubric Failure Mode Taxonomy) (paper), 2026-04-01, Quality signals detect failure modes with 75% accuracy, and 10 of 48 expert-authored rubrics weighted their criteria backwards, https://snorkel.ai/research/
  - FinQA RL environment (OpenEnv) (environment), 2026-03-30, A 4B model RL-trained on FinQA outperformed the 235B model from the same family (self-reported), https://snorkel.ai/blog/building-finqa-an-open-rl-environment-for-financial-reasoning-agents/
  - Benchmarking Agents in Insurance Underwriting Environments (Underwrite) (paper), 2026-01-31, Top models hallucinate domain knowledge despite tool access and show reliability gaps across repeated runs, https://snorkel.ai/research-paper/benchmarking-agents-in-insurance-underwriting-environments/
  - OSWorld 2.0 (benchmark), 2026, In the paper, the best system (Claude Opus 4.8, max thinking) completes 20.6% outright with a 54.8% partial score. Snorkel's leaderboard reports Claude Fable 5.1 at about 45% binary., https://snorkel.ai/research-paper/os-world-2/
  - Snorkel: Rapid Training Data Creation with Weak Supervision (paper), 2017, Subject-matter experts built models 2.8x faster, with predictive performance 45.5% higher on average than seven hours of hand labeling, https://arxiv.org/abs/1711.10160
  - Snorkel leaderboards (leaderboard), https://snorkel.ai/leaderboard/
- products:
  - Expert Data-as-a-Service: Managed development of expert datasets for frontier and enterprise model training and evaluation: agentic tasks, environments, rubrics and evals. Delivered by an expert contributor community (PhDs and domain specialists) working with Snorkel's internal Agentic Data Platform. A May 2025 announcement introduced 'Snorkel Evaluate and Expert Data-as-a-Service' for enterprises. The Series E release and Reuters date the launch of the frontier DaaS business to September 2025., https://snorkel.ai/
  - RL environments: Computer-use, terminal-based and enterprise-domain environments. The enterprise environments are simulated companies (insurance, finance, manufacturing, legal, sales/marketing) used for evaluation and RL training., https://snorkel.ai/blog/enterprise-environments-ai-agents/
  - Terminal-Bench+: A proprietary, training-scale curriculum of expert-authored coding and terminal tasks (50,000+ tasks per Snorkel). Samples available on request., https://snorkel.ai/blog/curriculum-learning-coding-agents/
  - Evaluation systems & benchmarks: Task-specific rubrics, programmatic graders and runnable environments, plus Snorkel-hosted public leaderboards., https://snorkel.ai/leaderboard/
  - Specialized agents: Custom AI systems built on specialized data for high-stakes enterprise workflows., https://snorkel.ai/
  - Open Benchmarks Grants: A $3M grants program funding open agent benchmarks and datasets (Agents' Last Exam, OSWorld 2.0, Terminal-Bench 2.1, Continual Learning Bench, SlopCode Bench, Senior SWE-Bench, MedPAIR and others). The steering committee includes Chris Ré, Fred Sala, Karthik Narasimhan, Ludwig Schmidt, Yu Su and Lewis Tunstall., https://benchmarks.snorkel.ai/
  - FinQA environment (open, OpenEnv): An open financial-reasoning RL environment on OpenEnv, published as the Hugging Face Space snorkelai/finqa-env., https://huggingface.co/spaces/snorkelai/finqa-env
  - Snorkel Flow (legacy platform): Snorkel's original programmatic data-labeling and development platform, sold as software before the 2025 move to data-as-a-service., https://snorkel.ai/press/snorkel-ai-launches-snorkel-flow-to-help-firms-harness-their-data-for-custom-ai-solutions/
- notes_for_researchers:
  - Open RL environment: FinQA on Meta PyTorch / Hugging Face OpenEnv, with four MCP tools, a binary reward with fuzzy numeric matching, and integration with TRL, TorchForge, veRL, SkyRL and Unsloth. The data, HF snorkelai/finqa-data, is Apache-2.0 and labelled evaluation only ('Do not train on it').
  - Snorkel co-authors, maintains or hosts several agent benchmarks: Terminal-Bench (2.0 to 4.0), OSWorld 2.0 (led by XLANG Lab), Agents' Last Exam (led by UC Berkeley RDI, arXiv 2606.05405), Senior SWE-bench (with Princeton and UW-Madison, Harbor-native), LibraryDesignBench (arXiv 2609.36730) and SlopCodeBench. Many of these are grant-funded collaborations rather than Snorkel-only work.
  - Published RL and data research: RLVR in Low Data Regimes (arXiv 2604.18381, which Snorkel lists as MLSys 2026), RIFT rubric failure taxonomy (arXiv 2604.01375), Shrinking the Generation-Verification Gap with Weak Verifiers (NeurIPS 2025) and the Underwrite insurance agent benchmark (arXiv 2602.00456, CAIS).
  - Self-reported RL-training results on Snorkel's own environments: Qwen3-30B on the insurance enterprise environment (+8.1 BFCL Multi-Turn, +5.8 τ³-bench overall, +14.0 τ³-bench airline), and a 4B model on FinQA beating the 235B model from the same family.
  - Research leadership: Chief Scientist Frederic Sala (UW-Madison) and co-founder and Head of Research Paroma Varma. The company claims 250+ peer-reviewed papers cited more than 25,000 times.
- notes_for_program_managers:
  - Scale: $350M Series E at $3.5B (2026-09-22, co-led by Insight Partners and S32). The company says ARR passed $375M, and TechCrunch reports the same figure. Reuters reports a run rate above $350M, up from about $20M a year earlier. Disclosed funding totals about $585M, plus an undisclosed 2025 strategic investment from Accenture.
  - Frontier-lab ties are self-claimed. The homepage partner logo wall shows Google, Anthropic, OpenAI, Mistral AI, Microsoft and AWS. The CEO says Snorkel partners with 'frontier labs, hyperscalers, neolabs ... and U.S. government agencies' without naming them. Reuters quotes the CEO on demand 'from frontier AI labs', but no press source names a lab customer.
  - Engagement model: a managed 'frontier AI data factory' that delivers expert tasks, environments, rubrics and evals. It combines synthetic generation and hundreds of QC agents with expert contributors, so it is not a pure human marketplace. TechCrunch notes that expert payments sit in cost of goods sold.
  - Government and enterprise signals: completed a Defense Innovation Unit challenge (2025-12), won the Army xTech AI Grand Challenge (2025-08), and has a strategic partnership and investment with Accenture for financial services (2025-08).
  - Maturity: founded in 2019 out of the Stanford AI Lab and moved from Snorkel Flow software to data-as-a-service in 2025. LinkedIn places it in the 201-500 band, and Greenhouse lists 41 open roles, about 12 of them in DaaS functions.
- notes_for_data_ops:
  - Security: the SafeBase trust center at security.snorkel.ai lists SOC 2 Type II, HIPAA, NIST CSF and AWS Qualified Software. The SOC 2 report, HIPAA report, pen-test report, security whitepaper and DPA are available through the portal. Infrastructure runs on AWS.
  - Publicly shown formats: OpenEnv environments with MCP tool servers in Docker (FinQA), Harbor-format benchmark datasets (Senior SWE-bench on GitHub and Harbor Hub), and Hugging Face datasets and Spaces under the snorkelai org.
  - Code and data: the GitHub org github.com/snorkel-ai (senior-swe-bench-v2026.06, terminal-bench, long-context-eval) and Hugging Face at huggingface.co/snorkelai.
  - Snorkel has not published delivery formats, APIs or licensing for its commercial datasets, environments or Terminal-Bench+. Confirm export formats, harness compatibility and data-use terms during procurement.
- sources:
  - https://snorkel.ai/ (accessed 2026-10-05)
  - https://snorkel.ai/press/snorkel-ai-raises-350m-to-scale-the-data-factory-for-frontier-ai/ (accessed 2026-10-05)
  - https://techcrunch.com/2026/09/22/snorkel-ai-triples-valuation-to-3-5b-as-demand-for-ai-training-data-booms/ (accessed 2026-10-05)
  - https://snorkel.ai/press/snorkel-ai-valued-at-3-5-billion-amid-surging-demand-for-complex-ai-training-data-2/ (accessed 2026-10-05)
  - https://snorkel.ai/press/training-data-provider-snorkel-ai-raises-350m-at-3-5b-valuation/ (accessed 2026-10-05)
  - https://snorkel.ai/blog/data-2-0-and-the-research-era-of-ai-data/ (accessed 2026-10-05)
  - https://snorkel.ai/company/ (accessed 2026-10-05)
  - https://snorkel.ai/blog/introducing-application-studio/ (accessed 2026-10-05)
  - https://snorkel.ai/blog/85-million-series-c-accelerating-data-centric-ai-enterprise/ (accessed 2026-10-05)
  - https://snorkel.ai/press/snorkel-ai-scores-35m-series-b-to-automate-data-labeling-in-machine-learning-apps/ (accessed 2026-10-05)
  - https://snorkel.ai/press/meet-the-stanford-ai-lab-alums-that-raised-15-million-to-optimize-machine-learning/ (accessed 2026-10-05)
  - https://snorkel.ai/press/snorkel-ai-raises-100-million-to-build-better-evaluators-for-ai-models/ (accessed 2026-10-05)
  - https://snorkel.ai/press/accenture-invests-in-snorkel-ai-to-help-financial-services-firms-transform-data-into-ai-solutions/ (accessed 2026-10-05)
  - https://snorkel.ai/press/snorkel-ai-completes-defense-innovation-unit-challenge/ (accessed 2026-10-05)
  - https://snorkel.ai/wp-json/wp/v2/press?per_page=50 (accessed 2026-10-05)
  - https://snorkel.ai/research/ (accessed 2026-10-05)
  - https://snorkel.ai/blog/building-finqa-an-open-rl-environment-for-financial-reasoning-agents/ (accessed 2026-10-05)
  - https://huggingface.co/datasets/snorkelai/finqa-data (accessed 2026-10-05)
  - https://huggingface.co/snorkelai (accessed 2026-10-05)
  - https://github.com/snorkel-ai (accessed 2026-10-05)
  - https://snorkel.ai/blog/enterprise-environments-ai-agents/ (accessed 2026-10-05)
  - https://snorkel.ai/blog/curriculum-learning-coding-agents/ (accessed 2026-10-05)
  - https://snorkel.ai/blog/rl-environments-for-llm-agents-design-rewards-and-validation/ (accessed 2026-10-05)
  - https://snorkel.ai/blog/2026-the-year-of-environments/ (accessed 2026-10-05)
  - https://snorkel.ai/blog/terminal-bench-4-continuous-qa/ (accessed 2026-10-05)
  - https://snorkel.ai/leaderboard/terminal-bench-4-0 (accessed 2026-10-05)
  - https://snorkel.ai/blog/osworld-2-0-why-computer-use-agents-fail-most-tasks/ (accessed 2026-10-05)
  - https://snorkel.ai/press/snorkel-ai-highlights-first-wave-of-open-benchmarks-grants-projects/ (accessed 2026-10-05)
  - https://benchmarks.snorkel.ai/ (accessed 2026-10-05)
  - https://snorkel.ai/press/researchers-from-snorkel-ai-princeton-and-uw-madison-release-senior-swe-bench/ (accessed 2026-10-05)
  - https://senior-swe-bench.snorkel.ai/ (accessed 2026-10-05)
  - https://snorkel.ai/research-paper/agents-last-exam/ (accessed 2026-10-05)
  - https://snorkel.ai/research-paper/os-world-2/ (accessed 2026-10-05)
  - https://snorkel.ai/research-paper/can-agents-design-libraries-for-agents/ (accessed 2026-10-05)
  - https://arxiv.org/abs/2609.36730 (accessed 2026-10-05)
  - https://arxiv.org/abs/2604.18381 (accessed 2026-10-05)
  - https://arxiv.org/abs/2604.01375 (accessed 2026-10-05)
  - https://arxiv.org/abs/2602.00456 (accessed 2026-10-05)
  - https://arxiv.org/abs/1711.10160 (accessed 2026-10-05)
  - https://snorkel.ai/blog/medpair-measuring-whether-physicians-and-ai-agree-on-what-matters-in-medical-qa/ (accessed 2026-10-05)
  - https://snorkel.ai/press/fortune-openai-astra-evaluation-metrics/ (accessed 2026-10-05)
  - https://security.snorkel.ai/ (accessed 2026-10-05)
  - https://job-boards.greenhouse.io/snorkelai (accessed 2026-10-05)
  - https://snorkel.ai/join-us/ (accessed 2026-10-05)
  - https://www.linkedin.com/company/snorkel-ai (accessed 2026-10-05)
  - https://snorkel.ai/author/alex-ratner/ (accessed 2026-10-05)
  - https://snorkel.ai/author/christopher-re/ (accessed 2026-10-05)
  - https://snorkel.ai/author/paroma-varma/ (accessed 2026-10-05)
  - https://snorkel.ai/author/braden-hancock/ (accessed 2026-10-05)
  - https://snorkel.ai/author/henry-ehrenberg/ (accessed 2026-10-05)
  - https://snorkel.ai/author/fred-sala/ (accessed 2026-10-05)

## Modal (not ranked, incumbent / infrastructure / open-source)
- slug: modal
- segment: Adjacent: execution infrastructure
- website: https://modal.com
- focus_areas: Coding
- positioning: Modal (Modal Labs) is a New York-based, Python-native serverless cloud for AI workloads. It offers on-demand GPU/CPU compute, multi-node clusters with RDMA, managed LLM inference endpoints, and fast-booting sandboxes, including full VM sandboxes, which Modal says scale to 1M concurrent. It sells execution infrastructure, not RL environments. AI labs, agent companies and RL-data vendors use it to run RL rollouts, post-training and large fleets of parallel sandboxed environments. It integrates with Anthropic's Claude Managed Agents and Claude Science, Cognition's Devin Outposts and the OpenAI Agents SDK, and ships an open-source RL/SFT library (Modal Dojo).
- best_fit: Buyers needing serverless, autoscaling compute to run untrusted code, RL training loops, and thousands of parallel sandboxed agent/coding environments without managing infrastructure.
- overall_confidence: high
- founded_year: 2021 [confirmed], source: https://techcrunch.com/2026/02/11/ai-inference-startup-modal-labs-in-talks-to-raise-at-2-5b-valuation-sources-say/ ; founded Jan 2021 corroborated by startupintros.com / Sacra (accessed 2026-06-07)
- status: active [confirmed], source: https://modal.com/blog/modal-series-c (accessed 2026-06-07)
- hq_location: New York, NY, USA [confirmed], source: https://modal.com/company (accessed 2026-06-07)
- other_locations: San Francisco, CA, USA; Stockholm, Sweden; London, UK [confirmed], source: https://modal.com/blog/modal-europe-expansion (accessed 2026-10-05)
- distributed_remote: yes [estimated], source: https://modal.com/company (accessed 2026-06-07), three offices (NYC, SF, Stockholm) with roles spanning all locations
- current_headcount: ~150 employees (TechCrunch, Sept 2026); 120+ per official Series C post (May 2026) [reported], source: https://techcrunch.com/2026/09/28/source-inference-provider-modal-labs-closing-in-on-750m-round-at-15-75b-valuation/ (accessed 2026-10-05)
- headcount_band: 51-200 [confirmed], source: LinkedIn/Tracxn ~153 employees ; https://modal.com/blog/modal-series-c (120+) (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: 30+ open roles across NYC, SF, Stockholm and London [reported], source: https://modal.com/company (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: https://modal.com/blog/multi-token-residual-prediction (accessed 2026-10-05)
- researcher_count: unknown [unknown]
- researcher_backgrounds: CEO Erik Bernhardsson: ex-Spotify (music recommendations), ex-CTO Better.com, creator of open-source Annoy and Luigi; CTO Akshat Bubna: ex-Scale AI engineer, MIT [reported], source: https://erikbern.com/about.html ; https://techcrunch.com/2026/02/11/ai-inference-startup-modal-labs-in-talks-to-raise-at-2-5b-valuation-sources-say/ ; startupintros.com founder profiles (accessed 2026-06-07)
- published_papers_or_benchmarks: Multi-token Residual Prediction (arXiv 2605.18817; co-authored by Modal Research, 2026-05); LLM Engineer's Almanac inference benchmarks (modal.com/llm-almanac; code: modal-labs/stopwatch); Quail open-source AI-SQL inference engine with CMU Full Stack Data Lab (2026-09); FlashAttention-4 inference contributions (2026-06); SOTA speculative decoding / open-source DFlash speculators with Z Lab and SGLang (2026-06); Trillion-token / trillion-parameter inference engineering write-up (2026-09); GPU Glossary (modal.com/gpu-glossary) [confirmed], source: https://modal.com/blog/multi-token-residual-prediction (accessed 2026-10-05)
- total_raised: ~$466M total disclosed [reported], source: https://modal.com/company ; corroborated by Sacra and startupintros.com (accessed 2026-06-07)
- last_round: Series C, $355M, closed ~2026-05-21 [confirmed], source: https://modal.com/blog/modal-series-c ; https://siliconangle.com/2026/05/21/serverless-ai-infrastructure-startup-modal-labs-seals-355m-funding-round/ ; https://www.datacenterdynamics.com/en/news/modal-labs-secures-funding/ ; https://www.finsmes.com/2026/05/modal-raises-355m-in-series-c-funding-at-post-money-valuation-of-4-65-billion.html (accessed 2026-06-07)
- notable_investors: General Catalyst (Series C co-lead); Redpoint Ventures (Series C co-lead; earlier Series A lead); Lux Capital (Series B lead); Accel (Series C participant; reported lead of the ~$750M round, Sept 2026); Amplify Partners; Menlo Ventures; Bain Capital Ventures; Creandum [reported], source: https://modal.com/company (accessed 2026-10-05)
- valuation: $4.65B post-money (Series C, May 2026, confirmed); a ~$750M round at ~$15.75B led by Accel was reported as nearly closed (TechCrunch, 2026-09-28; single source, not confirmed by Modal) [reported], source: https://techcrunch.com/2026/09/28/source-inference-provider-modal-labs-closing-in-on-750m-round-at-15-75b-valuation/ (accessed 2026-10-05)
- revenue_signals: ~$300M annualized revenue run rate (May 2026), grown ~5x since September; ~$50M ARR reported Feb 2026 [reported], source: https://modal.com/blog/modal-series-c ; https://www.datacenterdynamics.com/en/news/modal-labs-secures-funding/ ; https://techcrunch.com/2026/02/11/ai-inference-startup-modal-labs-in-talks-to-raise-at-2-5b-valuation-sources-say/ (accessed 2026-06-07)
- soc2: Type II [confirmed], source: https://modal.com/docs/guide/security (SOC 2 Type II audit completed; report via trust.modal.com) (accessed 2026-06-07)
- other_certifications: HIPAA: supports HIPAA-compliant workloads via BAA on Enterprise plan (no formal HIPAA certification exists); PCI handled via Stripe (Level 1) [confirmed], source: https://modal.com/docs/guide/security (accessed 2026-06-07)
- security_page: https://trust.modal.com (security portal); https://modal.com/docs/guide/security [confirmed], source: https://modal.com/docs/guide/security (accessed 2026-06-07)
- what_they_sell: infra [confirmed], source: https://modal.com/ (accessed 2026-06-07)
- open_source: no [confirmed], source: https://modal.com/ (accessed 2026-06-07), core serverless platform is proprietary/managed; only client SDKs are open source
- license: Client SDKs open source (Python client Apache-2.0, JS/TS/Go libmodal MIT, examples MIT); platform itself proprietary [confirmed], source: https://github.com/modal-labs/modal-client ; https://github.com/modal-labs/libmodal (accessed 2026-06-07)
- deployment_model: managed-hosted (serverless cloud); Python/JS/TS/Go SDKs and API; managed LLM endpoints with token-based pricing (Auto/Managed Endpoints) [confirmed], source: https://modal.com/blog/introducing-auto-endpoints (accessed 2026-10-05)
- maturity: GA [confirmed], source: https://modal.com/ (accessed 2026-06-07)
- notable_customers: Cognition (Devin) (self-claimed); Applied Compute (self-claimed); Physical Intelligence (self-claimed); Suno (self-claimed); Ramp (self-claimed); DoorDash (self-claimed); Decagon (self-claimed); Chai Discovery (self-claimed); Reducto (self-claimed); Runway (self-claimed); Black Forest Labs (self-claimed); Legora (self-claimed); Linear (self-claimed); Snorkel (self-claimed); 1X (self-claimed); Imbue (self-claimed); Blend (self-claimed); Mason AI (self-claimed); Meta (Code World Models RL sandboxes) (self-claimed, frontier-lab tie) [reported], source: https://modal.com/blog/vm-sandboxes-agent-computers (accessed 2026-10-05)
- founding_team:
  - Erik Bernhardsson, Co-founder & CEO, Built the first version of Spotify's music recommendation system (Discover Weekly, Radio, Related Artists). Former CTO of Better.com, where he led a ~300-person tech team. Created Luigi, Annoy and ann-benchmarks. M.Sc. in Physics from KTH Stockholm.
  - Akshat Bubna, Co-founder & CTO, Former engineer at Scale AI; MIT (reported in the baseline's third-party sources, not re-verified in this pass).
- funding_history:
  - 2026-09-28 undisclosed $750M led by Accel, valuation $15.75B, source: https://techcrunch.com/2026/09/28/source-inference-provider-modal-labs-closing-in-on-750m-round-at-15-75b-valuation/
- recent_developments (last 6 months):
  - 2026-10-01 [product] Runtime by Modal: VM Sandboxes, Modal Clusters and Sticky Sessions reach GA; Endpoint Candidates enters private beta, source: https://modal.com/blog/runtime-product-update-sandbox-endpoints
  - 2026-10-01 [product] Sandbox Sidecars: a trust boundary next to agent sandboxes, source: https://modal.com/blog/introducing-sandbox-sidecars-trust-boundary
  - 2026-09-28 [funding] TechCrunch: Modal close to raising $750M at a $15.75B valuation, led by Accel, source: https://techcrunch.com/2026/09/28/source-inference-provider-modal-labs-closing-in-on-750m-round-at-15-75b-valuation/
  - 2026-09-24 [open-source] Quail: open-source AI-SQL inference engine built with CMU's Full Stack Data Lab, source: https://modal.com/blog/quail-billion-tpm
  - 2026-09-23 [research] Engineering write-up: serving trillion-parameter Kimi K2.6 at trillions of tokens, source: https://modal.com/blog/trillion-tokens-trillion-parameters
  - 2026-09-15 [product] New controls for unauthenticated web endpoints, source: https://modal.com/changelog/new-controls-for-unauthenticated-web-endpoints
  - 2026-09-14 [product] Product update: Sandbox Sidecars (public alpha), VM Sandboxes (public beta), filesystem API and directory snapshots GA, new open-weight model endpoints, source: https://modal.com/blog/product-updates-sidecars-models-dashboard
  - 2026-09-02 [hire] Modal opens a London office and hires a VP EMEA, source: https://modal.com/blog/modal-europe-expansion
  - 2026-08-14 [product] Changelog: Sandbox filesystem API GA; V2 Sandboxes opt-in (directory snapshots GA on 2026-08-12), source: https://modal.com/changelog/sandbox-filesystem-api-is-now-generally-available
  - 2026-07-29 [legal] Modal statement on the Hugging Face agent incident, source: https://modal.com/blog/a-note-on-the-hugging-face-agent-incident
  - 2026-07-21 [partnership] Modal is a launch partner for Cognition's Devin Outposts and open-sources modal-devin, source: https://modal.com/blog/devin-outposts-run-devin-in-modal-sandoxes
  - 2026-07-16 [product] Rebuilt sandbox scheduler runs 1 million concurrent sandboxes (beta), source: https://modal.com/blog/scaling-to-1-million-concurrent-sandboxes-in-seconds
  - 2026-07-15 [partnership] Day-zero hosting of Thinking Machines' Inkling model as a Managed Endpoint, source: https://modal.com/blog/inkling-by-thinking-machines-labs-now-available-on-modal
  - 2026-07-01 [research] Multi-token Residual Prediction (MRP) paper co-authored by Modal Research, source: https://modal.com/blog/multi-token-residual-prediction
  - 2026-06-30 [partnership] Modal integrates with Anthropic's Claude Science, source: https://modal.com/blog/modal-integration-brings-scalable-compute-to-claude-science
  - 2026-06-24 [research] State-of-the-art speculative decoding write-up; open-source DFlash speculators, source: https://modal.com/blog/achieve-sota-specdec
  - 2026-06-23 [product] Modal Auto Endpoints launched, source: https://modal.com/blog/introducing-auto-endpoints
  - 2026-06-11 [open-source] Modal upstreams inference optimizations to FlashAttention-4, source: https://modal.com/blog/flash-attention-4-faster
  - 2026-06-01 [open-source] Modal argues RL is an infrastructure problem and open-sources Modal Training Gym (now modal-dojo), source: https://modal.com/blog/reinforcement-learning-infrastructure-problem
  - 2026-05-21 [funding] Series C: $355M at $4.65B post-money (already in baseline), source: https://modal.com/blog/modal-series-c
  - 2026-05-20 [customer] Applied Compute case study: running RL on Modal, source: https://modal.com/blog/applied-compute-reinforcement-learning
  - 2026-05-19 [partnership] Integration with Anthropic's Claude Managed Agents, source: https://modal.com/blog/introducing-claude-managed-agents-with-modal-sandboxes
  - 2026-04-29 [research] AE Studio case study: an RL theorem-proving workflow on Modal, source: https://modal.com/blog/building-an-rl-theorem-proving-workflow-on-modal
  - 2026-04-15 [product] Modal sandbox extension for the OpenAI Agents SDK, source: https://modal.com/blog/building-with-modal-and-the-openai-agent-sdk
  - 2026-04-14 [open-source] Autoscaling autoresearch, plus open-source Modal Auto-Research Skills, source: https://modal.com/blog/autoscaling-autoresearch
  - 2026-04-10 [acquisition] Modal acquires Butter (agent sandbox and computer-use startup), source: https://modal.com/blog/butter-is-joining-modal
  - 2026-04-08 [customer] Physical Intelligence runs real-time robot inference on Modal, source: https://modal.com/blog/physical-intelligence-runs-real-time-remote-inference-for-robotic-control-on-modal
  - 2026-04-07 [product] RTX Pro 6000 Blackwell GPUs, Sandbox filesystem API beta, SDK 1.4.0, source: https://modal.com/blog/product-updates-rtx-pro-6000-command-k-sandbox-fs-api-and-more
- benchmarks_and_research:
  - Quail (QUery-Aware Inference Layer) (paper), 2026-09-24, Over 1B tokens per minute per H100; 1.84x faster than vLLM (geometric mean across tasks on the AI-SQL benchmark). Vendor-reported., https://modal.com/blog/quail-billion-tpm
  - State-of-the-art speculative decoding with DFlash (paper), 2026-06-24, p50 latency cut from about 290 ms to about 190 ms on a customer voice-AI workload, 60 ms faster than proprietary providers (vendor-reported)., https://modal.com/blog/achieve-sota-specdec
  - FlashAttention-4 inference contributions (paper), 2026-06-11, Up to 4.37x throughput for small query lengths from KV-dimension parallelism (vendor-reported)., https://modal.com/blog/flash-attention-4-faster
  - Multi-token Residual Prediction (MRP) (paper), 2026-05-12, Up to 1.4x lossless speedup in SGLang (arXiv abstract; Modal's blog says up to 1.56x); +22.6 points on HumanEval and +17.7 on GSM8K under aggressive decoding., https://arxiv.org/abs/2605.18817
  - LLM Engineer's Almanac (benchmark), Out of the box, vLLM and SGLang perform comparably; TensorRT-LLM can be faster but needs significant tuning., https://modal.com/llm-almanac
  - GPU Glossary (dataset), https://modal.com/gpu-glossary
- products:
  - Modal Sandboxes: Fast-starting containers for untrusted code and agent or RL environments, isolated with gVisor or an optional VM runtime. Additions in the window: filesystem API (GA 2026-08), directory snapshots (GA 2026-08-12), and a rebuilt scheduler that Modal says reaches 1M concurrent sandboxes (beta, 2026-07)., https://modal.com/docs/guide/sandbox
  - VM Sandboxes: Full Linux VMs for agents, with Docker and Docker-in-Docker, local databases, graphical environments and kernel-level access. They use the same APIs as Sandboxes. Linear, Legora and Snorkel are named as users (self-claimed)., https://modal.com/blog/vm-sandboxes-agent-computers
  - Sandbox Sidecars: Isolated containers next to a Sandbox for credential injection, egress proxying, redaction and monitoring. Public alpha on 2026-09-14. On 2026-10-01 Modal's Runtime roundup called them GA while the dedicated post said beta., https://modal.com/blog/introducing-sandbox-sidecars-trust-boundary
  - Modal Clusters: Multi-node training and inference through the @modal.clustered decorator, with gang scheduling and RDMA up to 6.4 Tbps. Modal names Decagon (post-training), 1X (pre-training) and Runway (inference) as users., https://modal.com/blog/modal-clusters-generally-available
  - Modal Functions (serverless GPU/CPU compute): Python-native serverless compute with autoscaling GPUs for inference, training, fine-tuning and batch jobs. SDKs for Python, JS/TS and Go., https://modal.com/
  - Modal Auto Endpoints / Managed Endpoints: One-command LLM inference deployment with an OpenAI-compatible API, engine-level metrics and DFlash speculative decoding. Shared endpoints use token-based pricing. Endpoint Candidates, for testing serving setups against mirrored traffic, entered private beta on 2026-10-01., https://modal.com/blog/introducing-auto-endpoints
  - Modal Dojo (formerly Modal Training Gym): Experimental MIT-licensed library for RL and SFT post-training on Modal. A job is defined by a reward function, a model and an environment., https://github.com/modal-projects/modal-dojo
  - modal-devin (Devin Outposts integration): Open-source integration that runs Cognition's Devin sessions in Modal Sandboxes., https://modal.com/blog/devin-outposts-run-devin-in-modal-sandoxes
  - Claude Managed Agents integration: Runs tool calls from Anthropic's Claude Managed Agents in Modal Sandboxes., https://modal.com/blog/introducing-claude-managed-agents-with-modal-sandboxes
  - Claude Science integration: Connects a Modal workspace to Anthropic's Claude Science workbench so that GPU and multi-CPU scientific workloads run on Modal., https://modal.com/blog/modal-integration-brings-scalable-compute-to-claude-science
- notes_for_researchers:
  - Modal has a named research group: the MRP paper's co-authors are listed as 'Modal Research'. Its published inference work includes Quail (with CMU), the FlashAttention-4 PRs, DFlash speculators (with Z Lab and SGLang), the trillion-parameter serving write-up, and the LLM Engineer's Almanac benchmarks with open code (modal-labs/stopwatch).
  - RL tooling: Modal Dojo (formerly Training Gym, MIT) defines RL or SFT jobs from a reward function, model and environment. Modal says it supports customers doing SFT and RL on GLM 4.7 and Kimi K2.6. AE Studio published a reproducible Lean theorem-proving RL case study (ES vs GRPO).
  - Open releases: modal-dojo, modal-auto-research-skills, modal-devin, Quail and the DFlash speculators. The core platform is proprietary.
  - RL use is published as vendor case studies: Applied Compute (rollouts, evals and inference with thousands of parallel sandboxes), Snorkel (millions of agent simulations a month on VM Sandboxes) and Decagon (post-training on Modal Clusters).
- notes_for_program_managers:
  - Modal sells infrastructure for running RL environments. It does not sell environments or data.
  - Frontier-lab ties: two Anthropic integrations (Claude Managed Agents, May 2026; Claude Science, June 2026), day-zero hosting of Thinking Machines' Inkling (July 2026), and launch partner for Cognition's Devin Outposts (July 2026). The baseline's Meta/CWM sandbox usage is still self-claimed.
  - Scale signals: $355M Series C at $4.65B (May 2026). A ~$750M round at $15.75B led by Accel was reported by TechCrunch on 2026-09-28 from a single source and is unconfirmed. Annualized revenue was over $300M in May 2026, and TechCrunch puts headcount at about 150.
  - Offices in NYC (HQ), SF, Stockholm and London; London opened in September 2026 with a VP EMEA. Modal acquired Butter in April 2026 to strengthen its sandbox team. Modal's first Runtime conference was held on 2026-10-01.
- notes_for_data_ops:
  - SOC 2 Type 2, with the report on trust.modal.com. HIPAA via BAA on the Enterprise plan. The security docs list no ISO 27001 or GDPR certification.
  - Isolation uses gVisor, with an optional VM runtime; VM Sandboxes went GA on 2026-10-01. Function inputs and outputs are kept for at most 7 days.
  - July 2026 Hugging Face incident: a customer's public, unauthenticated code-execution endpoint was abused. Modal says its isolation held. In September it added controls to block unauthenticated endpoints workspace-wide, with audit-log events and tagging.
  - Enterprise controls added August-September 2026: environment budgets, RBAC user groups with SCIM sync, SSO IdP XML upload, regional proxies, Private Environments, and a public changelog at modal.com/changelog. Sidecars can sit in front of agent egress as a proxy.
  - Integrations: Python, JS/TS and Go SDKs; Claude Managed Agents; Claude Science; Devin Outposts; the OpenAI Agents SDK sandbox extension (April 2026).
- sources:
  - https://techcrunch.com/2026/09/28/source-inference-provider-modal-labs-closing-in-on-750m-round-at-15-75b-valuation/ (accessed 2026-10-05)
  - https://modal.com/company (accessed 2026-10-05)
  - https://modal.com/sitemap.xml (accessed 2026-10-05)
  - https://modal.com/blog/runtime-product-update-sandbox-endpoints (accessed 2026-10-05)
  - https://modal.com/blog/vm-sandboxes-agent-computers (accessed 2026-10-05)
  - https://modal.com/blog/modal-clusters-generally-available (accessed 2026-10-05)
  - https://modal.com/blog/introducing-sandbox-sidecars-trust-boundary (accessed 2026-10-05)
  - https://modal.com/blog/modal-europe-expansion (accessed 2026-10-05)
  - https://modal.com/blog/a-note-on-the-hugging-face-agent-incident (accessed 2026-10-05)
  - https://modal.com/changelog/new-controls-for-unauthenticated-web-endpoints (accessed 2026-10-05)
  - https://modal.com/changelog (accessed 2026-10-05)
  - https://modal.com/changelog/sandbox-filesystem-api-is-now-generally-available (accessed 2026-10-05)
  - https://modal.com/blog/scaling-to-1-million-concurrent-sandboxes-in-seconds (accessed 2026-10-05)
  - https://modal.com/blog/devin-outposts-run-devin-in-modal-sandoxes (accessed 2026-10-05)
  - https://modal.com/blog/introducing-claude-managed-agents-with-modal-sandboxes (accessed 2026-10-05)
  - https://modal.com/blog/modal-integration-brings-scalable-compute-to-claude-science (accessed 2026-10-05)
  - https://modal.com/blog/inkling-by-thinking-machines-labs-now-available-on-modal (accessed 2026-10-05)
  - https://modal.com/blog/butter-is-joining-modal (accessed 2026-10-05)
  - https://modal.com/blog/applied-compute-reinforcement-learning (accessed 2026-10-05)
  - https://modal.com/blog/reinforcement-learning-infrastructure-problem (accessed 2026-10-05)
  - https://github.com/modal-projects/training-gym (accessed 2026-10-05)
  - https://modal.com/blog/building-an-rl-theorem-proving-workflow-on-modal (accessed 2026-10-05)
  - https://modal.com/blog/building-with-modal-and-the-openai-agent-sdk (accessed 2026-10-05)
  - https://modal.com/blog/physical-intelligence-runs-real-time-remote-inference-for-robotic-control-on-modal (accessed 2026-10-05)
  - https://modal.com/blog/product-updates-sidecars-models-dashboard (accessed 2026-10-05)
  - https://modal.com/blog/product-updates-rtx-pro-6000-command-k-sandbox-fs-api-and-more (accessed 2026-10-05)
  - https://modal.com/blog/introducing-auto-endpoints (accessed 2026-10-05)
  - https://modal.com/blog/achieve-sota-specdec (accessed 2026-10-05)
  - https://modal.com/blog/flash-attention-4-faster (accessed 2026-10-05)
  - https://modal.com/blog/trillion-tokens-trillion-parameters (accessed 2026-10-05)
  - https://modal.com/blog/quail-billion-tpm (accessed 2026-10-05)
  - https://modal.com/blog/multi-token-residual-prediction (accessed 2026-10-05)
  - https://arxiv.org/abs/2605.18817 (accessed 2026-10-05)
  - https://modal.com/blog/autoscaling-autoresearch (accessed 2026-10-05)
  - https://modal.com/blog/modal-series-c (accessed 2026-10-05)
  - https://modal.com/llm-almanac (accessed 2026-10-05)
  - https://modal.com/gpu-glossary (accessed 2026-10-05)
  - https://modal.com/docs/guide/security (accessed 2026-10-05)
  - https://github.com/erikbern (accessed 2026-10-05)
  - https://erikbern.com/about.html (accessed 2026-10-05)
  - https://modal.com/ (accessed 2026-06-07)
  - https://modal.com/blog/announcing-our-series-b (accessed 2026-06-07)
  - https://modal.com/blog/soc2type2 (accessed 2026-06-07)
  - https://modal.com/blog/hipaa (accessed 2026-06-07)
  - https://techcrunch.com/2026/02/11/ai-inference-startup-modal-labs-in-talks-to-raise-at-2-5b-valuation-sources-say/ (accessed 2026-06-07)
  - https://www.datacenterdynamics.com/en/news/modal-labs-secures-funding/ (accessed 2026-06-07)
  - https://siliconangle.com/2026/05/21/serverless-ai-infrastructure-startup-modal-labs-seals-355m-funding-round/ (accessed 2026-06-07)
  - https://techstartups.com/2026/05/21/modal-labs-raises-355m-quadrupling-valuation-to-4-65b-as-ai-infrastructure-demand-surges/ (accessed 2026-06-07)
  - https://github.com/modal-labs (accessed 2026-06-07)
  - https://github.com/modal-labs/modal-client (accessed 2026-06-07)
  - https://www.linkedin.com/company/modal-labs (accessed 2026-06-07)

## Mercor (not ranked, incumbent / infrastructure / open-source)
- slug: mercor
- segment: Incumbents also building RL environments
- website: https://www.mercor.com/
- focus_areas: Coding, Enterprise Workflows, Long-Horizon
- positioning: Mercor is a venture-backed expert marketplace and AI-training-data company. It supplies RLHF data, evaluations and reinforcement-learning environments to frontier AI labs and enterprises, drawing on a professional expert network: the homepage cites 30k+ experts, and Mercor's Deeptune post claims more than five million. It started as an AI-recruiting platform, moved into human data and RL, bought Sepal AI (Feb 2026), and announced the acquisition of Deeptune (Jul 2026), which builds simulated enterprise-app 'training gyms'. It publishes the APEX family of professional-task agent benchmarks. Mercor says it reached $2B ARR in June 2026, despite a March 2026 supply-chain data breach.
- best_fit: Buyers who want one managed partner for large-scale expert-validated human data, professional-domain evals (finance, law, medicine, accounting, SWE) and enterprise-app RL environments with expert tasks and verifiers.
- overall_confidence: high
- founded_year: 2023 [confirmed], source: https://en.wikipedia.org/wiki/Mercor (accessed 2026-06-07); corroborated by https://research.contrary.com/company/mercor (accessed 2026-06-07)
- status: active [confirmed], source: https://techcrunch.com/2025/10/27/mercor-quintuples-valuation-to-10b-with-350m-series-c/ (accessed 2026-06-07)
- hq_location: San Francisco, CA, USA (181 Fremont) [confirmed], source: https://en.wikipedia.org/wiki/Mercor (accessed 2026-06-07); https://www.linkedin.com/company/mercor-ai (accessed 2026-06-07)
- other_locations: New York City, NY, USA; London, UK; Mexico City, Mexico (select roles) [confirmed], source: https://www.mercor.com/careers/ (accessed 2026-10-05)
- distributed_remote: yes [reported], source: https://en.wikipedia.org/wiki/Mercor (accessed 2026-06-07; global contractor network across 6 continents)
- current_headcount: Full-time staff not publicly disclosed (Wikipedia: ~300 employees, 2025; LinkedIn band 51-200 per baseline). The homepage claims 30k+ experts, and a Mercor blog post claims a network of 5M+ domain experts. 107 open roles as of 2026-10-05. [reported], source: https://www.mercor.com/careers/ (accessed 2026-10-05)
- headcount_band: 51-200 [reported], source: https://www.linkedin.com/company/mercor-ai (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: 107 (careers page, 2026-10-05) [confirmed], source: https://www.mercor.com/careers/ (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: https://www.mercor.com/research/ (accessed 2026-06-07)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Founders Brendan Foody, Adarsh Hiremath, Surya Midha are Thiel Fellows and college dropouts (not research-lab veterans); Sundeep Jain (President, hired May 2025) ex-Uber CPO/SVP Eng and ex-Google VP Product [reported], source: https://en.wikipedia.org/wiki/Mercor (accessed 2026-06-07); https://research.contrary.com/company/mercor (accessed 2026-06-07)
- published_papers_or_benchmarks: APEX (AI Productivity Index) v1 / APEX-1; APEX-Agents; APEX-Agents 1.1; APEX-SWE (with Cognition); APEX-Accounting (with Ramp); SWE-Marathon-Ext; Human Baselines for Benchmarks (accounting); ApexAgents SkyRL 397B RL training recipe; ACE (AI Consumer Index): listed on mercor.com/research in Jun 2026, not shown on research or APEX pages as of 2026-10-05 [confirmed], source: https://www.mercor.com/apex/ (accessed 2026-10-05)
- total_raised: ~$492M across 4 rounds (Seed ~$3M+, Series A $30M, Series B $100M, Series C $350M) [reported], source: https://tracxn.com/d/companies/mercor (accessed 2026-06-07); corroborated by https://techcrunch.com/2025/10/27/mercor-quintuples-valuation-to-10b-with-350m-series-c/ (accessed 2026-06-07)
- last_round: Series C, $350M, October 2025 [confirmed], source: https://techcrunch.com/2025/10/27/mercor-quintuples-valuation-to-10b-with-350m-series-c/ (accessed 2026-06-07)
- notable_investors: Felicis Ventures (led Series C and Series B); Benchmark; General Catalyst; Robinhood Ventures [confirmed], source: https://techcrunch.com/2025/10/27/mercor-quintuples-valuation-to-10b-with-350m-series-c/ (accessed 2026-06-07)
- valuation: $10B (Series C, Oct 2025) [confirmed], source: https://techcrunch.com/2025/10/27/mercor-quintuples-valuation-to-10b-with-350m-series-c/ (accessed 2026-06-07); https://www.cnbc.com/2025/10/27/ai-hiring-startup-mercor-funding.html (accessed 2026-06-07)
- revenue_signals: Company-stated $2B ARR in June 2026, up from a $1B run rate four months earlier (Foody to Fortune, 2026-07-09). Reported H1 2026 revenue ~$614M, ~91% from foundation-model companies, gross margin ~33% in Q2 2026 (The Information, via a secondary outlet). The homepage claims $4M+ in daily payouts. [reported], source: https://fortune.com/2026/07/09/ai-unicorn-mercor-acquires-deeptune-brendan-foody-investor-a16z-openai-anthropic/ (accessed 2026-10-05)
- soc2: unknown [unknown], source: https://trust.mercor.com/ (accessed 2026-06-07; trust center exists but certification details not extractable from public page; note: press reports Mercor's compliance vendor Delve was accused of issuing fraudulent SOC 2 reports, casting doubt on any prior claimed certification)
- other_certifications: unknown [unknown]
- security_page: https://trust.mercor.com/ [confirmed], source: https://trust.mercor.com/ (accessed 2026-06-07)
- what_they_sell: mixed [confirmed], source: https://www.mercor.com/research/ (accessed 2026-06-07)
- open_source: partial (open eval harness Archipelago, APEX datasets, apex-swe, apex-evals, SkyRL training recipe and weights; core services proprietary) [confirmed], source: https://github.com/Mercor-Intelligence (accessed 2026-10-05)
- license: mixed: Archipelago Apache-2.0; apex-swe MIT; apex-evals CC-BY-4.0; commercial services proprietary [confirmed], source: https://github.com/Mercor-Intelligence (accessed 2026-10-05)
- deployment_model: managed-hosted (managed service / marketplace; RL environments delivered as a service to labs) [estimated], source: https://www.mercor.com/research/ (accessed 2026-06-07)
- maturity: GA [reported], source: https://techcrunch.com/2025/10/27/mercor-quintuples-valuation-to-10b-with-350m-series-c/ (accessed 2026-06-07)
- notable_customers: OpenAI (verified, frontier-lab tie); Anthropic (verified, frontier-lab tie); Google DeepMind (reported, frontier-lab tie); Meta (reported (contracts reportedly paused Apr 2026; current status unconfirmed), frontier-lab tie); Nvidia (reported, frontier-lab tie) [reported], source: https://fortune.com/2026/07/09/ai-unicorn-mercor-acquires-deeptune-brendan-foody-investor-a16z-openai-anthropic/ (accessed 2026-10-05)
- founding_team:
  - Brendan Foody, Co-founder & CEO, Enrolled at Georgetown, then dropped out. Thiel Fellow, and met his co-founders on the Bellarmine College Prep debate team. Mercor's careers page says he was named a TIME100 AI Innovator in 2026. He invested personally in Deeptune before the acquisition.
  - Adarsh Hiremath, Co-founder & CTO, Attended Harvard, then dropped out. Thiel Fellow and Bellarmine College Prep debate teammate of Foody and Midha.
  - Surya Midha, Co-founder & Board Chairman, Attended Georgetown, then dropped out. Thiel Fellow. Wikipedia lists him as board chairman.
  - Sundeep Jain, President (non-founder), Former Uber CPO and SVP Engineering, and former Google VP Product Management (Search Ads). Hired as Mercor's first president in May 2025. No 2026 source confirms he still holds the role.
  - Kevin Shiau, Chief Financial Officer (non-founder, joined 2026), Former Tesla finance and operations leader (battery-storage business). Earlier at First Solar and in Citigroup investment banking, then at an early-stage AI robotics company.
- recent_developments (last 6 months):
  - 2026-10-01 [research] Human-baselines study: frontier AI outperforms junior CPAs on accounting tasks, source: https://www.mercor.com/blog/human-baselines-for-benchmarks-ai-now-outperforms-junior-accountants/
  - 2026-09-30 [research] Engineering post: 'How we 20x LLM eval throughput', source: https://www.mercor.com/blog/how-we-20x-llm-eval-throughput/
  - 2026-09-08 [benchmark] APEX-Agents 1.1 released, source: https://www.mercor.com/blog/introducing-apex-agents-1-1/
  - 2026-09-01 [open-source] Open RL training recipe: Qwen 397B trained on APEX-Agents tasks with SkyRL and Harbor, source: https://www.mercor.com/blog/training-frontier-knowledge-work-agents-a-397b-rl-training-guide-with-skyrl/
  - 2026-08-11 [research] SWE-Marathon-Ext: can AI build a whole SaaS product?, source: https://www.mercor.com/blog/swe-marathon-ext-can-ai-build-a-whole-saas-product/
  - 2026-07-31 [benchmark] APEX-Accounting benchmark launched in partnership with Ramp, source: https://www.mercor.com/blog/introducing-the-ai-productivity-index-for-accounting/
  - 2026-07-22 [other] Reported H1 2026 revenue ~$614M, ~91% from foundation-model companies, source: https://finance.biggo.com/news/865c23cb-3e0d-4c21-aa0d-6bf2a7fc8d1a
  - 2026-07-09 [acquisition] Mercor to acquire Deeptune, an a16z-backed builder of RL 'training gyms', source: https://www.mercor.com/blog/mercor-to-acquire-deeptune/
  - 2026-07-09 [funding] Mercor reported in early talks to raise at ~$20B valuation, source: https://siliconangle.com/2026/07/09/mercor-buys-deeptune-build-training-environments-ai-agents/
  - 2026-07-09 [customer] Mercor says it reached $2B ARR in June 2026; 'every frontier lab has expanded' since the breach, source: https://fortune.com/2026/07/09/ai-unicorn-mercor-acquires-deeptune-brendan-foody-investor-a16z-openai-anthropic/
  - 2026-06-25 [hire] Mercor hires Kevin Shiau (ex-Tesla) as CFO, source: https://www.mercor.com/blog/why-i-joined-mercor-kevin-shiau/
  - 2026-06-25 [legal] Mercor publishes 'Update on Mercor security incident', source: https://www.mercor.com/blog/
  - 2026-04-09 [legal] Breach fallout: Meta reportedly pauses Mercor contracts indefinitely; contractor lawsuits filed, source: https://techcrunch.com/2026/04/09/after-data-breach-10b-valued-startup-mercor-is-having-a-month/
- benchmarks_and_research:
  - Human Baselines: AI vs junior accountants (paper), 2026-10-01, CPAs averaged 37%. Claude Opus 5 scored 100%, at $0.21 versus $10.35 per criterion met., https://www.mercor.com/blog/human-baselines-for-benchmarks-ai-now-outperforms-junior-accountants/
  - APEX-Agents 1.1 (benchmark), 2026-09-08, Leader at launch: Claude Fable 5.1 at 68.6% Pass@1, https://www.mercor.com/blog/introducing-apex-agents-1-1/
  - ApexAgents-SkyRL-Recipe (paper), 2026-09-01, Qwen3.5-397B-A17B Pass@1 on APEX-Agents rose from 16.11% to 27.29%, https://github.com/Mercor-Intelligence/ApexAgents-SkyRL-Recipe
  - SWE-Marathon-Ext (benchmark), 2026-08-11, Pass rates from 74% (GPT-5.6 Sol) to 27% (GLM-5.2) across 8 models, https://www.mercor.com/blog/swe-marathon-ext-can-ai-build-a-whole-saas-product/
  - APEX-Accounting (benchmark), 2026-07-31, At launch the top model (Claude Fable 5) scored 56.4%, and 58% of tasks were never fully solved. Current leaderboard top: Claude Opus 5.5 Max at 61.8% ± 4.0% (accessed 2026-10-05)., https://mercor.com/apex/apex-accounting-leaderboard/
  - APEX-Agents (benchmark), 2026-01, Leaderboard top: Gemini 4 Argon at 82.2% ± 4.4% (accessed 2026-10-05). The leaderboard's metric may differ from the 68.6% Pass@1 reported at the v1.1 launch., https://mercor.com/apex/apex-agents-leaderboard/
  - APEX-SWE (benchmark), 2026-01, Leaderboard top: Opus 5.5 Max at 67.6% ± 5.9% Pass@1 (accessed 2026-10-05), https://www.mercor.com/apex/apex-swe-leaderboard/
  - APEX (AI Productivity Index) v1 / APEX-1 (benchmark), 2025-09, Leaderboard top: Gemini 4 Argon at 75.7% ± 2.3% (accessed 2026-10-05), https://mercor.com/apex/apex-v1-leaderboard/
  - Archipelago (agent eval harness) (environment), https://github.com/Mercor-Intelligence
  - Partner benchmarks hosted by Mercor: FrontierCode (Cognition) and Harvey LAB (Harvey) (leaderboard), https://www.mercor.com/apex/
- products:
  - Human data & evals: Expert-sourced training data, rubrics and evaluations. The homepage claims '30k+ experts', a $108/hr average contracted rate and $4M+ in daily payouts. The Deeptune post claims a network of 'more than five million domain experts'., https://www.mercor.com/
  - RL environments (incl. Deeptune platform): Simulated enterprise-app environments ('training gyms') plus expert-written tasks and verifiers for agent RL. Deeptune's platform was added through the acquisition announced 2026-07-09., https://www.mercor.com/blog/mercor-to-acquire-deeptune/
  - APEX benchmarks & leaderboards: APEX-Agents, APEX-Accounting (with Ramp), APEX-SWE (with Cognition) and APEX-1 leaderboards, plus hosted partner and extended benchmarks., https://www.mercor.com/apex/
  - Enterprise AI / Deploy Agents: Agent deployment and workflow-mapping offering for enterprises, listed in site navigation. The careers page has a separate 'Enterprise Agents' team., https://www.mercor.com/
  - Data monetization: Workflow-data licensing with automated anonymization, listed in site navigation., https://www.mercor.com/
  - Archipelago (open source): Apache-2.0 Python harness for running and evaluating agents against RL environments., https://github.com/Mercor-Intelligence
- notes_for_researchers:
  - The APEX family has papers on arXiv (2509.25721, 2601.14242, 2601.08806, 2607.27189). Datasets on Hugging Face: Mercor/APEX-v1, mercor/apex-agents, mercor/apex-agents-v1.1, mercor/APEX-SWE, mercor/apex-accounting-harbor.
  - APEX-Agents 1.1 (Sep 2026) uses a new judge that penalizes hedged multi-answer outputs, so v1.0 and v1.1 scores are not comparable. State the version when citing.
  - The Sep 2026 SkyRL recipe releases its training script, Qwen3.5-397B and Qwen3.6-35B RL weights, and eval traces. Mercor reports that data quality mattered more than algorithm choices.
  - The Mercor-Intelligence GitHub org publishes Archipelago (Apache-2.0), apex-swe (MIT), apex-evals (CC-BY-4.0) and ApexAgents-SkyRL-Recipe, plus Harbor and aira-dojo forks.
  - The Oct 2026 human-baseline study gives CPA reference scores for APEX-style accounting tasks.
- notes_for_program_managers:
  - Frontier-lab ties: Fortune (Jul 2026) names Anthropic and OpenAI. Foody claims all Mag Seven except Tesla are customers. The Information, via PYMNTS, reports Nvidia as a Nemotron data customer. Meta reportedly paused contracts after the Mar 2026 breach, and whether it resumed is unconfirmed.
  - Scale signals: company-stated ~$2B ARR in June 2026. The Information (via a secondary outlet) reports ~$614M H1 2026 revenue with ~91% from foundation-model companies, so customer concentration is high.
  - Two RL-environment acquisitions in 2026: Sepal AI in February and Deeptune, announced in July. Deeptune appears as its own NYC team on the careers page.
  - Funding: the ~$20B raise (and Nvidia's possible participation) is reported talks only. No close was announced as of 2026-10-05, so $10B (Series C, Oct 2025) is the last confirmed valuation.
  - Leadership: Kevin Shiau (ex-Tesla) joined as CFO in June 2026.
- notes_for_data_ops:
  - The published RL recipe uses Docker images for world environments, MCP servers for tools and Harbor for environment and trial management, so Harbor- and MCP-compatible pipelines are publicly demonstrated.
  - The trust center at trust.mercor.com shows no readable SOC 2 or ISO certification. The Mar 2026 LiteLLM supply-chain breach reportedly exposed PII, source code and API keys and led to contractor lawsuits. Ask for the June 2026 incident update and current attestations during due diligence.
  - The careers page (2026-10-05) lists 6 open security roles.
- sources:
  - https://www.mercor.com/blog/ (accessed 2026-10-05)
  - https://www.mercor.com/blog/mercor-to-acquire-deeptune/ (accessed 2026-10-05)
  - https://siliconangle.com/2026/07/09/mercor-buys-deeptune-build-training-environments-ai-agents/ (accessed 2026-10-05)
  - https://fortune.com/2026/07/09/ai-unicorn-mercor-acquires-deeptune-brendan-foody-investor-a16z-openai-anthropic/ (accessed 2026-10-05)
  - https://www.forbes.com/sites/richardnieva/2026/07/09/mercor-fundraise/ (accessed 2026-10-05)
  - https://www.pymnts.com/news/investment-tracker/2026/nvidia-weighs-investment-in-round-valuing-mercor-at-20-billion/ (accessed 2026-10-05)
  - https://www.theinformation.com/articles/nvidia-discusses-funding-ai-data-supplier-mercor-20-billion-valuation (accessed 2026-10-05)
  - https://finance.biggo.com/news/865c23cb-3e0d-4c21-aa0d-6bf2a7fc8d1a (accessed 2026-10-05)
  - https://techcrunch.com/2026/04/09/after-data-breach-10b-valued-startup-mercor-is-having-a-month/ (accessed 2026-10-05)
  - https://www.mercor.com/blog/why-i-joined-mercor-kevin-shiau/ (accessed 2026-10-05)
  - https://www.mercor.com/research/ (accessed 2026-10-05)
  - https://www.mercor.com/apex/ (accessed 2026-10-05)
  - https://mercor.com/apex/apex-accounting-leaderboard/ (accessed 2026-10-05)
  - https://mercor.com/apex/apex-agents-leaderboard/ (accessed 2026-10-05)
  - https://www.mercor.com/apex/apex-swe-leaderboard/ (accessed 2026-10-05)
  - https://mercor.com/apex/apex-v1-leaderboard/ (accessed 2026-10-05)
  - https://www.mercor.com/blog/introducing-the-ai-productivity-index-for-accounting/ (accessed 2026-10-05)
  - https://www.mercor.com/blog/training-frontier-knowledge-work-agents-a-397b-rl-training-guide-with-skyrl/ (accessed 2026-10-05)
  - https://www.mercor.com/blog/introducing-apex-agents-1-1/ (accessed 2026-10-05)
  - https://www.mercor.com/blog/swe-marathon-ext-can-ai-build-a-whole-saas-product/ (accessed 2026-10-05)
  - https://www.mercor.com/blog/how-we-20x-llm-eval-throughput/ (accessed 2026-10-05)
  - https://www.mercor.com/blog/human-baselines-for-benchmarks-ai-now-outperforms-junior-accountants/ (accessed 2026-10-05)
  - https://github.com/Mercor-Intelligence (accessed 2026-10-05)
  - https://www.mercor.com/careers/ (accessed 2026-10-05)
  - https://www.mercor.com/ (accessed 2026-10-05)
  - https://en.wikipedia.org/wiki/Mercor (accessed 2026-10-05)
  - https://research.contrary.com/company/mercor (accessed 2026-10-05)
  - https://trust.mercor.com/ (accessed 2026-10-05)
  - https://techcrunch.com/2025/10/27/mercor-quintuples-valuation-to-10b-with-350m-series-c/ (accessed 2026-06-07)
  - https://www.orrick.com/en/News/2026/02/Mercor-Acquires-Sepal-AI (accessed 2026-06-07)
  - https://theaiinsider.tech/2025/10/29/mercor-raises-350m-to-scale-expert-driven-ai-training-reaching-10b-valuation/ (accessed 2026-06-07)
  - https://tracxn.com/d/companies/mercor/__764DkS7wJgmA1B8PuOw3_4HUbgJcaKjh8xY9UxvBIpY/funding-and-investors (accessed 2026-06-07)
  - https://techcrunch.com/2026/03/31/mercor-says-it-was-hit-by-cyberattack-tied-to-compromise-of-open-source-litellm-project/ (accessed 2026-06-07)
  - https://newsletter.semianalysis.com/p/rl-environments-and-rl-for-science (accessed 2026-06-07)

## Prime Intellect (not ranked, incumbent / infrastructure / open-source)
- slug: prime-intellect
- segment: Open-source & open environments
- website: https://www.primeintellect.ai/
- focus_areas: Coding, Enterprise Workflows, Long-Horizon, Math
- positioning: Prime Intellect sells an open-source-first stack for agentic RL: the Environments Hub (2,500+ community environments plus 365k+ unified SWE, terminal and search tasks), the verifiers, prime-rl and renderers libraries, Lab hosted RL post-training and evals (GA), Prime Sandboxes microVMs (GA), Prime Inference, and GPU compute. After a $130M Series A in July 2026 (TechCrunch reports a $1B valuation) it states $100M+ annualized revenue and 6,000+ customers. It also trains the open INTELLECT models and is the post-training partner in NVIDIA's Nemotron Coalition.
- best_fit: Teams wanting open, community-shared RL environments plus managed RL post-training/compute to build or fine-tune agentic models without locked-down proprietary tooling.
- overall_confidence: high
- founded_year: 2023 [reported], source: https://tracxn.com/d/companies/prime-intellect/__c00KIKAVH9b1POsp5fnIynw_8joG2ie8qsqmCTAB6Bc (accessed 2026-06-07) - multiple aggregators state founded 2023 by Vincent Weisser & Johannes Hagemann; no primary registry confirmation
- status: active [confirmed], source: https://www.primeintellect.ai/ (accessed 2026-06-07); active GitHub releases and blog posts through 2026
- hq_location: San Francisco, USA [confirmed], source: https://www.linkedin.com/company/primeintellect-ai (accessed 2026-06-07) - public snippet lists San Francisco, US
- other_locations: New York City, USA (roles listed; office not separately confirmed) [reported], source: https://www.primeintellect.ai/careers (accessed 2026-10-05)
- distributed_remote: yes [reported], source: https://wellfound.com/company/prime-intellect/jobs (accessed 2026-06-07) - several roles list remote/flexible location
- current_headcount: 11-50 (LinkedIn public snippet, as of 2026-06-07); Sacra reported 23 FTE (date unclear, likely 2025) [reported], source: https://www.linkedin.com/company/primeintellect-ai (accessed 2026-06-07); https://sacra.com/c/prime-intellect/ (accessed 2026-06-07)
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/primeintellect-ai (accessed 2026-06-07) - public snippet '11-50 employees'; consistent with Sacra 23 FTE
- headcount_growth: 229% YoY (per Sacra; period unspecified, likely 2024-2025, stale) [reported], source: https://sacra.com/c/prime-intellect/ (accessed 2026-06-07) - single-source aggregator, period not stated
- open_roles_count: 30 open roles (careers page, 2026-10-05) [confirmed], source: https://www.primeintellect.ai/careers (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: https://arxiv.org/abs/2512.16144 (accessed 2026-06-07) - INTELLECT-3 Technical Report authored by 'Prime Intellect Team'; hiring Research Engineers
- researcher_count: unknown [unknown]
- researcher_backgrounds: Johannes Hagemann (co-founder/CTO) - ex-Aleph Alpha (distributed training); Will Brown - created Verifiers RL environments library; research lead (not a founder); Vincent Weisser (co-founder/CEO) - background in DeSci/Web3 (e.g. VitaDAO) [reported], source: https://nextomoro.com/johannes-hagemann/ (accessed 2026-06-07); https://github.com/PrimeIntellect-ai/verifiers (accessed 2026-06-07) - 'Originally created by Will Brown (@willccbb)'; https://www.vincentweisser.com/ (accessed 2026-06-07)
- published_papers_or_benchmarks: INTELLECT-3 Technical Report (https://arxiv.org/abs/2512.16144); INTELLECT-2 (https://www.primeintellect.ai/blog/intellect-2); Measuring Autonomous AI Research / NanoGPT speedrun benchmark (https://primeintellect.ai/blog/measuring-autonomous-research); General Agent synthetic environment (https://primeintellect.ai/blog/general-agent); Systematic Reward Hacking study (https://primeintellect.ai/blog/reward-hacking); True Agents Model the World / ECHO (https://primeintellect.ai/blog/true-agents-model-the-world); Universal offline sandbox escape disclosure (https://primeintellect.ai/blog/universal-offline-sandbox-escape); Scaling Agentic RL: 365k+ SWE/terminal/search tasks (https://primeintellect.ai/blog/scaling-agentic-rl) [confirmed], source: https://primeintellect.ai/blog (accessed 2026-10-05)
- total_raised: Over $150M (company-stated in the Series A announcement, Jul 2026): $5.5M seed (2024) + $15M (2025) + $130M Series A (2026) [confirmed], source: https://www.primeintellect.ai/blog/series-a (accessed 2026-10-05)
- last_round: Series A, $130M, announced 2026-07-08, led by Radical Ventures with NVIDIA Ventures, Intel Capital and Dell Technologies Capital (Iconiq per TechCrunch); $1B valuation reported by TechCrunch [confirmed], source: https://www.primeintellect.ai/blog/series-a (accessed 2026-10-05)
- notable_investors: Radical Ventures (led $130M Series A, 2026); NVIDIA Ventures; Intel Capital; Dell Technologies Capital; Iconiq (reported by TechCrunch); Founders Fund (led $15M round, 2025); Menlo Ventures; Distributed Global (co-led seed); CoinFund (co-led seed); John Schulman (angel); Aaron Levie (angel); Dwarkesh Patel (angel); Matthew Prince (angel); Harrison Chase (angel); Brendan Foody (angel); Aravind Srinivas (angel, reported by TechCrunch); Andrej Karpathy (angel); Clem Delangue (angel); Tri Dao (angel); Emad Mostaque (angel); Dylan Patel (angel) [confirmed], source: https://www.primeintellect.ai/blog/series-a (accessed 2026-10-05)
- valuation: $1B (Series A, Jul 2026; reported by TechCrunch, not disclosed by the company) [reported], source: https://techcrunch.com/2026/07/08/prime-intellect-raises-130m-series-a-to-help-enterprises-build-their-own-ai-agents/ (accessed 2026-10-05)
- revenue_signals: Company-stated (Jul 2026): over $100M annualized revenue reached in under a year, and over 6,000 customers (self-reported, not audited; TechCrunch repeats the $100M run rate) [confirmed], source: https://www.primeintellect.ai/blog/series-a (accessed 2026-10-05)
- soc2: not stated (no SOC 2 claim on the homepage or security page as of 2026-10-05) [confirmed], source: https://www.primeintellect.ai/security (accessed 2026-10-05)
- other_certifications: unknown [unknown]
- security_page: https://www.primeintellect.ai/security (vulnerability disclosure and bug bounty policy only; no certifications listed) [confirmed], source: https://www.primeintellect.ai/security (accessed 2026-10-05)
- what_they_sell: mixed [confirmed], source: https://www.primeintellect.ai/ (accessed 2026-10-05)
- open_source: yes [confirmed], source: https://github.com/PrimeIntellect-ai/verifiers (accessed 2026-06-07) - MIT-licensed library; https://github.com/PrimeIntellect-ai/prime-rl (accessed 2026-06-07)
- license: MIT (Verifiers library); other repos vary [confirmed], source: https://github.com/PrimeIntellect-ai/verifiers (accessed 2026-06-07) - MIT license confirmed; created by Will Brown (@willccbb), maintained by PrimeIntellect-ai
- deployment_model: managed-hosted + API (hosted training/inference, on-demand GPU); open-source frameworks self-hostable [confirmed], source: https://www.primeintellect.ai/ (accessed 2026-06-07); https://github.com/PrimeIntellect-ai/verifiers (accessed 2026-06-07)
- maturity: GA: Lab GA 2026-05-07 (opened to all 2026-02-10), Prime Sandboxes GA 2026-09-23, Prime Inference launched 2026-10-02; company reports $100M+ annualized revenue and 6,000+ customers [confirmed], source: https://primeintellect.ai/blog/lab-is-open (accessed 2026-10-05)
- notable_customers: NVIDIA (integration/collaboration and Nemotron Coalition member; not strictly a buyer) (verified, frontier-lab tie); Ramp (case study; also named by TechCrunch) (verified); Zapier (case study; also named by TechCrunch) (verified); Flapping Airplanes (named as a customer by TechCrunch) (verified); Goodfire (case study) (self-claimed); Mixedbread (case study) (self-claimed); Extropic (case study) (self-claimed); Character AI (logo) (self-claimed); Arcee AI (logo) (self-claimed); Browserbase (logo; partnership announced 2026-03-30) (self-claimed); Inception (logo) (self-claimed); Standard Intelligence (logo) (self-claimed) [reported], source: https://www.primeintellect.ai/ (accessed 2026-10-05)
- founding_team:
  - Vincent Weisser, Co-founder & CEO, German entrepreneur from the decentralized-science (DeSci) community before co-founding Prime Intellect (per Wikipedia). The VitaDAO link in the baseline could not be re-verified from a primary source in this run.
  - Johannes Hagemann, Co-founder & CTO, Joined Aleph Alpha in early 2022, where he helped build a distributed training framework from scratch. Co-founded Prime Intellect with Weisser at the end of 2023 (per hagemann.ai).
- funding_history:
  - 2026-07-08 Series A $130M led by Radical Ventures, valuation $1B, source: https://www.primeintellect.ai/blog/series-a
- recent_developments (last 6 months):
  - 2026-10-02 [product] Prime Inference launches: fast, reliable serving for frontier open models, source: https://primeintellect.ai/blog/prime-inference
  - 2026-10-01 [customer] Extropic case study: a thermodynamic-ML research agent trained with RL on Prime Intellect, source: https://primeintellect.ai/case-study/extropic
  - 2026-09-23 [product] Prime Sandboxes reach general availability: microVMs for agentic RL, source: https://primeintellect.ai/blog/sandboxes
  - 2026-09-17 [customer] Goodfire case study: activation probes that catch reward hacking, built with verifiers and the Environments Hub, source: https://primeintellect.ai/case-study/goodfire
  - 2026-09-03 [research] GLM-5.2 RL weight transfer cut from 86.1s to 3.9s with NVIDIA NIXL and ModelExpress, source: https://primeintellect.ai/blog/nixl-modelexpress-weight-transfer
  - 2026-08-25 [research] Disclosure of a universal 'offline' sandbox escape through internet-connected inference APIs, source: https://primeintellect.ai/blog/universal-offline-sandbox-escape
  - 2026-08-15 [benchmark] Measuring Autonomous AI Research: a NanoGPT speedrun benchmark for frontier models, source: https://primeintellect.ai/blog/measuring-autonomous-research
  - 2026-08-13 [research] Prime Flash MoE: Blackwell-optimized MoE kernels, source: https://primeintellect.ai/blog/prime-flash-moe
  - 2026-08-13 [customer] Mixedbread case study: Toast 1 search agent trained with prime-rl, source: https://primeintellect.ai/case-study/mixedbread
  - 2026-08-07 [open-source] Multi-agent RL in prime-rl (verifiers 0.3.0, prime-rl 0.8.0), source: https://primeintellect.ai/blog/multi-agent-systems
  - 2026-08-05 [open-source] Prime Agent: an open-source, self-improving RLM agent harness, source: https://primeintellect.ai/blog/prime-agent
  - 2026-07-22 [open-source] Scaling Agentic RL: 365,000+ SWE, terminal and search tasks behind one API, source: https://primeintellect.ai/blog/scaling-agentic-rl
  - 2026-07-12 [open-source] verifiers v1 splits environments into tasksets, harnesses and runtimes, source: https://primeintellect.ai/blog/verifiers-v1
  - 2026-07-08 [funding] $130M Series A led by Radical Ventures; TechCrunch reports a $1B valuation, source: https://www.primeintellect.ai/blog/series-a
  - 2026-07-06 [open-source] prime-rl adds an algorithms layer, source: https://primeintellect.ai/blog/algorithms-layer
  - 2026-06-22 [open-source] prime-rl v0.6.0: RL at 1T-parameter scale, source: https://primeintellect.ai/blog/rl-at-1t-scale
  - 2026-06-10 [research] True Agents Model the World: ECHO, adding world-model prediction to RL, source: https://primeintellect.ai/blog/true-agents-model-the-world
  - 2026-06-04 [partnership] Joins the NVIDIA Nemotron Coalition; day-zero Nemotron 3 Ultra post-training on Lab, source: https://primeintellect.ai/blog/nemotron-coalition
  - 2026-05-30 [product] Hosted Evaluations released, source: https://primeintellect.ai/blog/hosted-evaluations
  - 2026-05-20 [research] Systematic reward-hacking study and the Prime Sprints community program, source: https://primeintellect.ai/blog/reward-hacking
  - 2026-05-18 [research] General Agent: a self-evolving synthetic agent environment, source: https://primeintellect.ai/blog/general-agent
  - 2026-05-12 [open-source] renderers: open-source token-level templating for agentic RL, source: https://primeintellect.ai/blog/renderers
  - 2026-05-07 [product] Lab moves from beta to general availability, source: https://primeintellect.ai/blog/lab-is-open
  - 2026-05-07 [customer] Ramp case study: FastAsk spreadsheet-search subagent trained with RL on Lab, source: https://primeintellect.ai/case-study/ramp
  - 2026-04-20 [customer] Zapier case study: AutomationBench as a continuous agent-improvement loop, source: https://primeintellect.ai/case-study/zapier
  - 2026-04-16 [benchmark] FrontierSWE (by Proximal) launches on the Environments Hub, source: https://primeintellect.ai/blog/frontier-swe
- benchmarks_and_research:
  - Universal offline sandbox escape (security research) (paper), 2026-08-25, Coordinated fixes and allowlists across frameworks; METR and UK AISI were informed before publication., https://primeintellect.ai/blog/universal-offline-sandbox-escape
  - Measuring Autonomous AI Research (NanoGPT speedrun benchmark) (benchmark), 2026-08-15, 153 runs, 18 models. Per the post, the best run (Claude Fable 5) reached 2,726 steps against a 3,290-step baseline and the 2,600-step human record, closing 81.7% of the gap., https://primeintellect.ai/blog/measuring-autonomous-research
  - Scaling Agentic RL: 365k+ SWE / terminal / search tasks (environment), 2026-07-22, Vendor-reported training example: GLM-4.5-Air on Scale-SWE reached 0.554 pass@10., https://primeintellect.ai/blog/scaling-agentic-rl
  - True Agents Model the World (ECHO) (paper), 2026-06-10, Per the post, learning is faster with little overhead, and the benefit is clearer at 110B than at 4B., https://primeintellect.ai/blog/true-agents-model-the-world
  - Environments Hub leaderboards / Hosted Evaluations (leaderboard), 2026-05-30, https://primeintellect.ai/blog/hosted-evaluations
  - Systematic Reward Hacking study (paper), 2026-05-20, Hacks with a 0.16% baseline frequency still get exploited ('no rarity floor'); moderate task difficulty slows hacking., https://primeintellect.ai/blog/reward-hacking
  - General Agent (synthetic self-evolving environment) (environment), 2026-05-18, SFT on 4,417 traces: BFCL-v3 from 18.9% to 52.3%; MCP-Atlas from 0.6% to 12.1%., https://primeintellect.ai/blog/general-agent
  - INTELLECT-3 Technical Report (paper), 2025-12, https://arxiv.org/abs/2512.16144
  - INTELLECT-2 (paper), 2025, https://www.primeintellect.ai/blog/intellect-2
  - SYNTHETIC-2 (reasoning traces dataset) (dataset), https://www.primeintellect.ai/
- products:
  - Lab (hosted RL post-training platform): End-to-end platform to configure environments, evaluate, train (hosted RL with LoRA) and deploy agentic models. Per-token training pricing. 14 models from 1B to 70B at GA, plus day-zero Nemotron 3 Ultra post-training., https://primeintellect.ai/blog/lab-is-open
  - Environments Hub: Community registry of RL environments (2,500+ per the homepage), plus 365k+ unified agentic tasks and third-party benchmarks such as FrontierSWE., https://app.primeintellect.ai/dashboard/environments
  - Hosted Evaluations: Run Hub benchmarks as hosted evals, with leaderboards and an eval viewer., https://primeintellect.ai/blog/hosted-evaluations
  - Prime Sandboxes: MicroVM sandboxes (a guest kernel per sandbox) for agentic RL and evals. CLI and SDK access, tier-free usage-based pricing., https://primeintellect.ai/blog/sandboxes
  - Prime Inference: OpenAI-compatible serving for open frontier models (GLM-5.3 on GB200 NVL72) with failover and SLAs. Batch and dedicated deployments are on the roadmap., https://primeintellect.ai/blog/prime-inference
  - Compute (on-demand GPUs and reserved clusters): On-demand and spot GPUs (H100, H200, GH200, B200, B300, A100 and others) and multi-node clusters of up to 512 GPUs with SLURM/K8s and InfiniBand. Reserved-cluster quotes are sourced across 50+ providers., https://www.primeintellect.ai/
  - verifiers (open source, MIT): Library for RL environments and evals. v1 adds tasksets, harnesses and runtimes, Harbor support and alpha NeMo Gym/OpenEnv support; 0.3.0 adds multi-agent support; 0.3.1 adds sandbox egress allow and deny lists., https://github.com/PrimeIntellect-ai/verifiers
  - prime-rl (open source): Async RL training framework: v0.6.0 runs at 1T-parameter scale, an algorithms layer was added 2026-07-06, and 0.8.0 adds multi-agent RL., https://github.com/PrimeIntellect-ai/prime-rl
  - renderers (open source): Token-level chat templating library for multi-turn RL., https://github.com/PrimeIntellect-ai/renderers
  - Prime Agent (open source): Self-improving RLM agent harness with a Continual Harness., https://github.com/PrimeIntellect-ai/prime-agent
  - Prime CLI / SDK: CLI and Python SDK for compute, sandboxes, environments, evals and training., https://docs.primeintellect.ai/cli-reference/introduction
- notes_for_researchers:
  - Fully open RL stack: verifiers (MIT, about 4.7k GitHub stars), prime-rl (async RL with a per-environment algorithms layer covering GRPO, MaxRL, on-policy and self-distillation and ECHO; multi-agent support since 0.8.0; v0.6.0 deep dive at 1T scale with 131k context), renderers for token-level templating, and the Prime Agent harness.
  - Open research in 2026: the NanoGPT-speedrun autonomous-research benchmark (Aug), the General Agent synthetic environment (May), the reward-hacking dynamics study (May), ECHO world-modeling (Jun), and the universal offline sandbox escape disclosure (Aug).
  - Prime Sprints awards $5K+ in monthly compute credits for community environment research (first track: reward hacking).
  - 365k+ agentic tasks (SWE, terminal, search) unified from 23 public tasksets, with about 135k prebuilt images and verified Hugging Face re-uploads.
- notes_for_program_managers:
  - Scale signals, company-stated (Jul 2026): over $100M annualized revenue reached in under a year, over 6,000 customers and over $150M raised. The $130M Series A was led by Radical Ventures with NVIDIA Ventures, Intel Capital and Dell Technologies Capital. TechCrunch reports a $1B valuation and Iconiq's participation.
  - Self-serve plus forward-deployed: Lab and Sandboxes are GA and self-serve, and the careers page lists Applied Research (Forward-Deployed), Solutions Architect, Technical Account Manager and Forward Deployed AI Strategy Lead roles.
  - Ecosystem ties: NVIDIA Nemotron Coalition member (Jun 2026) with day-zero Nemotron 3 Ultra post-training; NVIDIA technical collaboration (Mar 2026); John Schulman (Thinking Machines) is an angel investor.
  - Named case studies are all on the vendor's own site, so treat them as self-claimed: Ramp, Zapier, Goodfire, Mixedbread, Extropic. TechCrunch independently names Ramp, Zapier and Flapping Airplanes as customers. Homepage logos also include Character AI, Inception, Arcee, Browserbase and Standard Intelligence.
  - The product now spans compute, sandboxes, environments, evals, training and inference, so expect a single-vendor pitch.
- notes_for_data_ops:
  - Integration formats: verifiers v1 supports Harbor tasksets (e.g. Terminal-Bench 2); subprocess, Docker, Prime Sandboxes and Modal runtimes; OpenAI Chat/Responses and Anthropic Messages dialects; and alpha NeMo Gym and OpenEnv support.
  - Prime Sandboxes: hardware-isolated microVMs with a guest kernel each, Docker Compose support, a default limit of 1,024 concurrent sandboxes, and CLI/SDK access.
  - Prime Inference exposes an OpenAI-compatible API at api.pinference.ai/api/v1.
  - Security: the public /security page is a vulnerability disclosure policy with a bug bounty, 2-business-day acknowledgment and an email channel. No SOC 2 or other certification is stated publicly. Pipelines that run 'offline' evals should move to the patched framework versions listed in the 2026-08-25 sandbox-escape post (e.g. verifiers v0.3.1, Inspect v0.3.255).
- sources:
  - https://www.primeintellect.ai/blog/series-a (accessed 2026-10-05)
  - https://techcrunch.com/2026/07/08/prime-intellect-raises-130m-series-a-to-help-enterprises-build-their-own-ai-agents/ (accessed 2026-10-05)
  - https://en.wikipedia.org/wiki/Prime_Intellect (accessed 2026-10-05)
  - https://primeintellect.ai/blog (accessed 2026-10-05)
  - https://primeintellect.ai/blog/prime-inference (accessed 2026-10-05)
  - https://primeintellect.ai/blog/sandboxes (accessed 2026-10-05)
  - https://primeintellect.ai/case-study/extropic (accessed 2026-10-05)
  - https://primeintellect.ai/case-study/goodfire (accessed 2026-10-05)
  - https://primeintellect.ai/case-study/mixedbread (accessed 2026-10-05)
  - https://primeintellect.ai/case-study/ramp (accessed 2026-10-05)
  - https://primeintellect.ai/case-study/zapier (accessed 2026-10-05)
  - https://primeintellect.ai/blog/nixl-modelexpress-weight-transfer (accessed 2026-10-05)
  - https://primeintellect.ai/blog/universal-offline-sandbox-escape (accessed 2026-10-05)
  - https://primeintellect.ai/blog/measuring-autonomous-research (accessed 2026-10-05)
  - https://primeintellect.ai/blog/prime-flash-moe (accessed 2026-10-05)
  - https://primeintellect.ai/blog/multi-agent-systems (accessed 2026-10-05)
  - https://primeintellect.ai/blog/prime-agent (accessed 2026-10-05)
  - https://primeintellect.ai/blog/scaling-agentic-rl (accessed 2026-10-05)
  - https://primeintellect.ai/blog/verifiers-v1 (accessed 2026-10-05)
  - https://primeintellect.ai/blog/algorithms-layer (accessed 2026-10-05)
  - https://primeintellect.ai/blog/rl-at-1t-scale (accessed 2026-10-05)
  - https://primeintellect.ai/blog/true-agents-model-the-world (accessed 2026-10-05)
  - https://primeintellect.ai/blog/nemotron-coalition (accessed 2026-10-05)
  - https://primeintellect.ai/blog/nemotron-3 (accessed 2026-10-05)
  - https://primeintellect.ai/blog/hosted-evaluations (accessed 2026-10-05)
  - https://primeintellect.ai/blog/reward-hacking (accessed 2026-10-05)
  - https://primeintellect.ai/blog/general-agent (accessed 2026-10-05)
  - https://primeintellect.ai/blog/renderers (accessed 2026-10-05)
  - https://primeintellect.ai/blog/lab-is-open (accessed 2026-10-05)
  - https://primeintellect.ai/blog/lab (accessed 2026-10-05)
  - https://primeintellect.ai/blog/frontier-swe (accessed 2026-10-05)
  - https://www.primeintellect.ai/ (accessed 2026-10-05)
  - https://www.primeintellect.ai/security (accessed 2026-10-05)
  - https://www.primeintellect.ai/careers (accessed 2026-10-05)
  - https://www.vincentweisser.com/ (accessed 2026-10-05)
  - https://github.com/vincentweisser (accessed 2026-10-05)
  - https://github.com/JohannesHa (accessed 2026-10-05)
  - https://hagemann.ai (accessed 2026-10-05)
  - https://github.com/PrimeIntellect-ai/verifiers (accessed 2026-10-05)
  - https://www.primeintellect.ai/blog/environments (accessed 2026-06-07)
  - https://www.primeintellect.ai/blog/fundraise (accessed 2026-06-07)
  - https://www.prnewswire.com/news-releases/prime-intellect-secures-5-5m-in-seed-funding-co-led-by-distributed-global-and-coinfund-302124585.html (accessed 2026-06-07)
  - https://tracxn.com/d/companies/prime-intellect/__c00KIKAVH9b1POsp5fnIynw_8joG2ie8qsqmCTAB6Bc/funding-and-investors (accessed 2026-06-07)
  - https://sacra.com/c/prime-intellect/ (accessed 2026-06-07)
  - https://www.linkedin.com/company/primeintellect-ai (accessed 2026-06-07)
  - https://github.com/PrimeIntellect-ai/prime-rl (accessed 2026-06-07)
  - https://www.primeintellect.ai/case-study/zapier (accessed 2026-06-07)
  - https://www.primeintellect.ai/case-study/ramp (accessed 2026-06-07)
  - https://www.primeintellect.ai/blog/intellect-3 (accessed 2026-06-07)
  - https://arxiv.org/pdf/2512.16144 (accessed 2026-06-07)
  - https://arxiv.org/pdf/2601.16443 (accessed 2026-06-07)
  - https://nextomoro.com/johannes-hagemann/ (accessed 2026-06-07)
  - https://www.fastaijobs.com/companies/prime-intellect (accessed 2026-06-07)
  - https://wellfound.com/company/prime-intellect/jobs (accessed 2026-06-07)
  - https://www.sequoiacap.com/podcast/building-the-github-for-rl-environments-prime-intellects-will-brown-johannes-hagemann/ (accessed 2026-06-07)

## Daytona (not ranked, incumbent / infrastructure / open-source)
- slug: daytona
- segment: Adjacent: execution infrastructure
- website: https://www.daytona.io/
- focus_areas: Coding, Computer Use
- positioning: Daytona provides secure, elastic, programmatic sandboxes (Linux containers, Linux and Windows VMs, and GPU sandboxes) that AI agents spin up in under ~90ms (vendor claim) to run untrusted AI-generated code in isolated, stateful runtimes with snapshot and fork primitives. It is a managed service with a Bring-Your-Own-Compute option and has been closed source since June 2026. It is an integrated sandbox provider for Anthropic Claude Managed Agents, OpenAI Agents SDK Sandbox Agents and Cognition Devin Outposts, and targets code execution, computer use and RL/eval rollouts.
- best_fit: Buyers needing fast, isolated, programmatic sandboxes to safely execute AI-generated code or run agent tool/computer-use loops at scale.
- overall_confidence: high
- founded_year: 2023 [reported], source: https://www.linkedin.com/company/daytonaio (accessed 2026-06-07); pre-seed Nov 2023 per https://www.daytona.io/dotfiles/daytona-secures-5m-to-simplify-development-environments (accessed 2026-06-07)
- status: active [confirmed], source: https://www.daytona.io/ (accessed 2026-06-07); raised Series A Feb 2026
- hq_location: New York, NY, United States [reported], source: https://tracxn.com/d/companies/daytona (accessed 2026-06-07); https://www.daytona.io/dotfiles/daytona-lights-up-times-square (accessed 2026-06-07). Note: a LinkedIn public snippet also surfaced San Francisco, CA as an associated location
- other_locations: San Francisco, CA (work location for open roles); Croatia (work location for open roles) [reported], source: https://www.daytona.io/careers (accessed 2026-10-05)
- distributed_remote: yes [reported], source: https://www.daytona.io/careers (accessed 2026-06-07), roles open across NY, SF, and Croatia
- current_headcount: ~20 (Feb 2026, company-stated at Series A); third-party trackers list ~63 associated profiles and an 11-50 band as of 2026-06-07 [reported], source: https://www.prnewswire.com/news-releases/daytona-raises-24m-series-a-to-give-every-agent-a-computer-302680740.html and https://tracxn.com/d/companies/daytona and https://www.linkedin.com/company/daytonaio (accessed 2026-06-07)
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/daytonaio (accessed 2026-06-07), LinkedIn size band; company stated ~20 at Series A
- headcount_growth: unknown [unknown]
- open_roles_count: 11 [reported], source: https://www.daytona.io/careers (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: https://www.daytona.io/dotfiles/snapshots-as-search-states-go-explore-with-daytona (accessed 2026-10-05)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Founders Ivan Burazin (CEO), Vedran Jukic (CTO), Goran Draganic (Chief Architect) previously co-founded Codeanywhere, a cloud IDE/dev-environment company; Burazin also founded the Shift conference (acquired by Infobip) and was Infobip's Chief Developer Experience Officer; Daniel Thi Graviet, ML Researcher at Daytona (author of 2026 research posts on sandbox snapshots and environment startup) [reported], source: https://www.daytona.io/dotfiles/snapshots-as-search-states-go-explore-with-daytona (accessed 2026-10-05)
- published_papers_or_benchmarks: Searching Over Sandbox States for Coding Agents: Go-Explore with Daytona snapshots on Terminal-Bench (research blog, 2026-09-18; not peer-reviewed); OSWorld Windows tasks on Daytona Windows sandboxes (vendor-run eval, 2026-07-20; not peer-reviewed); Kubernetes Is the Wrong Primitive for Diverse Workloads (vendor infra study, 2026-07-13; not peer-reviewed) [confirmed], source: https://www.daytona.io/dotfiles/snapshots-as-search-states-go-explore-with-daytona (accessed 2026-10-05)
- total_raised: ~$79M total disclosed: $2M pre-seed (Nov 2023) + $5M seed (Jun 2024) + $24M Series A (Feb 2026) + $48.3M equity sold per SEC Form D filed 2026-08-17 (of a $60M offering; stage undisclosed) [reported], source: https://formds.com/issuers/daytona-platforms-inc (accessed 2026-10-05)
- last_round: Undisclosed-stage equity round (aggregator-labelled Series B): $48.3M sold of a $60M offering, SEC Form D filed 2026-08-17 (first sale 2026-07-31, 6 investors). Lead investor and valuation not disclosed. [confirmed], source: https://www.sec.gov/Archives/edgar/data/1977563/000197756326000002/xslFormDX01/primary_doc.xml (accessed 2026-10-05)
- notable_investors: FirstMark Capital (Series A lead; Matt Turck on board and listed as a director on the Aug 2026 Form D); Pace Capital; Upfront Ventures (seed lead, Series A participant); E2VC; Darkmode; Datadog (strategic); Figma Ventures (strategic); 500 Global (seed); Aug 2026 round: investors not publicly named; Form D reports 6 investors (Kevin Zhang also listed as a director; affiliation not stated in the filing) [reported], source: https://www.sec.gov/Archives/edgar/data/1977563/000197756326000002/xslFormDX01/primary_doc.xml (accessed 2026-10-05)
- valuation: unknown [unknown]
- revenue_signals: Company-claimed: ~$1M forward revenue run rate within three months, doubled six weeks later (as of Series A, Feb 2026) [reported], source: https://www.prnewswire.com/news-releases/daytona-raises-24m-series-a-to-give-every-agent-a-computer-302680740.html (accessed 2026-06-07), vendor-stated, unaudited
- soc2: Type II (issued 2026-07-28 by Insight Assurance, unqualified opinion, period 2025-12-16 to 2026-03-15; Security, Availability, Confidentiality); Type I also listed [reported], source: https://trust.daytona.io/ (accessed 2026-10-05)
- other_certifications: HIPAA (report listed in Trust Center); ISO/IEC 27001 (listed in Trust Center; certified vs in-progress not stated); GDPR (listed in Trust Center) [reported], source: https://trust.daytona.io/ (accessed 2026-06-07)
- security_page: https://trust.daytona.io/ [confirmed], source: https://trust.daytona.io/ (accessed 2026-06-07)
- what_they_sell: infra [confirmed], source: https://www.daytona.io/ (accessed 2026-06-07)
- open_source: no (core platform closed source since June 2026; the legacy repo was archived 2026-10-03 and is unmaintained; SDKs and docs remain public at github.com/daytona) [confirmed], source: https://www.daytona.io/dotfiles/updates/daytona-is-going-closed-source (accessed 2026-10-05)
- license: Proprietary (current platform); AGPL-3.0 applies only to the archived, unmaintained legacy repo daytonaio/daytona [confirmed], source: https://github.com/daytonaio/daytona (accessed 2026-10-05)
- deployment_model: managed-hosted plus Bring Your Own Compute (customer-run runners in custom regions under Daytona's control plane, deployed with Helm on Kubernetes). Open-source self-hosting is no longer maintained. [confirmed], source: https://www.daytona.io/docs/en/bring-your-own-compute/ (accessed 2026-10-05)
- maturity: GA [reported], source: https://www.prnewswire.com/news-releases/daytona-raises-24m-series-a-to-give-every-agent-a-computer-302680740.html (accessed 2026-06-07), commercial product with paying customers and revenue run rate
- notable_customers: Anthropic, Claude Managed Agents self-hosted sandbox launch partner (platform integration), not a disclosed Daytona customer (verified, frontier-lab tie); OpenAI, Daytona is a hosted sandbox provider for OpenAI Agents SDK Sandbox Agents (v0.14.0, Apr 2026) and has an OpenAI developer-docs provider page (platform integration), not a disclosed Daytona customer (verified, frontier-lab tie); Cognition, Devin Outposts partner integration (listed on Cognition's Outposts docs), not a disclosed customer contract (verified); Clay (Sculptor GTM agent runs in production on Claude Managed Agents + Daytona) (verified); Cursor, Self-Hosted Machines integration for Cursor Cloud Agents (per Daytona blog) (self-claimed); Yutori, Navigator n2 computer-use playground on Daytona sandboxes (post by a Yutori staff author on Daytona's blog) (self-claimed); Anchor Browser, browser-agent partnership (post by Anchor's CEO on Daytona's blog) (self-claimed); Brainbase, Universal Harness API uses Daytona as default sandbox (per Daytona blog) (self-claimed); LangChain (self-claimed); Turing (self-claimed); Writer (self-claimed); SambaNova (self-claimed) [reported], source: https://github.com/openai/openai-agents-python/releases/tag/v0.14.0 (accessed 2026-10-05)
- founding_team:
  - Ivan Burazin, Co-founder & CEO, Co-founded Codeanywhere, an early cloud IDE (2009). Founded the Shift developer conference, which Infobip acquired, and then served as Infobip's Chief Developer Experience Officer.
  - Vedran Jukic, Co-founder & CTO, Co-founder of Codeanywhere. Tech lead, with his GitHub bio naming him Daytona co-founder and CTO.
  - Goran Draganic, Co-founder & Chief Architect, Co-founder of Codeanywhere.
- funding_history:
  - 2026-08-17 undisclosed $48.274M, source: https://www.sec.gov/Archives/edgar/data/1977563/000197756326000002/xslFormDX01/primary_doc.xml
- recent_developments (last 6 months):
  - 2026-10-03 [open-source] Legacy daytonaio/daytona GitHub repo archived (read-only), source: https://github.com/daytonaio/daytona
  - 2026-09-18 [research] Daytona research: searching over sandbox states for coding agents (Go-Explore on Terminal-Bench), source: https://www.daytona.io/dotfiles/snapshots-as-search-states-go-explore-with-daytona
  - 2026-09-02 [partnership] Cursor Self-Hosted Machines on Daytona, source: https://www.daytona.io/dotfiles/cursor-self-hosted-machines-on-daytona
  - 2026-08-26 [customer] Yutori's Navigator n2 computer-use playground runs on Daytona sandboxes, source: https://www.daytona.io/dotfiles/real-computers-for-computer-use
  - 2026-08-17 [funding] SEC Form D: Daytona sells $48.3M of equity in a $60M offering, source: https://www.sec.gov/Archives/edgar/data/1977563/000197756326000002/xslFormDX01/primary_doc.xml
  - 2026-07-28 [other] SOC 2 Type II report issued (Insight Assurance, unqualified opinion), source: https://trust.daytona.io/
  - 2026-07-27 [research] Daytona explores CRAB (agent-aware checkpointing) on Daytona snapshots, source: https://www.daytona.io/dotfiles/agent-aware-snapshots-with-daytona
  - 2026-07-22 [partnership] Anchor Browser x Daytona: production-grade browsers for agents, source: https://www.daytona.io/dotfiles/production-grade-browsers-for-agents-anchor-daytona
  - 2026-07-21 [partnership] Devin Outposts on Daytona (Cognition), source: https://www.daytona.io/dotfiles/devin-outposts-on-daytona
  - 2026-07-21 [partnership] Brainbase Universal Harness API uses Daytona as the default sandbox, source: https://www.daytona.io/dotfiles/brainbases-universal-harness-api-on-daytona
  - 2026-07-20 [benchmark] OSWorld Windows tasks run on Daytona Windows sandboxes, source: https://www.daytona.io/dotfiles/osworld-on-daytona-windows-sandboxes
  - 2026-07-13 [research] Daytona study: Kubernetes is the wrong primitive for diverse short-lived workloads, source: https://www.daytona.io/dotfiles/kubernetes-is-the-wrong-primitive-for-diverse-workloads
  - 2026-06-11 [open-source] Daytona is going closed source, source: https://www.daytona.io/dotfiles/updates/daytona-is-going-closed-source
  - 2026-06-03 [product] Windows sandbox class and GPU type selector (v0.184.0), source: https://www.daytona.io/changelog
  - 2026-05-25 [product] API moves to cursor-based pagination (breaking change), source: https://www.daytona.io/dotfiles/updates/daytona-is-moving-to-cursor-based-pagination
  - 2026-04-30 [other] Security update for CVE-2026-31431 ('Copy Fail') Linux kernel flaw, source: https://www.daytona.io/dotfiles/updates/security-update-cve-2026-31431-copy-fail
  - 2026-04-29 [partnership] Daytona sandboxes available through Stripe Projects, source: https://www.daytona.io/dotfiles/daytona-stripe-projects
  - 2026-04-15 [partnership] OpenAI Agents SDK launches Sandbox Agents with Daytona as a hosted sandbox provider, source: https://github.com/openai/openai-agents-python/releases/tag/v0.14.0
  - 2026-04-14 [product] Sandbox fork and snapshot endpoints and SDK (v0.165.0 / v0.166.0), source: https://www.daytona.io/changelog
  - 2026-04-13 [other] Security advisory: API credential exposure in sandboxes, source: https://www.daytona.io/dotfiles/updates/security-advisory-api-credential-exposure-in-sandboxes
- benchmarks_and_research:
  - Searching Over Sandbox States for Coding Agents (Go-Explore with Daytona snapshots on Terminal-Bench) (paper), 2026-09-18, Five Terminal-Bench tasks: independent retry 17/25 vs snapshot branching 6/25. Stateful staged-service-repair task: full snapshot 4/5 vs 0/5 for clean restart and git-diff restore., https://www.daytona.io/dotfiles/snapshots-as-search-states-go-explore-with-daytona
  - OSWorld (Windows tasks) on Daytona Windows sandboxes (paper), 2026-07-20, Claude Opus 4.6 averaged 31.54% vs GPT-5.4 at 29.13% (Excel 45.45% vs 27.27%; Word 33.12% vs 55.34%). Sandbox created in 10.5s; desktop ready in about 32s., https://www.daytona.io/dotfiles/osworld-on-daytona-windows-sandboxes
- products:
  - Daytona Sandboxes (Linux containers): Programmatic, stateful, isolated sandboxes for running AI-generated code, with sub-90ms creation (vendor claim). SDKs for Python, TypeScript, Go, Ruby and Java (Java SDK since v0.161.0, 2026-04-03)., https://www.daytona.io/docs/en/sandboxes
  - Windows and Linux VM sandboxes: A Windows sandbox class (v0.184.0) with snapshot and fork support (v0.186.0), plus Linux VMs, for computer-use agents., https://www.daytona.io/changelog
  - GPU sandboxes: GPU sandboxes with a GPU type selector, a dedicated physical GPU per sandbox, and regional GPU quotas., https://www.daytona.io/changelog
  - Sandbox snapshots and forking: Fork and snapshot API endpoints and experimental SDK methods for branching sandbox state., https://www.daytona.io/changelog
  - Bring Your Own Compute (custom regions / self-hosted runners): Runs sandbox workloads on customer-owned runner nodes managed by Daytona's control plane, through custom regions with an optional snapshot manager, sandbox proxy and SSH gateway, deployed with official Helm charts on Kubernetes., https://www.daytona.io/docs/en/bring-your-own-compute/
  - OpenAI Agents SDK integration: Daytona as a hosted sandbox backend for OpenAI Agents SDK Sandbox Agents (openai-agents[daytona])., https://www.daytona.io/docs/en/guides/openai-agents/openai-agents-sdk-with-sandboxes
  - Claude Managed Agents self-hosted sandbox integration: Daytona as a self-hosted sandbox provider for Anthropic Claude Managed Agents., https://www.daytona.io/docs/en/guides/claude/claude-managed-agents/
  - Devin Outposts on Daytona: Runs Cognition Devin's tool calls in Daytona Linux or Windows sandboxes (requires Devin Enterprise). Listed as a partner on Cognition's Outposts docs., https://www.daytona.io/docs/en/guides/devin/devin-outposts/
  - Cursor Self-Hosted Machines on Daytona: Runs Cursor Cloud Agents in Daytona sandboxes through Self-Hosted Pools (requires Cursor Enterprise)., https://www.daytona.io/dotfiles/cursor-self-hosted-machines-on-daytona
  - Daytona via Stripe Projects: One-command provisioning of Daytona accounts and sandbox credentials through Stripe Projects, with Stripe Shared Payment Tokens., https://www.daytona.io/dotfiles/daytona-stripe-projects
- notes_for_researchers:
  - The fork and snapshot API (since Apr 2026) branches full sandbox state (filesystem, databases, services), which is useful for tree search and RL rollouts. Daytona's own Terminal-Bench study found full snapshots mattered on a stateful task (4/5 vs 0/5) but independent retries did better on file-centric tasks (17/25 vs 6/25).
  - Harbor, the official Terminal-Bench 2.0 harness, supports Daytona as an execution environment through --env daytona for high-concurrency runs.
  - Computer-use evaluation: Daytona published a reproducible OSWorld Windows run (46 tasks), and Yutori's n2 playground runs OSWorld-image Linux desktops on Daytona with noVNC.
  - The platform code is no longer open source, so reproducibility depends on the hosted service or Bring Your Own Compute. The archived AGPL repo is unsupported.
- notes_for_program_managers:
  - Frontier-lab ties are platform integrations, not disclosed customer contracts. Daytona is a sandbox provider for Anthropic Claude Managed Agents and for OpenAI Agents SDK Sandbox Agents. It is also a Cognition Devin Outposts partner.
  - Scale signal: $48.3M of equity sold (Form D, Aug 2026; $60M offering) about six months after the $24M Series A. Stage, lead investor and valuation are undisclosed. Hiring is focused on GPU and Windows sandbox teams.
  - Strategy shift: closed source since June 2026, citing security. The legacy AGPL repo is archived and unmaintained.
  - Customer references beyond Clay (verified) are mostly vendor-published: LangChain, Turing, Writer, SambaNova, the Cursor integration, Yutori, Anchor and Brainbase.
- notes_for_data_ops:
  - SDKs in Python, TypeScript, Go, Ruby and Java. Integrations include the openai-agents[daytona] extra, a Claude Managed Agents guide, Devin Outposts, Cursor Self-Hosted Pools, Harbor/Terminal-Bench, and Stripe Projects provisioning.
  - trust.daytona.io lists SOC 2 Type II (Insight Assurance, issued 2026-07-28), SOC 2 Type I, HIPAA, ISO/IEC 27001 and GDPR. Reports are available on request, and all are vendor-hosted.
  - Security history: an April 2026 advisory on API credentials readable inside sandboxes with sudo was fixed on 2026-04-09; rotate keys if you used affected snapshots before then. The CVE-2026-31431 ('Copy Fail') kernel flaw was patched fleet-wide within 12 hours, with no customer action required.
  - Breaking API change: sandbox listing moved to cursor-based pagination, and the old endpoint and page parameter stopped working on 2026-06-25 (SDK v0.180.0 or later required). Org webhooks and OpenTelemetry org metrics are available.
  - Bring Your Own Compute: customer-run runners in custom regions under Daytona's control plane, deployed with official Helm charts.
- sources:
  - https://www.sec.gov/Archives/edgar/data/1977563/000197756326000002/xslFormDX01/primary_doc.xml (accessed 2026-10-05)
  - https://formds.com/issuers/daytona-platforms-inc (accessed 2026-10-05)
  - https://www.alleywatch.com/2026/08/the-alleywatch-startup-daily-funding-report-8-17-2026/ (accessed 2026-10-05)
  - https://www.notacompany.com/bookmarks/daytona-raises-a-48-0m-series-b-round (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/updates/daytona-is-going-closed-source (accessed 2026-10-05)
  - https://github.com/daytonaio/daytona (accessed 2026-10-05)
  - https://trust.daytona.io/ (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/snapshots-as-search-states-go-explore-with-daytona (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/agent-aware-snapshots-with-daytona (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/kubernetes-is-the-wrong-primitive-for-diverse-workloads (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/osworld-on-daytona-windows-sandboxes (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/cursor-self-hosted-machines-on-daytona (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/devin-outposts-on-daytona (accessed 2026-10-05)
  - https://docs.devin.ai/cloud/outposts/overview.md (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/real-computers-for-computer-use (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/production-grade-browsers-for-agents-anchor-daytona (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/brainbases-universal-harness-api-on-daytona (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/daytona-stripe-projects (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/updates/security-advisory-api-credential-exposure-in-sandboxes (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/updates/security-update-cve-2026-31431-copy-fail (accessed 2026-10-05)
  - https://www.daytona.io/dotfiles/updates/daytona-is-moving-to-cursor-based-pagination (accessed 2026-10-05)
  - https://www.daytona.io/changelog (accessed 2026-10-05)
  - https://github.com/openai/openai-agents-python/releases/tag/v0.14.0 (accessed 2026-10-05)
  - https://pypi.org/project/openai-agents/ (accessed 2026-10-05)
  - https://developers.openai.com/api/docs/guides/agents-api/environments/providers/daytona.md (accessed 2026-10-05)
  - https://www.daytona.io/docs/en/guides/openai-agents/openai-agents-sdk-with-sandboxes (accessed 2026-10-05)
  - https://www.daytona.io/docs/en/bring-your-own-compute/ (accessed 2026-10-05)
  - https://github.com/laude-institute/harbor (accessed 2026-10-05)
  - https://www.daytona.io/careers (accessed 2026-10-05)
  - https://www.daytona.io/about (accessed 2026-10-05)
  - https://slush.org/2026-speakers/ivan-burazin (accessed 2026-10-05)
  - https://github.com/vedranjukic (accessed 2026-10-05)
  - https://www.daytona.io/ (accessed 2026-06-07)
  - https://www.prnewswire.com/news-releases/daytona-raises-24m-series-a-to-give-every-agent-a-computer-302680740.html (accessed 2026-06-07)
  - https://www.daytona.io/dotfiles/daytona-raises-24m-series-a-to-give-every-agent-a-computer (accessed 2026-06-07)
  - https://www.daytona.io/dotfiles/daytona-secures-5m-to-simplify-development-environments (accessed 2026-06-07)
  - https://www.linkedin.com/company/daytonaio (accessed 2026-06-07)
  - https://www.anthropic.com/engineering/managed-agents (accessed 2026-06-07)
  - https://www.daytona.io/docs/en/guides/claude/claude-managed-agents/ (accessed 2026-06-07)
  - https://tracxn.com/d/companies/daytona/__TzaXWUoUzJqHEQmWu6SWgVcuHYFltYtBs_uhDgw84Ss (accessed 2026-06-07)

## Turing (not ranked, incumbent / infrastructure / open-source)
- slug: turing
- segment: Incumbents also building RL environments
- website: https://www.turing.com/
- focus_areas: Coding, Enterprise Workflows, Private Codebases
- positioning: Turing is a large incumbent supplier of expert human data to frontier AI labs. It claims 5M+ experts and work with '9/9 frontier labs' (both self-claimed). In 2026 it moved further into verifiable agent environments and evaluations. It launched Turing Frontier in April, set up the Turing Frontier Research Lab, and published benchmarks including CyberStrike (Harbor format, public on Hugging Face), KernelQuest, CEO Bench, SciCode++ and PLSQLBench (with Oracle, per Turing). It also contributed tasks to Terminal-Bench 3.0 and hired a research-heavy CTO, Ece Kamar (ex-Microsoft Research).
- best_fit: Buyers needing very large-scale human coding and reasoning data, expert labor, and evaluations over real codebases from an established incumbent rather than a focused environment startup.
- overall_confidence: medium
- founded_year: 2018 [reported], source: Widely reported founding year (Jonathan Siddharth & Vijay Krishnan); accessed 2026-06-07
- status: active [confirmed], source: https://www.turing.com/ (accessed 2026-06-07)
- hq_location: San Francisco, CA, USA [confirmed], source: https://www.turing.com/company (accessed 2026-10-05)
- other_locations: unknown [unknown]
- distributed_remote: yes [reported], source: Turing operates a global remote engineering/expert network (accessed 2026-06-07)
- current_headcount: unknown [unknown]
- headcount_band: 200+ [reported], source: Large organization with a global contractor/expert network (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: unknown [unknown]
- has_researchers: yes [reported], source: Turing AGI/research org and expert network (accessed 2026-06-07)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Microsoft Research (CTO Ece Kamar, ex-CVP AI Frontiers Lab); Harvard CS PhD [confirmed], source: https://go.turing.com/press/turing-names-ece-kamar-chief-technology-officer (accessed 2026-10-05)
- published_papers_or_benchmarks: CyberStrike; KernelQuest; CEO Bench; SciCode++; PLSQLBench (with Oracle, per Turing); Open-MM-RL (MMRL); SDLCbench; CompanyBench; EKWbench; Medical Bench; OdysseysMega; Terminal-Bench 3.0 (task contributor); Turing Terminal-Bench Data; Turing Frontier Agentic STEM [confirmed], source: https://bench.turing.com/ (accessed 2026-10-05)
- total_raised: $300M+ (company-stated) [confirmed], source: https://www.turing.com/ (accessed 2026-10-05)
- last_round: Series E, $111M (2025-03-06) [confirmed], source: https://www.turing.com/press (accessed 2026-10-05)
- notable_investors: unknown [unknown]
- valuation: unknown [unknown]
- revenue_signals: The company reported its annual revenue run-rate nearly tripled to $300M in a profitable year (press release 2025-01-28) [confirmed], source: https://www.turing.com/press (accessed 2026-10-05)
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: unknown [unknown]
- what_they_sell: human data, expert networks, benchmarks/evaluations and verifier-graded agentic environments/datasets for frontier labs [confirmed], source: https://www.turing.com/research (accessed 2026-10-05)
- open_source: partial [confirmed], source: https://huggingface.co/datasets/TuringFrontierResearchLab/CyberStrike (accessed 2026-10-05)
- license: CC BY-NC-ND 4.0 (CyberStrike dataset); other datasets proprietary [confirmed], source: https://huggingface.co/datasets/TuringFrontierResearchLab/CyberStrike (accessed 2026-10-05)
- deployment_model: unknown [unknown]
- maturity: GA [reported], source: Established commercial company (accessed 2026-06-07)
- notable_customers: Anthropic (self-claimed, frontier-lab tie); Google (Gemini) (self-claimed, frontier-lab tie); Nvidia (self-claimed) [reported], source: https://www.turing.com/ (accessed 2026-10-05)
- founding_team:
  - Jonathan Siddharth, Co-Founder and CEO, Co-founder and CEO of Turing, listed as co-founder on the company leadership page.
  - Vijay Krishnan, Co-Founder and Head of Growth, Co-founder of Turing, now Head of Growth, according to the company leadership page.
  - Ece Kamar, Chief Technology Officer (since 2026-09), Former Corporate VP leading the AI Frontiers Lab at Microsoft Research. Harvard CS PhD and a BS from Sabanci University. The company page spells her name 'Ece Kumar'; the official release uses 'Kamar'.
  - Robin Bordoli, President and Global Chief Revenue Officer (since 2026-07), Former CRO at Weights & Biases (through its acquisition by CoreWeave) and former CEO of Figure Eight (acquired by Appen). Earlier in go-to-market roles at Marketo and Jive Software. Stanford MBA and Cambridge MEng.
- recent_developments (last 6 months):
  - 2026-09-18 [research] SciCode++ scientific coding dataset introduced, source: https://www.turing.com/blog/scicode-plus-plus-scientific-coding-dataset
  - 2026-09-09 [hire] Patrick McKinney named Chief Information Security Officer, source: https://go.turing.com/press/turing-names-patrick-mckinney-chief-information-security-officer
  - 2026-09-07 [benchmark] CEO Bench: evaluating frontier agents on real company work, source: https://www.turing.com/blog/ceo-bench-frontier-agents-real-company-work
  - 2026-09-01 [hire] Ece Kamar named Chief Technology Officer, source: https://go.turing.com/press/turing-names-ece-kamar-chief-technology-officer
  - 2026-09 [benchmark] KernelQuest benchmark: can agents write Triton GPU kernels?, source: https://www.turing.com/blog/kernelquest-benchmark-ai-agents-triton-kernels
  - 2026-09 [benchmark] bench.turing.com hub lists nine Turing benchmarks, source: https://bench.turing.com/
  - 2026-08-16 [research] PLSQLBench: executable PL/SQL benchmark, which Turing says it built with Oracle, source: https://arxiv.org/abs/2608.15931
  - 2026-08-13 [benchmark] Terminal-Bench 3.0: Turing contributes tasks and publishes a 'confident failure' analysis, source: https://www.turing.com/blog/terminal-bench-3-confident-failure
  - 2026-08 [open-source] Turing Frontier Research Lab releases the CyberStrike cybersecurity agent benchmark, source: https://www.turing.com/blog/cyberstrike-cybersecurity-benchmark-ai-agents
  - 2026-07-14 [hire] Robin Bordoli named President and Global Chief Revenue Officer, source: https://go.turing.com/press/robin-bordoli-named-president-global-cro
  - 2026-07 [research] Turing Frontier Research Lab publishes the Turing Terminal-Bench Data and Turing Frontier Agentic STEM training sets, source: https://www.turing.com/research
  - 2026-04-09 [product] Turing announces Turing Frontier, connecting AI labs with elite domain experts, source: https://www.turing.com/press
- benchmarks_and_research:
  - SciCode++ (dataset), 2026-09-18, LoRA SFT of Qwen 3.5 9B: +10.3% relative on SciCode, +9.1% on SciCode-Verified, https://bench.turing.com/scicode/
  - CEO Bench (benchmark), 2026-09-07, Best model GPT-5.6 Terra averaged 41.3/100, https://bench.turing.com/ceo-bench/
  - KernelQuest (benchmark), 2026-09, Claude Opus 5: 67.0% (±5.3) pass rate, https://labs.turing.com/benchmarks/kernelquest
  - Medical Bench (benchmark), 2026-09, https://bench.turing.com/medical/
  - OdysseysMega (benchmark), 2026-09, https://bench.turing.com/odysseys-mega/
  - PLSQLBench (with Oracle, per Turing) (paper), 2026-08-16, Eight LLMs evaluated; consistent gaps in schema grounding, PL/SQL dialect fidelity, control flow, exception handling and cross-turn consistency, https://arxiv.org/abs/2608.15931
  - Terminal-Bench 3.0 (task contributor) (benchmark), 2026-08-13, In 3,600 trials, almost all failures were well-formed artifacts with wrong content, and agents' self-verification repeated their own misreading of the task, https://bench.turing.com/terminal-bench-3/
  - CyberStrike (benchmark), 2026-08, Top model GPT-5.6 Sol: 31.5% mean per-task pass rate, https://labs.turing.com/benchmarks/cyberstrike
  - SDLCbench (benchmark), 2026-08, Leader Fable 5: 0.51 reward, 40% of tasks fully passed, https://bench.turing.com/sdlc/
  - EKWbench (benchmark), 2026-08, https://bench.turing.com/ekw/
  - Turing Terminal-Bench Data (dataset), 2026-07, GPT-5.5 48.6%, Claude Opus 4.8 41.3%, Gemini 3.5 Flash 38.3%, https://www.turing.com/research
  - Turing Frontier Agentic STEM (dataset), 2026-07, Claude Opus 4.8 43.7%, GPT-5.5 40.6%, Gemini 3.5 Flash 21%, https://www.turing.com/research
  - CompanyBench (benchmark), 2026-06, https://bench.turing.com/companybench/
  - Open-MM-RL (MMRL) (dataset), 2026-05, https://bench.turing.com/open-mm-rl/
  - bench.turing.com (Turing Benches hub) (leaderboard), https://bench.turing.com/
- products:
  - Turing Frontier: Offering that connects AI labs with elite domain experts. Only the headline is verified; the full release returned 403., https://www.turing.com/press
  - Turing Frontier Research Lab: Research arm that publishes expert-written benchmarks and verifier-graded training datasets and environments for frontier agents., https://labs.turing.com/
  - Turing Benches: Public dashboards for Turing's benchmarks (STEM, SWE, enterprise knowledge work, computer use)., https://bench.turing.com/
  - Turing DataHub: Distribution point for Turing datasets such as MMRL and SciCode, according to bench.turing.com., https://data-hub.turing.com/
  - Turing expert talent platform: Portal that recruits domain experts and engineers for remote AI training and evaluation work., https://work.turing.com/
- notes_for_researchers:
  - The Turing Frontier Research Lab publishes benchmark write-ups with failure-mode analyses, such as the Terminal-Bench 3.0 confident-failure study and KernelQuest's rejection of shortcut runs.
  - Open release: CyberStrike on Hugging Face (CC BY-NC-ND 4.0, non-commercial, no derivatives). Turing says MMRL is on Hugging Face as TuringEnterprises/MMRL, but that page returned 401 when checked. Most other datasets, such as SciCode++, are proprietary, with samples on request.
  - SciCode++ includes a published post-training result: LoRA SFT on Qwen 3.5 9B gave +10.3% relative on SciCode.
  - PLSQLBench (arXiv 2608.15931) is described by Turing as joint research with Oracle.
- notes_for_program_managers:
  - An incumbent at scale. The company page claims 5M+ domain experts and work with '9/9 frontier labs' (self-claimed). The recruiting homepage shows Anthropic, Gemini and Nvidia under 'Work With' (self-claimed logos).
  - Turing's site states $300M+ raised. The last round found is the $111M Series E (2025-03-06, outside this refresh window), and the opened pages name no investors or valuation. A January 2025 release headline reported a ~$300M revenue run-rate in a profitable year. Web search was unavailable, so newer funding press could not be checked.
  - Leadership expanded in 2026: Robin Bordoli (President/CRO, ex-W&B and ex-Figure Eight), Ece Kamar (CTO, ex-Microsoft Research AI Frontiers) and Patrick McKinney (CISO, ex-Invisible Technologies).
- notes_for_data_ops:
  - CyberStrike ships in Harbor format: instruction.md, a containerized environment, deterministic tests with reward config, an oracle solution and task.toml metadata.
  - SDLCbench runs against MCP backend environments, and Turing contributes tasks to Terminal-Bench 3.0.
  - A CISO was appointed in 2026-09, but none of the pages opened mention SOC 2 or ISO certification, so this is still unverified.
- sources:
  - https://www.turing.com/company (accessed 2026-10-05)
  - https://www.turing.com/press (accessed 2026-10-05)
  - https://www.turing.com/ (accessed 2026-10-05)
  - https://go.turing.com/press/turing-names-ece-kamar-chief-technology-officer (accessed 2026-10-05)
  - https://go.turing.com/press/robin-bordoli-named-president-global-cro (accessed 2026-10-05)
  - https://go.turing.com/press/turing-names-patrick-mckinney-chief-information-security-officer (accessed 2026-10-05)
  - https://www.turing.com/blog (accessed 2026-10-05)
  - https://www.turing.com/blog/cyberstrike-cybersecurity-benchmark-ai-agents (accessed 2026-10-05)
  - https://huggingface.co/datasets/TuringFrontierResearchLab/CyberStrike (accessed 2026-10-05)
  - https://huggingface.co/TuringFrontierResearchLab (accessed 2026-10-05)
  - https://www.turing.com/blog/kernelquest-benchmark-ai-agents-triton-kernels (accessed 2026-10-05)
  - https://www.turing.com/blog/ceo-bench-frontier-agents-real-company-work (accessed 2026-10-05)
  - https://www.turing.com/blog/terminal-bench-3-confident-failure (accessed 2026-10-05)
  - https://www.turing.com/blog/introducing-terminal-bench-3-0 (accessed 2026-10-05)
  - https://www.turing.com/blog/scicode-plus-plus-scientific-coding-dataset (accessed 2026-10-05)
  - https://www.turing.com/blog/frontier-ai-newsletter-84 (accessed 2026-10-05)
  - https://arxiv.org/abs/2608.15931 (accessed 2026-10-05)
  - https://www.turing.com/research (accessed 2026-10-05)
  - https://bench.turing.com/ (accessed 2026-10-05)
  - https://bench.turing.com/open-mm-rl/ (accessed 2026-10-05)
  - https://bench.turing.com/sdlc/ (accessed 2026-10-05)

## micro1 (not ranked, incumbent / infrastructure / open-source)
- slug: micro1
- segment: Incumbents also building RL environments
- website: https://www.micro1.ai/
- focus_areas: Enterprise Workflows
- positioning: micro1 is a San Francisco AI training-data company founded in 2022 by Ali Ansari. It began as an AI recruiter (Zara) and now uses that system to vet expert contractors. It sells expert human data, frontier evaluations and RL environments to AI labs under the Realm brand, an enterprise agent-evaluation layer called Cortex, and expert-demonstrated robotics data, all run on its flow data platform. To make its 'RL gyms' realistic it pays companies for operational data, and it made a $12.5M late bid for Spirit Airlines' records. TechCrunch reported a $500M gross run rate in August 2026, and Forbes reported a raise of more than $100M at a $4B valuation in September 2026, with frontier labs taking part.
- best_fit: Labs and enterprises that want a large managed vendor for expert-authored, rubric-graded agent tasks and evals in professional domains (legal, finance, tax, healthcare), for RL environments seeded with real enterprise data, or for robotics demonstration data.
- overall_confidence: medium
- founded_year: 2022 [reported], source: https://finance.yahoo.com/news/scale-ai-rival-micro1-hits-134611177.html (accessed 2026-10-05)
- status: active [confirmed], source: https://www.micro1.ai/research (accessed 2026-10-05)
- hq_location: San Francisco, CA, USA [reported], source: https://www.forbes.com.au/news/uncategorized/this-25-year-old-raised-us100-million-for-his-ai-startup-at-a-us4-billion-valuation/ (accessed 2026-10-05)
- other_locations: unknown [unknown]
- distributed_remote: unknown [unknown]
- current_headcount: Full-time staff not disclosed. LinkedIn lists a 51-200 band, but about 10,400 people list micro1 on LinkedIn, likely including contractors. [reported], source: https://www.linkedin.com/company/micro1/ (accessed 2026-10-05)
- headcount_band: 51-200 [reported], source: https://www.linkedin.com/company/micro1/ (accessed 2026-10-05)
- headcount_growth: unknown [unknown]
- open_roles_count: unknown [unknown], source: https://www.micro1.ai/careers (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: https://www.micro1.ai/research/pii-transformation-for-enterprise-datasets (accessed 2026-10-05)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Andrew Maas, VP of AI; Yongchao Zhou, Director of Research; Rumi Elias Calles, Director of Research Engineering; Arian Sadeghi, VP Robotics; Mark Esposito, Chief Economist (co-authors with Harvard's Liu Zhang); Stanford's Stefano Ermon, said by Ansari to lead micro1's research lab (Stanford Daily, 2025-10; not confirmed on micro1 pages) [reported], source: https://www.micro1.ai/research/ai-models-now-introduce-safety-risks-in-the-physical-world (accessed 2026-10-05)
- published_papers_or_benchmarks: Realm: Legal reasoning benchmark (2026-04); Realm: Financial reasoning benchmark (2026-05); Realm: Tax reasoning benchmark; Realm: Pathology-report reasoning benchmark (v1 2026-06, v2 2026-09); Crosby-micro1 RedlineBench; CortexRetrievalBench; LongExtractionBench; No Last Mile: A Theory of the Human Data Market (arXiv 2603.00932); The Benchmark Ceiling (arXiv 2607.01254); PII Transformation for Enterprise Datasets (flow-transform 1.0); AI Models Now Introduce Safety Risks in the Physical World (2026-09) [confirmed], source: https://www.micro1.ai/research (accessed 2026-10-05)
- total_raised: $138M+ (est. sum: $3.3M pre-seed/seed 2023 + $35M Series A 2025-09 + $100M+ round 2026-09) [estimated], source: https://www.forbes.com.au/news/uncategorized/this-25-year-old-raised-us100-million-for-his-ai-startup-at-a-us4-billion-valuation/ (accessed 2026-10-05)
- last_round: $100M+ at $4B valuation, 2026-09 (stage not disclosed) [reported], source: https://www.forbes.com.au/news/uncategorized/this-25-year-old-raised-us100-million-for-his-ai-startup-at-a-us4-billion-valuation/ (accessed 2026-10-05)
- notable_investors: 01 Advisors (Dick Costolo, Adam Bain; led Series A; Bain on board); Two unnamed frontier AI labs (2026 round, reported); Two unnamed xAI co-founders (2026 round, reported); Dream Ventures (led 2023 round); Jason Calacanis (2023); Joshua Browder (2023; board member); Cory Levy (2023) [reported], source: https://techcrunch.com/2025/09/12/micro1-a-competitor-to-scale-ai-raises-funds-at-500m-valuation/ (accessed 2026-10-05)
- valuation: $4B (2026-09, reported) [reported], source: https://www.forbes.com.au/news/uncategorized/this-25-year-old-raised-us100-million-for-his-ai-startup-at-a-us4-billion-valuation/ (accessed 2026-10-05)
- revenue_signals: $500M gross annual run rate (2026-08), with net estimated at $150M-$200M (TechCrunch, citing a person familiar with the company). Earlier: about $7M ARR at the start of 2025, $50M in 2025-09, $100M+ in 2025-12. [reported], source: https://techcrunch.com/2026/08/20/ai-data-startup-micro1-reaches-500m-gross-run-rate-amid-ai-training-boom/ (accessed 2026-10-05)
- soc2: unknown [unknown], source: https://www.micro1.ai/government (accessed 2026-10-05)
- other_certifications: CDAO Tradewinds Solutions Marketplace 'Awardable' status (procurement status, not a security certification) [confirmed], source: https://www.micro1.ai/government (accessed 2026-10-05)
- security_page: unknown [unknown]
- what_they_sell: human data + environments [confirmed], source: https://www.micro1.ai/realm (accessed 2026-10-05)
- open_source: partial (LongExtractionBench code and sample data; RedlineBench dataset with Crosby; Realm, Cortex and flow are proprietary) [confirmed], source: https://github.com/micro1-research (accessed 2026-10-05)
- license: mixed: longextract-bench MIT; RedlineBench dataset CC-BY-4.0 (Crosby); commercial products proprietary [confirmed], source: https://github.com/micro1-research (accessed 2026-10-05)
- deployment_model: managed service (expert data, evals and RL environments delivered to labs and enterprises) [estimated], source: https://www.micro1.ai/cortex (accessed 2026-10-05)
- maturity: GA [reported], source: https://techcrunch.com/2026/08/20/ai-data-startup-micro1-reaches-500m-gross-run-rate-amid-ai-training-boom/ (accessed 2026-10-05)
- notable_customers: Microsoft (verified, frontier-lab tie); Amazon (self-claimed, frontier-lab tie); 1X (self-claimed); Box (self-claimed) [reported], source: https://www.forbes.com.au/news/uncategorized/this-25-year-old-raised-us100-million-for-his-ai-startup-at-a-us4-billion-valuation/ (accessed 2026-10-05)
- founding_team:
  - Ali Ansari, Founder & CEO, Born in Iran and raised in Los Angeles. Studied computer science at UC Berkeley, and his micro1 bio adds Stanford computer science with a focus on reinforcement learning (he told the Stanford Daily in 2025 that he was a master's student). Before micro1 he ran a textbook-resale business, a math-tutoring platform and a software agency. Forbes calls him a solo founder.
  - Andrew Maas, VP of AI (non-founder), VP of AI at micro1, co-author of the September 2026 flow-transform PII paper and the robotics safety research. Prior background is not stated on micro1 pages.
  - Yongchao Zhou, Director of Research (non-founder), Director of Research at micro1, per the September 2026 research byline. Prior background is not stated on micro1 pages.
  - Kathy Wang, Chief Information Security Officer (non-founder), micro1's CISO. Her 'Why I joined micro1' post says the expert data passing through micro1 is among the most sensitive in the AI supply chain. Prior roles are not stated on the page.
- funding_history:
  - 2026-09-22 undisclosed $100M, valuation $4B, source: https://www.forbes.com.au/news/uncategorized/this-25-year-old-raised-us100-million-for-his-ai-startup-at-a-us4-billion-valuation/
  - 2025-09-12 Series A $35M led by 01 Advisors (Dick Costolo, Adam Bain), valuation $500M, source: https://techcrunch.com/2025/09/12/micro1-a-competitor-to-scale-ai-raises-funds-at-500m-valuation/
  - 2023-10-24 Pre-seed / seed $3.3M led by Dream Ventures, valuation $30M, source: https://pulse2.com/micro1-ai-recruiting-company-raises-3-3-million/
- recent_developments (last 6 months):
  - 2026-09-24 [research] micro1 publishes flow-transform 1.0 for corpus-level PII transformation, source: https://www.micro1.ai/research/pii-transformation-for-enterprise-datasets
  - 2026-09-22 [funding] micro1 raises more than $100M at a $4B valuation (reported), source: https://www.forbes.com.au/news/uncategorized/this-25-year-old-raised-us100-million-for-his-ai-startup-at-a-us4-billion-valuation/
  - 2026-09-22 [product] micro1 introduces flow, its data platform combining experts and models, source: https://micro1.ai/blog/introducing-flow
  - 2026-09-18 [research] micro1 research: frontier models introduce safety risks when controlling robots, source: https://www.micro1.ai/research/ai-models-now-introduce-safety-risks-in-the-physical-world
  - 2026-09-04 [other] micro1 presses its $12.5M late bid for bankrupt Spirit Airlines' records against Google's $10M winning offer, source: https://thenextweb.com/news/micro1-12-5m-counterbid-spirit-airlines-records-google-deidentification-cjeu-srb-edpb-opinion-28-2024
  - 2026-09-01 [partnership] micro1 says it was selected to support the DOE Genesis Mission, source: https://micro1.ai/blog/micro1-supporting-the-doe-genesis
  - 2026-08-20 [other] TechCrunch: micro1 reaches a $500M gross run rate, source: https://techcrunch.com/2026/08/20/ai-data-startup-micro1-reaches-500m-gross-run-rate-amid-ai-training-boom/
  - 2026-08-02 [product] Flow for mission data: machine-first annotation of defense EO/IR video, source: https://micro1.ai/blog/data-for-our-defense-customers
  - 2026-07-13 [research] 'The Benchmark Ceiling' paper by micro1's Chief Economist, source: https://www.micro1.ai/research/the-benchmark-ceiling
  - 2026-06-30 [open-source] LongExtractionBench code and sample data released, source: https://github.com/micro1-research
  - 2026-05-15 [benchmark] Realm: Financial Reasoning Benchmark, source: https://www.micro1.ai/benchmark/realm-financial
  - 2026-04-16 [other] micro1 named to Forbes' AI 50 'Brink' list, source: https://www.micro1.ai/newsroom
  - 2026-04-10 [benchmark] Realm: Legal Reasoning Benchmark, source: https://www.micro1.ai/benchmark/realm-legal
- benchmarks_and_research:
  - PII Transformation for Enterprise Datasets (flow-transform 1.0) (paper), 2026-09-24, TQI 88.9 with agentic review vs 74.9 for NVIDIA NeMo Anonymizer. Detection F1 0.829 vs 0.678. Self-reported., https://www.micro1.ai/research/pii-transformation-for-enterprise-datasets
  - AI Models Now Introduce Safety Risks in the Physical World (paper), 2026-09-18, Both models failed both tasks, scoring 6/15 to 9/15, https://www.micro1.ai/research/ai-models-now-introduce-safety-risks-in-the-physical-world
  - Realm: Pathology-report reasoning benchmark (v2) (benchmark), 2026-09, Claude Fable 5.1 leads the main scoreboard at 78.0%, ahead of Muse Spark 1.3 (75.8%) and Claude Opus 5 (75.2%). A separate summary table lists Claude Opus 5.5 at 77.0%. Extraction criteria are met 93% of the time, interpretation criteria 73%., https://www.micro1.ai/benchmark/realm-pathology-report-v2
  - The Benchmark Ceiling (paper), 2026-07-13, https://www.micro1.ai/research/the-benchmark-ceiling
  - LongExtractionBench (benchmark), 2026-06-30, Reducto Deep Extract completed 225/225 at 99.6% precision and recall. Claude Opus 4.8 and Gemini 3.1 Pro completed 116 and 112 of 225., https://www.micro1.ai/benchmark/long-extraction
  - Realm: Pathology-report reasoning benchmark (v1) (benchmark), 2026-06-22, Claude Opus 4.8 82.6%, GPT-5.5 76.3%, Gemini 3.5 Flash 75.7%, https://www.micro1.ai/benchmark/realm-pathology-report
  - Realm: Financial reasoning benchmark (benchmark), 2026-05-15, GPT-5.5 leads on mean reward. No model clears 50% on judgment-call tasks, and VC tasks average 22.8%., https://www.micro1.ai/benchmark/realm-financial
  - Realm: Legal reasoning benchmark (benchmark), 2026-04-10, Claude Opus 4.7 and GPT-5.5 are statistically tied (0.007 apart). Gemini 3.1 Pro scores about two-thirds of either., https://www.micro1.ai/benchmark/realm-legal
  - No Last Mile: A Theory of the Human Data Market (paper), 2026-03-01, Predicts a long-run structured-labor share of 5-7%, https://www.micro1.ai/research/no-last-mile
  - Realm: Tax reasoning benchmark (benchmark), 2026, GPT-5.4 leads with mean reward 0.368 and pass@3 of 28%. 44% of criteria go unsolved by every model., https://www.micro1.ai/benchmark/realm-tax
  - Crosby-micro1 RedlineBench (benchmark), 2026, Turn-weighted scores: GPT-5.5 50.5%, Claude Fable 5 47.3%, Gemini 3.5 Flash 45.1%, Claude Opus 4.8 44.4%, https://www.micro1.ai/benchmark/crosby-micro1-redlinebench
  - CortexRetrievalBench (benchmark), 2026, Success@1 ranges from 42.1% (BM25) to 69.3% (OpenAI dense retriever). Authority Success@10 ranges from 92.1% to 99.3%., https://www.micro1.ai/benchmark/cortex-retrieval-bench
- products:
  - Realm: Sold as 'Frontier evaluations and RL environments for AI Labs': expert-authored, sandboxed agent tasks with rubric-based rewards across 100+ fields (finance, medical, legal, coding, STEM, VLM, audio), delivered as a managed service. Forbes describes these as RL gyms built with real enterprise data., https://www.micro1.ai/realm
  - Cortex: Evaluation and improvement layer for enterprise agents in production: eval design, failure diagnosis by domain experts, targeted training data and reliability monitoring., https://www.micro1.ai/cortex
  - Robotics data: Expert-demonstrated POV and third-person capture, 2D/3D annotation and QC for embodied-AI labs., https://www.micro1.ai/robotics
  - flow: Data platform that pairs expert judgment with model-assisted task generation (flow gen), QC (flow qc), grading (flow grader) and orchestration. Variants include flow-transform for PII and Flow for mission data (defense EO/IR)., https://micro1.ai/blog/introducing-flow
  - Data partnerships: micro1 pays companies for operational data such as SOPs, knowledge bases, workflow and decision data, and human feedback. Tiers are $100k+, $500k+ and $1M+. Data goes through PII removal, restricted access and isolated pipelines, and is used for Realm evaluations and RL environments., https://www.micro1.ai/data-partnerships
  - Zara (AI interviewer) and expert network: AI recruiter that vets the expert contractors. The government page cites '130,000+ deeply vetted candidates across 100+ domains and 60+ languages'., https://www.micro1.ai/experts
  - Government: Agent buildouts and human data for government buyers. micro1 holds 'Awardable' status on the CDAO Tradewinds Solutions Marketplace., https://www.micro1.ai/government
- notes_for_researchers:
  - The Realm benchmark pages (legal, financial, tax, pathology v1 and v2) document the task design: sandboxed agents with shell, file I/O and web search; 35-60 expert rubric items per legal task, and 1,071 criteria in total for pathology v2; and an LLM judge giving a reward from 0 to 1. Most of these pages do not link a public dataset or paper.
  - Open releases are limited: LongExtractionBench (github.com/micro1-research/longextract-bench, MIT; Hugging Face micro1-inc/longextract-bench-50), RedlineBench with Crosby (Hugging Face crosbylegal/RedlineBench, CC-BY-4.0) and a small Hugging Face dataset, micro1-inc/Prospera_Benchmark. No open environments or model weights were found.
  - Papers with arXiv versions: 'No Last Mile' (2603.00932) and 'The Benchmark Ceiling' (2607.01254). The flow-transform PII paper and the robotics-safety study (both September 2026) appear only on micro1.ai. In a 2025 Stanford Daily interview, Ansari said Stanford's Stefano Ermon leads micro1's research lab. No 2026 micro1 page confirms this.
- notes_for_program_managers:
  - Scale: TechCrunch, citing an anonymous source, reports a $500M gross run rate (August 2026) and estimates net at $150M-$200M. Earlier figures: about $7M ARR at the start of 2025, $50M in September 2025 and $100M+ in December 2025.
  - Frontier-lab ties: Forbes (September 2026) names unnamed frontier labs, Microsoft, Amazon and 1X as customers, and reports that two frontier labs and two xAI co-founders joined the $100M+ round at $4B. TechCrunch (December 2025) described Microsoft as one of the AI labs micro1 works with. micro1's own site names no lab.
  - Engagement is a managed service: Realm for labs, Cortex for enterprises and Robotics for embodied AI, all run on the flow platform. The main named public reference is a self-published Box case study (enterprise agent eval data), which has no Box quote.
  - micro1 makes its environments realistic by buying enterprise data: data partnerships at $100k to $1M+, and a $12.5M late bid for Spirit Airlines' records that was unresolved in coverage through late September 2026. Check data provenance and consent terms.
- notes_for_data_ops:
  - No public trust center and no SOC 2 or ISO 27001 claim was found on the micro1.ai home, Realm, government or data-partnerships pages, and /security and trust.micro1.ai were not found. micro1 does have a named CISO, Kathy Wang. Ask for attestations directly.
  - Privacy controls stated on the data-partnerships page: agreed data scope, PII removal and anonymization, restricted access, isolated processing pipelines, and deletion on request. flow-transform 1.0 is the de-identification tool.
  - No public API docs or delivery-format specs were found. Government standing: CDAO Tradewinds 'Awardable' status, and micro1 says it was selected to support the DOE Genesis Mission (September 2026).
- sources:
  - https://www.micro1.ai/ (accessed 2026-10-05)
  - https://www.micro1.ai/realm (accessed 2026-10-05)
  - https://www.micro1.ai/cortex (accessed 2026-10-05)
  - https://www.micro1.ai/robotics (accessed 2026-10-05)
  - https://www.micro1.ai/data-partnerships (accessed 2026-10-05)
  - https://www.micro1.ai/government (accessed 2026-10-05)
  - https://www.micro1.ai/research (accessed 2026-10-05)
  - https://www.micro1.ai/research/no-last-mile (accessed 2026-10-05)
  - https://www.micro1.ai/research/ai-models-now-introduce-safety-risks-in-the-physical-world (accessed 2026-10-05)
  - https://www.micro1.ai/newsroom (accessed 2026-10-05)
  - https://micro1.ai/blog/introducing-flow (accessed 2026-10-05)
  - https://micro1.ai/blog/micro1-supporting-the-doe-genesis (accessed 2026-10-05)
  - https://micro1.ai/blog/data-for-our-defense-customers (accessed 2026-10-05)
  - https://www.micro1.ai/research/pii-transformation-for-enterprise-datasets (accessed 2026-10-05)
  - https://www.micro1.ai/research/the-benchmark-ceiling (accessed 2026-10-05)
  - https://www.micro1.ai/benchmark/realm-legal (accessed 2026-10-05)
  - https://www.micro1.ai/benchmark/realm-financial (accessed 2026-10-05)
  - https://www.micro1.ai/benchmark/realm-tax (accessed 2026-10-05)
  - https://www.micro1.ai/benchmark/realm-pathology-report (accessed 2026-10-05)
  - https://www.micro1.ai/benchmark/realm-pathology-report-v2 (accessed 2026-10-05)
  - https://www.micro1.ai/benchmark/crosby-micro1-redlinebench (accessed 2026-10-05)
  - https://huggingface.co/datasets/crosbylegal/RedlineBench (accessed 2026-10-05)
  - https://www.micro1.ai/benchmark/cortex-retrieval-bench (accessed 2026-10-05)
  - https://www.micro1.ai/benchmark/long-extraction (accessed 2026-10-05)
  - https://github.com/micro1-research (accessed 2026-10-05)
  - https://huggingface.co/micro1-inc (accessed 2026-10-05)
  - https://www.micro1.ai/ali-ansari (accessed 2026-10-05)
  - https://www.micro1.ai/team/why-i-joined-micro1-2 (accessed 2026-10-05)
  - https://www.micro1.ai/case-study/box (accessed 2026-10-05)
  - https://www.micro1.ai/series-a (accessed 2026-10-05)
  - https://www.forbes.com.au/news/uncategorized/this-25-year-old-raised-us100-million-for-his-ai-startup-at-a-us4-billion-valuation/ (accessed 2026-10-05)
  - https://techcrunch.com/2026/08/20/ai-data-startup-micro1-reaches-500m-gross-run-rate-amid-ai-training-boom/ (accessed 2026-10-05)
  - https://techcrunch.com/2025/12/04/micro1-a-scale-ai-competitor-touts-crossing-100m-arr/ (accessed 2026-10-05)
  - https://techcrunch.com/2025/09/12/micro1-a-competitor-to-scale-ai-raises-funds-at-500m-valuation/ (accessed 2026-10-05)
  - https://finance.yahoo.com/news/scale-ai-rival-micro1-hits-134611177.html (accessed 2026-10-05)
  - https://pulse2.com/micro1-ai-recruiting-company-raises-3-3-million/ (accessed 2026-10-05)
  - https://e.vnexpress.net/news/tech/personalities/from-50-ebay-flip-to-2-5b-ai-startup-how-iran-born-ali-ansari-is-closing-in-on-billionaire-status-at-25-5040490.html (accessed 2026-10-05)
  - https://stanforddaily.com/2025/10/16/micro1-founder-ali-ansari-on-ai-and-human-intelligence/ (accessed 2026-10-05)
  - https://thenextweb.com/news/micro1-12-5m-counterbid-spirit-airlines-records-google-deidentification-cjeu-srb-edpb-opinion-28-2024 (accessed 2026-10-05)
  - https://www.aa.com.tr/en/science-technology/ai-firms-set-sights-on-data-from-bankrupt-companies/4073573 (accessed 2026-10-05)
  - https://time.com/article/2026/08/25/google-spirit-airlines-ai-data-RL/ (accessed 2026-10-05)
  - https://www.linkedin.com/company/micro1/ (accessed 2026-10-05)

## Deeptune (not ranked, incumbent / infrastructure / open-source)
- slug: deeptune
- segment: Commercial vendors
- website: https://deeptune.com
- focus_areas: Coding, Computer Use, Enterprise Workflows, Private Codebases
- positioning: Deeptune was a New York-based startup building managed reinforcement-learning environments ('training gyms') for computer-use and code, where AI agents practice and are evaluated on realistic digital knowledge-work tasks (simulating tools like Slack and Salesforce). It sold these pre-built environments primarily to frontier AI labs and raised a $43M Series A led by a16z (March 2026). In July 2026 it was acquired by Mercor; the team joined Mercor and Deeptune's environment platform now sits under Mercor. It is therefore no longer ranked as an independent vendor.
- best_fit: Frontier labs and model teams needing ready-made, managed RL environments for training/evaluating computer-use and coding agents.
- overall_confidence: high
- founded_year: ~2025 per official Series A blog ('since we started one year ago', Mar 2026); aggregators (Tracxn, CB Insights) and LinkedIn list 2022, conflict, leaning 2025 per primary source [reported], source: https://deeptune.com/blog/series-a/ ; https://www.linkedin.com/company/trydeeptune accessed 2026-06-07
- status: acquired (by Mercor, 2026-07-09); operates as the 'Deeptune Environments' team within Mercor [confirmed], source: https://www.mercor.com/blog/mercor-to-acquire-deeptune/ (accessed 2026-10-05)
- hq_location: New York, NY, USA [confirmed], source: https://fortune.com/2026/03/19/andreessen-horowitz-ai-startups-deeptune-series-a/ accessed 2026-06-07
- other_locations: India (remote Member of Technical Staff role posted; a job posting, not a confirmed office) [reported], source: https://jobs.ashbyhq.com/deeptune accessed 2026-06-07
- distributed_remote: partial, NYC team described as in-person; some remote roles posted (e.g. India) [reported], source: https://jobs.ashbyhq.com/deeptune ; https://fortune.com/2026/03/19/andreessen-horowitz-ai-startups-deeptune-series-a/ accessed 2026-06-07
- current_headcount: ~20-25 (Fortune: 'roughly 20-person'; LinkedIn snippet: 25) [reported], source: https://fortune.com/2026/03/19/andreessen-horowitz-ai-startups-deeptune-series-a/ ; https://www.linkedin.com/company/trydeeptune accessed 2026-06-07
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/trydeeptune accessed 2026-06-07
- headcount_growth: unknown [unknown]
- open_roles_count: Standalone Ashby board is empty. Hiring now runs under Mercor (e.g. Member of Technical Staff and Strategic Projects Lead, 'Deeptune', posted 2026-08-10, NYC). [reported], source: https://jobs.generalcatalyst.com/companies/mercor/jobs/89404694-strategic-projects-lead-deeptune (accessed 2026-10-05)
- has_researchers: yes [reported], source: https://a16z.com/announcement/investing-in-deeptune/ accessed 2026-06-07
- researcher_count: unknown [unknown]
- researcher_backgrounds: Team includes engineers/operators from Anthropic, Scale AI, Palantir, Hebbia, Glean, Retool, Modal (per company/press); CEO/co-founder Tim Lupo: ex-Hebbia founding engineer, USC CS & Business; Co-founder/CTO Lukas Schmit (per Crunchbase and press) [reported], source: https://fortune.com/2026/03/19/andreessen-horowitz-ai-startups-deeptune-series-a/ ; https://www.crunchbase.com/person/lukas-schmit ; https://www.linkedin.com/in/timlup/ accessed 2026-06-07
- published_papers_or_benchmarks: No own papers/benchmarks published; a16z post references third-party benchmarks OSWorld and Terminal-Bench as targets Deeptune environments help improve (cites Opus 4.6 ~72.7% and GPT-5.4 ~75% on OSWorld) [reported], source: https://a16z.com/announcement/investing-in-deeptune/ accessed 2026-06-07
- total_raised: ~$43M (latest/likely only round). NOTE: value reflects the Series A round; lifetime total is uncertain, aggregators conflict (Tracxn/CB Insights $46.1M, PitchBook $42.2M) [reported], source: https://deeptune.com/blog/series-a/ ; https://tracxn.com/d/companies/deeptune/__oi3tMd7lIKO3Yo_B0aq6tBQHcjZSfLMnyHdwbaOa0SE accessed 2026-06-07
- last_round: Series A, $43M, announced March 19, 2026 (Crunchbase lists round date Feb 23, 2026) [confirmed], source: https://deeptune.com/blog/series-a/ ; https://a16z.com/announcement/investing-in-deeptune/ ; https://fortune.com/2026/03/19/andreessen-horowitz-ai-startups-deeptune-series-a/ accessed 2026-06-07
- notable_investors: Andreessen Horowitz (a16z, lead); 776; Abstract Ventures; Inspired Capital; Noam Brown (angel, OpenAI); Brendan Foody (angel, Mercor CEO); Yash Patil (angel, Applied Compute CEO) [confirmed], source: https://deeptune.com/blog/series-a/ ; https://fortune.com/2026/03/19/andreessen-horowitz-ai-startups-deeptune-series-a/ accessed 2026-06-07
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: unknown [unknown]
- what_they_sell: environments [confirmed], source: https://a16z.com/announcement/investing-in-deeptune/ accessed 2026-06-07
- open_source: no [estimated]
- license: unknown [unknown]
- deployment_model: managed-hosted environments ('training gyms'), now sold through Mercor alongside its expert network [reported], source: https://www.mercor.com/blog/mercor-to-acquire-deeptune/ (accessed 2026-10-05)
- maturity: acquired; product continues inside Mercor [confirmed], source: https://jobs.generalcatalyst.com/companies/mercor/jobs/89404711-member-of-technical-staff-deeptune (accessed 2026-10-05)
- notable_customers: Leading/frontier AI labs (unnamed; company claims '100s of gyms' built for frontier research labs) (self-claimed, frontier-lab tie) [reported], source: https://deeptune.com/ (accessed 2026-10-05)
- founding_team:
  - Tim Lupo, Co-founder & CEO (joined Mercor with the team after the July 2026 acquisition), USC (per LinkedIn). Formerly a founding engineer at Hebbia per the baseline's LinkedIn read; this could not be re-confirmed on the public profile.
  - Lukas Schmit, Co-founder (reported by the aggregator Tracxn and marked there as a former role; the CTO title is unverified), unknown
- recent_developments (last 6 months):
  - 2026-08-10 [hire] Mercor hiring for its 'Deeptune' RL environments team in NYC, source: https://jobs.generalcatalyst.com/companies/mercor/jobs/89404711-member-of-technical-staff-deeptune
  - 2026-07-09 [acquisition] Mercor acquires Deeptune; team joins Mercor in New York, source: https://www.mercor.com/blog/mercor-to-acquire-deeptune/
  - 2026-07-09 [acquisition] Fortune: Foody says his angel check into Deeptune was made with acquisition in mind, source: https://fortune.com/2026/07/09/ai-unicorn-mercor-acquires-deeptune-brendan-foody-investor-a16z-openai-anthropic/
  - 2026-07-09 [acquisition] SiliconANGLE: Mercor buys Deeptune to build training environments for AI agents, source: https://siliconangle.com/2026/07/09/mercor-buys-deeptune-build-training-environments-ai-agents/
- products:
  - Deeptune training gyms (now 'Deeptune Environments' within Mercor): Managed RL environments ('training gyms') that simulate enterprise software such as Slack, Salesforce, spreadsheets and code. Each comes with problems, datasets and infrastructure for training computer-use and coding agents, and integrates in 'a few lines of code' (company claim). Since July 2026 they are offered through Mercor., https://deeptune.com/
- notes_for_researchers:
  - Deeptune has published no papers, benchmarks or open-source releases of its own. The only benchmarks mentioned are third-party targets (OSWorld, Terminal-Bench), cited in a16z's investment post.
  - Environments simulate enterprise apps (Slack, Salesforce, spreadsheets, code) for RL on computer-use and coding agents. Mercor and SiliconANGLE say hundreds of enterprise applications have been recreated (company claim).
- notes_for_program_managers:
  - Acquired by Mercor on 2026-07-09, terms undisclosed. Engagement now goes through Mercor, which Fortune reports supplies Anthropic and OpenAI. No source names these labs as Deeptune customers.
  - Continues as the 'Deeptune Environments' team inside Mercor in NYC, and was still hiring Deeptune-specific roles as of 2026-08-10.
- notes_for_data_ops:
  - No public docs, API reference, integration format, or security/SOC 2 page was found for Deeptune. Expect Mercor's security and contracting terms to apply after the acquisition.
  - For vendor security review: SiliconANGLE reports that attackers breached Mercor in March 2026 through malware in the open-source LiteLLM library, and that Lapsus$ claimed to hold about 4TB of data, including contractor PII.
- sources:
  - https://www.mercor.com/blog/mercor-to-acquire-deeptune/ (accessed 2026-10-05)
  - https://fortune.com/2026/07/09/ai-unicorn-mercor-acquires-deeptune-brendan-foody-investor-a16z-openai-anthropic/ (accessed 2026-10-05)
  - https://siliconangle.com/2026/07/09/mercor-buys-deeptune-build-training-environments-ai-agents/ (accessed 2026-10-05)
  - https://jobs.generalcatalyst.com/companies/mercor/jobs/89404711-member-of-technical-staff-deeptune (accessed 2026-10-05)
  - https://jobs.generalcatalyst.com/companies/mercor/jobs/89404694-strategic-projects-lead-deeptune (accessed 2026-10-05)
  - https://deeptune.com/ (accessed 2026-10-05)
  - https://jobs.ashbyhq.com/deeptune (accessed 2026-10-05)
  - https://fortune.com/2026/03/19/andreessen-horowitz-ai-startups-deeptune-series-a/ (accessed 2026-10-05)
  - https://deeptune.com/blog/series-a/ (accessed 2026-10-05)
  - https://a16z.com/announcement/investing-in-deeptune/ (accessed 2026-10-05)
  - https://www.linkedin.com/in/timlup/ (accessed 2026-10-05)
  - https://tracxn.com/d/companies/deeptune/__oi3tMd7lIKO3Yo_B0aq6tBQHcjZSfLMnyHdwbaOa0SE (accessed 2026-10-05)
  - https://dealroom.co/news/137987-mercor-buys-a16z-backed-deeptune-months-after-founder-invested-in-it/ (accessed 2026-10-05)
  - https://futurumgroup.com/insights/mercors-acquisition-of-deeptune-a-strategic-move-for-ai-training-environments/ (accessed 2026-10-05)
  - https://siliconangle.com/2026/03/19/deeptune-raises-43m-accelerate-ai-learning-virtual-training-gyms/ (accessed 2026-06-07)
  - https://www.linkedin.com/company/trydeeptune (accessed 2026-06-07)
  - https://x.com/a16z/status/2034694854123692462 (accessed 2026-06-07)

## General Reasoning (not ranked, incumbent / infrastructure / open-source)
- slug: general-reasoning
- segment: Open-source & open environments
- website: https://www.gr.inc/
- focus_areas: Coding, Long-Horizon
- positioning: General Reasoning is a London AI research lab (legal entity General Reasoning, Inc., US) working on models that operate over much longer time horizons. It builds OpenReward, a platform for serving RL environments (the vendor states 380+), and the open Open Reward Standard. It also ships open tooling (Firehorse, inspect-openreward, Harbor support), the KellyBench long-horizon benchmark and BackSearch, a paid point-in-time search API. It says its own language models launch in Q4 2026.
- best_fit: Teams wanting open, portable RL environments (including Harbor tasks) and long-horizon agent benchmarks they can self-host or call via API, plus leak-free point-in-time search for forecasting and backtesting evals.
- overall_confidence: medium
- founded_year: 2025 [confirmed], source: https://uk.linkedin.com/company/general-reasoning (accessed 2026-06-07); https://www.gr.inc/careers (Chapter I Founded 2025); SEC Form D filed 2025-07-11 https://www.formds.com/issuers/general-reasoning-inc
- status: active [confirmed], source: https://www.gr.inc/careers (accessed 2026-06-07); actively hiring, recent product launches
- hq_location: London, United Kingdom (Shoreditch), operating research hub; legal entity General Reasoning, Inc. registered in San Francisco/US per SEC Form D [confirmed], source: https://www.gr.inc/careers (accessed 2026-06-07); https://uk.linkedin.com/company/general-reasoning; https://www.formds.com/issuers/general-reasoning-inc
- other_locations: unknown [reported], source: https://www.gr.inc/careers (accessed 2026-06-07); only London 'Chapter I' currently exists, additional chapters 'coming soon'
- distributed_remote: no [estimated], source: https://www.gr.inc/careers (accessed 2026-06-07); all roles tied to London chapter, geographic-chapter model emphasizes in-person community
- current_headcount: ~10 (LinkedIn public snippet; '11-50' band) [reported], source: https://uk.linkedin.com/company/general-reasoning (accessed 2026-06-07)
- headcount_band: 11-50 [reported], source: https://uk.linkedin.com/company/general-reasoning (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: 3 [confirmed], source: https://www.gr.inc/careers (accessed 2026-06-07); 3 Member of Technical Staff roles (Head of Rollouts, Infrastructure, Research Scientist)
- has_researchers: yes [confirmed], source: https://www.gr.inc/research (accessed 2026-06-07)
- researcher_count: ~6 named individuals on site (founders + research/eng staff); counted from research/team page [estimated], source: https://www.gr.inc/research (accessed 2026-06-07)
- researcher_backgrounds: Ross Taylor (co-founder/CEO) - former head of reasoning at Meta AI (Llama 2/Llama 3), research lead on Galactica; co-founded Papers with Code (acquired by Meta 2019); Chengxi Taylor (co-founder/President) - serial founder (4 tech companies), Oxford MBA, CFA; leads investment and strategic partnerships; KellyBench co-author; Kip Parker (co-founder) - KellyBench co-author; Form D director; Research staff named as KellyBench authors: Thomas Grady (also a Form D director), Iliyan Zarov, Henry Course [confirmed], source: https://ai.engineer/orgs/general-reasoning (accessed 2026-10-05)
- published_papers_or_benchmarks: KellyBench: long-horizon sequential decision-making benchmark (release 2026-04-09; arXiv 2604.27865, 2026-04-30) - https://arxiv.org/abs/2604.27865; OpenReward / Open Reward Standard - https://www.gr.inc/releases/introducing-openreward ; https://openrewardstandard.io/; Firehorse open harness library (Apache-2.0) - https://github.com/GeneralReasoning/firehorse; inspect-openreward (Inspect integration) - https://github.com/OpenRewardAI/inspect-openreward; BackSearch point-in-time search for leak-free forecasting evals - https://www.gr.inc/releases/introducing-backsearch; Galactica, Llama 2, Llama 3 (founders' prior work at Meta) [confirmed], source: https://arxiv.org/abs/2604.27865 (accessed 2026-10-05)
- total_raised: ~$10.9M (amount sold per SEC Form D 2025-07-11; corroborated by Crunchbase) [reported], source: https://www.formds.com/issuers/general-reasoning-inc (accessed 2026-06-07); Crunchbase profile
- last_round: Equity offering, $10,904,992 sold, filed 2025-07-11 (SEC Form D; stage not labeled, reported as ~2025 seed-stage by Crunchbase) [reported], source: https://www.formds.com/issuers/general-reasoning-inc (accessed 2026-06-07); Crunchbase lists $10.9M Aug 2025
- notable_investors: Entrepreneur First [reported], source: https://joinef.substack.com/p/portfolio-news-march-2026 (accessed 2026-10-05)
- valuation: unknown [unknown]
- revenue_signals: Published usage pricing for BackSearch ($10 per 1,000 searches, $2 per 1,000 fetches; only successful requests billed); no revenue figures disclosed [confirmed], source: https://www.gr.inc/releases/introducing-backsearch (accessed 2026-10-05)
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: unknown [unknown]
- what_they_sell: environments [confirmed], source: https://www.gr.inc/releases/introducing-openreward (accessed 2026-06-07)
- open_source: yes [confirmed], source: https://www.gr.inc/releases/introducing-openreward (accessed 2026-06-07); Open Reward Standard described as open-source specification, openrewardstandard.io
- license: Apache-2.0 (firehorse and sonic-moe-fork repos); Open Reward Standard and inspect-openreward described as open-source, specific license not stated [reported], source: https://github.com/GeneralReasoning (accessed 2026-10-05)
- deployment_model: managed-hosted / self-hosted / API [confirmed], source: https://www.gr.inc/releases/introducing-openreward (accessed 2026-06-07)
- maturity: Early commercial / beta (OpenReward launched as public beta with training as an early research preview; BackSearch sold pay-as-you-go) [estimated], source: https://www.gr.inc/releases/introducing-openreward (accessed 2026-10-05)
- notable_customers: Meta (self-claimed, frontier-lab tie); Google DeepMind (self-claimed, frontier-lab tie); OpenAI (self-claimed, frontier-lab tie); NVIDIA (self-claimed, frontier-lab tie); Qwen (Alibaba) (self-claimed, frontier-lab tie); Nebius (self-claimed); Sarvam (self-claimed); Eigent (self-claimed); EpochAI (self-claimed); Tencent AI Lab (self-claimed) [reported], source: https://openreward.ai/ (accessed 2026-10-05)
- founding_team:
  - Ross Taylor, Co-founder & CEO, Former head of the reasoning team at Meta AI (Llama 2, Llama 3) and research lead on Galactica. Co-founded the startup behind Papers with Code, which Meta acquired in 2019. Earlier worked at HM Treasury and on quantitative football modelling. Studied at Warwick and Cambridge.
  - Chengxi Taylor, Co-founder & President, Leads investment and strategic partnerships at GR; her site says she has managed partnerships with Meta and Google DeepMind. Founded four tech companies (3D printing, AR, AI) and SEEN, angel investor via SixND, Oxford MBA, CFA. KellyBench co-author. Likely the 'Chengxi Wang' listed as director/executive on the 2025 Form D (inference from the name match).
  - Kip Parker, Co-founder, Named as co-founder by Entrepreneur First and listed as co-founder on the AI Engineer org page. Appears as director 'Kip Gabriel Parker' on the 2025 SEC Form D. KellyBench co-author. No other public background found.
- recent_developments (last 6 months):
  - 2026-10 [product] Homepage announces a new generation of GR language models for Q4 2026, source: https://www.gr.inc/
  - 2026-07-31 [other] Ross and Chengxi Taylor's 'Scaling to Long Horizons' talk (AI Engineer World's Fair 2026) published, source: https://www.youtube.com/watch?v=2bvtay8wGYI
  - 2026-07-24 [product] BackSearch launched: point-in-time web search and fetch over a frozen archive, source: https://www.gr.inc/releases/introducing-backsearch
  - 2026-04-30 [research] KellyBench paper posted to arXiv (2604.27865), source: https://arxiv.org/abs/2604.27865
  - 2026-04-27 [open-source] inspect-openreward: OpenReward environments inside the UK AISI Inspect eval framework, source: https://www.gr.inc/releases/inspect-openreward
  - 2026-04-21 [open-source] Firehorse released: open library for sampling agent trajectories on OpenReward environments, source: https://www.gr.inc/releases/introducing-firehorse
  - 2026-04-21 [product] Native Harbor task support in OpenReward, source: https://www.gr.inc/releases/deploy-harbor-environments
  - 2026-04-10 [other] Financial Times and The Times cover KellyBench; TechSpot and Decrypt follow, source: https://www.gr.inc/news
  - 2026-04-09 [benchmark] KellyBench released: every frontier model evaluated loses money over a simulated Premier League season, source: https://www.gr.inc/releases/introducing-kellybench
- benchmarks_and_research:
  - KellyBench (benchmark), 2026-04-09, Every frontier model evaluated lost money over the season and many hit ruin. Current release-page leaderboard: Claude Opus 4.7 -3.7% mean ROI, then GPT-5.4 -7.9%. The paper abstract (2026-04-30) reports about -8% for the best model at that time., https://www.gr.inc/releases/introducing-kellybench
  - KellyBench environment and leaderboard (OpenReward) (leaderboard), 2026-04, https://openreward.ai/GeneralReasoning/KellyBench
  - Open Reward Standard (ORS) (paper), 2026-03-24, https://openrewardstandard.io/
  - OpenReward environment catalog (environment), 2026-03-24, https://openreward.ai/
- products:
  - OpenReward: Platform for hosting and serving RL environments as API endpoints for agent training and evaluation. Launched as a public beta, with training support as an early research preview. The vendor states 380+ environments. Docs at docs.openreward.ai; status page at status.openreward.ai., https://openreward.ai/
  - Open Reward Standard (ORS): Open-source HTTP protocol for agent-environment interaction. License not stated., https://openrewardstandard.io/
  - KellyBench: Long-horizon decision-making benchmark environment and leaderboard., https://openreward.ai/GeneralReasoning/KellyBench
  - Firehorse: Apache-2.0 library that samples trajectories from Claude Code, Codex, Gemini CLI and ReSum harnesses on OpenReward environments and writes JSONL logs., https://github.com/GeneralReasoning/firehorse
  - inspect-openreward: Open-source package that runs OpenReward environments inside the UK AISI Inspect eval framework., https://github.com/OpenRewardAI/inspect-openreward
  - BackSearch: Point-in-time web search and fetch API over a frozen archive (news, SEC filings, arXiv, Wikipedia, web) for leak-free forecasting and backtesting evals. Pay-as-you-go: $10 per 1,000 searches, $2 per 1,000 fetches., https://www.gr.inc/releases/introducing-backsearch
- notes_for_researchers:
  - KellyBench (arXiv 2604.27865) tests long-horizon, non-stationary decision-making; the environment and leaderboard are open on OpenReward. The paper abstract and the release-page leaderboard give different best-model numbers (about -8% vs -3.7%) because the leaderboard was updated later with newer models.
  - In ORS, agents act on environments only through tool calls; the protocol adds rewards, episode termination and task splits.
  - Research direction stated in the 2026 World's Fair talk: RL-trained context compaction, value critics against gradient variance and sparse rewards, and rollout infrastructure for trajectories lasting days or weeks.
  - GR says it will release its own long-horizon language models in Q4 2026. It has no public models or datasets on Hugging Face yet.
  - Public GitHub repos include env-swe-gym, env-r2e-gym, LAB-Bench, env-zebra and citecast, plus sonic-moe-fork (Apache-2.0).
- notes_for_program_managers:
  - No new funding disclosed. The only filing is the 2025-07-11 SEC Form D: $10,904,992 sold, 8 investors, no stage label. Entrepreneur First lists GR as a portfolio company.
  - Small London team led by Meta's former Llama 2/3 reasoning lead. Three open roles: Head of Rollouts, Infrastructure, Research Scientist.
  - Organizations shown as environment providers or partners on OpenReward and in the launch post (Meta, Google DeepMind, Qwen, Nebius, Sarvam, NVIDIA, OpenAI, Eigent, EpochAI, Tencent AI Lab) are vendor-shown contributors, not confirmed paying customers.
  - BackSearch has published pay-as-you-go pricing. OpenReward pricing is not published on its homepage.
  - A Q4 2026 model launch is planned. If you use OpenReward to evaluate competing models, check vendor neutrality.
- notes_for_data_ops:
  - Native Harbor task support: connect a GitHub repo and OpenReward builds and redeploys Docker images per task, rebuilding only what changed.
  - Integrations: Inspect (inspect-openreward), Firehorse (Claude Code, Codex, Gemini CLI, ReSum harnesses; JSONL logs), Tinker, Miles and Slime trainers (named in the launch post), and the OpenAI Responses API (BackSearch).
  - Deployment: managed hosting, self-hosted, or API.
  - Docs at docs.openreward.ai; public status page at status.openreward.ai.
  - No SOC 2, ISO certification or security/trust page found.
- sources:
  - https://www.gr.inc/ (accessed 2026-10-05)
  - https://www.gr.inc/news (accessed 2026-10-05)
  - https://www.gr.inc/careers (accessed 2026-10-05)
  - https://www.gr.inc/research (accessed 2026-10-05)
  - https://www.gr.inc/releases/introducing-openreward (accessed 2026-10-05)
  - https://www.gr.inc/releases/introducing-kellybench (accessed 2026-10-05)
  - https://arxiv.org/abs/2604.27865 (accessed 2026-10-05)
  - https://openreward.ai/GeneralReasoning/KellyBench (accessed 2026-10-05)
  - https://www.gr.inc/releases/introducing-firehorse (accessed 2026-10-05)
  - https://github.com/GeneralReasoning/firehorse (accessed 2026-10-05)
  - https://www.gr.inc/releases/deploy-harbor-environments (accessed 2026-10-05)
  - https://www.gr.inc/releases/inspect-openreward (accessed 2026-10-05)
  - https://github.com/OpenRewardAI/inspect-openreward (accessed 2026-10-05)
  - https://www.gr.inc/releases/introducing-backsearch (accessed 2026-10-05)
  - https://openreward.ai/ (accessed 2026-10-05)
  - https://openrewardstandard.io/ (accessed 2026-10-05)
  - https://github.com/GeneralReasoning (accessed 2026-10-05)
  - https://huggingface.co/GeneralReasoning (accessed 2026-10-05)
  - https://ai.engineer/orgs/general-reasoning (accessed 2026-10-05)
  - https://www.youtube.com/watch?v=2bvtay8wGYI (accessed 2026-10-05)
  - https://www.startuphub.ai/ai-news/ai-research/2026/general-reasoning-founders-on-scaling-ai-models-to-long-horizons (accessed 2026-10-05)
  - https://rossjtaylor.com/ (accessed 2026-10-05)
  - https://chengxitaylor.com/ (accessed 2026-10-05)
  - https://joinef.substack.com/p/portfolio-news-march-2026 (accessed 2026-10-05)
  - https://www.formds.com/issuers/general-reasoning-inc (accessed 2026-10-05)
  - https://decrypt.co/364406/ai-beat-sports-betting-market-top-8-models (accessed 2026-10-05)
  - https://www.ft.com/content/544cbd80-492e-4ee8-a8b4-66e447361651 (accessed 2026-10-05)
  - https://uk.linkedin.com/company/general-reasoning (accessed 2026-06-07)
  - https://www.linkedin.com/in/rosstaylor90/ (accessed 2026-06-07)
  - https://www.interconnects.ai/p/interviewing-ross-taylor-on-llm-reasoning (accessed 2026-06-07)

## E2B (not ranked, incumbent / infrastructure / open-source)
- slug: e2b
- segment: Adjacent: execution infrastructure
- website: https://e2b.dev/
- focus_areas: Coding, Computer Use
- positioning: E2B provides open-source Firecracker-microVM sandboxes for AI agents. It is offered as a hosted API, as BYOC, and since September 2026 as a self-hostable single-node package (E2B Embed). In 2026 it became a supported sandbox provider in the OpenAI Agents SDK and published integrations for Devin Outposts and Cursor Self-Hosted Machines. Vendor case studies show it used for RL rollouts (Paper Instruments) and large-scale evals (Arena).
- best_fit: Teams that need secure, isolated, snapshot-able runtimes for LLM-generated code, coding agents, massively parallel RL rollouts or eval harnesses, including deployments inside their own environment via BYOC or E2B Embed.
- overall_confidence: medium
- founded_year: 2023 [reported], source: https://tracxn.com/d/companies/e2b (accessed 2026-06-07); https://www.crunchbase.com/organization/e2b-1c91 (accessed 2026-06-07)
- status: active [confirmed], source: https://e2b.dev/ (accessed 2026-06-07); https://e2b.dev/blog/series-a (accessed 2026-06-07)
- hq_location: San Francisco, USA [confirmed], source: https://www.linkedin.com/company/e2b-dev (accessed 2026-06-07); https://e2b.dev/blog/series-a (accessed 2026-06-07)
- other_locations: Prague, Czech Republic [reported], source: https://www.linkedin.com/company/e2b-dev (accessed 2026-06-07)
- distributed_remote: yes [estimated], source: Team distributed across US (San Francisco) and Czech Republic (Prague) per LinkedIn; https://www.linkedin.com/company/e2b-dev (accessed 2026-06-07)
- current_headcount: 11-50 (LinkedIn company size band; ~37 employees listed on LinkedIn as of 2026-06-07) [reported], source: https://www.linkedin.com/company/e2b-dev (accessed 2026-06-07)
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/e2b-dev (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: unknown [unknown], source: https://e2b.dev/careers (accessed 2026-10-05)
- has_researchers: no [estimated], source: Team and open roles appear engineering/infra/DevRel/GTM focused, not research scientists; https://e2b.dev/careers (accessed 2026-06-07)
- researcher_count: unknown [unknown]
- researcher_backgrounds: unknown [unknown]
- published_papers_or_benchmarks: unknown [unknown]
- total_raised: ~$32M (official Series A blog states $32M total) [reported], source: https://e2b.dev/blog/series-a (accessed 2026-06-07) - $11.5M seed + $21M Series A = $32M; some third-party aggregators (Tracxn) report ~$35M+
- last_round: Series A, $21M, July 2025 (announced 2025-07-28) [confirmed], source: https://e2b.dev/blog/series-a (accessed 2026-06-07); https://www.prnewswire.com/news-releases/e2b-raises-a-21m-series-a-to-offer-cloud-for-ai-agents-to-fortune-100-302514540.html (accessed 2026-06-07)
- notable_investors: Insight Partners (Series A lead); Decibel (seed lead); Sunflower Capital; Kaya / Kaya VC; Scott Johnston (former Docker CEO, angel) [confirmed], source: https://e2b.dev/blog/series-a (accessed 2026-06-07); https://www.insightpartners.com/ideas/e2b-raises-a-21m-series-a-to-offer-cloud-for-ai-agents-to-fortune-100/ (accessed 2026-06-07)
- valuation: unknown [unknown]
- revenue_signals: unknown revenue. Self-claimed usage signals: 1B+ sandbox starts, 100k+ teams have run a sandbox, signups from 94 of the Fortune 100 (careers page, 2026-10); Arena case study cites up to 600k sandboxes/day. No revenue figure independently verified. [reported], source: https://e2b.dev/careers (accessed 2026-10-05)
- soc2: claimed-unverified (homepage states 'E2B Cloud is SOC 2 Type II compliant'; trust center also claims it) [reported], source: https://e2b.dev/ (accessed 2026-10-05)
- other_certifications: ISO 27001 (claimed-unverified); HIPAA (claimed-unverified); GDPR (claimed-unverified) [reported], source: https://trust.e2b.dev/ (accessed 2026-06-07) - claimed on vendor trust center only; not independently verified
- security_page: https://trust.e2b.dev/ [confirmed], source: https://trust.e2b.dev/ (accessed 2026-06-07)
- what_they_sell: infra [confirmed], source: https://e2b.dev/ (accessed 2026-06-07) - secure cloud sandboxes (Firecracker microVMs) for running AI-generated code and AI agents; RL training is one stated use case
- open_source: yes [confirmed], source: https://github.com/e2b-dev/E2B (accessed 2026-06-07)
- license: Apache-2.0 [confirmed], source: https://github.com/e2b-dev/E2B (accessed 2026-06-07)
- deployment_model: managed-hosted cloud API (US, EU, APAC regions), BYOC in customer AWS/GCP/Azure VPC, and self-hosted via open-source E2B Embed (Apache-2.0, single machine; Docker Compose, Terraform, Kubernetes); private cloud listed as in development [confirmed], source: https://e2b.dev/enterprise (accessed 2026-10-05)
- maturity: GA [confirmed], source: https://e2b.dev/ (accessed 2026-06-07) - usage-based pricing, published SDKs, enterprise customers
- notable_customers: Perplexity (self-claimed); Hugging Face (self-claimed); Groq (self-claimed); Manus (self-claimed); Lindy (self-claimed); Genspark (self-claimed); Anything (self-claimed); Arena (LMArena) (self-claimed); Paper Instruments (self-claimed); Rogo (self-claimed); StackAI (self-claimed) [reported], source: https://e2b.dev/customers (accessed 2026-10-05)
- founding_team:
  - Vasek (Vaclav) Mlejnsky, Co-founder & CEO, Czech founder. Studied at Charles University's Faculty of Mathematics and Physics (Matfyz), focusing on computer vision. Built iPhone games with Tomas Valenta in secondary school, and the two built several developer tools before E2B.
  - Tomas Valenta, Co-founder & CTO, Czech founder. Studied at Charles University's Faculty of Mathematics and Physics (Matfyz), focusing on computer vision. Built iPhone games and later developer tools with Mlejnsky before co-founding E2B.
- recent_developments (last 6 months):
  - 2026-10-01 [customer] Arena case study: up to 600,000 E2B sandboxes per day, source: https://e2b.dev/customers/arena.md
  - 2026-09-30 [open-source] E2B Embed: open-source (Apache-2.0) single-machine self-hosted E2B, source: https://e2b.dev/resources/introducing-e2b-embed
  - 2026-09-10 [product] Guide: build an agent workbench on OpenAI's Agents API with E2B, source: https://e2b.dev/resources/build-an-agent-workbench-on-openais-agents-api
  - 2026-09-07 [product] Workspaces, Cursor Self-Hosted Machines templates, and an NVIDIA NeMo Gym integration guide, source: https://docs.e2b.dev/changelog
  - 2026-09-03 [customer] Paper Instruments case study: thousands of concurrent RL rollouts on E2B, source: https://e2b.dev/customers/paperinstruments.md
  - 2026-08-24 [product] Bring Your Own Proxy (SOCKS5 egress) and sandbox list filters, source: https://docs.e2b.dev/changelog
  - 2026-08-17 [product] Workload identity tokens in the SDKs and a Rust-based Python HTTP transport, source: https://docs.e2b.dev/changelog
  - 2026-08-03 [product] Legacy access tokens disabled; Fedora, Alpine, Arch and NixOS template base images, source: https://docs.e2b.dev/changelog
  - 2026-07-21 [product] E2B publishes a guide to running Devin Outposts sessions on E2B microVMs, source: https://e2b.dev/resources/devin-outposts
  - 2026-07-21 [product] Sandbox forking and snapshot name filters released, source: https://docs.e2b.dev/changelog
  - 2026-06-12 [customer] Rogo case study: Claude Managed Agents for orchestration plus E2B for code execution, source: https://e2b.dev/customers/rogo.md
  - 2026-06-10 [partnership] E2B sandboxes available through Stripe Projects (developer preview partner), source: https://e2b.dev/resources/e2b-sandboxes-are-now-available-through-stripe-projects
  - 2026-06-08 [customer] StackAI case study: replaced its in-house sandbox infrastructure with E2B, source: https://e2b.dev/customers/stackai.md
  - 2026-04-30 [other] E2B says its sandboxes are not affected by the Copy Fail Linux kernel bug (CVE-2026-31431), source: https://e2b.dev/resources/not-affected-by-copy-fail-heres-why
  - 2026-04-15 [partnership] E2B becomes a supported sandbox provider in the OpenAI Agents SDK (Python), source: https://e2b.dev/resources/e2b-is-now-in-agents-sdk
- products:
  - E2B Cloud (Sandbox SDK / API): Managed Firecracker microVM sandboxes for AI agents. Regions: US, EU, APAC. Python/JS SDKs and CLI. The homepage lists 'parallel environments for reinforcement learning, evaluations, and benchmarks' as a use case., https://e2b.dev/
  - E2B BYOC: E2B runs the control plane and the data plane sits in the customer's VPC on AWS, GCP or Azure. Sandbox traffic does not pass through E2B Cloud., https://e2b.dev/enterprise
  - E2B Private Cloud (in development): Control plane and data plane both inside the customer boundary, for air-gapped or sovereign networks. Listed as in development and seeking design partners., https://e2b.dev/enterprise
  - E2B Embed: Apache-2.0, single-machine self-hosted package of the E2B stack (Docker Compose, Terraform on GCP/AWS, Kubernetes). No account or license key required. Exports OpenTelemetry., https://e2b.dev/resources/introducing-e2b-embed
  - E2B Runtime (open source): Go runtime behind every E2B stack: Cloud, Enterprise and self-hosted (~1.7k GitHub stars)., https://github.com/e2b-dev
  - Code Interpreter SDK: Python and JS/TS SDK for running AI-generated code (~2.4k stars)., https://github.com/e2b-dev/code-interpreter
  - E2B Desktop: Graphical virtual-desktop sandbox for computer-use agents (~1.5k stars)., https://github.com/e2b-dev/desktop
  - MCP in E2B Sandbox (with Docker MCP Catalog): Gives sandboxes access to 200+ MCP tools from Docker's MCP Catalog., https://e2b.dev/resources/docker-e2b-partner-to-introduce-mcp-support-in-e2b-sandbox
  - E2B for Research program: One-time $20,000 in usage credits plus Pro tier for first-time academic or nonprofit research teams. Eligible uses include agent evals, RL rollouts and code-generation benchmarks. Capacity is limited, and no launch date is published., https://e2b.dev/research.md
  - Fragments: Open-source Next.js template for AI-generated apps (~6.4k stars)., https://github.com/e2b-dev/fragments
  - Surf: Open-source computer-use agent demo on E2B Desktop (~0.9k stars)., https://github.com/e2b-dev/surf
- notes_for_researchers:
  - E2B has not authored any benchmarks or papers. It is infrastructure that other teams' evals and RL runs depend on: Hugging Face's Open R1 replication, Arena's WebDev, Code and Agent Arena evaluations, and Paper Instruments' RL rollouts. All of these come from vendor case studies.
  - Memory and filesystem snapshots and live-sandbox forking (2026-07-21; CLI fork 2026-09-21) let you pre-warm running servers and branch trajectories from an exact in-memory state.
  - An NVIDIA NeMo Gym integration guide was added in September 2026 (changelog).
  - The E2B for Research program offers academic and nonprofit teams a one-time $20k in credits for evals, RL rollouts and code-gen benchmarks. Acceptance is not guaranteed.
  - The core stack is open source under Apache-2.0: the E2B SDK repo (~14.2k stars), the Runtime (~1.7k) and E2B Embed for single-node self-hosting.
- notes_for_program_managers:
  - Frontier-lab ties are integrations, not disclosed contracts. E2B is a supported sandbox provider in the OpenAI Agents SDK (2026-04, Python only) and published an OpenAI Agents API workbench guide (2026-09). Rogo's vendor case study pairs E2B with Anthropic's Claude Managed Agents (2026-06).
  - Coding-agent ties are E2B-published integration guides and templates for Devin Outposts (2026-07) and Cursor Self-Hosted Machines (2026-09). Neither is confirmed by Cognition or Cursor.
  - Scale signals, all self-claimed: 1B+ sandbox starts, 100k+ teams, signups from 94 of the Fortune 100 (careers page), and Arena running up to 600k sandboxes a day.
  - No new funding round was found after the July 2025 $21M Series A led by Insight Partners. The official total is $32M.
- notes_for_data_ops:
  - Deployment options: managed cloud (US/EU/APAC), BYOC (AWS/GCP/Azure; data plane in the customer VPC), self-hosted E2B Embed (single machine), and private cloud (in development).
  - Breaking changes: legacy access tokens were disabled on 2026-08-01 and API-key validation moved server-side (2026-08-31). 'Team' was renamed to 'project' in the CLI and SDKs (2026-07-27). Workspaces arrived in September 2026, along with public prj_ project IDs (2026-09-28).
  - Security and network controls added in 2026: workload identity tokens, wildcard domain transform rules, SOCKS5 bring-your-own-proxy egress, and OpenTelemetry export from Embed.
  - The homepage states 'E2B Cloud is SOC 2 Type II compliant'. This is a vendor-only claim and has not been independently verified.
  - MCP is available through the Docker MCP Catalog partnership. The SDKs automatically retry HTTP 429 responses with exponential backoff (changelog 2026-09-14).
- sources:
  - https://e2b.dev/ (accessed 2026-10-05)
  - https://e2b.dev/enterprise (accessed 2026-10-05)
  - https://e2b.dev/careers (accessed 2026-10-05)
  - https://docs.e2b.dev/changelog (accessed 2026-10-05)
  - https://e2b.dev/resources/product.md (accessed 2026-10-05)
  - https://e2b.dev/resources/updates.md (accessed 2026-10-05)
  - https://e2b.dev/sitemap.md (accessed 2026-10-05)
  - https://e2b.dev/research.md (accessed 2026-10-05)
  - https://e2b.dev/resources/introducing-e2b-embed (accessed 2026-10-05)
  - https://e2b.dev/resources/devin-outposts (accessed 2026-10-05)
  - https://e2b.dev/resources/e2b-is-now-in-agents-sdk (accessed 2026-10-05)
  - https://e2b.dev/resources/e2b-sandboxes-are-now-available-through-stripe-projects (accessed 2026-10-05)
  - https://e2b.dev/resources/build-an-agent-workbench-on-openais-agents-api (accessed 2026-10-05)
  - https://e2b.dev/resources/not-affected-by-copy-fail-heres-why (accessed 2026-10-05)
  - https://e2b.dev/resources/docker-e2b-partner-to-introduce-mcp-support-in-e2b-sandbox (accessed 2026-10-05)
  - https://e2b.dev/customers (accessed 2026-10-05)
  - https://e2b.dev/customers/paperinstruments.md (accessed 2026-10-05)
  - https://e2b.dev/customers/arena.md (accessed 2026-10-05)
  - https://e2b.dev/customers/rogo.md (accessed 2026-10-05)
  - https://e2b.dev/customers/stackai.md (accessed 2026-10-05)
  - https://github.com/e2b-dev (accessed 2026-10-05)
  - https://e2b.dev/blog/series-a (accessed 2026-10-05)
  - https://cc.cz/cc25/e2b/ (accessed 2026-10-05)
  - https://cc.cz/na-stredni-vyvijeli-hry-pro-iphone-ted-zari-s-umelou-inteligenci-cesi-ziskali-za-dva-roky-700-milionu/ (accessed 2026-10-05)
  - https://api.github.com/users/mlejva (accessed 2026-10-05)
  - https://api.github.com/users/ValentaTomas (accessed 2026-10-05)
  - https://github.com/e2b-dev/E2B (accessed 2026-06-07)
  - https://trust.e2b.dev/ (accessed 2026-06-07)
  - https://trust.e2b.dev/controls (accessed 2026-06-07)
  - https://www.linkedin.com/company/e2b-dev (accessed 2026-06-07)
  - https://www.insightpartners.com/ideas/e2b-raises-a-21m-series-a-to-offer-cloud-for-ai-agents-to-fortune-100/ (accessed 2026-06-07)
  - https://www.prnewswire.com/news-releases/e2b-raises-a-21m-series-a-to-offer-cloud-for-ai-agents-to-fortune-100-302514540.html (accessed 2026-06-07)
  - https://venturebeat.com/ai/how-e2b-became-essential-to-88-of-fortune-100-companies-and-raised-21-million (accessed 2026-06-07)
  - https://tracxn.com/d/companies/e2b/__U7C82j6Wk3VH-rgW0n4LFnUqqq-LuBw6rnIcnLGz2yU/funding-and-investors (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/e2b-1c91 (accessed 2026-06-07)
  - https://www.glassdoor.com/Jobs/E2B-Jobs-E10750951.htm (accessed 2026-06-07)

## Runloop (not ranked, incumbent / infrastructure / open-source)
- slug: runloop
- segment: Adjacent: execution infrastructure
- website: https://runloop.ai/
- focus_areas: Coding
- positioning: Runloop sells cloud VM-isolated devboxes and sandboxes for AI agents. Around them it now offers agent lifecycle tooling: the Agent API, Axons and Broker event streams, Agent Gateway and MCP Hub for credential isolation, Benchmark Job Orchestration with W&B Weave, Reflex background agents, and VPC deployment. It is execution and eval infrastructure for agent builders and training workloads, not an RL-data or environments vendor.
- best_fit: Teams building or training coding agents that need scalable, sandboxed execution environments to run agents, evals (SWE-Bench, etc.), and RFT/SFT loops.
- overall_confidence: medium
- founded_year: 2024 [reported], source: https://www.crunchbase.com/organization/runloop-ai (founded 2024); https://venturebeat.com/ai/runloop-lands-7m-to-power-ai-coding-agents-with-cloud-based-devboxes (accessed 2026-06-07)
- status: active [confirmed], source: https://runloop.ai/ (accessed 2026-06-07)
- hq_location: San Francisco, CA, USA [confirmed], source: https://www.prnewswire.com/news-releases/runloop-raises-7m-seed-round-to-bring-enterprise-grade-infrastructure-to-ai-coding-agents-302516898.html (accessed 2026-06-07)
- other_locations: unknown [unknown]
- distributed_remote: Appears in-office (careers page features an employee testimonial about in-person office culture; no explicit remote policy or office location stated) [estimated], source: https://runloop.ai/careers (accessed 2026-10-05)
- current_headcount: ~16-17 employees (Tracxn 16 as of 2026-01-31; PitchBook 17; press cited 12 in Jul 2025) [reported], source: https://tracxn.com/d/companies/runloopai (accessed 2026-06-07; 16 as of 2026-01-31); https://pitchbook.com/profiles/company/629631-73 (17); press release 12 (Jul 2025). Note: startupintros aggregator showed an erroneous 1,001-5,000 LinkedIn band, disregarded.
- headcount_band: 11-50 [reported], source: https://tracxn.com/d/companies/runloopai (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: unknown [unknown]
- has_researchers: no [estimated], source: https://runloop.ai/about (accessed 2026-06-07; team described as infrastructure engineers from Google, Stripe, Vercel, AWS, Meta, Scale AI; no research/PhD scientists noted)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Founder/CEO Jonathan Wall: ex-Google (co-founded Google Wallet), co-founded Index (acquired by Stripe); deeper Google File System / Founders Award detail is vendor-stated; Senior engineers are alumni of Google, Stripe, Vercel, AWS, Meta, and Scale AI (per press) [reported], source: https://runloop.ai/about (accessed 2026-06-07); https://venturebeat.com/ai/runloop-lands-7m-to-power-ai-coding-agents-with-cloud-based-devboxes (accessed 2026-06-07)
- published_papers_or_benchmarks: No Runloop-authored papers/benchmarks found. Platform supports running third-party academic coding benchmarks: SWE-Bench, R2E-Gym, SWE-Smith, Multi-SWE (these are external benchmarks, not Runloop publications) [reported], source: https://runloop.ai/ (accessed 2026-06-07)
- total_raised: $7M [confirmed], source: https://www.prnewswire.com/news-releases/runloop-raises-7m-seed-round-to-bring-enterprise-grade-infrastructure-to-ai-coding-agents-302516898.html (accessed 2026-06-07); corroborated by Crunchbase ($7M, 1 round)
- last_round: Seed, $7M, 2025-07-30 [confirmed], source: https://www.crunchbase.com/funding_round/runloop-ai-seed--bd4cd02b (accessed 2026-06-07); https://www.prnewswire.com/news-releases/runloop-raises-7m-seed-round-to-bring-enterprise-grade-infrastructure-to-ai-coding-agents-302516898.html
- notable_investors: The General Partnership (lead); Blank Ventures; Exponent Founders Capital; Nascent; Roneil Rumburg (angel) [reported], source: Official press names only The General Partnership (lead) + Blank Ventures: https://www.prnewswire.com/news-releases/runloop-raises-7m-seed-round-to-bring-enterprise-grade-infrastructure-to-ai-coding-agents-302516898.html. Additional participants (Exponent Founders Capital, Nascent, Roneil Rumburg; AWS Startups per one source) listed only on aggregators: https://www.crunchbase.com/organization/runloop-ai, https://startupintros.com/orgs/runloop-ai (accessed 2026-06-07)
- valuation: unknown [unknown], source: Not publicly disclosed (Crunchbase financials obfuscated as of 2026-06-07)
- revenue_signals: Vendor-claimed: >200% customer growth and >100% revenue growth since March 2025; 'a few dozen customers' [reported], source: https://venturebeat.com/ai/runloop-lands-7m-to-power-ai-coding-agents-with-cloud-based-devboxes (accessed 2026-06-07; figures self-reported by vendor)
- soc2: claimed-unverified [reported], source: Vendor states 'SOC2 certified' on homepage and press (https://runloop.ai/, https://www.prnewswire.com/news-releases/runloop-unveils-enterprise-grade-sandboxes-for-ai-coding-agents-302460834.html, accessed 2026-06-07) but no report type, audit registry entry, or independent attestation located
- other_certifications: HIPAA (vendor-claimed; HIPAA logo shown on homepage); GDPR (vendor-claimed in 2025 press; not mentioned on current homepage) [reported], source: https://runloop.ai/ (accessed 2026-10-05)
- security_page: https://trust.oneleet.com/runloop [confirmed], source: https://runloop.ai/ (accessed 2026-10-05)
- what_they_sell: infra [confirmed], source: https://runloop.ai/ (accessed 2026-06-07); https://docs.runloop.ai/docs/devboxes/overview
- open_source: yes (SDKs/CLI only; core platform proprietary) [confirmed], source: https://github.com/runloopai (accessed 2026-06-07; MIT-licensed TS and Python SDKs, CLI)
- license: MIT (SDKs/CLI; core platform is proprietary/closed-source) [confirmed], source: https://github.com/runloopai (accessed 2026-06-07)
- deployment_model: managed-hosted / API (VPC deployment option claimed for enterprise) [confirmed], source: https://docs.runloop.ai/docs/devboxes/overview (accessed 2026-06-07); https://runloop.ai/ (VPC option)
- maturity: GA [confirmed], source: https://runloop.ai/blog/accrual-moves-coding-agents-to-the-cloud-with-runloop (accessed 2026-10-05)
- notable_customers: Trajectory (vendor case study 2026-05-27; 10,000+ concurrent devboxes) (self-claimed); ION (vendor case study 2026-06-11; migrated in 3 days, founder quoted) (self-claimed); Accrual (vendor case study 2026-08-21 and homepage feature; ARC platform, co-founder/CTO quoted) (self-claimed); Detail.dev (CEO quote in 2025 vendor press release) (self-claimed); Unnamed 'major model laboratories' (2025 vendor claim; no lab named) (self-claimed, frontier-lab tie) [reported], source: https://runloop.ai/blog/accrual-moves-coding-agents-to-the-cloud-with-runloop (accessed 2026-10-05)
- founding_team:
  - Jonathan Wall, Founder & CEO, Ex-Google: technical lead on the Google File System team and recipient of the Google Founders Award (both vendor-stated), and co-founder of Google Wallet (2011). Co-founder and CTO of Index (2015), which Stripe acquired. Founded Runloop in 2023 per the about page.
- recent_developments (last 6 months):
  - 2026-08-21 [customer] Case study: Accrual built its ARC remote coding-agent platform on Runloop in 7 days, source: https://runloop.ai/blog/accrual-moves-coding-agents-to-the-cloud-with-runloop
  - 2026-07-16 [partnership] Kernel Managed Auth integration pattern for agent purchases, source: https://runloop.ai/blog/trust-your-agent-with-a-credit-card-with-runloop-and-kernel
  - 2026-06-11 [customer] Case study: ION migrated to Runloop in three days, source: https://runloop.ai/blog/ion-case-study
  - 2026-05-27 [customer] Case study: Trajectory ran 10,000+ concurrent devboxes, source: https://runloop.ai/blog/runloop-trajectory-launch-partner-announcement
  - 2026-05-20 [product] Transparent Proxy for Runloop Tunnels, source: https://runloop.ai/blog/release-transparent-proxy-for-tunnels
  - 2026-05-05 [open-source] Remote Agents SDK for Axons released (open source, MIT), source: https://runloop.ai/blog/axons-remote-agent-sdk-launch
  - 2026-04-30 [partnership] Runloop joins Stripe Projects, demoed at the Stripe Sessions 2026 keynote, source: https://www.prnewswire.com/news-releases/runloop-joins-stripe-projects-providing-essential-infrastructure-for-the-agentic-commerce-era-302759081.html
  - 2026-04-27 [product] Benchmark Job Orchestration launched with Weights & Biases Weave integration, source: https://runloop.ai/media/runloop-launches-ai-benchmark-orchestration-platform-with-weights-biases-integration
  - 2026-04-27 [product] Axons: distributed event streams for agents, source: https://runloop.ai/blog/axon-release-part-1
  - 2026-04-15 [partnership] Runloop becomes a sandbox provider for the OpenAI Agents SDK, source: https://runloop.ai/blog/runloop-openai-agents-sdk-release
  - 2026-04-13 [product] Runloop Agent API released, source: https://runloop.ai/blog/agent-api-release
  - 2026-04-02 [product] Tunnels add Wake on HTTP to spin up agents on demand, source: https://runloop.ai/blog/tunnels-now-support-wake-on-http-this-means-that-y
- products:
  - Devboxes / Sandboxes: VM-isolated Linux sandboxes for running agents and agent tools. Homepage claims (vendor-stated): 10k+ parallel sandboxes, 10GB images starting in under 2 seconds., https://docs.runloop.ai/docs/devboxes/overview
  - Blueprints and Snapshots: Reusable devbox environment templates, plus save/suspend/resume of devbox state., https://docs.runloop.ai/docs/overview/what-is-runloop
  - Agent API: Defines, versions and deploys agents onto Devboxes from Git, npm, PyPI or object storage., https://runloop.ai/blog/agent-api-release
  - Axons and Broker: Persistent event streams for agent sessions with durable history and structured state. Broker connects Axons to agents running in devboxes, including ACP agents and Claude Code., https://docs.runloop.ai/docs/axons/overview
  - Benchmark Job Orchestration: Runs agent benchmark scenarios in parallel across models, with baselines, regression detection and trace export to W&B Weave., https://runloop.ai/media/runloop-launches-ai-benchmark-orchestration-platform-with-weights-biases-integration
  - Public Benchmarks: On-demand runs of standard third-party coding-agent benchmarks (SWE-Bench Verified named). These benchmarks are not Runloop-authored., https://runloop.ai/media/runloop-launches-public-benchmarks-industry-standard-testing-for-ai-coding-agents
  - Custom Benchmarks (with Fermatix.ai): Private, organization-specific benchmarks for evaluating agents on proprietary code and business logic., https://runloop.ai/media/runloop-ai-and-fermatix-ai-partner-to-introduce-custom-benchmarks-for-ai-agents
  - Agent Gateway: Proxies LLM API calls and injects credentials on the server side, so agents never see raw API keys., https://docs.runloop.ai/docs/devboxes/agent-gateways
  - MCP Hub: A single endpoint to MCP tool servers, with per-server opaque tokens and tool-level access control. Real credentials stay on the server., https://docs.runloop.ai/docs/devboxes/mcp-hub
  - Tunnels (Wake on HTTP, Transparent Proxy): Exposes devbox services, wakes suspended devboxes on incoming HTTP requests, and supports customer-branded domains., https://runloop.ai/blog/tunnels-now-support-wake-on-http-this-means-that-y
  - Reflex: Runloop's platform for background coding agents, listed on the homepage. Its open-source SDKs are in reflex-os (MIT): API client, React chat UI kit and CLI. Launch date not found., https://github.com/runloopai/reflex-os
  - Deploy to VPC: Deploys Runloop infrastructure into the customer's own AWS, GCP or Azure account (vendor-stated)., https://runloop.ai/media/runloop-ai-launches-deploy-to-vpc-with-full-enterprise-control
  - SDKs, CLI and Remote Agents SDK (open source, MIT): Python and TypeScript SDKs, rl-cli, and remote-agents-sdk for Axon-backed ACP and Claude Code agents. All MIT-licensed., https://github.com/runloopai
- notes_for_researchers:
  - We found no benchmarks, datasets or papers authored by Runloop. It is execution and eval infrastructure: it hosts third-party benchmarks (Public Benchmarks, 2025-06, naming SWE-Bench Verified) and private Custom Benchmarks (with Fermatix.ai, 2025-10).
  - Benchmark Job Orchestration (2026-04-27) runs agent evals in parallel in codebase, terminal and browser environments and sends traces to W&B Weave. The Agent API (2026-04-13) lets you pin agent versions while keeping environments fixed.
  - The homepage lists 'RL Fine Tuning' as a use case. The Trajectory case study (2026-05-27) documents a continual-learning workload at 10,000+ concurrent devboxes, including transport-layer lessons (HTTP/2 multiplexing in SDK 1.20.2).
- notes_for_program_managers:
  - The last known funding is still the $7M seed (2025-07-30). No newer round was found, but no full press or funding search was possible in this run, so treat this as unconfirmed.
  - Frontier-lab ties are integrations, not confirmed contracts. OpenAI's Agents SDK ships a 'runloop' sandbox extra alongside E2B, Modal, Daytona and Vercel, and the Remote Agents SDK supports Claude Code. The 2025 claim about unnamed 'model labs' is still unverified.
  - Named customer case studies, all vendor-published: Trajectory, ION, Accrual. Partnerships and integrations: Weights & Biases (eval traces), Stripe Projects (demoed at Stripe Sessions 2026), and Kernel (tutorial-level integration).
- notes_for_data_ops:
  - Integrations: REST API, MIT-licensed Python and TypeScript SDKs, rl-cli, OpenAI Agents SDK provider, MCP Hub, ACP and Claude Code remote agents, W&B Weave.
  - Security: SOC 2 is claimed on the site and in the docs, and the homepage shows a HIPAA logo. A trust center exists at trust.oneleet.com/runloop, but its contents could not be read, so no report type or audit date was verified. Agent Gateway and MCP Hub keep credentials out of agents, and egress network policies exist.
  - Deployment: managed cloud, or VPC in AWS, GCP or Azure (vendor-stated). Axons provide durable session history for audit and replay.
- sources:
  - https://runloop.ai/ (accessed 2026-10-05)
  - https://runloop.ai/about (accessed 2026-10-05)
  - https://cdn.prod.website-files.com/68f6566c0817e720021136f4/6967dde8ec6fc94795e95cda_Jonathan-profile.png (accessed 2026-10-05)
  - https://runloop.ai/blog (accessed 2026-10-05)
  - https://runloop.ai/runloop-in-the-media (accessed 2026-10-05)
  - https://www.prnewswire.com/news-releases/runloop-joins-stripe-projects-providing-essential-infrastructure-for-the-agentic-commerce-era-302759081.html (accessed 2026-10-05)
  - https://runloop.ai/media/runloop-joins-stripe-projects-providing-essential-infrastructure-for-the-agentic-commerce-era (accessed 2026-10-05)
  - https://runloop.ai/media/runloop-launches-ai-benchmark-orchestration-platform-with-weights-biases-integration (accessed 2026-10-05)
  - https://ciofirst.com/runloop-unveils-benchmark-orchestration-platform-to-strengthen-trust-in-enterprise-ai-agents/ (accessed 2026-10-05)
  - https://runloop.ai/blog/tunnels-now-support-wake-on-http-this-means-that-y (accessed 2026-10-05)
  - https://runloop.ai/blog/agent-api-release (accessed 2026-10-05)
  - https://runloop.ai/blog/runloop-openai-agents-sdk-release (accessed 2026-10-05)
  - https://raw.githubusercontent.com/openai/openai-agents-python/main/pyproject.toml (accessed 2026-10-05)
  - https://docs.runloop.ai/docs/tutorials/openai-agentssdk-runloop (accessed 2026-10-05)
  - https://runloop.ai/blog/axon-release-part-1 (accessed 2026-10-05)
  - https://runloop.ai/blog/axons-remote-agent-sdk-launch (accessed 2026-10-05)
  - https://github.com/runloopai/remote-agents-sdk (accessed 2026-10-05)
  - https://runloop.ai/blog/release-transparent-proxy-for-tunnels (accessed 2026-10-05)
  - https://runloop.ai/blog/runloop-trajectory-launch-partner-announcement (accessed 2026-10-05)
  - https://runloop.ai/blog/ion-case-study (accessed 2026-10-05)
  - https://runloop.ai/blog/trust-your-agent-with-a-credit-card-with-runloop-and-kernel (accessed 2026-10-05)
  - https://runloop.ai/blog/accrual-moves-coding-agents-to-the-cloud-with-runloop (accessed 2026-10-05)
  - https://runloop.ai/media/runloop-launches-public-benchmarks-industry-standard-testing-for-ai-coding-agents (accessed 2026-10-05)
  - https://runloop.ai/media/runloop-ai-and-fermatix-ai-partner-to-introduce-custom-benchmarks-for-ai-agents (accessed 2026-10-05)
  - https://docs.runloop.ai/docs/overview/what-is-runloop (accessed 2026-10-05)
  - https://docs.runloop.ai/docs/devboxes/overview (accessed 2026-10-05)
  - https://docs.runloop.ai/docs/axons/overview (accessed 2026-10-05)
  - https://docs.runloop.ai/docs/devboxes/agent-gateways (accessed 2026-10-05)
  - https://docs.runloop.ai/docs/devboxes/mcp-hub (accessed 2026-10-05)
  - https://docs.runloop.ai/docs/overview/release-notes (accessed 2026-10-05)
  - https://github.com/runloopai (accessed 2026-10-05)
  - https://github.com/runloopai/reflex-os (accessed 2026-10-05)
  - https://trust.oneleet.com/runloop (accessed 2026-10-05)
  - https://runloop.ai/careers (accessed 2026-10-05)
  - https://www.prnewswire.com/news-releases/runloop-raises-7m-seed-round-to-bring-enterprise-grade-infrastructure-to-ai-coding-agents-302516898.html (accessed 2026-06-07)
  - https://www.prnewswire.com/news-releases/runloop-unveils-enterprise-grade-sandboxes-for-ai-coding-agents-302460834.html (accessed 2026-06-07)
  - https://venturebeat.com/infrastructure/runloop-lands-7m-to-power-ai-coding-agents-with-cloud-based-devboxes (accessed 2026-06-07)
  - https://www.crunchbase.com/funding_round/runloop-ai-seed--bd4cd02b (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/runloop-ai (accessed 2026-06-07)
  - https://tracxn.com/d/companies/runloopai (accessed 2026-06-07)
  - https://pitchbook.com/profiles/company/629631-73 (accessed 2026-06-07)

## Mechanize (not ranked, incumbent / infrastructure / open-source)
- slug: mechanize
- segment: Commercial vendors
- website: https://www.mechanize.work/
- focus_areas: Coding, Private Codebases
- positioning: Mechanize is a San Francisco vendor founded in April 2025 by ex-Epoch AI researchers. It builds a small number of high-fidelity, long-horizon RL environments and evals for frontier coding agents, and publishes the open-source GBA Eval benchmark. It raised $9.1M at a $500M post-money valuation in April 2026. In September 2026 Business Insider reported that Google had completed a license-and-hire deal: co-founder and CEO Tamay Besiroglu and more than a dozen staff joined Google DeepMind, and talks had been reported at more than $1.5B. Mechanize continues under CEO Guive Assadi.
- best_fit: A frontier lab that wants a small set of deep, hard software-engineering RL environments and evals. Buyers should first confirm how much environment-building capacity remains after the September 2026 departures to Google DeepMind, and what Google's non-exclusive license covers.
- overall_confidence: medium
- founded_year: 2025 [confirmed], source: https://www.mechanize.work/announcing-mechanize-inc/ (founded April 17, 2025; accessed 2026-06-07)
- status: acqui-hired [confirmed], source: https://www.businessinsider.com/google-completes-deal-for-ai-agents-startup-mechanize-2026-9 (accessed 2026-10-05)
- hq_location: San Francisco, USA [confirmed], source: https://www.linkedin.com/company/mechanize-inc (SF HQ) and https://tracxn.com/d/companies/mechanize (San Francisco; accessed 2026-06-07)
- other_locations: unknown [unknown]
- distributed_remote: no [reported], source: https://www.mechanize.work/apply (roles listed in-person San Francisco; accessed 2026-06-07)
- current_headcount: About 35 (Aug 2026, per TNW relaying Business Insider), before more than a dozen staff moved to Google DeepMind (Sept 2026). Current size unknown. [reported], source: https://thenextweb.com/news/google-may-spend-1-5bn-on-a-35-person-startup-to-catch-up-on-ai-coding (accessed 2026-10-05)
- headcount_band: 11-50 [reported], source: https://www.linkedin.com/company/mechanize-inc (11-50 band) and Tracxn 38 employees (accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: 3: Software Engineer ($300K–$400K), Research Engineer, Alignment ($300K–$400K) and Recruiter ($200K); all in person in San Francisco [confirmed], source: https://www.mechanize.work/apply (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: Founders are ex-Epoch AI researchers; hiring Research Engineer (Alignment) per https://www.mechanize.work/apply (accessed 2026-06-07)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Founders Tamay Besiroglu (Epoch AI co-founder), Matthew Barnett and Ege Erdil are Epoch AI alumni; Besiroglu and more than a dozen staff moved to Google DeepMind (Sept 2026, reported). Barnett's and Erdil's status after the deal is not reported; Erdil was active at Mechanize as of June 2026. [reported], source: https://www.businessinsider.com/google-completes-deal-for-ai-agents-startup-mechanize-2026-9 (accessed 2026-10-05)
- published_papers_or_benchmarks: GBA Eval: 24-hour Game Boy Advance emulator benchmark for coding agents, open source, with a public leaderboard (23 entries by July 2026; top score Claude Opus 5 at 0.796) (gbaeval.com; github.com/mechanize-work/gba-eval); gba-eval-attempts and per-model gba-eval-attempt-* repos: published agent codebases with checkpoint histories [confirmed], source: https://github.com/mechanize-work/gba-eval (accessed 2026-10-05)
- total_raised: $9.1M disclosed (April 24, 2026) plus an earlier undisclosed angel round (April 2025) [reported], source: https://www.mechanize.work/press-releases/ ($9.1M, April 24, 2026); https://tracxn.com/d/companies/mechanize (angel round April 28, 2025, amount undisclosed) (accessed 2026-06-07)
- last_round: $9.1M, 2026-04-24, at $500M post-money, from Marco Mascorro, Adam D'Angelo and Devendra Chaplot. Mechanize states no stage; press calls it a seed. [confirmed], source: https://www.mechanize.work/mechanize-raises-9-1m (accessed 2026-10-05)
- notable_investors: Nat Friedman; Daniel Gross; Patrick Collison; Adam D'Angelo; Marco Mascorro; Dwarkesh Patel; Sholto Douglas; Devendra Chaplot; Alex Atallah; Marcus Abramovitch [confirmed], source: https://www.mechanize.work/ (accessed 2026-10-05)
- valuation: $500M post-money (April 2026 round); a Google license-and-hire deal was later reported in talks at more than $1.5B (Aug 2026), with final terms undisclosed [confirmed], source: https://www.mechanize.work/mechanize-raises-9-1m (accessed 2026-10-05)
- revenue_signals: unknown [unknown]
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: unknown [unknown]
- what_they_sell: environments [confirmed], source: https://www.mechanize.work/ ('Environments and evals for frontier coding agents'; accessed 2026-06-07)
- open_source: partial: the GBA Eval benchmark harness and attempt repos are public on GitHub; the commercial environments are proprietary [confirmed], source: https://github.com/mechanize-work/gba-eval (accessed 2026-10-05)
- license: GBA Eval: MIT (spec, corpus, candidates, harness, quickstart); GPL-3.0 for the Mesen2 reference components; test ROMs keep their upstream licenses [confirmed], source: https://github.com/mechanize-work/gba-eval (accessed 2026-10-05)
- deployment_model: unknown [unknown]
- maturity: unknown [unknown]
- notable_customers: Anthropic (self-claimed, frontier-lab tie) [reported], source: https://techcrunch.com/2025/09/21/silicon-valley-bets-big-on-environments-to-train-ai-agents/ ('Mechanize has already been working with Anthropic on RL environments, two sources familiar with the matter told TechCrunch'; both companies declined to comment) (accessed 2026-06-07). Anonymous-sourced, unconfirmed by either party, not independently verified.
- status_change: Acqui-hired by Google (Google DeepMind) (2026-09) This is a reverse acqui-hire: Google hired staff and took a technology license, but did not buy the company outright. On 2026-08-05 Business Insider (Katie Roof) reported that Google was in talks for a deal worth more than $1.5B. The deal would hire some Mechanize staff and include a non-exclusive license to its technology. Four people familiar with the talks were the source. Google and Mechanize both declined to comment. On 2026-09-11 Business Insider (Charles Rollet) reported that Google had completed the deal, based on public LinkedIn postings. Co-founder and former CEO Tamay Besiroglu is now a research scientist at Google DeepMind. More than a dozen former Mechanize employees now work at Google, mostly on midtraining. Former chief of staff Guive Assadi now describes himself as Mechanize's CEO on LinkedIn. Final terms were not disclosed; the $1.5B+ figure comes from the August talks report. Neither Google nor Mechanize has made a primary announcement. Mechanize still appears to operate on its own: its website is live and lists 3 open roles. Confidence: reported., source: https://www.businessinsider.com/google-completes-deal-for-ai-agents-startup-mechanize-2026-9
- founding_team:
  - Tamay Besiroglu, Co-founder; former CEO, Co-founded Epoch AI, a research institute that tracks AI progress. He left Mechanize for Google DeepMind in the 2026 license-and-hire deal (reported).
  - Matthew Barnett, Co-founder, Former Epoch AI researcher and co-author of Mechanize's founding announcement. His role after the Google deal is not publicly reported.
  - Ege Erdil, Co-founder, Former Epoch AI researcher. He appeared on Mechanize's first podcast episode in June 2026. His role after the Google deal is not publicly reported.
  - Guive Assadi, CEO, Mechanize's former chief of staff. Business Insider reports him as CEO after Besiroglu left for Google DeepMind. No further background was verified.
- funding_history:
  - 2026-04-24 undisclosed $9.1M, valuation $500M, source: https://www.mechanize.work/mechanize-raises-9-1m
- recent_developments (last 6 months):
  - 2026-09-11 [acquisition] Business Insider: Google completes the Mechanize talent deal; Besiroglu and more than a dozen staff join Google DeepMind, source: https://www.businessinsider.com/google-completes-deal-for-ai-agents-startup-mechanize-2026-9
  - 2026-09-11 [hire] Former chief of staff Guive Assadi now Mechanize CEO, source: https://www.businessinsider.com/google-completes-deal-for-ai-agents-startup-mechanize-2026-9
  - 2026-08-05 [acquisition] Business Insider: Google in talks for a deal worth more than $1.5B for Mechanize's talent and technology, source: https://www.businessinsider.com/google-mechanize-deal-talent-tech-ai-coding-2026-8
  - 2026-07 [benchmark] GBA Eval leaderboard grows to 23 entries; Claude Opus 5 leads at 0.796, source: https://gbaeval.com/results/leaderboard.json
  - 2026-06-16 [other] Mechanize launches a podcast on evals and RL environments, source: https://www.mechanize.work/podcast/how-bad-data-teaches-models-to-write-terrible-code/
  - 2026-05 [benchmark] Mechanize launches GBA Eval, an open benchmark for 24-hour emulator builds by coding agents, source: https://github.com/mechanize-work/gba-eval
  - 2026-04-24 [funding] Mechanize raises $9.1M at a $500M post-money valuation, source: https://www.mechanize.work/mechanize-raises-9-1m
- benchmarks_and_research:
  - GBA Eval leaderboard (leaderboard), 2026-07, Claude Opus 5 leads at 0.796 overall (as of 2026-07-26)., https://gbaeval.com/leaderboard
  - GBA Eval (benchmark), 2026-05, Top overall score: 0.796 for Claude Opus 5 (graded 2026-07-26), ahead of Claude Fable 5 at 0.745 and Claude Opus 4.8 at 0.709; GPT-5.5 scored 0.532. Source: the official leaderboard data at gbaeval.com/results/leaderboard.json., https://gbaeval.com/
  - gba-eval-attempts (dataset), 2026-05, https://github.com/mechanize-work/gba-eval-attempts
- products:
  - RL environments and evals for frontier coding agents: Long-horizon software-engineering tasks with automated graders, bundled into environments that frontier AI labs use for RL training or capability evals. Official 'working here' page: one engineer owns each task end to end, and a typical task takes about a week from idea to submission. Most of that effort goes into making the grader robust, fair and deterministic., https://www.mechanize.work/
  - GBA Eval: Public, open-source benchmark and leaderboard in which coding agents get 24 hours to build a Game Boy Advance emulator., https://gbaeval.com/
- notes_for_researchers:
  - GBA Eval (github.com/mechanize-work/gba-eval) is an open-source, long-horizon SWE eval: agents get 24 hours to write a GBA emulator, graded against Mesen2 on replays (60%), procedural tests (20%) and audio (20%). The harness is MIT-licensed; the Mesen2 parts are GPL-3.0.
  - The official leaderboard data (gbaeval.com/results/leaderboard.json) includes per-checkpoint score trajectories, token counts and wall-clock hours for each model. Claude Opus 5 leads at 0.796 as of 2026-07-26. The newest grading in the data is from 2026-07-26.
  - Per-model attempt repos are public, and the GBA Eval blog documents Grok Build 0.1 hard-coding example ROMs to fake CPU tests. Both are useful for studying reward hacking and grader robustness.
  - Mechanize's essays (May–Nov 2025: 'Cheap RL tasks will waste compute', 'Sweatshop data is over', 'The upcoming GPT-3 moment for RL') argue for a few expensive, high-fidelity environments over many cheap ones. Its June 2026 podcast covers eval quality.
- notes_for_program_managers:
  - Major change: Business Insider reported on 2026-09-11 that Google had completed a license-and-hire deal. Founder-CEO Tamay Besiroglu and more than a dozen staff moved to Google DeepMind. Talks in August were reported at more than $1.5B, with a non-exclusive license; final terms are undisclosed. Former chief of staff Guive Assadi now calls himself CEO. Neither company has announced the deal.
  - Before engaging, ask how much environment-building capacity remains after the deal, and whether Google's license or former staff create conflicts for other frontier-lab buyers.
  - The Anthropic relationship rests only on anonymous-sourced TechCrunch reporting from 2025. Neither company has confirmed it.
  - Funding: $9.1M on 2026-04-24 at a $500M post-money valuation, from Marco Mascorro, Adam D'Angelo and Devendra Chaplot (official press release). Headcount was reported at about 35 in August 2026 (TNW) or 46 at the April round (TechFundingNews). Both figures predate the departures to Google.
- notes_for_data_ops:
  - GBA Eval runs on Docker and git LFS. smoke-test.sh is a 30-second preflight and grade.sh does the grading; candidates are compiled to WebAssembly.
  - Official process note: each commercial task is owned by one engineer and takes about a week. Most of that effort goes into making the grader deterministic and fair.
  - No public SOC 2 report, security page, API or documented integration formats for commercial environments were found.
- sources:
  - https://www.mechanize.work/ (accessed 2026-10-05)
  - https://www.mechanize.work/mechanize-raises-9-1m (accessed 2026-10-05)
  - https://www.mechanize.work/press-releases/ (accessed 2026-10-05)
  - https://www.mechanize.work/announcing-mechanize-inc/ (accessed 2026-10-05)
  - https://www.mechanize.work/apply (accessed 2026-10-05)
  - https://www.mechanize.work/what-working-here-is-like/ (accessed 2026-10-05)
  - https://www.mechanize.work/blog/ (accessed 2026-10-05)
  - https://www.mechanize.work/podcast/how-bad-data-teaches-models-to-write-terrible-code/ (accessed 2026-10-05)
  - https://github.com/mechanize-work (accessed 2026-10-05)
  - https://github.com/mechanize-work/gba-eval (accessed 2026-10-05)
  - https://github.com/mechanize-work/gba-eval-attempts (accessed 2026-10-05)
  - https://gbaeval.com/ (accessed 2026-10-05)
  - https://gbaeval.com/results/leaderboard.json (accessed 2026-10-05)
  - https://www.businessinsider.com/google-mechanize-deal-talent-tech-ai-coding-2026-8 (accessed 2026-10-05)
  - https://www.businessinsider.com/google-completes-deal-for-ai-agents-startup-mechanize-2026-9 (accessed 2026-10-05)
  - https://thenextweb.com/news/google-may-spend-1-5bn-on-a-35-person-startup-to-catch-up-on-ai-coding (accessed 2026-10-05)
  - https://techfundingnews.com/google-mechanize-1-5b-ai-deal/ (accessed 2026-10-05)
  - https://www.linkedin.com/company/mechanize-inc (accessed 2026-06-07)
  - https://techcrunch.com/2025/09/21/silicon-valley-bets-big-on-environments-to-train-ai-agents/ (accessed 2026-06-07)
  - https://techcrunch.com/2025/04/19/famed-ai-researcher-launches-controversial-startup-to-replace-all-human-workers-everywhere/ (accessed 2026-06-07)
  - https://tracxn.com/d/companies/mechanize/__Zj76EY9-Iuwd7s9iR3MxAfLvnovzofl38AL66MgNwrQ (accessed 2026-06-07)
  - https://pitchbook.com/profiles/company/846556-03 (accessed 2026-06-07)
  - https://the-decoder.com/mechanize-is-building-digital-offices-to-train-ai-agents-to-fully-automate-computer-work/ (accessed 2026-06-07)

## Cua (not ranked, incumbent / infrastructure / open-source)
- slug: cua
- segment: Adjacent: execution infrastructure
- website: https://cua.ai/
- focus_areas: Computer Use
- positioning: Cua (trycua, YC X25) builds open-core 'computer-use 2.0' infrastructure: an MIT, agent-neutral background driver for macOS, Windows and Linux (CLI/MCP), local and cloud sandboxes, elastic Cua Fleets for parallel eval and RL training (cloud, BYOC, on-prem), Lume macOS virtualization, the Cua Spaces Mac app, small CUA-S1 decision models, and Cua-Bench for verifiable tasks, leaderboards and RL trajectory export. SOC 2 Type I.
- best_fit: Teams that need agent-neutral real-desktop environments (macOS/Windows/Linux apps) at fleet scale to evaluate computer-use agents or generate RL/SFT trajectories, including professional-software tasks such as EDA.
- overall_confidence: medium
- founded_year: 2025 [confirmed], source: https://www.ycombinator.com/companies/cua (accessed 2026-06-07)
- status: active [confirmed], source: https://cua.ai/ (accessed 2026-06-07); https://github.com/trycua/cua (accessed 2026-06-07)
- hq_location: San Francisco, CA, USA [reported], source: https://www.ycombinator.com/companies/cua (lists SF; accessed 2026-06-07). Note: Tracxn lists Dover, DE (likely state of incorporation), so a single authoritative HQ is not fully resolved.
- other_locations: unknown [unknown]
- distributed_remote: unknown [unknown]
- current_headcount: ~8 (8 current team members named on cua.ai/about; YC profile still shows team size 3) [confirmed], source: https://cua.ai/about (accessed 2026-10-05)
- headcount_band: 1-10 [confirmed], source: https://cua.ai/about (accessed 2026-10-05)
- headcount_growth: unknown [unknown]
- open_roles_count: 3 (Founding Technical GTM Lead; Founding Engineer, Infra & Agent Systems; Research Intern Summer 2026) [confirmed], source: https://www.ycombinator.com/companies/cua (accessed 2026-10-05)
- has_researchers: yes [reported], source: https://www.ycombinator.com/companies/cua (Research Intern role; accessed 2026-06-07); https://github.com/trycua/cua (Cua-Bench RL/eval work; accessed 2026-06-07)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Francesco Bonacci, founder: ex-Microsoft (Xbox, Microsoft AI), Windows Agent Arena co-author; Dillon DuPont, research lead: ex-Microsoft, Windows Agent Arena co-author; Dan Zhao, research: Windows Agent Arena co-author; Kai Mei, research (background not detailed on the About page) [confirmed], source: https://cua.ai/about (accessed 2026-10-05)
- published_papers_or_benchmarks: Cua-Bench: MIT computer-use task collection and harness with RL/SFT trajectory export (https://cua.ai/cuabench); Cua-Bench EDA Leaderboard, KiCad v1: 25 tasks authored by Snorkel AI (https://cua.ai/cuabench/leaderboard; https://snorkel.ai/cua-bench-computer-use-agent-benchmark/); Gemini 3.5 Flash computer-use evaluation on Cua-Bench (https://cua.ai/blog/evaluating-gemini-3.5-flash-on-computer-use); Windows Agent Arena (Microsoft, 2024; co-authored by Bonacci, DuPont and Zhao before Cua) (https://arxiv.org/abs/2409.08264) [confirmed], source: https://cua.ai/cuabench (accessed 2026-10-05)
- total_raised: ~$500K (pre-seed/seed, ~June 2025) [reported], source: https://tracxn.com/d/companies/cua-ai (aggregator; accessed 2026-06-07); https://www.linkedin.com/company/cua-ai (accessed 2026-06-07). No primary press release or official funding announcement located; figure rests on aggregators only.
- last_round: Pre-seed/seed, ~$500K, ~June 2025 [reported], source: https://tracxn.com/d/companies/cua-ai (lists ~$500K, round ~June 2025; accessed 2026-06-07). Round-type label (pre-seed vs seed) and exact date inconsistent across aggregator snippets; no primary announcement found.
- notable_investors: Y Combinator (X25 batch) [reported], source: https://www.ycombinator.com/companies/cua (YC X25 batch membership confirmed; accessed 2026-06-07). YC as a named round investor is corroborated only by aggregators; no primary round announcement located.
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: SOC 2 Type I (report received May 2026, available on request); SOC 2 Type II observation period in progress [confirmed], source: https://cua.ai/trust (accessed 2026-10-05)
- other_certifications: unknown [unknown]
- security_page: https://cua.ai/trust [confirmed], source: https://cua.ai/trust (accessed 2026-10-05)
- what_they_sell: infra [confirmed], source: https://cua.ai/ (accessed 2026-06-07); https://github.com/trycua/cua (accessed 2026-06-07)
- open_source: yes [confirmed], source: https://github.com/trycua/cua (accessed 2026-06-07)
- license: MIT (default: SDK, CLI, Driver, Lume, Bench); Cua Spaces FSL-1.1-MIT (source-available, becomes MIT 2 years after release); optional extensions AGPL-3.0 / Apache-2.0; commercial hosting per COMMERCIAL.md [confirmed], source: https://github.com/trycua/cua (accessed 2026-10-05)
- deployment_model: Local open-source sandboxes (GUI Docker/QEMU/Apple VZ, Lume) + managed Cua Fleet cloud (run.cua.ai) + BYOC + on-prem + SDK/CLI/MCP [confirmed], source: https://cua.ai/pricing (accessed 2026-10-05)
- maturity: GA: open-source components plus a usage-metered Cua Fleet. Cua Driver stable on macOS/Windows/Linux (v0.33.x by Oct 2026). Cua Spaces newly launched (v0.x, Oct 2026). Fleets macOS in preview. [confirmed], source: https://cua.ai/pricing (accessed 2026-10-05)
- notable_customers: Snorkel AI (KiCad benchmark built on Cua-Bench) (verified); Factory (Droid Control integrates cua-driver) (verified); H Company (partner; helped audit Driver regression matrix) (self-claimed); Qwen Code (Driver adopter / ecosystem integration) (self-claimed); Hermes (Driver adopter) (self-claimed); Clicky (first Driver adopter) (self-claimed); Google (early access to Gemini 3.5 Flash Computer Use tool; not a customer) (self-claimed, frontier-lab tie); OpenAI (homepage 'ecosystem integration'; link is an OpenAI-staff Tart image PR that does not name Cua) (self-claimed, frontier-lab tie) [reported], source: https://cua.ai/blog/computer-use-2-ai-engineer-worlds-fair (accessed 2026-10-05)
- founding_team:
  - Francesco Bonacci, Founder (YC lists Founder/CTO; cua.ai About page lists him under Product), Ex-Microsoft (Xbox, Microsoft AI). Co-author of Microsoft's Windows Agent Arena benchmark (2024). Built Lume before founding Cua in March 2025.
  - Dillon DuPont, Research lead, early team (joined May 2025; the About page does not call him a co-founder), Ex-Microsoft. Co-author of Windows Agent Arena (2024). Leads Cua research (Gemini 3.5 Flash eval; co-author of CUA-S1).
- recent_developments (last 6 months):
  - 2026-10-01 [product] Cua launches Cua Spaces v0.1.0, a source-available Mac app for watching and supervising agent desktops, source: https://github.com/trycua/cua/releases/tag/cua-spaces-v0.1.0
  - 2026-09-25 [product] Cua Driver adds compositor-native background computer use on Hyprland (Omarchy), source: https://github.com/trycua/cua/blob/main/blog/omarchy-cua-driver.md
  - 2026-09-18 [open-source] Cua releases CUA-S1, small open 'System One' models for narrow computer-use decisions, source: https://news.ycombinator.com/item?id=49767564
  - 2026-08-11 [research] Lume Metal capability shim speeds llama.cpp inference inside macOS VMs by 11-16x, source: https://cua.ai/blog/gpu-passthrough-macos-vms
  - 2026-08-06 [product] Cua Driver 0.19.0 adds extension-free browser use via Chrome DevTools Protocol, source: https://cua.ai/blog/extension-free-browser-use
  - 2026-07-13 [product] Cua publishes a recap of its 'Computer-Use 2.0' talk at AI Engineer World's Fair: Cua Driver 0.8, the Cua-Bench catalog and Cua Fleets, source: https://cua.ai/blog/computer-use-2-ai-engineer-worlds-fair
  - 2026-06-24 [benchmark] Cua evaluates Gemini 3.5 Flash computer use on Cua-Bench's KiCad suite, with early access from Google, source: https://cua.ai/blog/evaluating-gemini-3.5-flash-on-computer-use
  - 2026-06-18 [product] Cua Driver comes to Linux (X11/XWayland; native Wayland in preview), source: https://cua.ai/blog/inside-linux-computer-use
  - 2026-06-15 [partnership] Snorkel AI publishes a 25-task KiCad (EDA) benchmark built on Cua-Bench, source: https://snorkel.ai/cua-bench-computer-use-agent-benchmark/
  - 2026-05-27 [product] Cua Driver adds Windows background computer use with synthetic per-agent cursors, source: https://cua.ai/blog/inside-windows-computer-use
  - 2026-05 [other] Cua receives SOC 2 Type I report; Type II observation period under way, source: https://cua.ai/trust
  - 2026-04-23 [open-source] Cua open-sources Cua Driver for macOS: background, multi-cursor computer use for any agent, source: https://cua.ai/blog/inside-macos-window-internals
- benchmarks_and_research:
  - Cua-Bench EDA Leaderboard (KiCad, v1) (leaderboard), 2026-06, Claude Fable 5: 6/25 full + 2 partial, mean reward 0.267. GPT-5.5: 6/25 full, 0.240. Gemini 3.5 Flash: 5/25 full + 3 partial, 0.267. Claude Sonnet 4.5 and Haiku 4.5: 0.253 each. Every full pass was an edit to an existing schematic., https://cua.ai/cuabench/leaderboard
  - Cua-Bench (benchmark), 2025-12, The best frontier agent tested clears 6 of 25 expert KiCad tasks. No model reliably builds a schematic from a blank canvas., https://cua.ai/cuabench
  - Windows Agent Arena (Microsoft; co-authored by Cua's Bonacci, DuPont and Dan Zhao before Cua) (paper), 2024-09-12, The Navi agent reached 19.5% success in the Windows domain, against 74.5% for unassisted humans., https://arxiv.org/abs/2409.08264
- products:
  - Cua Driver: MIT open-source background computer-use driver for macOS, Windows and Linux (including Hyprland), exposed over CLI and MCP. Works with Claude Code, Codex and custom loops. Adds extension-free browser use via CDP from v0.19. Latest cua-driver-rs release is v0.33.4 (2026-10-05)., https://github.com/trycua/cua
  - Cua Sandbox / SDK: Local sandboxes (GUI Docker, QEMU, Apple VZ) plus cloud sandboxes. The SDK is documented for Python, TypeScript, Swift and Rust; the repo also lists Kotlin bindings., https://cua.ai/docs
  - Cua Fleets: Usage-metered elastic fleets with warm sandbox pools for parallel eval and RL training. Linux, Windows and Android, with macOS in preview. Console at run.cua.ai. BYOC and on-prem are available., https://cua.ai/pricing
  - Cua-Bench: Computer-use task collection and harness: evaluation, leaderboards and trajectory export for RL/SFT., https://cua.ai/cuabench
  - Lume: Local macOS and Linux VMs on Apple Silicon via Virtualization.framework. Includes an open-source Metal capability shim for faster in-VM inference., https://github.com/trycua/cua
  - Cua Spaces: Source-available (FSL-1.1-MIT) Mac app for watching, taking over and sharing files with agent desktops (Spaces)., https://github.com/trycua/cua/tree/main/apps/cua-spaces
  - CUA-S1: Small MIT-licensed 'System One' decision models for narrow UI decisions. The first release, CUA-S1-FORMS (706k parameters), handles forms., https://news.ycombinator.com/item?id=49767564
- notes_for_researchers:
  - Cua-Bench is MIT-licensed and supports trajectory export for RL/SFT. The KiCad EDA leaderboard (25 tasks, authored by Snorkel AI) tops out at a mean reward of about 0.27, far from saturated. Every full pass was an edit, and no model built a schematic from scratch (https://cua.ai/cuabench/leaderboard).
  - Cua publishes detailed engineering write-ups (macOS SkyLight, Windows synthetic cursors, Linux AT-SPI/XTEST, Hyprland) and reports null results, e.g. OSWorld 2.0 browser-tools test at +2.55 pp, not significant (p = 0.5).
  - Bonacci, DuPont and Dan Zhao co-authored Microsoft's Windows Agent Arena benchmark before Cua, so the team has real benchmark-building pedigree.
- notes_for_program_managers:
  - Small YC X25 team. YC still lists team size 3, but cua.ai/about names 8 current members across SF, Toronto, Vancouver, New York and New Jersey. No funding beyond the ~$500K June 2025 seed (aggregator-reported: YC, Mark Kahn).
  - Frontier-lab ties are vendor-stated: early access to Google's Gemini 3.5 Flash Computer Use tool. The homepage lists OpenAI under 'ecosystem integrations', but the link goes to an OpenAI-staff PR in cirruslabs/macos-image-templates ('Include Metal Capabilities Helper') that does not name Cua. Treat it as an integration claim, not a customer.
  - Third-party-verified relationships: Snorkel AI (KiCad benchmark on Cua-Bench) and Factory (Droid Control docs install cua-driver). Hermes, Qwen Code, H Company and Clicky are self-reported adopters or partners.
  - Licensing is open-core: MIT for the SDK, CLI, Driver, Lume and Bench; FSL-1.1-MIT for Cua Spaces; AGPL-3.0 and Apache-2.0 for optional perception extensions; hosted commercial use needs review per COMMERCIAL.md.
  - Hiring (YC): Founding Technical GTM Lead; Founding Engineer, Infra & Agent Systems; Research Intern (Summer 2026).
- notes_for_data_ops:
  - SOC 2 Type I report received May 2026, available on request via enterprise@cua.ai. Type II observation period in progress. No other certifications listed (https://cua.ai/trust).
  - Deployment: local open-source sandboxes (Docker/QEMU/Apple VZ, Lume), Cua Fleet cloud via run.cua.ai, BYOC and on-prem. The Fleet price per vCPU-hour and per GiB-hour is published on https://cua.ai/pricing.
  - Integration surfaces: Cua Driver over CLI and MCP; SDKs in Python, TypeScript, Swift and Rust (Kotlin bindings in the repo).
- sources:
  - https://cua.ai/ (accessed 2026-10-05)
  - https://cua.ai/about (accessed 2026-10-05)
  - https://cua.ai/pricing (accessed 2026-10-05)
  - https://cua.ai/trust (accessed 2026-10-05)
  - https://cua.ai/blog (accessed 2026-10-05)
  - https://cua.ai/blog/inside-macos-window-internals (accessed 2026-10-05)
  - https://cua.ai/blog/inside-windows-computer-use (accessed 2026-10-05)
  - https://cua.ai/blog/inside-linux-computer-use (accessed 2026-10-05)
  - https://cua.ai/blog/evaluating-gemini-3.5-flash-on-computer-use (accessed 2026-10-05)
  - https://cua.ai/blog/computer-use-2-ai-engineer-worlds-fair (accessed 2026-10-05)
  - https://cua.ai/blog/extension-free-browser-use (accessed 2026-10-05)
  - https://cua.ai/blog/gpu-passthrough-macos-vms (accessed 2026-10-05)
  - https://github.com/trycua/cua/blob/main/blog/omarchy-cua-driver.md (accessed 2026-10-05)
  - https://news.ycombinator.com/item?id=49767564 (accessed 2026-10-05)
  - https://github.com/trycua/cua (accessed 2026-10-05)
  - https://github.com/trycua/cua/releases (accessed 2026-10-05)
  - https://github.com/trycua/cua/releases/tag/cua-spaces-v0.1.0 (accessed 2026-10-05)
  - https://github.com/trycua/cua/tree/main/apps/cua-spaces (accessed 2026-10-05)
  - https://cua.ai/docs (accessed 2026-10-05)
  - https://cua.ai/cuabench (accessed 2026-10-05)
  - https://cua.ai/cuabench/leaderboard (accessed 2026-10-05)
  - https://snorkel.ai/cua-bench-computer-use-agent-benchmark/ (accessed 2026-10-05)
  - https://docs.factory.com/cli/features/droid-control (accessed 2026-10-05)
  - https://github.com/cirruslabs/macos-image-templates/pull/372 (accessed 2026-10-05)
  - https://www.ycombinator.com/companies/cua (accessed 2026-10-05)
  - https://github.com/francedot (accessed 2026-10-05)
  - https://arxiv.org/abs/2409.08264 (accessed 2026-10-05)
  - https://startupintros.com/orgs/cua (accessed 2026-10-05)
  - https://codenote.net/en/posts/codex-computer-use-cua-repl-vs-trycua-cua-driver/ (accessed 2026-10-05)
  - https://www.linkedin.com/company/cua-ai (accessed 2026-06-07)
  - https://tracxn.com/d/companies/cua-ai (accessed 2026-06-07)
  - https://cuabench.ai/ (accessed 2026-06-07)
  - https://news.ycombinator.com/item?id=43773563 (accessed 2026-06-07)
  - https://www.linkedin.com/in/francesco-bonacci-70428a121/ (accessed 2026-06-07)
  - https://www.linkedin.com/in/alessandro-puppo/ (accessed 2026-06-07)

## Good Start Labs (not ranked, incumbent / infrastructure / open-source)
- slug: good-start-labs
- segment: Open-source & open environments
- website: https://goodstartlabs.com
- focus_areas: Long-Horizon
- positioning: Good Start Labs is a 2025 Every spin-out that sells game-based RL environments and agent and human gameplay data to AI labs. It also partners with game publishers (Bad Cards, Arkadium GameLab) to turn play into licensed training data, in-game agents and leaderboards. It publishes transfer studies from games to real work (Diplomacy to customer support, the 1830 railroad game to finance research), arXiv papers and open repos, and runs the Diplomacy and Humor (Bad Cards) Arena leaderboards.
- best_fit: Buyers wanting open, game-based RL environments and benchmarks for long-horizon multi-agent reasoning and preference/humor evaluation.
- overall_confidence: medium
- founded_year: 2025 [confirmed], source: https://www.inovia.vc/active-companies/good-start-labs/ (accessed 2026-06-07); https://every.to/on-every/our-new-incubation-raised-3-6-million-to-teach-ais-to-play-games (accessed 2026-06-07)
- status: active [confirmed], source: https://goodstartlabs.com/ (accessed 2026-06-07); https://github.com/GoodStartLabs (accessed 2026-06-07); https://www.inovia.vc/active-companies/good-start-labs/ (accessed 2026-06-07)
- hq_location: Brooklyn, NY, USA [reported], source: https://www.inovia.vc/active-companies/good-start-labs/ (accessed 2026-06-07)
- other_locations: Toronto, ON, Canada [reported], source: https://www.inovia.vc/active-companies/good-start-labs/ (accessed 2026-06-07)
- distributed_remote: yes [confirmed], source: https://goodstartlabs.com/careers (accessed 2026-10-05)
- current_headcount: unknown (Hugging Face org now shows 7 members; LinkedIn band 2-10) [unknown], source: https://huggingface.co/GoodStartLabs (accessed 2026-10-05)
- headcount_band: 1-10 [reported], source: https://huggingface.co/GoodStartLabs (9 team members, accessed 2026-06-07); LinkedIn public snippet (size 2-10, accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: 2 (Research Scientist; Machine Learning Engineer; both Remote, US & Canada) [confirmed], source: https://goodstartlabs.com/careers (accessed 2026-10-05)
- has_researchers: yes [reported], source: https://huggingface.co/GoodStartLabs (research model/dataset publishing, accessed 2026-06-07); https://github.com/GoodStartLabs (research benchmarks, accessed 2026-06-07)
- researcher_count: unknown [unknown]
- researcher_backgrounds: Alex Duffy (Co-Founder & CEO): AI Camp co-founder and head of product, ex-VP of AI at Salt AI (drug discovery), led AI training at Every; co-creator of AI Diplomacy; Tyler Marques (Co-Founder & CTO): University of Waterloo; data engineering, infrastructure and applied AI/ML; Paper collaborators include Dinesh Manocha and Tianyi Zhou (UMD) and Matthew Lyle Olson (co-author on all three arXiv papers; affiliation not verified) [reported], source: https://ai.engineer/speakers/alex-duffy (accessed 2026-10-05)
- published_papers_or_benchmarks: Democratizing Diplomacy harness paper (arXiv 2508.07485); Data-Centric Interpretability for LLM-based Multi-Agent RL (arXiv 2602.05183); COS-PLAY (arXiv 2604.20987); Diplomacy Arena leaderboard; Humor / Bad Cards Arena (LOL Arena) leaderboard; StarSkirmish Bench (2026-10, with StarSkirmish/Kai McPheeters); 1830 railroad game to Finance-Agent transfer study (2026-08); Jev verifier cost study (2026-09); AI_Diplomacy open-source harness [confirmed], source: https://goodstartlabs.com/research (accessed 2026-10-05)
- total_raised: $3.6M [reported], source: https://every.to/on-every/our-new-incubation-raised-3-6-million-to-teach-ais-to-play-games (accessed 2026-06-07); https://www.inovia.vc/active-companies/good-start-labs/ (accessed 2026-06-07)
- last_round: Seed, $3.6M, October 2025 (Inovia describes it as pre-seed) [reported], source: https://every.to/on-every/our-new-incubation-raised-3-6-million-to-teach-ais-to-play-games (accessed 2026-06-07); https://www.inovia.vc/active-companies/good-start-labs/ (pre-seed, Oct 2025, accessed 2026-06-07); https://www.linkedin.com/posts/tirtavc_were-thrilled-to-announce-tirta-ventures-activity-7384641634371608576-pf7K (seed, accessed 2026-06-07)
- notable_investors: General Catalyst; Inovia Capital; Tirta Ventures; Every; angel investors from top AI labs (incl. DeepMind alumni) [reported], source: https://every.to/on-every/our-new-incubation-raised-3-6-million-to-teach-ais-to-play-games (General Catalyst, Inovia, Every, DeepMind angels, accessed 2026-06-07); https://www.linkedin.com/posts/tirtavc_were-thrilled-to-announce-tirta-ventures-activity-7384641634371608576-pf7K (Tirta Ventures co-lead, accessed 2026-06-07)
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: unknown [unknown]
- what_they_sell: environments and data (game RL environments, agent trajectory and human gameplay data, benchmarks, plus publisher data-licensing partnerships) [confirmed], source: https://goodstartlabs.com/products (accessed 2026-10-05)
- open_source: yes [confirmed], source: https://github.com/GoodStartLabs (accessed 2026-06-07); https://github.com/GoodStartLabs/AI_Diplomacy (accessed 2026-06-07)
- license: Mixed: OpenTinker fork Apache-2.0; catanatron fork GPL-3.0; AI_Diplomacy no license specified [reported], source: https://github.com/GoodStartLabs (accessed 2026-10-05)
- deployment_model: unknown [unknown]
- maturity: early commercial [estimated], source: https://goodstartlabs.com/research/why-we-do-what-we-do (accessed 2026-10-05)
- notable_customers: Arcee AI (Trinity Large trained on GSL Diplomacy environment 'among others'; logo on homepage; Arcee co-founders listed as GSL investors/advisors) (self-claimed, frontier-lab tie); OpenAI (logo in homepage 'Trusted by' row; no usage detail) (self-claimed, frontier-lab tie); Cohere (logo in homepage 'Trusted by' row; a Cohere employee is listed as investor/advisor) (self-claimed, frontier-lab tie); Arkadium (GameLab leaderboard partnership; not mentioned in GamesBeat launch coverage) (self-claimed); Bad Cards (game partner / LOL Arena data source) (self-claimed) [reported], source: https://goodstartlabs.com/ (accessed 2026-10-05)
- founding_team:
  - Alex Duffy, Co-founder & CEO, Co-founded AI Camp (head of product), then VP of AI at Salt AI leading drug-discovery research, then led AI training and consulting at Every. Co-created AI Diplomacy with Tyler Marques.
  - Tyler Marques, Co-founder & CTO, University of Waterloo. Data-engineering and infrastructure background in applied AI/ML and DevOps (per Inovia and his GitHub bio). Co-creator of AI Diplomacy and co-author of all three GSL arXiv papers.
- recent_developments (last 6 months):
  - 2026-10-01 [benchmark] "Keep Going": 48-hour livestream of GPT-6 Astra (Codex) and Claude Opus 5.5 (Claude Code) autonomously writing StarCraft: Brood War bots, plus StarSkirmish Bench, source: https://goodstartlabs.com/research/keep-going
  - 2026-09-15 [research] "Verification Is the Bottleneck": GSL tests TypeSafe's Jev classifier as a cheap RL verifier, source: https://goodstartlabs.com/research/verification-is-the-bottleneck
  - 2026-09-15 [other] Latent Space profile of Good Start Labs: "Can Skills Learned in Games Transfer to Real-World Work?", source: https://www.latent.space/p/good-start-labs
  - 2026-09-05 [benchmark] Humor (Bad Cards) Arena leaderboard updated: 14,459 rounds, 2,390 human judges, source: https://goodstartlabs.com/benchmarks
  - 2026-08-16 [customer] "Why do we do what we do": GSL says AI labs already train on its environments and data, source: https://goodstartlabs.com/research/why-we-do-what-we-do
  - 2026-08-04 [research] "What a Railroad Game Taught a Model About Finance": RL on the board game 1830 lifts Nemotron 30B on vals.ai Finance-Agent, source: https://goodstartlabs.com/research/what-a-railroad-game-taught-a-model-about-finance
  - 2026-07-30 [research] "Where an Agent's Intelligence Lives": COS-PLAY write-up (co-evolving harness and skill bank), source: https://goodstartlabs.com/research/where-an-agents-intelligence-lives
  - 2026-07-27 [open-source] Hugging Face org publishes diplomacy-235b-lora, 1830 Nemotron adapters and a run of chess models, source: https://huggingface.co/GoodStartLabs
  - 2026-07-15 [open-source] GoodStartLabs GitHub org adds a catanatron fork (Settlers of Catan simulator, GPL-3.0), source: https://github.com/GoodStartLabs
  - 2026-07 [partnership] "Measure What Matters. Get Paid for It.": GSL says it helped Arkadium launch GameLab, a public leaderboard of frontier models on casual games, source: https://goodstartlabs.com/research/measure-what-matters
  - 2026-04-22 [research] COS-PLAY paper on arXiv (2604.20987), co-authored by Duffy and Marques, source: https://arxiv.org/abs/2604.20987
- benchmarks_and_research:
  - StarSkirmish Bench (benchmark), 2026-10-01, GPT-6 Astra and Claude Opus 5.5 are functionally tied at the top; together with GPT-6 Sol they are a clear step above the other models tested., https://goodstartlabs.com/research/keep-going
  - 1830 railroad-game RL, transfer to vals.ai Finance-Agent (paper), 2026-08-04, Nemotron 30B went from 12.1% to 18.3% on Finance-Agent, and its rate of confident false claims fell from 83.0% to 64.5%., https://goodstartlabs.com/research/what-a-railroad-game-taught-a-model-about-finance
  - COS-PLAY: Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks (paper), 2026-04-22, Over 25.1% average reward improvement against four frontier LLM baselines on single-player games (Qwen3-8B vs GPT-5.4, per GSL's write-up)., https://arxiv.org/abs/2604.20987
  - Data-Centric Interpretability for LLM-based Multi-Agent Reinforcement Learning (paper), 2026-02-05, 90% of the SAE meta-features it found are significant, and system-prompt augmentation driven by its hypotheses improved agent performance by 14.2%., https://arxiv.org/abs/2602.05183
  - Democratizing Diplomacy: A Harness for Evaluating Any Large Language Model on Full-Press Diplomacy (paper), 2025-08-10, With an optimized game-state representation, a 24B model reliably completes full matches without fine-tuning., https://arxiv.org/abs/2508.07485
  - Diplomacy Arena / AI Diplomacy (leaderboard), 2025-06-05, The benchmarks index reports 200,000+ matches and a top model score of about 60%. The leaderboard was last updated 2026-01-15., https://goodstartlabs.com/benchmarks/diplomacy
  - Humor / Bad Cards Arena (LOL Arena) (leaderboard), 14,459 rounds and 2,390 judges. Chance is 41.1% and the top model scores about 52% (updated 2026-09-05)., https://goodstartlabs.com/benchmarks/lol-arena
- products:
  - RL environments (Intelligence): Interactive learning environments built from curated games with verifiable rewards, covering long-horizon planning, code, spatial reasoning, memory, negotiation and collaboration (for example the Diplomacy environment)., https://goodstartlabs.com/products
  - World Understanding data: Real human gameplay data across text, image and 3D worlds (physics, space, navigation, perception, motion, problem solving), with continuous feedback from live games., https://goodstartlabs.com/products
  - Benchmarks / leaderboards (Diplomacy, Humor/Bad Cards): Live public leaderboards. Negotiation, Theory of Mind, Tool Use, Spatial Reasoning and Deception Detection are listed as coming soon., https://goodstartlabs.com/benchmarks
  - Game publisher services: For publishers: data monetization and licensing to AI labs, data pipelines with player consent, in-game AI agents, public benchmarks and RL advisory. Named partners are Bad Cards and Arkadium GameLab., https://goodstartlabs.com/for-game-studios
  - AI_Diplomacy (open source): Open-source harness for frontier models playing Diplomacy (707 stars; no license specified)., https://github.com/GoodStartLabs/AI_Diplomacy
  - OpenTinker fork (open source): GSL's fork of open-tinker/OpenTinker, an RL infrastructure project (Apache-2.0). It is a fork, not an original GSL product., https://github.com/GoodStartLabs/OpenTinker
- notes_for_researchers:
  - Open papers: Democratizing Diplomacy (arXiv 2508.07485, NeurIPS 2025 workshop), interpretability of multi-agent RL in Diplomacy (arXiv 2602.05183, ICLR 2026 workshop), and COS-PLAY with UMD, USC and MBZUAI (arXiv 2604.20987, Agent Skills '26 poster). Matthew Lyle Olson co-authors all three.
  - Transfer results, all self-reported: Diplomacy RL improved Qwen 3 on Tau2 customer support and by more than 10% on Hanabi and Wordle. RL on 1830 lifted Nemotron 30B from 12.1% to 18.3% on vals.ai Finance-Agent, and the gain appeared only with a multi-turn terminal-agent setup.
  - Open code and checkpoints: AI_Diplomacy (no license stated), forks of OpenTinker (Apache-2.0) and catanatron (GPL-3.0), and 53 public Hugging Face models including diplomacy-235b-lora and nemotron30b-1830-grpo-adapters. No public datasets are currently listed.
  - GSL's August 2026 post says it is 'growing a team of researchers who built some of the field's foundational learning environments' and credits the Factorio Learning Environment to Jack Hopkins. It does not explicitly say Hopkins has joined GSL, so treat that as unconfirmed.
- notes_for_program_managers:
  - Seed-stage team. No new round found since the October 2025 $3.6M raise (Inovia and General Catalyst leading; Tirta Ventures also an investor). Two open roles, both Remote (US & Canada): Research Scientist ($220-300K) and ML Engineer ($165-250K).
  - The homepage 'Trusted by' row shows Cohere, Arcee and OpenAI logos. These are self-claimed. The about page lists Arcee's Mark McQuade and Lucas Atkins and Cohere's Teresa Grandia as investors/advisors, so some logos may reflect investor ties rather than paid contracts. The one specific usage claim is that Arcee's 400B Trinity Large trained on GSL's Diplomacy environment 'among others'.
  - A two-sided model: it sells environments and trajectory data to AI labs, and partners with game publishers on data licensing, agents and leaderboards. All partner figures are self-claimed: Arkadium GameLab; Bad Cards with 2M+ users; a Bad Cards agent running 20,000+ games a week; a 13% engagement lift.
- notes_for_data_ops:
  - The products page publishes no integration formats (Docker, MCP, API). Engagement is through the site's contact form.
  - No SOC 2, security or trust page found. The publisher page says pipelines have 'player consent designed in from the start', and Latent Space reports that data is anonymized and stripped of PII. Neither gives detail.
  - Check licensing before reuse: open repos are mixed (Apache-2.0 fork, GPL-3.0 fork, AI_Diplomacy unspecified).
- sources:
  - https://goodstartlabs.com/ (accessed 2026-10-05)
  - https://goodstartlabs.com/products (accessed 2026-10-05)
  - https://goodstartlabs.com/benchmarks (accessed 2026-10-05)
  - https://goodstartlabs.com/benchmarks/diplomacy (accessed 2026-10-05)
  - https://goodstartlabs.com/research (accessed 2026-10-05)
  - https://goodstartlabs.com/research/keep-going (accessed 2026-10-05)
  - https://goodstartlabs.com/research/verification-is-the-bottleneck (accessed 2026-10-05)
  - https://goodstartlabs.com/research/why-we-do-what-we-do (accessed 2026-10-05)
  - https://goodstartlabs.com/research/what-a-railroad-game-taught-a-model-about-finance (accessed 2026-10-05)
  - https://goodstartlabs.com/research/where-an-agents-intelligence-lives (accessed 2026-10-05)
  - https://goodstartlabs.com/research/measure-what-matters (accessed 2026-10-05)
  - https://goodstartlabs.com/for-game-studios (accessed 2026-10-05)
  - https://goodstartlabs.com/careers (accessed 2026-10-05)
  - https://goodstartlabs.com/about (accessed 2026-10-05)
  - https://www.latent.space/p/good-start-labs (accessed 2026-10-05)
  - https://gamesbeat.com/arkadium-launches-gamelab-to-make-ai-ready-for-the-real-world/ (accessed 2026-10-05)
  - https://arxiv.org/abs/2604.20987 (accessed 2026-10-05)
  - https://arxiv.org/abs/2602.05183 (accessed 2026-10-05)
  - https://arxiv.org/abs/2508.07485 (accessed 2026-10-05)
  - https://github.com/GoodStartLabs (accessed 2026-10-05)
  - https://huggingface.co/GoodStartLabs (accessed 2026-10-05)
  - https://github.com/Tylermarques (accessed 2026-10-05)
  - https://every.to/@AlxAi (accessed 2026-10-05)
  - https://ai.engineer/speakers/alex-duffy (accessed 2026-10-05)
  - https://www.inovia.vc/active-companies/good-start-labs/ (accessed 2026-06-07)
  - https://www.inovia.vc/founders/company-founders/tyler-marques/ (accessed 2026-06-07)
  - https://every.to/on-every/our-new-incubation-raised-3-6-million-to-teach-ais-to-play-games (accessed 2026-06-07)
  - https://tracxn.com/d/companies/goodstartlabs/funding-and-investors (accessed 2026-06-07)
  - https://www.linkedin.com/company/good-start-labs (accessed 2026-06-07)
  - https://startups.gallery/companies/good-start-labs (accessed 2026-06-07)
  - https://every.to/diplomacy (accessed 2026-06-07)

## Surge AI (not ranked, incumbent / infrastructure / open-source)
- slug: surge-ai
- segment: Incumbents also building RL environments
- website: https://surgehq.ai/
- focus_areas: Enterprise Workflows, Long-Horizon
- positioning: Surge AI is a bootstrapped human-data and RLHF leader for frontier labs (reported $1.2B 2024 revenue). In 2026 it built a large in-house benchmark and RL-environment program (DAYJOB, GDP.pdf/.xlsx, HANDBOOK.md, Chartography, CoreCraft, and the Tuesday Work Index composite), published post-training transfer studies, launched Post-Training Runs and an enterprise offering, and released sample harnesses under Apache-2.0. Anthropic reports results on Surge's GDP.pdf in its Claude Sonnet 5 system card, and Microsoft publicly credits Surge's expert raters. OpenAI's adoption of GDP.pdf is so far claimed only by Surge.
- best_fit: Frontier and enterprise teams that need expert human data or evals, plus off-the-shelf long-horizon professional RL environments (healthcare, finance, office/policy, spreadsheets, coding), optionally with Surge-run post-training.
- overall_confidence: high
- founded_year: 2020 [reported], source: https://en.wikipedia.org/wiki/Surge_AI accessed 2026-06-07 (founded 2020 by Edwin Chen); corroborated by CB Insights/Craft. Note: Sacra describes a '2021 launch'; getlatka lists '2018' (outlier). Single primary source, so reported.
- status: active [confirmed], source: https://surgehq.ai/ accessed 2026-06-07; https://en.wikipedia.org/wiki/Surge_AI accessed 2026-06-07
- hq_location: San Francisco, California, USA [confirmed], source: https://en.wikipedia.org/wiki/Surge_AI accessed 2026-06-07; Craft.co lists 2193 Fillmore St, San Francisco accessed 2026-06-07
- other_locations: New York, NY; Seattle, WA [estimated], source: https://surgehq.ai/careers (accessed 2026-10-05)
- distributed_remote: yes [reported], source: https://surgehq.ai/careers accessed 2026-06-07 (open roles listed Remote); ~1M distributed annotator/contractor network per https://en.wikipedia.org/wiki/Surge_AI accessed 2026-06-07
- current_headcount: ~110-121 full-time employees (2025); ~1M annotators/contractors [reported], source: https://en.wikipedia.org/wiki/Surge_AI accessed 2026-06-07 (110 employees, 2025); getlatka/Inc. cite ~121 FTE; ~1M annotators. Pitchbook ~250 is an outlier and not relied upon.
- headcount_band: 51-200 [confirmed], source: https://www.linkedin.com/company/surge-ai accessed 2026-06-07 (public band 51-200); corroborated by Craft.co accessed 2026-06-07. Refers to full-time staff, not the ~1M contractor network.
- headcount_growth: unknown [unknown]
- open_roles_count: 59 [reported], source: https://surgehq.ai/careers (accessed 2026-10-05)
- has_researchers: yes [confirmed], source: https://surgehq.ai/research accessed 2026-06-07 (published benchmarks/RL environments; arXiv:2602.16179 EnterpriseBench CoreCraft); https://surgehq.ai/careers lists Research Scientist roles, accessed 2026-06-07
- researcher_count: unknown [unknown], source: Research roles/fellowship advertised but team count not disclosed. accessed 2026-06-07
- researcher_backgrounds: Founder/CEO Edwin Chen: former research scientist at Google, Facebook and Twitter; formerly at Clarium Capital; studied math, CS and linguistics at MIT; Recurring authors on 2026 papers: Sushant Mehta, Suhaas Garre, Logan Ritchie, Liudas Panavas (backgrounds not disclosed). DAYJOB lists domain-expert co-authors from healthcare and finance. [reported], source: https://arxiv.org/abs/2610.01306 (accessed 2026-10-05)
- published_papers_or_benchmarks: DAYJOB (Healthcare/Finance) - arXiv:2610.01306; Cross-Benchmark Transfer from RL on Agentic Coding Tasks - arXiv:2610.00890; GDP.xlsx - https://surgehq.ai/blog/gdp-xlsx; Tuesday Work Index leaderboard - https://surgehq.ai/benchmarks; Chartography - arXiv:2608.10677; Post-Training on Office Work Improves Software Engineering - arXiv:2608.01604; Cross-Benchmark Generalization in Long-Horizon Agents - arXiv:2608.00181; HANDBOOK.md - arXiv:2607.25398; GDP.pdf - arXiv:2607.11192; ComplexConstraints and Beyond: Expert Rubrics for RLVR - arXiv:2606.09118; Riemann-Bench - arXiv:2604.06802; EnterpriseBench CoreCraft - arXiv:2602.16179; The Hierarchy of Agentic Capabilities - arXiv:2601.09032; Antidote and Hemingway-bench leaderboards - https://surgehq.ai/benchmarks; AdvancedIF - arXiv:2511.10507 and Multimodal RewardBench 2 - arXiv:2512.16899 (Meta papers listed on Surge's research page) [confirmed], source: https://surgehq.ai/research (accessed 2026-10-05)
- total_raised: $0 outside funding (bootstrapped), per Surge's claim reported by Forbes as of Oct 2026. The July 2025 raise talks have no disclosed close. [reported], source: https://www.forbes.com/profile/edwin-chen/ (accessed 2026-10-05)
- last_round: None disclosed. Bootstrapped. Reported July 2025 talks (~$1B at $15-25B+) have no disclosed close as of Oct 2026. [reported], source: https://www.forbes.com/profile/edwin-chen/ (accessed 2026-10-05)
- notable_investors: unknown [unknown], source: No confirmed investors; round not confirmed closed. Bloomberg named only POTENTIAL investors (Andreessen Horowitz, Warburg Pincus, TPG) with J.P. Morgan facilitating secondary. https://www.bloomberg.com/news/articles/2025-07-30/scale-rival-surge-ai-in-talks-for-funding-at-25-billion-value accessed 2026-06-07
- valuation: ~$25B (reported July 2025 fundraise talks; 'at least $25B' per Bloomberg, '>$15B' per Reuters); unconfirmed/not closed [reported], source: https://www.bloomberg.com/news/articles/2025-07-30/scale-rival-surge-ai-in-talks-for-funding-at-25-billion-value accessed 2026-06-07. getlatka's $20B and a LinkedIn $30B claim are inconsistent/weak and not relied upon.
- revenue_signals: $1.2B revenue (2024); ~$1.4B ARR claimed (Aug 2025, weak source) [reported], source: $1.2B 2024 corroborated by https://en.wikipedia.org/wiki/Surge_AI and https://sacra.com/c/surge-ai/ accessed 2026-06-07. The ~$1.4B ARR figure is from getlatka.com (low-reliability aggregator) only. Note: Meta reportedly ~$150M/yr, Google ~$100M/yr (Sacra).
- soc2: unknown [unknown], source: No public trust/security page or SOC 2 statement found. accessed 2026-06-07
- other_certifications: unknown [unknown]
- security_page: unknown [unknown], source: No dedicated security/trust page found. accessed 2026-06-07
- what_they_sell: mixed [confirmed], source: https://surgehq.ai/ and https://surgehq.ai/research accessed 2026-06-07 (human data/RLHF labeling platform plus agentic RL environments). Core business is human data; environments are an adjacent/secondary line.
- open_source: partial [confirmed], source: https://github.com/surge-ai (accessed 2026-10-05)
- license: Apache-2.0 (benchmark repos dayjob, handbook, gdp-xlsx); MIT (surge-python SDK); full datasets and environments proprietary [confirmed], source: https://github.com/surge-ai (accessed 2026-10-05)
- deployment_model: managed-hosted [estimated], source: https://surgehq.ai/ accessed 2026-06-07 (managed human-data platform/service; no self-hosted/on-prem offering disclosed)
- maturity: GA [estimated], source: https://surgehq.ai/ accessed 2026-06-07; revenue-generating platform ($1.2B 2024) serving frontier labs per https://sacra.com/c/surge-ai/ accessed 2026-06-07
- notable_customers: Anthropic (verified, frontier-lab tie); Google (verified, frontier-lab tie); Meta (verified, frontier-lab tie); Microsoft (Microsoft AI; MAI-Thinking-1 human evals, named by Microsoft) (verified, frontier-lab tie); OpenAI (former; relationship reportedly ended Sept 2025) (verified, frontier-lab tie) [reported], source: https://microsoft.ai/news/introducing-mai-thinking-1/ (accessed 2026-10-05)
- founding_team:
  - Edwin Chen, Founder and CEO, Previously a research scientist at Google, Facebook and Twitter (per his own site), and worked at Peter Thiel's hedge fund Clarium (per Forbes). Studied math, computer science and linguistics at MIT. Forbes lists him as the youngest member of The Forbes 400.
- recent_developments (last 6 months):
  - 2026-10-05 [other] Forbes profile: Surge still claims no outside funding; Chen owns about 75% and is worth $18B, source: https://www.forbes.com/profile/edwin-chen/
  - 2026-10-01 [benchmark] GDP.xlsx spreadsheet-reasoning benchmark launched, source: https://surgehq.ai/blog/gdp-xlsx
  - 2026-10-01 [benchmark] DAYJOB: A Benchmark for Long-Horizon Professional Work (arXiv 2610.01306), source: https://arxiv.org/abs/2610.01306
  - 2026-10-01 [research] Cross-Benchmark Transfer from RL on Agentic Coding Tasks (arXiv 2610.00890), source: https://arxiv.org/abs/2610.00890
  - 2026-09-24 [benchmark] Surge launches the Tuesday Work Index, a composite 'AI at work' leaderboard, source: https://surgehq.ai/blog/tuesday-frontier-work-index
  - 2026-09 [open-source] Surge open-sources benchmark harnesses and sample tasks on GitHub and Hugging Face (dayjob, handbook, gdp-xlsx), source: https://github.com/surge-ai
  - 2026-08-28 [partnership] Surge says it built the human-researcher baseline for Anthropic's automated alignment researchers study, source: https://surgehq.ai/blog/anthropic-automated-alignment-research
  - 2026-08-12 [customer] Microsoft AI used Surge's professional raters for MAI-Thinking-1 human evaluations, source: https://microsoft.ai/news/introducing-mai-thinking-1/
  - 2026-08-11 [benchmark] Chartography: a benchmark for professional chart understanding (arXiv 2608.10677), source: https://arxiv.org/abs/2608.10677
  - 2026-08-03 [research] Post-Training on Office Work Improves Software Engineering (arXiv 2608.01604), source: https://arxiv.org/abs/2608.01604
  - 2026-07-31 [research] Cross-Benchmark Generalization in Long-Horizon Agents (arXiv 2608.00181), source: https://arxiv.org/abs/2608.00181
  - 2026-07-28 [benchmark] HANDBOOK.md: a benchmark for long-context agentic instruction following (arXiv 2607.25398), source: https://arxiv.org/abs/2607.25398
  - 2026-07-13 [benchmark] GDP.pdf paper: grounded multimodal reasoning over professional PDFs (arXiv 2607.11192), source: https://arxiv.org/abs/2607.11192
  - 2026-07 [benchmark] Surge says OpenAI reported GDP.pdf in its GPT-5.6 release (GPT-5.6 Sol 30.7%), source: https://surgehq.ai/blog/openai-gpt-5-6-gdp-pdf-benchmark
  - 2026-06-30 [benchmark] Anthropic's Claude Sonnet 5 system card reports results on Surge's GDP.pdf, source: https://www-cdn.anthropic.com/480e0bb54327b9622282e9c39a83a4f490ed377e/Claude%20Sonnet%205%20System%20Card.pdf
  - 2026-06-08 [research] ComplexConstraints and Beyond: Expert Rubrics for RLVR (arXiv 2606.09118), source: https://arxiv.org/abs/2606.09118
  - 2026-05 [benchmark] Surge launches the Antidote leaderboard, graded by domain experts and pitched against LMArena, source: https://surgehq.ai/blog/introducing-antidote
  - 2026-04-08 [benchmark] Riemann-Bench: A Benchmark for Moonshot Mathematics (arXiv 2604.06802), source: https://arxiv.org/abs/2604.06802
- benchmarks_and_research:
  - DAYJOB (Healthcare, Finance) (environment), 2026-10-01, Claude Opus 5.5: 24.7% healthcare, 23.9% finance, https://surgehq.ai/blog/dayjob
  - GDP.xlsx (benchmark), 2026-10-01, Gemini 4 Argon: 38.3%, https://surgehq.ai/blog/gdp-xlsx
  - Cross-Benchmark Transfer from RL on Agentic Coding Tasks (paper), 2026-10-01, Kimi K2.7 Code: Terminal-Bench 2.1 from 67.4 to 82.0; SWE-Bench Pro from 60.1 to 64.8, https://arxiv.org/abs/2610.00890
  - Tuesday Work Index (leaderboard), 2026-09-24, Live leader: Claude Opus 5.5 at 83.9. Launch leader: Claude Fable 5 at 66.8., https://surgehq.ai/benchmarks
  - Chartography (benchmark), 2026-08-11, Paper: best 45.0% mean pass@1. Live leaderboard: Gemini 4 Argon 71.6%., https://surgehq.ai/blog/chartography
  - Post-Training on Office Work Improves Software Engineering (paper), 2026-08-03, +5.8 pass@1 on SWE-Bench Pro from 363 office tasks (abstract), https://arxiv.org/abs/2608.01604
  - Cross-Benchmark Generalization in Long-Horizon Agents (paper), 2026-07-31, +9.6 Toolathlon, +5.3 tau2-Bench, +5.8 SWE-Bench Pro (abstract), https://arxiv.org/abs/2608.00181
  - HANDBOOK.md (environment), 2026-07-28, Paper: best 36.2%. Live leaderboard: Claude Fable 5.1 38.8%., https://surgehq.ai/blog/handbook-md
  - ComplexConstraints (benchmark), 2026-06-08, Rubric RLVR on Qwen3-4B: +15.5 pp held-out. Live leaderboard: Claude Fable 5.1 77.4%., https://surgehq.ai/blog/complexconstraints-a-benchmark-for-entangled-instruction-following
  - Antidote leaderboard (leaderboard), 2026-05, Live leader: Claude Fable 5 (Adaptive/High) at 1099 Elo, https://surgehq.ai/blog/introducing-antidote
  - Riemann-Bench (benchmark), 2026-04-08, Paper: all frontier models below 10%. Live leaderboard: GPT 5.6 Sol 74.4% (scoring basis not reconciled with the paper)., https://surgehq.ai/blog/riemann-bench-a-benchmark-for-moonshot-mathematics
  - GDP.pdf (benchmark), 2026-04, Paper: best 30.7%. Live leaderboard: GPT 6 Astra 34.2%., https://surgehq.ai/blog/gdp-pdf-can-100b-ai-models-master-the-documents-that-run-the-world
  - EnterpriseBench: CoreCraft (environment), 2026-02-18, GLM 4.6 went from 25.37% to 36.76% after one epoch of training, https://surgehq.ai/blog/enterprisebench-corecraft
  - The Hierarchy of Agentic Capabilities (paper), 2026-01-13, Even the best models fail about 40% of tasks, https://surgehq.ai/blog/rl-envs-real-world
  - Multimodal RewardBench 2 (Meta FAIR; listed on Surge research page) (benchmark), 2025-12-18, Gemini 3 Pro 75-80% accuracy vs >90% for humans, https://arxiv.org/abs/2512.16899
  - AdvancedIF (Meta; Surge-contributed per Surge research page) (benchmark), 2025-11-13, RIFL: 6.7% absolute gain on AdvancedIF, https://arxiv.org/abs/2511.10507
  - GSM8K (OpenAI; listed on Surge research page) (dataset), 2021-10, https://arxiv.org/abs/2110.14168
  - Hemingway-bench (leaderboard), Launch top: Gemini 3 Flash. Live leader: Claude Fable 5 at 1110 Elo., https://surgehq.ai/blog/hemingway-bench-ai-writing-leaderboard
- products:
  - Off-the-shelf frontier data and RL environments: Homepage headline offering: 'New Frontier data and RL environments, off the shelf'., https://surgehq.ai/
  - Post-Training Runs: 'We train frontier models on Surge datasets, then study what they learn.' Published runs use Kimi K2.7, Qwen3.5-122B-A10B, Qwen3-4B-Thinking, GLM-4.7 and GLM-4.6, with experiments dated May to September 2026. Results are Surge's own and name no customers., https://surgehq.ai/post-training
  - Enterprise (Evaluate / Improve / Train): 'Frontier Data for Enterprise': custom benchmarks and production monitoring, diagnosis of system bottlenecks, and training on company-specific workflows. The page shows no customer logos or compliance badges., https://surgehq.ai/enterprise
  - Expert Workforce / human data (RLHF, evaluations): Core managed human-data service using expert raters. Example: the blind side-by-side human evals for Microsoft's MAI-Thinking-1., https://surgehq.ai/workforce
  - Surge benchmark suite and leaderboards: Public leaderboards: Tuesday Work Index, DAYJOB, GDP.pdf/.xlsx, Chartography, HANDBOOK.md, ComplexConstraints, Riemann-bench, EnterpriseBench CoreCraft, Antidote, Hemingway-bench. Some full task sets are available on request via benchmarks@surgehq.ai., https://surgehq.ai/benchmarks
  - surge-python SDK / Surge API: MIT-licensed Python SDK for the Surge AI API., https://github.com/surge-ai/surge-python
- notes_for_researchers:
  - In 2026 Surge has posted 11 arXiv papers, 9 of them between April and October. Recurring authors: Sushant Mehta, Suhaas Garre, Logan Ritchie and Liudas Panavas, with Edwin Chen. The papers cover benchmarks (Riemann-Bench, GDP.pdf, HANDBOOK.md, Chartography, DAYJOB) and transfer studies from RL on Surge environments.
  - Transfer claims, per the paper abstracts: office-work tasks gave +5.8 on SWE-Bench Pro; RL on 1,700 coding tasks took Terminal-Bench 2.1 from 67.4 to 82.0 (Kimi K2.7 Code); expert-rubric RLVR gave +15.5 pp on Qwen3-4B. All are vendor-run studies of the vendor's own data.
  - Anthropic's Claude Sonnet 5 system card says it 'was not able to reproduce Surge's reported numbers' on GDP.pdf. Its scores use a mean criteria pass rate, an Opus 4.7 judge and truncated (not dropped) PDFs, so they are not directly comparable to Surge's leaderboard.
  - Paper and live leaderboard scores can differ a lot. Chartography: 45.0% in the paper vs 71.6% live. Riemann-Bench: below 10% in the paper vs 74.4% live. Cite the source and date you use.
- notes_for_program_managers:
  - Still bootstrapped. Forbes (profile, Oct 2026) says Surge claims no outside funding and that Chen owns about 75%. No round from the July 2025 talks has been disclosed.
  - Confirmed by primary sources: Microsoft (named as human-eval partner for MAI-Thinking-1, 2026-08-12) and Anthropic using GDP.pdf in the Claude Sonnet 5 system card (2026-06-30). Self-claimed by Surge: OpenAI's GPT-5.6 citation of GDP.pdf, the Riemann-bench citation in Anthropic's Fable 5 / Mythos 5 card, and running the human baseline for Anthropic's alignment study.
  - Surge now sells off-the-shelf RL environments, post-training runs and an enterprise eval/training offering alongside its expert workforce. The careers page includes RL Environments Architect and 'Senior Software Engineer, Agentic Coding RL Envs' roles.
  - The careers page lists 59 distinct role pages (accessed 2026-10-05). Most are remote; executive-assistant roles are tied to New York, San Francisco and Seattle.
- notes_for_data_ops:
  - Open benchmark repos (dayjob, handbook, gdp-xlsx) are Apache-2.0. Per the repo READMEs, DAYJOB tasks use the Harbor layout (task.toml, instruction.md, environment/, tests/) with an OpenHands SDK harness. HANDBOOK.md uses Docker mock services and MCP servers.
  - Datasets live under the 'surgeai' Hugging Face org. 'surge-ai' on Hugging Face is an unrelated person.
  - Full task sets for DAYJOB and HANDBOOK.md are available by contacting benchmarks@surgehq.ai. Public repos contain samples only.
  - No public SOC 2 or trust page: surgehq.ai/security and /trust both return 404, and the enterprise page shows no compliance badges.
- sources:
  - https://surgehq.ai/research (accessed 2026-10-05)
  - https://surgehq.ai/ (accessed 2026-10-05)
  - https://surgehq.ai/post-training (accessed 2026-10-05)
  - https://surgehq.ai/enterprise (accessed 2026-10-05)
  - https://surgehq.ai/careers (accessed 2026-10-05)
  - https://surgehq.ai/benchmarks (accessed 2026-10-05)
  - https://surgehq.ai/benchmarks/riemann-bench (accessed 2026-10-05)
  - https://surgehq.ai/benchmarks/gdp-pdf (accessed 2026-10-05)
  - https://surgehq.ai/benchmarks/chartography (accessed 2026-10-05)
  - https://surgehq.ai/benchmarks/handbook (accessed 2026-10-05)
  - https://surgehq.ai/benchmarks/antidote (accessed 2026-10-05)
  - https://surgehq.ai/benchmarks/hemingway-bench (accessed 2026-10-05)
  - https://surgehq.ai/blog/tuesday-frontier-work-index (accessed 2026-10-05)
  - https://surgehq.ai/blog/gdp-xlsx (accessed 2026-10-05)
  - https://surgehq.ai/blog/dayjob (accessed 2026-10-05)
  - https://surgehq.ai/blog/gdp-pdf-can-100b-ai-models-master-the-documents-that-run-the-world (accessed 2026-10-05)
  - https://surgehq.ai/blog/introducing-antidote (accessed 2026-10-05)
  - https://surgehq.ai/blog/hemingway-bench-ai-writing-leaderboard (accessed 2026-10-05)
  - https://surgehq.ai/blog/complexconstraints-a-benchmark-for-entangled-instruction-following (accessed 2026-10-05)
  - https://surgehq.ai/blog/chartography (accessed 2026-10-05)
  - https://surgehq.ai/blog/handbook-md (accessed 2026-10-05)
  - https://surgehq.ai/blog/enterprisebench-corecraft (accessed 2026-10-05)
  - https://surgehq.ai/blog/rl-envs-real-world (accessed 2026-10-05)
  - https://surgehq.ai/blog/openai-gpt-5-6-gdp-pdf-benchmark (accessed 2026-10-05)
  - https://surgehq.ai/blog/anthropic-fable-5-mythos-5-cites-surge-benchmarks (accessed 2026-10-05)
  - https://www-cdn.anthropic.com/480e0bb54327b9622282e9c39a83a4f490ed377e/Claude%20Sonnet%205%20System%20Card.pdf (accessed 2026-10-05)
  - https://surgehq.ai/blog/anthropic-automated-alignment-research (accessed 2026-10-05)
  - https://www.anthropic.com/research/automated-researchers-mitigate-alignment-failures (accessed 2026-10-05)
  - https://surgehq.ai/blog/surge-human-evaluation-mai-thinking-1 (accessed 2026-10-05)
  - https://microsoft.ai/news/introducing-mai-thinking-1/ (accessed 2026-10-05)
  - http://export.arxiv.org/api/query?id_list=2604.06802,2606.09118,2607.11192,2607.25398,2608.00181,2608.01604,2608.10677,2610.00890,2610.01306,2601.09032,2602.16179 (accessed 2026-10-05)
  - https://arxiv.org/abs/2601.09032 (accessed 2026-10-05)
  - https://arxiv.org/abs/2511.10507 (accessed 2026-10-05)
  - https://arxiv.org/abs/2512.16899 (accessed 2026-10-05)
  - https://github.com/surge-ai (accessed 2026-10-05)
  - https://github.com/surge-ai/dayjob (accessed 2026-10-05)
  - https://github.com/surge-ai/handbook (accessed 2026-10-05)
  - https://huggingface.co/surgeai (accessed 2026-10-05)
  - https://github.com/echen (accessed 2026-10-05)
  - https://www.edwinchen.ai (accessed 2026-10-05)
  - https://www.forbes.com/profile/edwin-chen/ (accessed 2026-10-05)
  - https://en.wikipedia.org/wiki/Surge_AI (accessed 2026-10-05)
  - https://news.google.com/rss/search?q=%22Surge+AI%22+funding&hl=en-US&gl=US&ceid=US:en (accessed 2026-10-05)
  - https://surgehq.ai/blog/anthropic-surge-ai-rlhf-platform-train-llm-assistant-human-feedback (accessed 2026-06-07)
  - https://sacra.com/c/surge-ai/ (accessed 2026-06-07)
  - https://www.bloomberg.com/news/articles/2025-07-30/scale-rival-surge-ai-in-talks-for-funding-at-25-billion-value (accessed 2026-06-07)
  - https://getlatka.com/companies/surgehq.ai (accessed 2026-06-07)
  - https://www.linkedin.com/company/surge-ai (accessed 2026-06-07)
  - https://pitchbook.com/profiles/company/472331-53 (accessed 2026-06-07)

## Sepal AI (not ranked, incumbent / infrastructure / open-source)
- slug: sepal-ai
- segment: Commercial vendors
- website: https://www.sepalai.com/
- focus_areas: Enterprise Workflows, Long-Horizon, Math
- positioning: Sepal AI was a YC-backed (S24) San Francisco data-research company that built high-quality training data, expert-graded evaluation benchmarks, and reinforcement-learning environments for frontier LLMs, drawing on a network of 20k+ domain experts (PhDs, finance, medical, STEM). It was acquired by Mercor in February 2026; the team joined Mercor and its RL-environment and human-data work now sits under Mercor. It is therefore no longer ranked as an independent vendor.
- best_fit: Buyers needing expert-validated evaluation environments and RL/training data for complex domains (notably finance/spreadsheet analyst workflows and advanced science), note the team is now part of Mercor following the Feb 2026 acquisition.
- overall_confidence: medium
- founded_year: 2024 [confirmed], source: https://www.ycombinator.com/companies/sepal-ai (accessed 2026-06-07); founded 2024, YC S24
- status: acquired [confirmed], source: https://www.orrick.com/en/News/2026/02/Mercor-Acquires-Sepal-AI and https://www.linkedin.com/posts/mercor-ai_today-were-welcoming-sepal-ai-to-mercor-activity-7424897704905826304-N_PJ (accessed 2026-06-07); Mercor acquired Sepal AI, announced 2026-02-06, corroborated by founder and Mercor LinkedIn posts
- hq_location: San Francisco, USA [confirmed], source: https://www.ycombinator.com/companies/sepal-ai and https://www.linkedin.com/company/sepalai (accessed 2026-06-07)
- other_locations: unknown [unknown]
- distributed_remote: unknown [unknown]
- current_headcount: ~13-15 core team (YC team size 15; getlatka 13 for 2024). LinkedIn public snippet shows 51-200 band (likely includes contract experts) [reported], source: https://www.ycombinator.com/companies/sepal-ai (team size 15), https://getlatka.com/companies/sepalai.com (13, 2024) and https://www.linkedin.com/company/sepalai (51-200 band) (accessed 2026-06-07)
- headcount_band: unknown [unknown], source: Conflicting signals: YC team size 15 (suggests 11-50) vs LinkedIn 51-200 band (likely inflated by contract experts). Cannot cleanly resolve to a single band, so left unknown. (https://www.ycombinator.com/companies/sepal-ai, https://www.linkedin.com/company/sepalai, accessed 2026-06-07)
- headcount_growth: unknown [unknown]
- open_roles_count: unknown (sepalai.com/careers returns 404; Sepal no longer hires as a standalone company, and hiring presumably runs through Mercor) [estimated], source: https://www.sepalai.com/careers (accessed 2026-10-05)
- has_researchers: yes [reported], source: https://www.ycombinator.com/companies/sepal-ai and https://www.orrick.com/en/News/2026/02/Mercor-Acquires-Sepal-AI (accessed 2026-06-07); hires Applied Research Engineers, runs specialized research projects, engages 20k+ PhDs/domain experts
- researcher_count: unknown [unknown]
- researcher_backgrounds: Co-founders ex-Turing (built and scaled the LLM-trainer business; Robi Lin scaled trainers from 50 to 800+; Kat Hu managed 500+ AI trainers); Co-founder Robi Lin formerly at Bain & Co.; co-founder Kat Hu formerly a McKinsey consultant; Co-founder Fedor Paretsky: early engineer at Vercel, also Newfront and Mosaic; now works on RL environments after the Mercor acquisition (self-published on fedor.me); Engages a self-claimed network of 20k+ academic PhDs and domain professionals (STEM, medical, finance, business) [reported], source: https://fedor.me (accessed 2026-10-05)
- published_papers_or_benchmarks: SheetBench-50 (released 2025-10-01): public financial-analyst-grade spreadsheet benchmark of 50 tasks (29 data prep/hygiene, 16 derivations/extraction, 5 modeling/forecasts), built by HUD and Sepal AI together; tasks blind-validated by three finance professionals each, from PwC, Cisco, Charles Schwab and Fannie Mae [confirmed], source: https://www.hud.ai/case-studies/sheetbench-50 (accessed 2026-10-05)
- total_raised: ~$500K disclosed (single pre-seed round, Sep 2024) per aggregators; YC profile separately states 'several million dollars from leading investors' (unresolved conflict) [reported], source: https://www.crunchbase.com/organization/sepal-ai, https://startupintros.com/orgs/sepal-ai, https://dealigence.vc/company/sepal-ai (all report $500K, 1 round) vs https://www.ycombinator.com/companies/sepal-ai ('several million') (accessed 2026-06-07)
- last_round: Pre-seed, ~$500K, 2024-09-25 (per Crunchbase aggregator; not independently confirmed by a primary announcement) [reported], source: https://www.crunchbase.com/funding_round/sepal-ai-pre-seed--fda22c6a and https://dealigence.vc/company/sepal-ai (accessed 2026-06-07)
- notable_investors: Y Combinator; Metaplanet Holdings; SID Venture Partners; Sterling Road; Team Ignite Ventures [reported], source: https://startupintros.com/orgs/sepal-ai and https://www.crunchbase.com/organization/sepal-ai (via search) (accessed 2026-06-07); consistent across multiple funding aggregators but no primary press release
- valuation: unknown [unknown]
- revenue_signals: Third-party (getlatka) claims ~$2M revenue with a 13-person team in 2024, single unverified source, not vendor-confirmed [reported], source: https://getlatka.com/companies/sepalai.com (accessed 2026-06-07)
- soc2: unknown [unknown]
- other_certifications: unknown [unknown]
- security_page: unknown [unknown]
- what_they_sell: mixed [confirmed], source: https://www.ycombinator.com/companies/sepal-ai and https://www.orrick.com/en/News/2026/02/Mercor-Acquires-Sepal-AI (accessed 2026-06-07); training data, evaluation benchmarks, and RL environments backed by a 20k+ expert network
- open_source: no [estimated], source: No public OSS repos identified for Sepal itself; product is a data/eval/RL-environment service. SheetBench-50 is hosted on HUD's HuggingFace org (hud-evals), not Sepal's own OSS
- license: unknown [unknown]
- deployment_model: unknown [unknown]
- maturity: Discontinued as a standalone offering (absorbed into Mercor after the Feb 2026 acquisition) [estimated], source: https://www.ycombinator.com/companies/sepal-ai (accessed 2026-10-05)
- notable_customers: Top AI research labs (unnamed; frontier-lab ties referenced in Mercor acquisition rationale) (self-claimed, frontier-lab tie); Multiple Fortune 500 companies (unnamed) (self-claimed); HUD (co-builder/collaboration partner on SheetBench-50, NOT a customer) (verified) [reported], source: https://www.ycombinator.com/companies/sepal-ai (Fortune 500 + AI-lab claims, self-claimed) and https://www.hud.ai/case-studies/sheetbench-50 (HUD collaboration on benchmark, verified as a partner not a customer) (accessed 2026-06-07)
- founding_team:
  - Robi (Robert) Lin, Co-Founder, At Turing.com, built enterprise workflow products and scaled the LLM-trainer business from 50 to 800+ developers; previously at Bain & Co.
  - Kat Hu, Co-Founder, Built the go-to-market for Turing's foundational LLM-trainer business and managed 500+ AI trainers; previously at McKinsey.
  - Fedor Paretsky, Co-Founder, Early engineer at Vercel; built foundational infrastructure at Newfront; also lists Mosaic. Joined Mercor through the Sepal acquisition, per his personal site, which says he is 'currently working on RL environments'.
- benchmarks_and_research:
  - SheetBench-50 (benchmark), 2025-10-01, The case study publishes no headline score. It links to a HUD leaderboard and an example Claude scorecard., https://www.hud.ai/case-studies/sheetbench-50
- products:
  - Sepal AI expert data, evaluation and RL-environment services (legacy, now inside Mercor): Expert-sourced training data, expert-graded evaluation benchmarks and RL environments. The YC profile also describes a 'Cloud-Native Agent Dataset Factory' and a self-claimed network of 20k+ experts across STEM and professional services. Since the February 2026 acquisition these are no longer offered as a standalone product, and sepalai.com returned HTTP 404 on 2026-10-05., https://www.ycombinator.com/companies/sepal-ai
- notes_for_researchers:
  - Sepal's main public artifact is SheetBench-50 (2025-10-01, co-built with HUD): 50 expert-validated financial spreadsheet tasks with cell-level verification. It is on Hugging Face as hud-evals/SheetBench-50. The dataset card does not credit Sepal and notes a migration to HUD v6, so legacy v5-format rows may not run on current HUD.
  - No Sepal-authored papers or open-source repositories found.
- notes_for_program_managers:
  - Acquired by Mercor (announced 2026-02-06; terms undisclosed). Engagements now go through Mercor, not Sepal.
  - Mercor's announcement calls Sepal 'a trusted partner to frontier AI labs and early enterprise teams' but names no labs. The YC profile says Sepal had 'multiple Fortune 500s and top AI research labs as paying customers' (self-claimed) and links a Sepal X post saying it supported Anthropic's Claude 3.7 Sonnet launch with pre-launch testing. That claim is self-claimed and could not be opened for verification (X returned 402).
  - Funding: aggregators report one ~$500K pre-seed round (2024), while the YC profile says 'several million dollars'. Nothing new appeared in the April to October 2026 window, and no Sepal-branded news was found.
- notes_for_data_ops:
  - sepalai.com, including /careers, returned HTTP 404 on 2026-10-05, so there are no Sepal docs, APIs or security pages to evaluate. Use Mercor's materials instead.
  - No SOC 2 or other certification was ever published under the Sepal brand.
- sources:
  - https://www.ycombinator.com/companies/sepal-ai (accessed 2026-10-05)
  - https://www.orrick.com/en/News/2026/02/Mercor-Acquires-Sepal-AI (accessed 2026-10-05)
  - https://www.linkedin.com/posts/mercor-ai_today-were-welcoming-sepal-ai-to-mercor-activity-7424897704905826304-N_PJ (accessed 2026-10-05)
  - https://www.sepalai.com/ (accessed 2026-10-05)
  - https://www.hud.ai/case-studies/sheetbench-50 (accessed 2026-10-05)
  - https://huggingface.co/datasets/hud-evals/SheetBench-50 (accessed 2026-10-05)
  - https://fedor.me (accessed 2026-10-05)
  - https://github.com/f3d0r (accessed 2026-10-05)
  - https://www.mercor.com/blog/ (accessed 2026-10-05)
  - https://www.sepalai.com/careers (accessed 2026-06-07)
  - https://www.crunchbase.com/organization/sepal-ai (accessed 2026-06-07)
  - https://www.crunchbase.com/funding_round/sepal-ai-pre-seed--fda22c6a (accessed 2026-06-07)
  - https://startupintros.com/orgs/sepal-ai (accessed 2026-06-07)
  - https://www.linkedin.com/company/sepalai (accessed 2026-06-07)
  - https://getlatka.com/companies/sepalai.com (accessed 2026-06-07)
  - https://pitchbook.com/profiles/company/639950-14 (accessed 2026-06-07)

## Morph (not ranked, incumbent / infrastructure / open-source)
- slug: morph
- segment: Adjacent: execution infrastructure
- website: https://www.morph.so
- focus_areas: Coding
- positioning: Morph (Morph Labs) provides snapshot-based VM compute for AI agents via its Infinibranch / Liquid Metal technology, which can snapshot, branch, and restore entire computational environments in roughly 100-250ms to enable massively parallel, reversible ('Git for compute') agent rollouts, evaluations, and reasoning-time branching. It markets the platform (Morph Cloud) as infrastructure for running and scaling agent/RL verification environments rather than as an RL-environment dataset vendor itself.
- best_fit: Teams needing to fork, snapshot, and run thousands of parallel/reversible sandboxed VM environments for agent rollouts, evals, and verification at scale.
- overall_confidence: medium
- founded_year: 2023 [reported], source: https://tracxn.com/d/companies/morph-labs (accessed 2026-06-07); https://www.cbinsights.com/company/morph-labs (accessed 2026-06-07)
- status: active [confirmed], source: https://cloud.morph.so/docs/developers (accessed 2026-06-07); https://x.com/morph_labs/status/1966223824078479667 (Math Inc incubation announcement, accessed 2026-06-07)
- hq_location: San Francisco, USA [reported], source: https://www.cbinsights.com/company/morph-labs (accessed 2026-06-07); https://tracxn.com/d/companies/morph-labs (accessed 2026-06-07)
- other_locations: unknown [unknown]
- distributed_remote: yes [reported], source: https://www.morph.so/careers (accessed 2026-06-07), remote roles indicated
- current_headcount: ~9-11 employees (as of 2026) [estimated], source: https://rocketreach.co/morph-labs-management_b7384e18c7e2a4d3 (accessed 2026-06-07, ~9); https://pitchbook.com/profiles/company/44794-09 (accessed 2026-06-07, ~11)
- headcount_band: 1-10 [estimated], source: https://rocketreach.co/morph-labs-management_b7384e18c7e2a4d3 (accessed 2026-06-07), borderline with 11-50 per PitchBook (~11)
- headcount_growth: unknown [unknown]
- open_roles_count: unknown [unknown], source: https://www.morph.so/careers (accessed 2026-06-07), careers page exists and is actively recruiting researcher-engineers, but exact count not extractable (JS-rendered)
- has_researchers: yes [confirmed], source: https://www.morph.so/blog/liquid-metal (accessed 2026-06-07); https://github.com/morph-labs (accessed 2026-06-07)
- researcher_count: unknown (small team; at least CEO Jesse Han and Chief Scientist Christian Szegedy plus members of technical staff) [estimated], source: https://rocketreach.co/morph-labs-management_b7384e18c7e2a4d3 (accessed 2026-06-07)
- researcher_backgrounds: Jesse Han, founder/CEO; PhD work in formal mathematics (formalized independence of the continuum hypothesis in Lean); ex-OpenAI / Microsoft Research / AWS per profiles; Christian Szegedy, Chief Scientist; ex-xAI co-founder (Grok-3 code reasoning lead), ex-Google Research (deep learning/Inception, BatchNorm, adversarial examples, autoformalization) [reported], source: https://www.theinformation.com/briefings/xai-cofounder-joins-ai-code-startup-morph-chief-scientist (accessed 2026-06-07); https://www.startuphub.ai/ai-news/ai-video/2025/morph-labs-unveils-liquid-metal-powering-the-agentic-future (accessed 2026-06-07)
- published_papers_or_benchmarks: Trinity: autoformalization system (Morph Labs, announced 2025-06-12; original morph.so post now returns 410); Trinity/Gauss Lean verification environments infrastructure, built with Math Inc on Morph Cloud; MiniMuSiQue: multi-hop QA eval dataset (Hugging Face, Apache-2.0, 2023); Morph Prover v0 7B: early open model, no longer publicly available on Hugging Face [reported], source: https://github.com/orgs/morph-labs/repositories (accessed 2026-10-05)
- total_raised: unknown [unknown], source: Tracxn lists Morph Labs (Jesse Han) as 'Unfunded / has not raised any funding rounds yet' (accessed 2026-06-07). A seed/angel investment by Christian Szegedy is reported but amount undisclosed. NOTE: the widely-cited '$19-20M seed' (Crunchbase round 875f78ba, led by Dragonfly Capital, 2024-03-20) belongs to the unrelated Morph Ethereum L2 / blockchain project, NOT this company.
- last_round: unknown [unknown]
- notable_investors: Christian Szegedy (reported seed/angel investor; also Chief Scientist), amount undisclosed [reported], source: https://www.cogniscendo.com/p/the-superhuman-mathematician-5d1b (accessed 2026-06-07)
- valuation: unknown [unknown]
- revenue_signals: unknown [unknown]
- soc2: claimed-unverified [reported], source: https://trust.delve.co/morph (accessed 2026-06-07), trust portal snippet indicates a SOC 2 Type 2 report for 2025, but the portal is hosted by Delve, which in March 2026 was the subject of a TechCrunch/Substack investigation alleging fabrication of ~494 SOC 2 reports (allegations denied/unproven). Direct fetch returned 403/429; no independent audit-firm confirmation obtained. Treat as claimed-unverified pending independent verification.
- other_certifications: unknown [unknown], source: https://trust.delve.co/morph (accessed 2026-06-07), no ISO 27001 / HIPAA confirmed
- security_page: https://trust.delve.co/morph [reported], source: https://trust.delve.co/morph (accessed 2026-06-07)
- what_they_sell: infra [confirmed], source: https://cloud.morph.so/docs/developers (accessed 2026-06-07); https://www.morph.so/blog/infinibranch/ (accessed 2026-06-07)
- open_source: partial (SDKs only) [confirmed], source: https://github.com/orgs/morph-labs/repositories (accessed 2026-10-05)
- license: Apache-2.0 (morph-python-sdk, morph-typescript-sdk). Core platform proprietary. [confirmed], source: https://github.com/orgs/morph-labs/repositories (accessed 2026-10-05)
- deployment_model: managed-hosted (cloud platform, accessed via Python/TypeScript SDK, CLI, and API keys) [confirmed], source: https://cloud.morph.so/docs/developers (accessed 2026-06-07)
- maturity: GA [estimated], source: https://cloud.morph.so/docs/developers (accessed 2026-06-07), public self-serve signup, SDKs, and docs available; no explicit GA label found
- notable_customers: Math Inc (Math, Inc.), used Morph Cloud / Infinibranch to scale Lean verification environments for its Gauss autoformalization agent; company incubated at Morph (closely tied entity, not arm's-length customer) (self-claimed) [reported], source: https://x.com/morph_labs/status/1966223824078479667 (accessed 2026-06-07); https://www.math.inc/gauss (accessed 2026-06-07)
- founding_team:
  - Jesse Michael Han, Founder & CEO, Morph Labs. His GitHub profile now lists Math Inc as his company; he leads Math, Inc. as well., Formal-mathematics and Lean researcher. He formalized the independence of the continuum hypothesis in Lean (the flypitch project, pinned on his GitHub). He co-authored OpenAI's "Formal Mathematics Statement Curriculum Learning" (2022, with Polu and Sutskever). Aggregator profiles also list Microsoft Research and AWS; those are unverified.
  - Christian Szegedy, Chief Scientist (joined 2025, reported; also reported to be a seed investor). Current status not re-verified in this window., xAI co-founder and former Google Research scientist (Inception, Batch Normalization, adversarial examples). Early autoformalization researcher.
- recent_developments (last 6 months):
  - 2026-10-05 [open-source] Morph pulls most of its public research and open-source material: GitHub org down to two SDK repos, Morph Prover model and Trinity post gone, source: https://github.com/orgs/morph-labs/repositories
  - 2026-09-01 [product] morphcloud Python SDK/CLI v0.1.115 released, after 0.1.112 (2026-07-19), 0.1.113 (2026-07-29) and 0.1.114 (2026-08-30), source: https://pypi.org/pypi/morphcloud/json
  - 2026-04-05 [partnership] Math Inc's open-source OpenGauss pins a Morph-hosted one-click launch (morph.new/opengauss-0-2-2) in its README, source: https://github.com/math-inc/OpenGauss
- benchmarks_and_research:
  - Trinity (autoformalization system) (paper), 2025-06-12, Reported to have nearly finished formalizing de Bruijn's result that the abc conjecture is true almost always. Human mathematicians structured the problem. This is a third-party account of a self-reported result., https://www.cogniscendo.com/p/the-superhuman-mathematician-5d1b
  - MiniMuSiQue (dataset), 2023-11-30, https://huggingface.co/datasets/morph-labs/MiniMuSiQue
  - Gauss / Trinity environments (with Math Inc) (environment), https://www.math.inc/gauss
- products:
  - Morph Cloud (Infinibranch VM primitives): Snapshots and instances that can be snapshotted, branched and restored in under 250ms, according to the docs. Managed through the Python and TypeScript SDKs, the CLI and API keys. Free accounts are available at cloud.morph.so., https://cloud.morph.so/docs/developers
  - Morph Devboxes: Cloud development workspaces with VSCode/Cursor integration for development, prototyping and team collaboration. The platform also offers GitHub Actions on Morph-managed self-hosted runners and EFS, a shareable filesystem that can be mounted across Linux hosts., https://cloud.morph.so/docs/developers
  - Infinibranch Liquid Metal: Announced as "Infinibranch Liquid Metal on Morph Cloud: burst scalability for the AI agent era". The launch month comes from the sitemap lastmod (2025-10-14), not a stated publication date., https://www.morph.so/blog/liquid-metal
  - morphcloud Python SDK/CLI: Apache-2.0 SDK and CLI for Morph Cloud VMs. Latest version is 0.1.115, released 2026-09-01., https://pypi.org/project/morphcloud/
  - morph-typescript-sdk: Apache-2.0 TypeScript SDK for Morph Cloud VMs. Last updated 2026-03-05., https://github.com/morph-labs/morph-typescript-sdk
- notes_for_researchers:
  - Infinibranch snapshot/branch/restore (under 250ms per the docs) supports reversible, branching agent rollouts and test-time search over full VM state.
  - Math Inc credits Morph Cloud with scaling the Lean verification environments for Gauss to thousands of concurrent agents. This is a claim from a closely tied company. OpenGauss (MIT) pins a Morph-hosted launch link, but that link returned 404 on 2026-10-05.
  - Morph's earlier public artifacts (SWELancer-Benchmark setup, open-r1 reproduction, Morph Prover v0 7B, the Trinity post) are no longer public. MiniMuSiQue on Hugging Face is still available.
- notes_for_program_managers:
  - Small team (about 9 to 11 people per aggregators) led by Jesse Han. xAI co-founder Christian Szegedy was reported as Chief Scientist in 2025, and his current status was not re-verified.
  - No priced or disclosed funding round found for Morph Labs. The $19-20M Dragonfly-led seed belongs to the unrelated Morph Ethereum L2.
  - The only public customer reference is Math Inc, which was incubated at Morph and is led by the same CEO, so it is not arm's-length. No frontier-lab ties have been verified.
  - Watch for a change in focus: the CEO's GitHub now lists Math Inc, no Morph blog posts have appeared since October 2025, and public research material was removed between June and October 2026. SDK releases did continue through September 2026.
- notes_for_data_ops:
  - Self-serve accounts at cloud.morph.so. Python SDK/CLI (pip install morphcloud) and TypeScript SDK, both Apache-2.0. GitHub Actions runs on Morph-managed self-hosted runners.
  - The SDK supports Docker containers, SSH execution, file transfer and HTTP service exposure from VMs. EFS provides shared filesystems across hosts.
  - SOC 2 Type 2 is claimed only on the Delve-hosted trust portal (trust.delve.co/morph), which returned 429. Treat it as claimed-unverified.
- sources:
  - https://www.morph.so/sitemap.xml (accessed 2026-10-05)
  - https://cloud.morph.so/docs/developers (accessed 2026-10-05)
  - https://github.com/orgs/morph-labs/repositories (accessed 2026-10-05)
  - https://api.github.com/orgs/morph-labs (accessed 2026-10-05)
  - https://github.com/morph-labs/SWELancer-Benchmark (accessed 2026-10-05)
  - https://github.com/morph-labs/open-r1 (accessed 2026-10-05)
  - https://huggingface.co/api/organizations/morph-labs/overview (accessed 2026-10-05)
  - https://huggingface.co/morph-labs/morph-prover-v0-7b (accessed 2026-10-05)
  - https://huggingface.co/datasets/morph-labs/MiniMuSiQue (accessed 2026-10-05)
  - https://www.morph.so/blog/trinity (accessed 2026-10-05)
  - https://pypi.org/pypi/morphcloud/json (accessed 2026-10-05)
  - https://github.com/math-inc/OpenGauss (accessed 2026-10-05)
  - https://api.github.com/repos/math-inc/OpenGauss/commits?path=README.md&per_page=30 (accessed 2026-10-05)
  - https://api.github.com/repos/math-inc/OpenGauss (accessed 2026-10-05)
  - https://morph.new/opengauss-0-2-2 (accessed 2026-10-05)
  - https://www.math.inc/gauss (accessed 2026-10-05)
  - https://github.com/jesse-michael-han (accessed 2026-10-05)
  - https://jesse-michael-han.github.io/ (accessed 2026-10-05)
  - https://arxiv.org/abs/2202.01344 (accessed 2026-10-05)
  - https://www.cogniscendo.com/p/the-superhuman-mathematician-5d1b (accessed 2026-10-05)
  - https://www.theinformation.com/briefings/xai-cofounder-joins-ai-code-startup-morph-chief-scientist (accessed 2026-10-05)
  - https://x.com/ChrSzegedy (accessed 2026-10-05)
  - https://www.morph.so/blog/liquid-metal (accessed 2026-10-05)
  - https://www.morph.so/ (accessed 2026-06-07)
  - https://www.morph.so/blog/infinibranch/ (accessed 2026-06-07)
  - https://www.startuphub.ai/ai-news/ai-video/2025/morph-labs-unveils-liquid-metal-powering-the-agentic-future (accessed 2026-06-07)
  - https://github.com/morph-labs (accessed 2026-06-07)
  - https://x.com/morph_labs/status/1966223824078479667 (accessed 2026-06-07)
  - https://trust.delve.co/morph (accessed 2026-06-07)
  - https://tracxn.com/d/companies/morph-labs/__8ZZWlPNWnRct4CJf-JJjU940HzzVxh1CEZxBQBG6tXg (accessed 2026-06-07)
  - https://pitchbook.com/profiles/company/44794-09 (accessed 2026-06-07)
  - https://rocketreach.co/morph-labs-management_b7384e18c7e2a4d3 (accessed 2026-06-07)
  - https://www.cbinsights.com/company/morph-labs (accessed 2026-06-07)
  - https://www.morph.so/careers (accessed 2026-06-07)
  - https://www.morphllm.com/ (accessed 2026-06-07)