Collinear
CommercialCollinear AI runs a Simulation Lab of sandboxed, stateful environments with simulated users (NPCs), tools, tasks and verifiers for agent RL training and evaluation. It now positions itself mainly as a supplier of training data, environments and evals to frontier labs across cybersecurity, software engineering, computer use and complex agent behaviour. Its CWE-bench defensive-cyber benchmark is part of the Artificial Analysis Cyber Index.
What's new
- 28 Sep 2026benchmark
CWE-bench v1 released: 120 tasks, 73 CWEs, 8 languages
v1 has 120 held-out tasks covering 73 CWEs, all 10 OWASP Top 10 2025 categories and 8 languages, tested with 13,000+ agent rollouts. Agents run with no internet access during tasks; the post notes that one model still tried to get out using its own API key. Fixes are judged by a three-judge panel from different model families (GPT-5.4, Claude Sonnet 5, Gemini 3.8 Flash). The blog reports Grok 4.7 and GPT-6 Astra at 68% pass@1 and Claude Opus 5.5 at 67% for about a quarter of the cost ($0.79 vs about $2.80 per rollout).
Why it matters: The held-out cyber eval is maturing. cwe-bench.com also offers a separate training corpus of 4,000+ tasks across 215 CWEs (vendor-stated) and says eval tasks never appear in training deliveries.
- 2 Sep 2026benchmark
Collinear launches CWE-bench, a defensive cybersecurity benchmark for coding agents
The launch version had 100 held-out agentic audit-and-patch tasks across 54 CWE types and all 10 OWASP Top 10 2025 categories. Fable 5 led at 47% pass@1 at maximum reasoning, and 18 of the 100 tasks were unsolved by every model. Some tasks contain newly introduced vulnerabilities that published fixes do not cover, so memorised patches are not enough.
Why it matters: This is Collinear's flagship public benchmark and marks its move into cyber RL environments and data.
- Sep 2026partnership
CWE-bench included in the Artificial Analysis Cyber Index as CWE-Bench-AA
Artificial Analysis credits Collinear AI with developing CWE-bench and runs its own implementation, CWE-Bench-AA. That is one of three equally weighted evaluations in its Cyber Index, alongside DeepsecBench-AA and CyberGym-E2E-AA. The AA page carries no date. Collinear's 2026-09-28 v1 post confirms the inclusion, and cwe-bench.com says results are 'independently evaluated and published by Artificial Analysis'.
Why it matters: A third party has adopted a Collinear benchmark independently, which is a meaningful credibility signal.
- 30 Apr 2026research
Newsletter #11: exec dinner series, Sunnyvale research social, push toward agentic NPCs
Exec dinners on multi-turn evals drew research leaders from Apple, IBM Research, Two Sigma, NVIDIA, Datadog Research, Wells Fargo and Google. A Q2 research social in Sunnyvale had researchers from GDM, NVIDIA, Apple and xAI. Collinear is building NPCs with behavioural traits, agency, secrets and limited context into Simulation Lab. It says frontier models that 'were comfortably solving our environments a quarter ago are now hitting walls' (self-claimed).
Why it matters: Shows the direction of the environment roadmap: multi-agent work and stateful, realistic NPCs. Attending an event does not make a company a customer.
- 2 Apr 2026product
Collinear launches self-serve SimLab CLI for agent staging and testing
Blog post announcing a self-serve SimLab CLI. It offers sandboxed, isolated containers, pre-built simulators (e.g. Workday, Salesforce) or custom tools, programmatic task and verifier generation for long-horizon workflows, deterministic (non-LLM-judge) evaluation, and parallel rollouts. The post points to github.com/collinear-ai/simlab, but that repo returned 404 on 2026-10-05, so whether it is publicly available now is unverified.
Why it matters: This is the first self-serve way to use Simulation Lab and opens the product beyond managed engagements. Teams should check availability with the vendor.
- 1 Apr 2026research
YC-Bench paper posted to arXiv; benchmark open-sourced under MIT
YC-Bench simulates one year of running a startup, which takes hundreds of turns, to test whether agents stay strategically coherent. Of 12 models, only 3 consistently ended above the $200K starting capital. Claude Opus 4.6 led with $1.27M in average final funds and GLM-5 was second at $1.21M with much lower inference cost. The paper says adversarial-client detection is the main failure mode, accounting for 47% of bankruptcies. The code is MIT-licensed at github.com/collinear-ai/yc-bench.
Why it matters: This is an open, reproducible long-horizon agent benchmark from the vendor. Researchers can use it to evaluate planning and consistency.
Founding team
What practitioners should know
For researchers
- Publishes on arXiv regularly: YC-Bench (Apr 2026), TraitBasis (Oct 2025), Valley of Code Reasoning (Oct 2025, company lists it as NeurIPS), CatAttack (Mar 2025), VERITAS (Nov 2024) and self-rationalizing judges (Oct 2024).
- Open-source code on github.com/collinear-ai: yc-bench, spider (on/off-policy distillation, MIT), tau-trait (MIT) and CatAttack (MIT).
- The CWE-bench eval set is private to Collinear. Public scores are on cwe-bench.com, and the Artificial Analysis Cyber Index runs an independent implementation (CWE-Bench-AA).
- Current research direction is NPCs with traits, agency, secrets and limited context inside Simulation Lab, plus 'hard but fair' verifier design (blog posts from May to September 2026).
For program managers
- The media-kit boilerplate now reads: 'Collinear AI provides data to frontier labs for faster model improvement', covering cybersecurity, SWE, computer use and complex agent behaviour.
- An AWS case study (undated) quotes co-founder/CPO Soumyadeep Bakshi saying Collinear works with 'three of the world's top four AI labs' and with Amazon. These are self-claimed, and the labs are not named.
- The About page now lists B Capital, Storm Ventures and Hitachi Ventures alongside Engineering Capital and 112 Capital; Firestreak is no longer shown. No round has been announced. Caplight (an aggregator) reports about $10M raised, seed-stage.
- CWE-bench is part of the Artificial Analysis Cyber Index, which counts as third-party validation of the benchmark, not of customer work.
- Locations: Sunnyvale HQ, San Francisco office opening, and a Bengaluru team per the careers page.
For data ops
- The Trust Centre (Sprinto-hosted, trustcentre.collinear.ai) lists SOC 2 'Compliant', ISO 42001 v2023 'Compliant' and VAPT complete. The SOC 2 type (I or II) is not stated publicly; reports are presumably available on request.
- Docs are on Mintlify at docs.collinear.ai, with an llms.txt index. Integration formats are not documented in the intro pages.
- CWE-bench v1 runs agents in a sandbox with no internet access during tasks, graded by programmatic verifiers plus a three-judge panel across model families. Eval tasks are kept out of training deliveries.
- AWS case study: Collinear's RL training workflow runs on Amazon SageMaker AI (vendor/AWS-stated).
Benchmarks & research
Products
Scale & velocity
Research depth
Capital
Security & compliance
Product
Buyer analysis
How we verified this
Related vendors
Sources
- www.collinear.ai/ · 2026-10-05, Homepage: Simulation Lab, task libraries ('1,000+ tasks off the shelf'), AICPA SOC / ISO 42001 badges, Trust Centre link; no customer logos
- www.collinear.ai/about-us · 2026-10-05, Founders (Rajani, Bakshi) with headshots, investors, Sunnyvale HQ, SF office opening
- trustcentre.collinear.ai/ · 2026-10-05, Sprinto Trust Centre: SOC 2 Compliant, ISO 42001 v2023 Compliant, VAPT complete (read from raw HTML; WebFetch returned 403)
- www.collinear.ai/research · 2026-10-05, Papers and benchmarks list
- www.collinear.ai/careers · 2026-10-05, San Francisco, Sunnyvale and Bengaluru locations; roles not itemised
- www.collinear.ai/media-kit · 2026-10-05, Boilerplate ('provides data to frontier labs'), CEO bio
- cwe-bench.com/ · 2026-10-05, CWE-bench v1 details, live leaderboard, private held-out set, 4,000+ task / 215-CWE training corpus
- blog.collinear.ai/p/cwe-bench · 2026-10-05, CWE-bench launch post, 2026-09-02
- blog.collinear.ai/p/cwe-bench-v1 · 2026-10-05, CWE-bench v1 post, 2026-09-28
- www.prweb.com/releases/collinear-ai-launches-cwe-bench-to-test-frontier- · 2026-10-05, CWE-bench launch press release, 2026-09-02
- artificialanalysis.ai/evaluations/cwe-bench-aa · 2026-10-05, Artificial Analysis implementation of CWE-bench, credited to Collinear; part of the AA Cyber Index
- arxiv.org/abs/2604.01212 · 2026-10-05, YC-Bench paper, submitted 2026-04-01
- github.com/collinear-ai/yc-bench · 2026-10-05, YC-Bench repo, MIT licence
- github.com/collinear-ai · 2026-10-05, GitHub org: yc-bench, spider, tau-trait, CatAttack, simulations
- blog.collinear.ai/archive · 2026-10-05, Blog archive, used to check for missed announcements from April to October 2026
- blog.collinear.ai/p/simlab-the-self-serve-staging-playground · 2026-10-05, Self-serve SimLab CLI announcement, 2026-04-02
- blog.collinear.ai/p/collinear-newsletter-10-notes-on · 2026-10-05, Newsletter #10, 2026-03-24 (outside window): new Sunnyvale office, TraitBasis integrated into Together Evals
- blog.collinear.ai/p/collinear-newsletter-11-notes-on · 2026-10-05, Newsletter #11, 2026-04-30: exec dinners, research social, NPC research
- aws.amazon.com/solutions/case-studies/collinear-sagemaker/ · 2026-10-05, AWS case study (undated): Bakshi (Cofounder & CPO) quote; self-claimed Amazon and 3 of top 4 AI labs
- www.caplight.com/company/collinear-ai · 2026-10-05, Aggregator: ~$10M total raised, seed investors
- docs.collinear.ai/introduction · 2026-10-05, Product docs (Mintlify)
- blog.collinear.ai/p/rl-env-as-a-service · 2026-06-07, Environment-as-a-service positioning; API endpoints; domains (product dev, SWE, computer use, web)
- www.linkedin.com/company/collinearai · 2026-06-07, Public page: 11-50 employees (~28), founded 2024, HQ Mountain View / Sunnyvale CA, Software Development
- tracxn.com/d/companies/collinearai/__afhJh0xW8gZpWYbw5rCoeqG-KD4fTP3wJr3 · 2026-06-07, Founded 2023, ~18-23 employees (Jan-Apr 2026), lists as unfunded, conflicts with investor lists
- pitchbook.com/profiles/company/550057-87 · 2026-06-07, Investors: Khasm Labs, 11.2 Capital, Engineering Capital, Firestreak Ventures, ISV Startup Springboard (via search snippet)
- www.linkedin.com/in/nazneenrajani · 2026-06-07, Founder/CEO background: ex-Hugging Face Research Lead, ex-Salesforce, PhD UT Austin
- www.crunchbase.com/organization/collinear-ai · 2026-06-07, Crunchbase profile exists (HTTP 403, not directly accessible)