ComputeSDK Benchmarks

The default platform for cloud infrastructure benchmarks.

ComputeSDK Benchmarks publishes reproducible, provider-neutral performance measurements for the infrastructure that AI agents and developer platforms run on: sandboxes, object storage, headless browsers, and AI gateways. Every test is automated, every result is committed as JSON, and the methodology is public.

Live leaderboards and full methodology: https://www.computesdk.com/benchmarks

For LLMs and agents: see llms.txt for a machine-readable index of this repo, and results/schema.json for the JSON Schema of every benchmark result file.

Writing and running benchmarks

• WRITING_BENCHMARKS.md — step-by-step guide to authoring benchmarks.
• examples/ — runnable *.bench.ts examples covering every benchsdk capability.
• packages/create-bench/README.md — scaffold a new benchmark project in one command.
• packages/benchsdk-runner/README.md — runner API and bench run reference.
• .agents/skills/benchsdk-cli/SKILL.md — full bench CLI reference (auth, org, runs, results, artifacts, export).

Partners

Our partners support our independent benchmarks.

_{★ SILVER}

   

_{+ BRONZE}

           

_{BENCHMARKS POWERED BY}

Become a sponsor →

What We Measure

We benchmark the infrastructure that AI agents and developer platforms rely on:

• Sandboxes — concurrent Time to Interactive (TTI): API request to first successful command inside a fresh sandbox; Dax: a pinned OpenCode clone/install/typecheck workload.
• Object storage — upload/download latency and throughput across providers and file sizes.
• Headless browsers — session creation, navigation, and step throughput.
• AI gateways — cold/warm connection latency and time to first token.

Methodology

Each suite uses a shared open workload across its participating providers, with provider-specific setup and resource mappings. Scored benchmarks declare per-metric ceilings and weights, not one universal ceiling: for example, sandbox TTI uses 10,000 ms and Dax uses 200,000 ms. Ceilings set the scoring scale; execution deadlines and workload checks decide pass/fail. Scored metrics report median, P95, and P99, and the weighted metric score is multiplied by the benchmark's success rate. Some probes do not declare composite scoring. Tests run on suite-specific schedules, and published results are committed as JSON to this repo.

For full details on each suite, see:

• Sandbox TTI, Dax, and per-suite scoring methodology →
• Browser and throughput methodology →
• AI gateway methodology →

Transparency

• 📖 Open source — All benchmark code is public
• 📊 Raw data — Every result committed to repo as JSON
• 🔁 Reproducible — Anyone can run the same tests
• ⚙️ Automated — Suite-specific workflows; sandbox TTI and Dax run weekly on Fridays at 13:00 and 13:30 UTC, respectively
• 🛡️ Independent — Sponsors cannot influence results

Latest Benchmarks

Retired benchmarks: The sequential and staggered sandbox TTI tests have been retired. Historic results are still accessible in the results/ directory under sequential_tti/ and staggered_tti/.

Burst TTI

Pricing Comparison

Browser Sessions

Browser Step Throughput

Object Storage

Snapshot & Fork

DAX Sandbox Builds

The DAX resource configuration generally targets 8 vCPUs and 16 GiB RAM. Modal uses the VM runtime with its default CPU request of 0.125 physical cores and a burst limit of 8 physical cores (16 vCPU equivalent). Its 16 GiB memory request has no explicit memory limit, allowing memory bursting when host capacity is available. Modal bills CPU and memory at the greater of request or actual usage, including usage above the memory request.

AI Gateway - Anthropic

AI Gateway - OpenAI

AI Gateway - Gemini

AI Gateway - Kimi

Roadmap

• computesdk.com/benchmarks
• Add P95 & P99
• TTI n=100 test
• TTI n=100 concurrency test (staggered + burst)
• 100,000 concurrent sandbox stress test
• Cold start vs warm start metrics
• Multi-region testing
• Cost-per-sandbox-minute

Powered by ComputeSDK — We use ComputeSDK, a multi-provider SDK, to test all sandbox providers with the same code. One API, multiple providers, fair comparison.

MIT License