Methodology · Suite v4
How the benchmark works
agenticcommerce.tech measures whether AI coding models can build and operate real commerce software. It is a multi-task suite: each task stresses a different capability axis, is scored 0–100, and the scores combine into a per-model capability profile. Everything is reproducible.
One fixed prompt per task
Each task ships a frozen, versioned one-shot prompt. Every model receives the exact same prompt for a task — one attempt, a fixed time budget, no follow-up questions. Changing a prompt's wording requires a new task version.
Agentic run
The prompt is executed agentically via the Cursor SDK in a clean, empty workspace. The model writes real files on its own. We record telemetry: time to first action, tool calls, run time, token usage and status. The deliverable always lands in the run's app/ directory.
Deterministic harness
Each task carries a standalone harness that observes the deliverable working — driving a static app in a headless browser, or booting the deliverable's service against a provided mock API and inspecting the resulting server state. It emits a deterministic auto-score, a graded tier and per-task ops metrics. A missing result or contract violation scores zero.
Judge + aggregation
An LLM-as-judge scores the remaining subjective dimensions from the deliverable's source (and, for visual tasks, screenshots) against anchored rubrics; multiple judge samples are aggregated as a per-dimension median. Each task score composes 45 base + 20 excellence + 25 judge + 10 robustness points, aggregated into a per-model capability profile, a weighted global index, and separate Elo, reliability and efficiency axes.
The tasks
Five orthogonal tasks. Task A (the original v3 storefront) keeps the strict offline contract; the others provide fixtures and a harness so success is observed, not asserted by the model. Tasks without results yet are marked “soon”.
A · Premium Storefront
20B · Client-side WebGPU Product Q&A
20C · Microsoft Dynamics 365 Order Integration
20D · In-browser Fashion Fit Estimation
20E · Autonomous Buying Agent
20Scoring
Every task is scored 0–100 from four components: 45 base(deterministic core probes), 20 excellence (harder deterministic checks that separate the top of the field — e.g. multi-item optimization scenarios, strict Lighthouse/latency/ accuracy bands), 25 judge (LLM-as-judge on anchored rubrics, per-dimension median over ensemble samples) and 10 robustness (fault recovery, honesty, validation, privacy/allowlist hygiene). Components without data are shown as "n/a" and the remaining weights are renormalized — nothing is guessed. A separate agent score (0–100) per task captures what the model added on its own initiative.
Graded tiers
- 0 Does not run / output contract violated.
- low Runs, but no core capability works.
- mid Core capability works on the happy path.
- high Correct across the fixture set.
- full Correct + robust (errors, edge cases) + polished.
Global index 100
Weighted mean of the per-task scores over scored tasks (default weights equal, 20 each). Failed runs — no working deliverable, contract violation, run cancelled at budget — are excluded from the index and surface as a per-model reliability rate instead of distorting the quality ranking with 0-scores.
Elo (Bradley–Terry)
A second ranking view robust to score-calibration drift: every task contributes pairwise comparisons (higher task score wins), fitted with a Bradley–Terry model and mapped onto an Elo-like scale (1500 = field average, 400 points per decade of strength).
Efficiency & reliability axes
From run telemetry: efficiency min-max-normalizes total tokens, wall-clock time and tool calls against the field per task (0–100, higher = leaner); reliability is the share of run attempts that produced a valid, scorable deliverable. Both are reported next to — never mixed into — the quality index.
Reproducibility
Clean workspace per run, fixed budgets, pinned model identifiers, per-task host allowlists, and dependency-free fixtures (Node built-ins only). Service deliverables run on Node's standard library — no install at eval time. Every task harness is smoke-tested against a golden reference solution.