Rank #9 · 5/5 tasks scored
Claude Sonnet 4.6 (high)
AURUM MAISON delivers a cohesive luxury pen experience with an impressive raw WebGL2 PBR configurator, persistent cart/pricing, five-step checkout, and a multi-step agentic assistant that performs real UI actions. Floor requirements are largely met, but gallery view switching is cosmetic-only, there is no mobile navigation or wishlist, locale switching mostly reformats prices rather than translating copy, and the mobile buy path is a long scroll without a sticky CTA.
- Elo 1456
- Reliability 100% (5/5 runs)
- Efficiency 31.6/100
Capability profile
Five task axes (each normalized to its 0–100 task score).
Per-task scores
| Task | Score | Base | Excel. | Judge | Robust. | Tier | Elo |
|---|---|---|---|---|---|---|---|
| Premium Storefront | 88 | 43.2 | 14.8 | 20.8 | 9.2 | Full | 1774 |
| Client-side WebGPU Product Q&A | 64 | 22.1 | 12.6 | 24 | 5 | Mid | 1055 |
| Microsoft Dynamics 365 Order Integration | 95 | 45 | 18.4 | 21.3 | 10 | Full | 1711 |
| In-browser Fashion Fit Estimation | 48 | 7.5 | 6.7 | 23.3 | 10 | Low | 1163 |
| Autonomous Buying Agent | 96 | 45 | 18 | 23.3 | 10 | Full | 1403 |
Score composition per task: 45 base + 20 excellence + 25 judge + 10 robustness. "n/a" = no data for that component in this run (weights renormalized). Failed runs are reliability events, not 0-scores.
Task A — Premium Storefront
Score breakdown
Engineering in detail
Lighthouse
CLS 0.000 · LCP 2927ms · Page weight 314 KB · axe 3 (crit 1)
Live preview
The actual storefront generated by the model — interactive.Open in new tab ↗
Screenshots
Verified interactions
Behavior actually driven in the browser (not just present in the DOM).
- Add to cart workspass
- Dark mode togglesunknown
- Variant changes price/galleryunknown
- Search filters the catalogunknown
- AI assistant performs an actionpass
- Cart persists across reloadunknown
Structured data & SEO
- ✓ Meta description
- ✓ Canonical URL
- ✗ Open Graph image
- 100% of images have alt text (0/0)
Tokens & cost
Token usage is reported by the agent run. Cost is an estimate (tokens × configured rates); shows “—” until rates are set.
Runtime metrics
Deductions (−2)
- −1 Copy-paste template
- −1 Console errors
Feature matrix (24/25)
- 3D configurator (WebGL)present
- Product gallerypresent
- Image zoompresent
- Color variantspresent
- Size selectionpresent
- Live stockpresent
- Pricepresent
- Discountpresent
- Buy boxpresent
- Sticky behaviorpresent
- Reviewspresent
- Cross-sellingpresent
- Search / filterpresent
- Mobile navigationmissing
- Wishlistpresent
- Cartpresent
- Multi-step checkoutpresent
- Currency / locale switchpresent
- AI assistantpresent
- Dark modepresent
- Animationspresent
- Accessibility (basics)present
Per-task results
Each task this model also ran, with the same depth as Task A where the task allows it: static-app tasks show screenshots and a live preview; backend / agent tasks show the harness probe breakdown and key metrics.
Client-side WebGPU Product Q&A
Applied On-device AI · static app
A polished, production-quality assistant with real WebLLM token streaming, live TTFT/tokens-per-sec readouts, and visible field-level citations backed by a thorough deterministic fallback engine. Input is disabled during generation and there is no cancel control, but loading, WebGPU-unavailable, and error paths all degrade gracefully without dead-ends.
Live preview — interactive app
Loads the actual app generated by this model.Open in a new tab ↗
Probe breakdown — automated 37/75
- Harness hook present & well-formed__ask returns { answer, sources? }6/6passed
- In-scope factual answers grounded in catalog3/6 in-scope factual correct8/16failed
- Multi-fact reasoning answers0/2 multi-fact correct0/8failed
- Refuses out-of-scope questions0/2 out-of-scope refused0/7failed
- Refuses adversarial / fabrication bait2/2 adversarial refused7/7passed
- Answers cite their catalog source8/8 in-scope answers cited a source5/5passed
- On-device model initialization tierreported tier=unavailable, engine=deterministic-lookup (WebLLM failed: [Invalid ShaderModule (unlabeled)] is invalid due to a previous error. - While validating compute s), navigator.gpu=true3/10failed
- Latency within budget (TTFT + tokens/sec)ttftMs=5 (budget 30000), tokensPerSec=n/a2/4failed
- No off-allowlist traffic after loadoff-allowlist hosts: us.aws.cdn.hf.co0/6failed
- Robust to hostile input (empty / very long / rapid-fire)empty=true longInput=true rapidFire=true6/6passed
Key metrics
Microsoft Dynamics 365 Order Integration
Integration Engineering · backend / agent task
Clean four-module design with complete entity mappers, a solid retrying D365 client with upsert and idempotency, and rich validation errors with field-level details. Observability is adequate but minimal—failures are logged safely while successful syncs are not.
Backend / agent task — evaluated by deterministic harness probes against a mock service. There is no visual preview for this submission.
Probe breakdown — automated 80/80
- Happy-path order maps to the correct D365 entity graph18/18 graph checks passed18/18passed
- Per-field mapping completeness & accuracy vs goldmean field accuracy 100.0% over 4 orders20/20passed
- Idempotent on retry (one key => exactly one sales order)salesorders=1 (want 1), lines=2 (want 2), replayFlag=true12/12passed
- Recovers from injected faults (429/500/reset) with retry + backoffgraph 100%, faultsServed=4, soPosts=3, accPosts=212/12passed
- Customer upsert: lookup-or-create without duplicatesreusedNoDup=true, salesorderRefsSeed=true, newCreated=true8/8passed
- Structured 4xx on malformed input with no partial writes3/3 rejected with structured 4xx, partialWrites=false8/8passed
- No credentials hardcoded or loggedleakInSource=false, leakInLogs=false, readsProcessEnv=true2/2passed
Key metrics
In-browser Fashion Fit Estimation
On-device ML & Continual Learning · static app
Strong probabilistic UX with a confidence bar, tiered confidence copy, per-metric ranges, and a full size probability chart, plus explicit return-adjustment messaging. On-device privacy and upfront consent are clear and credible; the main gap is no dedicated low-confidence warning beyond the meter and color cues.
Live preview — interactive app
Loads the actual app generated by this model.Open in a new tab ↗
Probe breakdown — automated 20/70
- Loads & produces well-formed metricsnot well-formed ({"ok":true,"value":{"measurements":null,"size":null,"confidence":0,"error":"bad_image"},"ms":5})4/8failed
- Real on-device model + runtime loadedreal model (external=true, backend=webgpu)6/6passed
- Measurement accuracy vs gold (MAE)meanMAE=20cm per-metric={"chest":20,"waist":20,"hip":20,"inseam":20,"shoulder":20}0/16failed
- Recommended size top-1 accuracytop1=0/80/10failed
- Recommended size within ±1within1=0/80/6failed
- Online learning lowers holdout error (CORE)holdout err 5 -> 50/14failed
- Graceful no-person / bad-image / non-imagenoperson:bad_image✓ bad:bad_image✓ notimage:bad_image✓4/4passed
- No image egress / on-device onlyoffAllowlist=0 bigUploadsAfterEstimate=06/6passed
Key metrics
Autonomous Buying Agent
Agentic Planning & Tool Use · backend / agent task
The agent documents and executes a clear eight-step plan with strong pre-feasibility analysis, coupon/shipping optimization, and multiple recovery paths (OOS fallback, checkout coupon rejection). Success trade-off explanations are somewhat templated, but impossibility reporting is detailed and honest.
Backend / agent task — evaluated by deterministic harness probes against a mock service. There is no visual preview for this submission.
Probe breakdown — automated 85/85
- Agent runs and writes a valid report for every scenario8/8 — 01-happy-hoodie:ok 02-budget-tight-tee:ok 03-coupon-optimality-sneaker:ok 04-deadline-express-tee:ok 05-faults-recovery-hoodie:ok 06-impossible-budget-hoodie:ok 07-oos-size-hoodie:ok 08-impossible-stock-sneaker:ok6/6passed
- Scenario goal achieved end-to-end (or impossible handled correctly)8/8 — 01-happy-hoodie:ok 02-budget-tight-tee:ok 03-coupon-optimality-sneaker:ok 04-deadline-express-tee:ok 05-faults-recovery-hoodie:ok 06-impossible-budget-hoodie:ok 07-oos-size-hoodie:ok 08-impossible-stock-sneaker:ok24/24passed
- No placed order ever exceeds the scenario budget8/8 — 01-happy-hoodie:ok 02-budget-tight-tee:ok 03-coupon-optimality-sneaker:ok 04-deadline-express-tee:ok 05-faults-recovery-hoodie:ok 06-impossible-budget-hoodie:ok 07-oos-size-hoodie:ok 08-impossible-stock-sneaker:ok12/12passed
- Hard constraints satisfied (in-stock, size, quantity, deadline)6/6 — 01-happy-hoodie:ok 02-budget-tight-tee:ok 03-coupon-optimality-sneaker:ok 04-deadline-express-tee:ok 05-faults-recovery-hoodie:ok 07-oos-size-hoodie:ok13/13passed
- Best valid coupon applied for the purchased cart6/6 — 01-happy-hoodie:ok 02-budget-tight-tee:ok 03-coupon-optimality-sneaker:ok 04-deadline-express-tee:ok 05-faults-recovery-hoodie:ok 07-oos-size-hoodie:ok12/12passed
- Recovers from injected API faults and still completes the goal2/2 — 05-faults-recovery-hoodie:ok 07-oos-size-hoodie:ok10/10passed
- Impossible goals reported honestly with NO order placed2/2 — 06-impossible-budget-hoodie:ok 08-impossible-stock-sneaker:ok8/8passed
- Excellence scenario goal achieved end-to-end7/8 — x1-excellence-quantity-coupon:ok x2-excellence-coupon-required:ok x3-excellence-deadline-budget:ok x4-excellence-fault-storm:ok x5-excellence-impossible-deadline:ok x6-excellence-impossible-stock-depth:ok x7-excellence-multi-product-cart:x x8-excellence-zero-slack:ok9/10failed
- Excellence: optimal cart + coupon under tight budgets5/6 — x1-excellence-quantity-coupon:ok x2-excellence-coupon-required:ok x3-excellence-deadline-budget:ok x4-excellence-fault-storm:ok x7-excellence-multi-product-cart:x x8-excellence-zero-slack:ok3/4failed
- Excellence: survives heavy fault storms2/2 — x4-excellence-fault-storm:ok x8-excellence-zero-slack:ok3/3passed
- Excellence: subtle impossibilities handled honestly2/2 — x5-excellence-impossible-deadline:ok x6-excellence-impossible-stock-depth:ok3/3passed