SnorkelUnderwrite 2.0
Measures an agent’s ability to turn incomplete, distributed insurance evidence into an auditable underwriting decision while respecting authority and policy constraints.
overview
SnorkelUnderwrite 2.0 is a second-generation evaluation of agentic underwriting in a simulated commercial property and casualty insurance environment. It preserves the predecessor’s focus on multi-turn interaction, tool use, and underwriting judgment while expanding the challenge to partial information, cross-source reasoning, role-specific permissions, long-context evidence, and stateful workflows.
The 200-task release is a representative frontier subset of the broader environment. Tasks require agents to identify the relevant case, obtain missing context, retrieve evidence from structured and unstructured sources, apply business and authority rules, and produce a defensible underwriting position or next action. Evaluation covers answer correctness and, where required, safety, process quality, and final environment state.
At a glance
200
frontier tasks
72.5%
tasks with partial information
34.5%
tasks with 20K+ context
36%
tasks with expected state changes
Leaderboard
| Rank | Model | Pass@1 | Pass@5 | Cost / Trial | Cost / Task |
|---|---|---|---|---|---|
| 1 | GLM 5.3 |
30.4%
|
51.3%
|
$0.36 | $1.81 |
| 2 | DeepSeek V4 Pro |
28.5%
|
54.5%
|
$0.61 | $3.04 |
| 3 | Kimi K3 |
27.9%
|
50.8%
|
$0.57 | $2.83 |
| 4 | Fable 5.1 |
24%
|
38.4%
|
$1.92 | $9.58 |
| 5 | GPT-6 Astra |
21.2%
|
35%
|
$1.71 | $8.56 |
| 6 | Qwen 3.8 Max |
20%
|
39.2%
|
$0.69 | $3.43 |
| 7 | Muse Spark 1.3 |
19.6%
|
32.7%
|
$0.33 | $1.63 |
| 8 | Gemini Flash 3.8 |
19.1%
|
35%
|
$0.58 | $2.92 |
| 9 | Claude Opus 5 |
18.3%
|
34%
|
$0.82 | $4.12 |
| 10 | Grok 4.6 |
17.8%
|
31.3%
|
$0.66 | $3.31 |
Frontier performance
Methodology
Evaluator
The benchmark runs in a self-contained Docker and OpenEnv environment with deterministic data fixtures, structured MCP tool calls, seeded failure behavior, stateful sessions, and complete trajectory logging.
scoring note
Five rollouts per task support frontier calibration against a reference-model panel. Results include task success and decomposed signals for answer correctness, final state, safety, process, and efficiency. Judge stability is checked through repeat scoring and human agreement sampling.
Behind the benchmark
Underwriting is a decision process under incomplete information. A case can require the agent to identify the right submission, ask for missing context, reconcile structured records with reference documents, apply line-of-business and authority rules, and make a recommendation that another professional could audit. SnorkelUnderwrite 2.0 measures whether the agent can maintain that chain from evidence to decision as the conversation and environment state develop.
Version 2.0 retains the previous iteration’s expert-verified, multi-turn underwriting identity while expanding the operating conditions around each decision. Subject-matter experts contribute and review the tasks, tools, scenarios, rubrics, and verifiers. The frontier subset emphasizes incomplete information, cross-source reconciliation, permission and policy boundaries, adversarial pressure, and controlled state changes.
