Enterprise environments and training AI agents for real-world workflows

Most agent benchmarks still evaluate a thin slice of the job. The agent receives a task, produces an answer, gets scored, and the episode ends.

Enterprise workflows work differently. An underwriting agent may need to read policy documents, inspect customer records, call internal tools, ask a simulated user for missing information, update state, and follow approval rules.

A correct final sentence does not prove that the workflow was completed correctly.

Snorkel’s research team has been building these environments for domains such as insurance, finance, manufacturing, and sales and marketing. The work turns domain expert input into simulated companies with workflow systems that can support evaluation and reinforcement learning.

The central question is practical: how do you build a realistic enterprise environment that is solvable, difficult, and useful for training?

Enterprise environments give agents a place to learn

Enterprise agents need more than task prompts. They need a setting where actions have consequences.

An enterprise environment gives the agent that setting. It defines the tools the agent can call, the state those tools can change, the documents the agent can inspect, the users it can interact with, and the rules it has to follow, all within a simulated company.

A legal, healthcare, finance, or insurance agent can then be evaluated inside the workflow instead of only on the final answer.

The environment also creates the feedback signal. The agent acts. The environment changes. A judge scores the outcome. That score can be used for evaluation, and it can also support reinforcement learning by updating the policy that chooses future actions.

Figure: RL reward loop. The environment supplies the tools, state, and simulated user interaction that make agent behavior measurable. Judges convert the trajectory into a reward signal that can update the policy

For example, a model may know that a typical loan workflow allows approvals up to $10,000. A specific company may cap the same workflow at $1,000. The agent only gets the right outcome if it checks the environment instead of relying on the generic rule.

Public agent benchmarks test different parts of workflow behavior

Static benchmarks score a response against a reference answer. Agent environments expose the work around that response: tool calls, user turns, state changes, failed attempts, and final system state.

Tau-Bench shows that structure in customer service. The agent talks to a simulated user, chooses API tools, follows airline or retail policies, and tries to satisfy the user’s request.

The benchmark grades the final database state against the goal state. A seat-change task fails when the agent says the seat was changed but the database still shows the old seat.

Tau-Bench also measures consistency. Pass^k runs the same task multiple times and checks whether the agent succeeds across trials. 

Public environments show how to grade agent work: the agent acts, the environment changes, and the trajectory can be scored. Enterprise workflows make that setup harder because the agent needs information that public benchmarks and base LLMs do not contain.

Enterprise environments need domain-specific construction

Public benchmarks can reuse generic workflows and interfaces. Enterprise environments must instead reflect the systems, policies, data, and organizational boundaries of the business being modeled.

• The agent needs access to proprietary tools and APIs because the real work happens inside internal systems.
• Domain-specific policies define approval gates, compliance rules, and guardrails that can change the correct action.
• Internal and proprietary data supplies the thresholds, records, catalogs, and exceptions that may never appear in pretraining data.
• Real organizational structure determines permissions, access boundaries, and escalation paths.

These expert-defined details establish the environment’s boundary conditions: the tools, policies, data, permissions, and valid outcomes the generated environment must preserve and the agent must discover rather than infer from general domain knowledge.

Good environments balance solvability, realism, and difficulty

Enterprise environment design depends on a balance between three constraints.

• Solvability: the environment should contain enough consistent evidence and available actions for capable agents to complete the workflow and reach a defensible outcome.
• Realism: the environment should preserve messy records, permission boundaries, policy conflicts, and user ambiguity.
• Difficulty: the task should require discovery, composition, and judgment rather than broken infrastructure or missing information.

Solvability under realistic conditions

A task is solvable when the environment contains a valid path through the workflow. The required evidence must be discoverable, and the available tools must support the actions needed to complete the task.

A successful oracle trajectory confirms that at least one valid path exists. Running the same task across capable agents provides a stronger test. Their intermediate decisions may differ, but they should reach the same defensible outcome from the available evidence.

Persistent disagreement can reveal missing records, contradictory policies, unavailable actions, or multiple valid interpretations. Those cases need expert review before the task can provide a reliable evaluation or training signal.

Solvability does not require removing enterprise complexity. Records can remain incomplete, policies can contain exceptions, and simulated users can omit or revise information. The task remains solvable when the necessary evidence can be recovered and the available actions support completion.

How Snorkel tests whether generated tasks are valid

Expert-authored seeds define what the environment should contain. The generated tasks still need to satisfy three conditions: the workflow must support a valid solution, the answer must require evidence from inside the environment, and evaluation must detect failures beyond the final response.

Oracle Agents Validate a Path Through the Workflow

Snorkel uses oracle agents to explore the completed environment and execute workflows defined in the seed files. The oracle can inspect documents, query database state, call available tools, and complete the workflow end to end.

A successful run confirms that the environment supports at least one valid trajectory. The captured outcome and state changes can then support task construction and evaluation without exposing privileged information in the task prompt.

The task can also be tested across frontier models. Their tool use may differ, but capable agents should converge on the same defensible outcome. Persistent disagreement can expose missing evidence, contradictory rules, unavailable actions, or multiple valid interpretations that require expert review.

Proprietary Rules Require Grounded Discovery

A solvable task can still provide weak training signal when the model can answer from pretraining without using the environment.

Snorkel introduces plausible company-specific rules, terminology, artifacts, and exceptions that are absent from public data. Domain experts keep these details consistent with the underlying workflow.

• Operational rules can introduce non-standard thresholds, approval gates, or regulatory interpretations. The correct action depends on the rule inside the environment rather than the industry default.
• Domain artifacts can include proprietary catalogs, internal terminology, organizational structures, and product names. The agent must inspect the available documents rather than rely on public terminology.
• Data characteristics can include noisy records, edge cases, null fields, and legacy policy behavior. These details force the agent to reason over the environment’s actual state.

The design test is whether an answer from memory diverges from one grounded in the environment. The agent should have to inspect evidence and apply the rule it finds.

Evaluation separates different failure modes

A final answer cannot reveal every workflow failure. An agent may provide the expected response while leaving the database in the wrong state, violating a permission boundary, or skipping a required approval.

Evaluation therefore needs to inspect several parts of the trajectory:

These checks can use deterministic validation, model-based review, or a combination appropriate to the workflow. The exact evaluation implementation can change without altering the underlying requirement: success must account for the outcome, resulting state, safety constraints, and trajectory.

A correct answer cannot compensate for an incorrect state or a safety violation. Process evaluation should still permit harmless variation when different tool sequences satisfy the same evidence and workflow requirements.

Insurance Underwriting shows the full workflow system

Snorkel’s insurance-underwriting environment provides a concrete test of whether expert-defined constraints can produce solvable, stateful tasks and useful training signals.

The environment contains more than 50 database tables, 100 connected documents, 47 workflow tools, 2,000 generated tasks, and seven user personas. These components create dependencies across records, policies, tool access, user interaction, and workflow state.

The sample task is a new auto policy application. The user gives the agent a driver profile and coverage request. The agent has to ask for missing information, check the customer record, inspect the driving record, look up coverage rules, and submit a decision.

The proprietary rule changes the outcome. In this environment, drivers under 25 require manager approval. The correct action is to refer to the manager, even when the driver has a clean record.

The task is graded on the underwriting decision, resulting workflow state, compliance with the under-25 referral rule, and whether the agent gathered the required evidence using appropriate tools.

Early training results show that the environment produced useful workflow signal. A Qwen3-30B model trained on the insurance environment improved on multi-turn and tool-use evaluations:

• BFCL Multi-Turn: 33.3 → 41.4, a gain of 8.1 points.
• τ³-bench overall: 24.6 → 30.4, a gain of 5.8 points.
• τ³-bench airline: 32.0 → 46.0, a gain of 14.0 points.

The gains suggest that training on one enterprise environment can teach behavior that transfers beyond the original domain: asking for missing information, using tools, applying hidden rules, and maintaining state across a workflow.

Open questions for enterprise environments

Static evaluations can show whether a model produced an acceptable answer. They cannot show whether an agent gathered the right evidence, used permitted actions, maintained state, or recovered from ambiguity during a workflow.

• LLM judge reliability: semantic, safety, and process judges can grade behavior that exact checks miss, but their calibration and consistency matter when those judgments become reward signals.
• Adaptive difficulty: environments need tasks that stay challenging as agents improve. The harder problem is preserving difficulty without constantly re-authoring the environment.
• Process vs. outcome reward: final state is easier to verify, but workflow quality often depends on the path. Reward design has to decide how much weight to give tool choice, reasoning quality, and user interaction.

Trajectory-graded environments make these behaviors observable. They provide a stronger basis for studying whether agents can complete stateful, domain-specific workflows under realistic constraints, and for turning those observations into training signals.

Acknowledgments

Enterprise environment-building is a cross-functional effort. This work reflects contributions from Snorkel’s Forward Deployed Engineering, Technical Delivery Management, Research, and Delivery Operations teams.

Contributors include Joe Licata, Peter Marinov, Jeremy Chan, Aleem Lakdawala, Vrushank Kumar, Pedro Alves, Marwan Bounassif, Connor Young, Chris Glaze, Bhavishya Pohani, Ramya Ramakrishnan, Maria Mir, Jon Kovar, and Louisa Savageaux.