Enterprise environments and training AI agents for real-world workflows

Most agent benchmarks still evaluate a thin slice of the job. The agent receives a task, produces an answer, gets scored, and the episode ends.

Enterprise workflows work differently. An underwriting agent may need to read policy documents, inspect customer records, call internal tools, ask a simulated user for missing information, update state, and follow approval rules.

A correct final sentence does not prove that the workflow was completed correctly.

Enterprise environments give agents a world to act in. The environment defines the tools, documents, database state, user behavior, policies, permissions, and judges around a domain-specific workflow.

The agent’s behavior can then be evaluated as a trajectory: what it read, what it changed, which rule it followed, which rule it broke, and whether the final state matched the expected outcome.

Snorkel’s research team has been building these environments for enterprise domains such as insurance, finance, manufacturing, and sales and marketing. The work turns domain expert input into simulated workflow systems that can support evaluation and reinforcement learning.

The central question is practical: how do you build a realistic enterprise environment that is solvable, difficult, and useful for training?

Enterprise environments give agents a place to learn

Enterprise agents need more than task prompts. They need a setting where actions have consequences.

An enterprise environment gives the agent that setting. It defines the tools the agent can call, the state those tools can change, the documents the agent can inspect, the users it can interact with, and the rules it has to follow.

A legal, healthcare, finance, or insurance agent can then be evaluated inside the workflow instead of only on the final answer.

The environment also creates the feedback signal. The agent acts. The environment changes. A judge scores the outcome. That score can be used for evaluation, and it can also support reinforcement learning by updating the policy that chooses future actions.

Figure: RL reward loop. The environment supplies the tools, state, and simulated user interaction that make agent behavior measurable. Judges convert the trajectory into a reward signal that can update the policy

Static datasets can test whether a model produced the expected answer. Enterprise environments add the missing workflow around that answer: the tools the agent used, the records it changed, the policy it followed, and the state it left behind.

For example, a model may know that a typical loan workflow allows approvals up to $10,000. A specific company may cap the same workflow at $1,000. The agent only gets the right outcome if it checks the environment instead of relying on the generic rule.

Static datasets measure answers. Environments produce trajectories.

Static datasets work well when the unit of evaluation is a single response. The model receives a task or question. The benchmark compares the output against a reference answer. That format can test knowledge, reasoning, or task-specific accuracy.

Enterprise agents need a different unit of evaluation. The agent may need to call a tool, retrieve a policy document, ask the user for missing information, update a database field, and stop when the workflow requires approval. The answer is only one artifact from that process.

Simulation environments make the process observable. The environment can record the tool calls, documents consulted, state changes, user turns, intermediate failures, and final outcome. That trajectory can support evaluation because reviewers can inspect what the agent did. It can also support training because judges can turn the trajectory into a reward signal.

Public agent benchmarks test different parts of workflow behavior

Answer-based benchmarks score a model response against a reference answer. Simulation environments score the work around the response: tool calls, user turns, state changes, failed attempts, and final system state.

Tau-Bench shows that structure in customer service. The agent talks to a simulated user, chooses API tools, follows airline or retail policies, and tries to satisfy the user’s request.

The benchmark grades the final database state against the goal state. A seat-change task fails when the agent says the seat was changed but the database still shows the old seat.

Tau-Bench also measures consistency. Pass^k runs the same task multiple times and checks whether the agent succeeds across trials. 

Other notable benchmarks such as TerminalBench and OSWorld test the same environment pattern in different interfaces. TerminalBench puts the agent in a command-line environment with files, commands, errors, and executable checks. OSWorld puts the agent in desktop and web applications where screen state and interface actions determine completion.

Benchmark	Environment	Workflow behavior it makes visible
Tau-Bench	Airline and retail customer-service workflows with API tools, policies, and simulated users	Tool choice, user interaction, rule following, repeated reliability, and final database state
TerminalBench	Terminal-based environments with command-line access and executable tests	File inspection, command execution, debugging, recovery, and verification through tests
OSWorld	Desktop and web application environments with GUI state and multimodal observations	Screen interpretation, GUI navigation, application state changes, and task completion

Public environments show how to grade agent work: the agent acts, the environment changes, and the trajectory can be scored. Enterprise workflows make that setup harder because the agent needs information that public benchmarks and base LLMs do not contain.

Enterprise environments need domain-specific construction

Public environments can be reused because they cover generic work. A customer-service benchmark can model airline and retail support. A GUI benchmark can model standard browser and operating-system tasks. A coding benchmark can use public repositories and standard developer tools.

Enterprise workflows cannot be copied that way. The environment has to be built around the business that owns the workflow.

• The agent needs access to proprietary tools and APIs because the real work happens inside internal systems.
• Domain-specific policies define approval gates, compliance rules, and guardrails that can change the correct action.
• Internal and proprietary data supplies the thresholds, records, catalogs, and exceptions that may never appear in pretraining data.
• Real organizational structure determines permissions, access boundaries, and escalation paths.

Those details decide the task. A workflow that looks standard from the outside can change once the company’s internal systems and rules appear. The environment has to expose those details through tools, documents, state, and user interaction so the agent can discover the valid action instead of relying on general domain knowledge.

A useful enterprise environment therefore contains the business logic behind the task. The agent’s score should depend on whether it used the right tool, found the relevant internal information, respected the approval path, and left the workflow in the correct state.

Good enterprise environments must handle four recurring challenges

Enterprise environment quality depends on more than domain coverage. The environment has to produce tasks that are realistic enough to reflect business work, clear enough to grade, and difficult for the right reasons.

Realism vs. solvability

Enterprise workflows contain ambiguity. Records can be incomplete, policies can overlap, and user requests can leave out details. A useful environment should preserve enough of that mess to represent the real workflow.

Solvability keeps realism from turning into noise. If two capable agents can reasonably reach different answers from the same information, the task may be underspecified. A hard task should require discovery, composition, and judgment. It should not depend on missing information or contradictory rules.

Outcome vs. process grading

Final-state reward can show whether the workflow ended correctly. It cannot always show whether the agent used the right path.

Process grading fills that missing layer. The environment should capture tool choice, intermediate decisions, policy compliance, and user interaction quality. An agent can land on the right final state while still taking an action that a real workflow would reject.

User simulation quality

Simulated users shape the agent’s behavior. A weak user simulator can make interactions too clean, too predictable, or too forgiving. Enterprise environments need user variation: missing details, vague requests, corrections, misunderstandings, and domain shorthand.

Memorization vs. discovery

Pretraining can weaken the signal. If the model can answer from general domain knowledge, it may never use the environment’s tools or documents.

Enterprise environments need realistic information that is specific to the environment: internal rules, proprietary artifacts, private records, naming conventions, thresholds, and edge cases. The agent should have to discover the answer from the system in front of it.

Snorkel builds environments from expert seed files

Snorkel’s approach starts by asking domain experts and operators to describe the workflow boundary first, then uses a synthesis pipeline to generate the runtime environment around that boundary.

The expert does not hand-author every database row, document, tool, user, judge, and task. The expert defines the domain boundary through a small set of seed files.

Those seed files describe the workflow world the environment has to create. They specify the domain metadata, the workflows the agent should perform, the domain knowledge that gives the workflow context, and the proprietary information that makes the environment specific to one business.

Figure: Snorkel’s environment-building pipeline starts from four expert-authored seed files, then synthesizes a complete runtime environment with database state, documents, tools, persona-driven users, judges, and validated tasks.

Four seed files define the environment boundary

Seed files make environment generation operator-driven. The expert defines the workflow boundary first. The synthesis pipeline then generates the database, documents, tools, users, judges, and tasks inside that boundary.

Expert-written seed file	What the operator provides
Domain metadata	The domain name, database tables, document repository layout, and target tool surfaces.
Workflow definitions	The triggers, outcomes, input and output documents, tables used, required tool categories, and safety classifications for each workflow.
Domain knowledge	The business context behind the workflow: processes, roles, data, sensitive areas, and domain language.
Proprietary information	The non-standard rules, internal terminology, edge cases, thresholds, approval gates, and company-specific assumptions that the environment should test.

A multi-phase pipeline turns seed files into a runtime environment

The synthesis pipeline expands expert-written seed files into a complete environment in three blocks: foundation, interaction, and signal generation. The blocks build on each other, so early schema decisions shape the tools, documents, judges, and tasks that come later.

Figure: Ten synthesis phases turn expert-written seed files into a runtime environment. Foundation phases create specifications, primitives, databases, and documents. Interaction phases add tools, users, and judges. Signal and refinement phases generate tasks, persona variants, and expert feedback loops.

Foundation: specifications, primitives, database, and documents

The foundation phases turn seed files into the environment’s structure. The pipeline creates the initial spec, identifies domain primitives, builds the database, and generates documents aligned to that database.

Primitives matter because they define the entities, fields, states, and relationships the rest of the environment uses. A weak primitive layer can create downstream errors in tools, documents, judges, and tasks.

Interaction: tools, users, and judges

The interaction phases make the environment usable by an agent. The pipeline generates MCP tools for actions, persona-driven users for multi-turn conversations, and judges for evaluation.

The tools expose what the agent can do. The users supply missing information, ambiguity, and role-specific behavior. The judges score the answer, state changes, safety constraints, and process quality.

Signal and refinement: tasks, variants, and expert feedback

The final phases create the training and evaluation signal. Oracle agents discover tasks by solving workflows inside the environment. Persona variants turn those tasks into varied user interactions.

Expert feedback closes the loop. Operators can flag ambiguity, missing details, broken assumptions, or unrealistic behavior, and the pipeline can re-synthesize the affected artifacts.

Oracle agents turn workflows into validated tasks

Hand-authoring every task would make environment-building too slow. Snorkel uses oracle agents to discover tasks from inside the completed environment instead.

The process starts with a workflow definition from the seed files. The oracle agent gets full environment access, then explores the available tools, reads the documents, checks database state, and executes the workflow end to end. The environment records the tool chain, intermediate state changes, and final answer.

The captured solution becomes the basis for a validated task. Internal identifiers and privileged information are stripped from the prompt, so the final task reads like a realistic user request rather than a leaked solution. The task keeps the expected outcome and the process trace needed for grading.

Oracle-generated tasks are grounded in what the environment actually supports. The oracle solved the task first, so the task is solvable by construction. The expected tool chain is captured, so the process can be graded. The same workflow can then be rolled out across frontier models to calibrate difficulty.

Fictional proprietary rules force discovery over memorization

Enterprise environments should test whether the agent can use the tools and documents in front of it. A task loses value when the model can answer from pretraining alone.

Snorkel uses fictional proprietary information to create that test. The information has to be plausible to a domain expert, but specific to the generated environment. The agent should get one answer from general domain memory and a different answer from the environment’s documents, records, and rules.

1. Operational rules create the first source of divergence. The environment can include non-standard thresholds, approval gates, or regulatory interpretations. A general model may assume the industry default, while the correct answer depends on the rule inside the environment.
2. Domain artifacts create the second source of divergence. The environment can use proprietary catalogs, internal terminology, organizational structures, and product names. The agent has to inspect the documents instead of mapping the task onto public terminology.
3. Data characteristics make the environment realistic. Records can include noise, edge cases, boundary conditions, null fields, or legacy policy behavior. Those details force the agent to reason over the actual state instead of giving a generic answer.

The design test is simple: an answer from memory should diverge from an answer grounded in the environment. The correct behavior is discovery. The agent has to read the relevant document, call the right tool, inspect the state, and apply the rule it finds there.

Evaluation uses specialized judges for different failure modes

Enterprise environment grading works best when separate judges inspect separate failure modes. The environment has to check different parts of the workflow separately because an agent can get one part right while failing another.

Judge	Type	What it checks
Verifier	Deterministic	Whether the final answer matches the expected answer after type-aware normalization.
State hash	Deterministic	Whether the post-episode database and document state match the expected state.
Semantic judge	LLM-based	Whether a long-form answer satisfies the expected content for that response type.
Safety judge	LLM-based	Whether the agent avoided harmful actions, respected refusal rules, and gave an acceptable explanation.
Process judge	Deterministic + LLM-based	Whether the agent used the expected task method and handled the user interaction properly.

A task is only correct if all core checks pass:

✓ the answer is right
✓ the state ends up right
✓ no safety rule was broken

These checks work as a conjunctive gate. The agent cannot compensate for a broken safety rule with a correct final answer, and it cannot compensate for a wrong database state with a fluent explanation.

Process quality is scored separately. The process judge can reveal whether the agent used the right tools, followed the expected method, and handled the user interaction well.

Snorkel calibrates this judge conservatively because a workflow can allow harmless variation in the path while still satisfying answer, state, and safety requirements.

Good Environments Balance Solvability, Realism, and Difficulty

Enterprise environment design depends on a balance between three constraints.

• Solvability: two capable agents should reach the same correct answer from the same information.
• Realism: the environment should preserve messy records, permission boundaries, policy conflicts, and user ambiguity.
• Difficulty: the task should require discovery, composition, and judgment rather than broken infrastructure or missing information.

The balance matters because each constraint can distort the training signal. A fully realistic environment can become ambiguous. A fully solvable environment can become too clean. A difficult environment can become unhelpful if the difficulty comes from contradictions or unavailable information.

Good environments make the hard part intentional. The agent should fail because it missed a rule, used the wrong tool, skipped a document, or changed the wrong state. The agent should not fail because the environment gave it an impossible task.

Insurance underwriting shows the full workflow system

Snorkel’s insurance-underwriting environment shows what an enterprise environment looks like once the seed files, synthesis pipeline, oracle-generated tasks, proprietary rules, and judges come together in one domain.

The environment simulates commercial insurance work at scale, with the artifacts an underwriting agent needs to read records, apply rules, use tools, interact with users, change state, and receive graded feedback.

The setup includes the core artifacts needed to simulate underwriting work at scale:

• 50+ database tables for customer, policy, claims, coverage, and workflow state.
• 100+ documents connected to the database and policy logic.
• 47 MCP tools for record lookup, rule checks, decision submission, and workflow actions.
• 2,000+ target tasks generated from the environment.
• 7 personas covering senior and junior underwriting, actuarial, portfolio management, claims, adversarial brokerage, and privilege escalation.

The sample task is a new auto policy application. The user gives the agent a driver profile and coverage request. The agent has to ask for missing information, check the customer record, inspect the driving record, look up coverage rules, and submit a decision.

The proprietary rule changes the outcome. In this environment, drivers under 25 require manager approval. The correct action is to refer to the manager, even when the driver has a clean record.

The task exposes the full evaluation target:

• Answer: the decision must be one of approve, decline, or refer.
• State: the workflow state has to reflect the decision.
• Safety: the agent must refer drivers under 25 to a manager.
• Process: the agent should use the right tools and gather the needed user information.

Early training results show that the environment produced useful workflow signal. A Qwen3-30B model trained on the insurance environment improved on multi-turn and tool-use evaluations:

• BFCL Multi-Turn: 33.3 → 41.4, a gain of 8.1 points.
• τ³-bench overall: 24.6 → 30.4, a gain of 5.8 points.
• τ³-bench airline: 32.0 → 46.0, a gain of 14.0 points.

The gains suggest that training on one enterprise environment can teach behavior that transfers beyond the original domain: asking for missing information, using tools, applying hidden rules, and maintaining state across a workflow.

Open questions for enterprise environments

Enterprise environment quality determines how useful the training signal becomes. The current work shows that environments can produce multi-turn workflow signal, but several design questions remain open.

• LLM judges reliability: semantic, safety, and process judges can grade behavior that exact checks miss, but their calibration and consistency matter when those judgments become reward signals.
• Adaptive difficulty: environments need tasks that stay challenging as agents improve. The harder problem is preserving difficulty without constantly re-authoring the environment.
• Process vs. outcome reward: final state is easier to verify, but workflow quality often depends on the path. Reward design has to decide how much weight to give tool choice, reasoning quality, and user interaction.

The next step is scaling the environment-building loop: more verticals, faster expert turnaround, and tighter integration between environment generation and post-training runs.

Acknowledgments

Enterprise environment-building is a cross-functional effort. This work reflects contributions from Snorkel’s Forward Deployed Engineering, Technical Delivery Management, Research, and Delivery Operations teams.

Contributors include Joe Licata, Peter Marinov, Jeremy Chan, Aleem Lakdawala, Vrushank Kumar, Pedro Alves, Marwan Bounassif, Connor Young, Chris Glaze, Bhavishya Pohani, Ramya Ramakrishnan, Maria Mir, Jon Kovar, and Louisa Savageaux.