resources

Resource library

Explore our complete library of resources including blogs, benchmarks, research papers and more.
Image for Evaluating Coding Agent Capabilities with Terminal-Bench: Snorkel’s Role in Building the Next Generation Benchmark
Blog

Evaluating Coding Agent Capabilities with Terminal-Bench: Snorkel’s Role in Building the Next Generation Benchmark

Announcing a $3M commitment to launch Open Benchmarks Grants
September 30, 2025
Image for Closing the Evaluation Gap in Agentic AI
Blog

Closing the Evaluation Gap in Agentic AI

Announcing a $3M commitment to launch Open Benchmarks Grants

February 11, 2026
Image for Benchtalks #1: Alex Shaw (Terminal-Bench, Harbor) – Building the Benchmark Factory
Blog

Benchtalks #1: Alex Shaw (Terminal-Bench, Harbor) – Building the Benchmark Factory

Announcing a $3M commitment to launch Open Benchmarks Grants
March 31, 2026
Image for Building FinQA: An Open RL Environment for Financial Reasoning Agents
Blog

Building FinQA: An Open RL Environment for Financial Reasoning Agents

Announcing a $3M commitment to launch Open Benchmarks Grants
March 30, 2026
Image for The science of rubric design
Blog

The science of rubric design

Announcing a $3M commitment to launch Open Benchmarks Grants
September 11, 2025
of
Type: All Types
Sort: Newest
A computational approach to measure the linguistic characteristics of psychotherapy timing, responsiveness, and consistency
Although individual psychotherapy is generally effective for a range of mental health conditions, little is known about the momentto-moment language use of effective therapists. Increased access to computational power, coupled with a rise in computermediated communication (telehealth), makes feasible the large-scale analyses of language use during psychotherapy. Transparent methodological approaches are lacking, however. Here we present novel methods to increase the efficiency of efforts to examine language use in psychotherapy. We evaluate three important aspects of therapist language use - timing, responsiveness, and consistency - across five clinically relevant language domains: pronouns, time orientation, emotional polarity, therapist tactics, and paralinguistic...
Research Paper
A computational approach to measure the linguistic characteristics of psychotherapy timing, responsiveness, and consistency

Although individual psychotherapy is generally effective for a range of mental health conditions, little is known about the momentto-moment language use of effective therapists. Increased access to computational power, coupled with a rise in computermediated communication (telehealth), makes feasible the large-scale analyses of language use during psychotherapy. Transparent methodological approaches are lacking, however. Here we present novel methods to increase…

Oct 20, 2023

AS. Miner, et al.

Learn more about A computational approach to measure the linguistic characteristics of psychotherapy timing, responsiveness, and consistency
Perspective Toward Machine Learning Implementation in Pediatric Medicine: Mixed Methods Study
Background: Given the costs of machine learning implementation, a systematic approach to prioritizing which models to implement into clinical practice may be valuable. Objective: The primary objective was to determine the health care attributes respondents at 2 pediatric institutions rate as important when prioritizing machine learning model implementation. The secondary objective was to describe their perspectives on implementation using a qualitative approach. Methods: In this mixed methods study, we distributed a survey to health system leaders, physicians, and data scientists at 2 pediatric institutions. We asked respondents to rank the following 5 attributes in terms of implementation usefulness: the clinical...
Research Paper
Perspective Toward Machine Learning Implementation in Pediatric Medicine: Mixed Methods Study

Background: Given the costs of machine learning implementation, a systematic approach to prioritizing which models to implement into clinical practice may be valuable. Objective: The primary objective was to determine the health care attributes respondents at 2 pediatric institutions rate as important when prioritizing machine learning model implementation. The secondary objective was to describe their perspectives on implementation using a…

Oct 20, 2023

N. Alexander, et al.

Learn more about Perspective Toward Machine Learning Implementation in Pediatric Medicine: Mixed Methods Study
Bloom: A 176b-parameter open-access multilingual language model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59...
Research Paper
Bloom: A 176b-parameter open-access multilingual language model

Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language…

Oct 20, 2023

TL. Scao, et al.

Learn more about Bloom: A 176b-parameter open-access multilingual language model
Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine
Temporal dataset shift associated with changes in healthcare over time is a barrier to deploying machine learning-based clinical decision support systems. Algorithms that learn robust models by estimating invariant properties across time periods for domain generalization (DG) and unsupervised domain adaptation (UDA) might be suitable to proactively mitigate dataset shift. The objective was to characterize the impact of temporal dataset shift on clinical prediction models and benchmark DG and UDA algorithms on improving model robustness. In this cohort study, intensive care unit patients from the MIMIC-IV database were categorized by year groups (2008–2010, 2011–2013, 2014–2016 and 2017–2019). Tasks were predicting...
Research Paper
Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine

Temporal dataset shift associated with changes in healthcare over time is a barrier to deploying machine learning-based clinical decision support systems. Algorithms that learn robust models by estimating invariant properties across time periods for domain generalization (DG) and unsupervised domain adaptation (UDA) might be suitable to proactively mitigate dataset shift. The objective was to characterize the impact of temporal dataset…

Oct 20, 2023

LL Guo, et al.

Learn more about Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine
An Adaptive Method for Weak Supervision with Drifting Data
We introduce an adaptive method with formal quality guarantees for weak supervision in a non-stationary setting. Our goal is to infer the unknown labels of a sequence of data by using weak supervision sources that provide independent noisy signals of the correct classification for each data point. This setting includes crowdsourcing and programmatic weak supervision. We focus on the non-stationary case, where the accuracy of the weak supervision sources can drift over time, e.g., because of changes in the underlying data distribution. Due to the drift, older data could provide misleading information to infer the label of the current data...
Research Paper
An Adaptive Method for Weak Supervision with Drifting Data

We introduce an adaptive method with formal quality guarantees for weak supervision in a non-stationary setting. Our goal is to infer the unknown labels of a sequence of data by using weak supervision sources that provide independent noisy signals of the correct classification for each data point. This setting includes crowdsourcing and programmatic weak supervision. We focus on the non-stationary…

Oct 20, 2023

A. Mazzetto, et al.

Learn more about An Adaptive Method for Weak Supervision with Drifting Data
Fairness via explanation quality: Evaluating disparities in the quality of post hoc explanations
As post hoc explanation methods are increasingly being leveragedto explain complex models in high-stakes settings, it becomes critical to ensure that the quality of the resulting explanations is consistently high across all subgroups of a population. For instance, it should not be the case that explanations associated with instances belonging to, e.g., women, are less accurate than those associated with other genders. In this work, we initiate the study of identifying group-based disparities in explanation quality. To this end, we first outline several key properties that contribute to explanation quality—namely, fidelity (accuracy), stability, consistency, and sparsity—and discuss why and how...
Research Paper
Fairness via explanation quality: Evaluating disparities in the quality of post hoc explanations

As post hoc explanation methods are increasingly being leveragedto explain complex models in high-stakes settings, it becomes critical to ensure that the quality of the resulting explanations is consistently high across all subgroups of a population. For instance, it should not be the case that explanations associated with instances belonging to, e.g., women, are less accurate than those associated with…

Oct 20, 2023

J. Dai, et al.

Learn more about Fairness via explanation quality: Evaluating disparities in the quality of post hoc explanations
R2E2: low-latency path tracing of terabyte-scale scenes using thousands of cloud CPUs
In this paper we explore the viability of path tracing massive scenes using a “supercomputer” constructed on-the-fly from thousands of small, serverless cloud computing nodes. We present R2E2 (Really Elastic Ray Engine) a scene decomposition-based parallel renderer that rapidly acquires thousands of cloud CPU cores, loads scene geometry from a pre-built scene BVH into the aggregate memory of these nodes in parallel, and performs full path traced global illumination using an inter-node messaging service designed for communicating ray data. To balance ray tracing work across many nodes, R2E2 adopts a service-oriented design that statically replicates geometry and texture data from...
Research Paper
R2E2: low-latency path tracing of terabyte-scale scenes using thousands of cloud CPUs

In this paper we explore the viability of path tracing massive scenes using a “supercomputer” constructed on-the-fly from thousands of small, serverless cloud computing nodes. We present R2E2 (Really Elastic Ray Engine) a scene decomposition-based parallel renderer that rapidly acquires thousands of cloud CPU cores, loads scene geometry from a pre-built scene BVH into the aggregate memory of these nodes…

Oct 20, 2023

S Fouladi, et al.

Learn more about R2E2: low-latency path tracing of terabyte-scale scenes using thousands of cloud CPUs
DEEM’22: Data Management for End-to-End Machine Learning
The DEEM’22 workshop (Data Management for End-to-End Machine Learning) is held on Sunday June 12th, in conjunction with SIGMOD/PODS 2022. DEEM brings together researchers and practitioners at the intersection of applied machine learning, data management and systems research, with the goal to discuss the arisingdata management issues in ML application scenarios. The workshop solicits regular research papers (10 pages) describing preliminary and ongoing research results, including industrial experience reports of end-to-end ML deployments, related to DEEM topics. In addition, DEEM 2022 establishes a new paper category for reports on applications and tools (4 pages) as a forum for sharing interesting...
Research Paper
DEEM’22: Data Management for End-to-End Machine Learning

The DEEM’22 workshop (Data Management for End-to-End Machine Learning) is held on Sunday June 12th, in conjunction with SIGMOD/PODS 2022. DEEM brings together researchers and practitioners at the intersection of applied machine learning, data management and systems research, with the goal to discuss the arisingdata management issues in ML application scenarios. The workshop solicits regular research papers (10 pages) describing…

Oct 20, 2023

M. Boehm, et al.

Learn more about DEEM’22: Data Management for End-to-End Machine Learning
MOTOR: A Time-to-Event Foundation Model for Structured Medical Records
We present a self-supervised, time-to-event (TTE) foundation model called MOTOR (Many Outcome Time Oriented Representations) which is pretrained on timestamped sequences of events in electronic health records (EHR) and health insurance claims. TTE models are used for estimating the probability distribution of the time until a specific event occurs, which is an important task in medical settings. TTE models provide many advantages over classification using fixed time horizons, including naturally handling censored observations, but are challenging to train with limited labeled data. MOTOR addresses this challenge by pretraining on up to 55M patient records (9B clinical events). We evaluate MOTOR’s...
Research Paper
MOTOR: A Time-to-Event Foundation Model for Structured Medical Records

We present a self-supervised, time-to-event (TTE) foundation model called MOTOR (Many Outcome Time Oriented Representations) which is pretrained on timestamped sequences of events in electronic health records (EHR) and health insurance claims. TTE models are used for estimating the probability distribution of the time until a specific event occurs, which is an important task in medical settings. TTE models provide…

Oct 20, 2023

E. Steinberg, et al.

Learn more about MOTOR: A Time-to-Event Foundation Model for Structured Medical Records
1 27 28 29 64
Image
Image

Join our newsletter

For expert advice, the latest research, and exclusive events.
By submitting this form, I acknowledge I will receive email updates from Snorkel AI, and I agree to the Terms of Use and acknowledge that my information will be used in accordance with the Privacy Policy.