resources

Resource library

Explore our complete library of resources including blogs, benchmarks, research papers and more.
Image for Evaluating Coding Agent Capabilities with Terminal-Bench: Snorkel’s Role in Building the Next Generation Benchmark
Blog

Evaluating Coding Agent Capabilities with Terminal-Bench: Snorkel’s Role in Building the Next Generation Benchmark

Announcing a $3M commitment to launch Open Benchmarks Grants
September 30, 2025
Image for Closing the Evaluation Gap in Agentic AI
Blog

Closing the Evaluation Gap in Agentic AI

Announcing a $3M commitment to launch Open Benchmarks Grants

February 11, 2026
Image for Benchtalks #1: Alex Shaw (Terminal-Bench, Harbor) – Building the Benchmark Factory
Blog

Benchtalks #1: Alex Shaw (Terminal-Bench, Harbor) – Building the Benchmark Factory

Announcing a $3M commitment to launch Open Benchmarks Grants
March 31, 2026
Image for Building FinQA: An Open RL Environment for Financial Reasoning Agents
Blog

Building FinQA: An Open RL Environment for Financial Reasoning Agents

Announcing a $3M commitment to launch Open Benchmarks Grants
March 30, 2026
Image for The science of rubric design
Blog

The science of rubric design

Announcing a $3M commitment to launch Open Benchmarks Grants
September 11, 2025
of
Type: All Types
Sort: Newest
Assessing the accuracy of automatic speech recognition for psychotherapy
Accurate transcription of audio recordings in psychotherapy would improve therapy effectiveness, clinician training, and safety monitoring. Although automatic speech recognition software is commercially available, its accuracy in mental health settings has not been well described. It is unclear which metrics and thresholds are appropriate for different clinical use cases, which may range from population descriptions to individual safety monitoring. Here we show that automatic speech recognition is feasible in psychotherapy, but further improvements in accuracy are needed before widespread use. Our HIPAA-compliant automatic speech recognition system demonstrated a transcription word error rate of 25%. For depression-related utterances, sensitivity was 80%...
Research Paper
Assessing the accuracy of automatic speech recognition for psychotherapy

Accurate transcription of audio recordings in psychotherapy would improve therapy effectiveness, clinician training, and safety monitoring. Although automatic speech recognition software is commercially available, its accuracy in mental health settings has not been well described. It is unclear which metrics and thresholds are appropriate for different clinical use cases, which may range from population descriptions to individual safety monitoring. Here…

Jun 03, 2020

A. Miner, et al.

Learn more about Assessing the accuracy of automatic speech recognition for psychotherapy
Ivy: Instrumental Variable Synthesis for Causal Inference
A popular way to estimate the causal effect of a variable x on y from observational data is to use an instrumental variable (IV): a third variable z that affects y only through x. The more strongly z is associated with x, the more reliable the estimate is, but such strong IVs are difficult to find. Instead, practitioners combine more commonly available IV candidates—which are not necessarily strong, or even valid, IVs—into a single "summary" that is plugged into causal effect estimators in place of an IV. In genetic epidemiology, such approaches are known as allele scores. Allele scores require...
Research Paper
Ivy: Instrumental Variable Synthesis for Causal Inference

A popular way to estimate the causal effect of a variable x on y from observational data is to use an instrumental variable (IV): a third variable z that affects y only through x. The more strongly z is associated with x, the more reliable the estimate is, but such strong IVs are difficult to find. Instead, practitioners combine more…

Jun 02, 2020

Z. Kuang, et al.

Learn more about Ivy: Instrumental Variable Synthesis for Causal Inference
The accuracy vs. coverage trade-off in patient-facing diagnosis models
A third of adults in America use the Internet to diagnose medical concerns, and online symptom checkers are increasingly part of this process. These tools are powered by diagnosis models similar to clinical decision support systems, with the primary difference being the coverage of symptoms and diagnoses. To be useful to patients and physicians, these models must have high accuracy while covering a meaningful space of symptoms and diagnoses. To the best of our knowledge, this paper is the first in studying the trade-off between the coverage of the model and its performance for diagnosis. To this end, we learn...
Research Paper
The accuracy vs. coverage trade-off in patient-facing diagnosis models

A third of adults in America use the Internet to diagnose medical concerns, and online symptom checkers are increasingly part of this process. These tools are powered by diagnosis models similar to clinical decision support systems, with the primary difference being the coverage of symptoms and diagnoses. To be useful to patients and physicians, these models must have high accuracy…

May 30, 2020

A. Kannan, et al.

Learn more about The accuracy vs. coverage trade-off in patient-facing diagnosis models
Extracting chemical reactions from text using Snorkel
Enzymatic and chemical reactions are key for understanding biological processes in cells. Curated databases of chemical reactions exist but these databases struggle to keep up with the exponential growth of the biomedical literature. Conventional text mining pipelines provide tools to automatically extract entities and relationships from the scientific literature, and partially replace expert curation, but such machine learning frameworks often require a large amount of labeled training data and thus lack scalability for both larger document corpora and new relationship types. We developed an application of Snorkel, a weakly supervised learning framework, for extracting chemical reaction relationships from biomedical literature...
Research Paper
Extracting chemical reactions from text using Snorkel

Enzymatic and chemical reactions are key for understanding biological processes in cells. Curated databases of chemical reactions exist but these databases struggle to keep up with the exponential growth of the biomedical literature. Conventional text mining pipelines provide tools to automatically extract entities and relationships from the scientific literature, and partially replace expert curation, but such machine learning frameworks often…

May 27, 2020

E. Mallory, et al.

Learn more about Extracting chemical reactions from text using Snorkel
AMELIE speeds Mendelian diagnosis by matching patient phenotype and genotype to primary literature
The diagnosis of Mendelian disorders requires labor-intensive literature research. Trained clinicians can spend hours looking for the right publication(s) supporting a single gene that best explains a patient’s disease. AMELIE (Automatic Mendelian Literature Evaluation) greatly accelerates this process. AMELIE parses all 29 million PubMed abstracts and downloads and further parses hundreds of thousands of full-text articles in search of information supporting the causality and associated phenotypes of most published genetic variants. AMELIE then prioritizes patient candidate variants for their likelihood of explaining any patient’s given set of phenotypes. Diagnosis of singleton patients (without relatives’ exomes) is the most time-consuming scenario,...
Research Paper
AMELIE speeds Mendelian diagnosis by matching patient phenotype and genotype to primary literature

The diagnosis of Mendelian disorders requires labor-intensive literature research. Trained clinicians can spend hours looking for the right publication(s) supporting a single gene that best explains a patient’s disease. AMELIE (Automatic Mendelian Literature Evaluation) greatly accelerates this process. AMELIE parses all 29 million PubMed abstracts and downloads and further parses hundreds of thousands of full-text articles in search of information…

May 20, 2020

J. Birgmeier, et al.

Learn more about AMELIE speeds Mendelian diagnosis by matching patient phenotype and genotype to primary literature
Towards Curiosity-Driven Learning of Physical Dynamics
Throughout our lives, we as humans acquire an intuitive understanding of our physical environments, a capacity that supports our imagination and planning abilities. Driven by our own curiosity, we learn about object motion and properties via self-curated targeted experiments, that teach us what we do not know. Recently, neural network models have been proposed that learn forward object dynamics from observations like humans. Unlike humans, these models do not actively interact with surrounding objects but learn from human-curated datasets as passive observers. In this work-in-progress, we propose a closed-loop system that teaches itself about forward object dynamics without any human...
Research Paper
Towards Curiosity-Driven Learning of Physical Dynamics

Throughout our lives, we as humans acquire an intuitive understanding of our physical environments, a capacity that supports our imagination and planning abilities. Driven by our own curiosity, we learn about object motion and properties via self-curated targeted experiments, that teach us what we do not know. Recently, neural network models have been proposed that learn forward object dynamics from…

Apr 26, 2020

MJ. Lingelbach, et al.

Learn more about Towards Curiosity-Driven Learning of Physical Dynamics
Weakly Supervised Sequence Tagging from Noisy Rules
We propose a framework for training sequence tagging models with weak supervision consisting of multiple heuristic rules of unknown accuracy. In addition to supporting rules that vote on tags in the output sequence, we introduce a new type of weak supervision, called linking rules, that vote on how sequence elements should be grouped into spans with the same tag. These rules are an alternative to candidate span generators that require significantly more human effort. To estimate the accuracies of the rules and combine their conflicting outputs into training data, we introduce a new type of generative model, linked hidden Markov...
Research Paper
Weakly Supervised Sequence Tagging from Noisy Rules

We propose a framework for training sequence tagging models with weak supervision consisting of multiple heuristic rules of unknown accuracy. In addition to supporting rules that vote on tags in the output sequence, we introduce a new type of weak supervision, called linking rules, that vote on how sequence elements should be grouped into spans with the same tag. These…

Apr 03, 2020

E. Safranchik, et al.

Learn more about Weakly Supervised Sequence Tagging from Noisy Rules
Weakly Supervised Classification of Aortic Valve Malformations Using Unlabeled Cardiac MRI Sequences
This work formalizes a deep learning baseline for aortic valve classification and outlines a general strategy for using weak supervision to train machine learning models using unlabeled medical images at scale.
Research Paper
Weakly Supervised Classification of Aortic Valve Malformations Using Unlabeled Cardiac MRI Sequences

This work formalizes a deep learning baseline for aortic valve classification and outlines a general strategy for using weak supervision to train machine learning models using unlabeled medical images at scale.

Dec 20, 2019

J. Fries, et al, 2019

Learn more about Weakly Supervised Classification of Aortic Valve Malformations Using Unlabeled Cardiac MRI Sequences
Utilizing Weak Supervision to Infer Complex Objects in Autonomous Driving Data
This paper explores the applicability of weak supervision, or relying on higher level, noisier forms of supervision to label training data, specifically using data programming.
Research Paper
Utilizing Weak Supervision to Infer Complex Objects in Autonomous Driving Data

This paper explores the applicability of weak supervision, or relying on higher level, noisier forms of supervision to label training data, specifically using data programming.

Dec 19, 2019

Z. Wheng, et al, 2019

Learn more about Utilizing Weak Supervision to Infer Complex Objects in Autonomous Driving Data
1 60 61 62 65
Image
Image

Join our newsletter

For expert advice, the latest research, and exclusive events.
By submitting this form, I acknowledge I will receive email updates from Snorkel AI, and I agree to the Terms of Use and acknowledge that my information will be used in accordance with the Privacy Policy.