We define and advance data and environments to push the AI frontier

Built on 10+ years of pioneering research in data-centric AI,
including 250+ publications and benchmarks.

building benchmarks and collaborating with

Image
Image
Image
Image
Image
Image
Image
Image
Image
key research areas

Vision and impact

We help labs advance frontier models by working with domain experts to design and build complex, realistic datasets that drive model performance.

initiatives

Community and open science

Open benchmarks, conversations, and research for real-world AI performance.

Image

Open Benchmarks Grants

Backed by a $3M commitment, the program funds
open-source datasets, benchmarks, and evaluation artifacts that shape how frontier AI systems are built
and evaluated.

Image

Bench Talks

Our podcast series at the intersection of AI evaluation, data quality, and real-world impact.
Image

Reading Group

A recurring forum for researchers and practitioners to explore the latest frontier developments in AI while building meaningful connections within the community.

DEEP RESEARCH Expertise

Technical advisors and distinguished affiliates

Stephen Bach headshot

Stephen Bach

Brown University
Eliot Horowitz Assistant Professor, Computer Science Department
Jason Fries headshot

Jason Fries

Stanford University
Assistant Professor of Biomedical Data Science and of Medicine
Jared Dunnmon headshot

Jared Dunnmon

Co-Founder & Chief Scientist, Stealth Startup
Prev. Dir. of AI at DIU
Fred Sala headshot

Fred Sala

Chief Scientist
,
Snorkel AI
Assistant Professor @ University of Wisconsin-Madison
Chris Ré headshot

Chris Ré

Co-Founder
,
Snorkel AI
Professor @ Stanford University
Ludwig Schmidt headshot

Ludwig Schmidt

Stanford University · LAION
Stanford researcher and LAION collaborator
Karthik Narasimhan headshot

Karthik Narasimhan

Princeton University
Professor of Computer Science
Yu Su headshot

Yu Su

Ohio State University
Associate Professor of Computer Science and Engineering
Lewis Tunstall headshot

Lewis Tunstall

Hugging Face
Machine Learning Engineer
PUBLICATIONS

Browse research blogs
and academic papers

Type: All Types
Sort: Newest
Assessment of Extractability and Accuracy of Electronic Health Record Data for Joint Implant Registries
Are the data in a large US electronic health record (EHR) complete and accurate enough to track trends in implant use and to assess the durability of implants (hereafter referred to as implant survivorship)? In this cohort study, EHR records of patients who had total hip arthroplasty in all Veterans Health Administration hospitals since 2000 were automatically reviewed using novel software; 80% to 95% of hip replacement components used since 2014 were accurately identified, trends in implant use matched known national trends, and known poor implants were found to be negative outliers. This suggests that automated analysis of the EHR...
Research Paper
Assessment of Extractability and Accuracy of Electronic Health Record Data for Joint Implant Registries

Are the data in a large US electronic health record (EHR) complete and accurate enough to track trends in implant use and to assess the durability of implants (hereafter referred to as implant survivorship)? In this cohort study, EHR records of patients who had total hip arthroplasty in all Veterans Health Administration hospitals since 2000 were automatically reviewed using novel…

Mar 15, 2021

NJ. Giori, et al.

Learn more about Assessment of Extractability and Accuracy of Electronic Health Record Data for Joint Implant Registries
Blog
Measuring NLP Progress With Sebastian Ruder

In this episode of Science Talks, Sebastian Ruder, Research Scientist at DeepMind, shares his thoughts on making AI practical with Snorkel AI’s Braden Hancock. This conversation covers progress made in the NLP domain with emerging research, new benchmarks like SuperGLUE, rich repositories and news sources that keep you in the loop and on top of what’s new in NLP, and more.

Mar 10, 2021
Learn more about Measuring NLP Progress With Sebastian Ruder
Blog
Productionizing ML Research With Thomas Wolf

In this episode of ScienceTalks, Snorkel AI’s Braden Hancock Hugging Face’s Chief Science Officer, Thomas Wolf. Thomas shares his story about how he got into machine learning and discusses important design decisions behind the widely adopted Transformers library, as well as the challenges of bringing research projects into production. ScienceTalks is an interview series from Snorkel AI, highlighting some of the best work and ideas to make AI practical.

Feb 05, 2021
Learn more about Productionizing ML Research With Thomas Wolf
Cut out the annotator, keep the cutout: better segmentation with weak supervision
Constructing large, labeled training datasets for segmentation models is an expensive and labor-intensive process. This is a common challenge in machine learning, addressed by methods that require few or no labeled data points such as few-shot learning (FSL) and weakly-supervised learning (WS). Such techniques, however, have limitations when applied to image segmentation—FSL methods often produce noisy results and are strongly dependent on which few datapoints are labeled, while WS models struggle to fully exploit rich image information. We propose a framework that fuses FSL and WS for segmentation tasks, enabling users to train high-performing segmentation networks with very few hand-labeled...
Research Paper
Cut out the annotator, keep the cutout: better segmentation with weak supervision

Constructing large, labeled training datasets for segmentation models is an expensive and labor-intensive process. This is a common challenge in machine learning, addressed by methods that require few or no labeled data points such as few-shot learning (FSL) and weakly-supervised learning (WS). Such techniques, however, have limitations when applied to image segmentation—FSL methods often produce noisy results and are strongly…

Jan 12, 2021

S. Hooper, et al.

Learn more about Cut out the annotator, keep the cutout: better segmentation with weak supervision
Background Splitting: Finding Rare Classes in a Sea of Background
We focus on the problem of training deep image classification models for a small number of extremely rare categories. In this common, real-world scenario, almost all images belong to the background category in the dataset. We find that state-of-the-art approaches for training on imbalanced datasets do not produce accurate deep models in this regime. Our solution is to split the large, visually diverse background into many smaller, visually similar categories during training. We implement this idea by extending an image classification model with an additional auxiliary loss that learns to mimic the predictions of a pre-existing classification model on the...
Research Paper
Background Splitting: Finding Rare Classes in a Sea of Background

We focus on the problem of training deep image classification models for a small number of extremely rare categories. In this common, real-world scenario, almost all images belong to the background category in the dataset. We find that state-of-the-art approaches for training on imbalanced datasets do not produce accurate deep models in this regime. Our solution is to split the…

Jan 01, 2021

RT. Mullapudi, et al.

Learn more about Background Splitting: Finding Rare Classes in a Sea of Background
Language models are an effective representation learning technique for electronic health record data
Widespread adoption of electronic health records (EHRs) has fueled the development of using machine learning to build prediction models for various clinical outcomes. However, this process is often constrained by having a relatively small number of patient records for training the model. We demonstrate that using patient representation schemes inspired from techniques in natural language processing can increase the accuracy of clinical prediction models by transferring information learned from the entire patient population to the task of training a specific model, where only a subset of the population is relevant. Such patient representation schemes enable a 3.5% mean improvement in...
Research Paper
Language models are an effective representation learning technique for electronic health record data

Widespread adoption of electronic health records (EHRs) has fueled the development of using machine learning to build prediction models for various clinical outcomes. However, this process is often constrained by having a relatively small number of patient records for training the model. We demonstrate that using patient representation schemes inspired from techniques in natural language processing can increase the accuracy…

Jan 01, 2021

E. Steinberg, et al.

Learn more about Language models are an effective representation learning technique for electronic health record data
Leveraging Organizational Resources to Adapt Models to New Data Modalities
This work demonstrates how organizational resources, in the form of aggregate statistics, knowledge bases, and existing services can be used to connect new and existing data modalities.
Research Paper
Leveraging Organizational Resources to Adapt Models to New Data Modalities

This work demonstrates how organizational resources, in the form of aggregate statistics, knowledge bases, and existing services can be used to connect new and existing data modalities.

Nov 23, 2020

S. Suri, et al, 2020

Learn more about Leveraging Organizational Resources to Adapt Models to New Data Modalities
Parameterizing neural power spectra into periodic and aperiodic components
Electrophysiological signals exhibit both periodic and aperiodic properties. Periodic oscillations have been linked to numerous physiological, cognitive, behavioral and disease states. Emerging evidence demonstrates that the aperiodic component has putative physiological interpretations and that it dynamically changes with age, task demands and cognitive states. Electrophysiological neural activity is typically analyzed using canonically defined frequency bands, without consideration of the aperiodic (1/f-like) component. We show that standard analytic approaches can conflate periodic parameters (center frequency, power, bandwidth) with aperiodic ones (offset, exponent), compromising physiological interpretations. To overcome these limitations, we introduce an algorithm to parameterize neural power spectra as a combination...
Research Paper
Parameterizing neural power spectra into periodic and aperiodic components

Electrophysiological signals exhibit both periodic and aperiodic properties. Periodic oscillations have been linked to numerous physiological, cognitive, behavioral and disease states. Emerging evidence demonstrates that the aperiodic component has putative physiological interpretations and that it dynamically changes with age, task demands and cognitive states. Electrophysiological neural activity is typically analyzed using canonically defined frequency bands, without consideration of the aperiodic…

Nov 23, 2020

T. Donoghue, et al.

Learn more about Parameterizing neural power spectra into periodic and aperiodic components
Fast and Three-Rious: Speed up Weak Supervision With Triplet Methods
Introducing FlyingSquid, a weak supervision framework that runs orders of magnitude faster than previous weak supervision approaches and requires fewer assumptions
Research Paper
Fast and Three-Rious: Speed up Weak Supervision With Triplet Methods

Introducing FlyingSquid, a weak supervision framework that runs orders of magnitude faster than previous weak supervision approaches and requires fewer assumptions

Nov 20, 2020

D. Fu, et al, 2020

Learn more about Fast and Three-Rious: Speed up Weak Supervision With Triplet Methods
1 28 29 30 35
Image

Let’s research together

Join our team of leading researchers and help shape the future of AI.