We define and advance data and environments to push the AI frontier
building benchmarks and collaborating with
Featured research
Vision and impact
We help labs advance frontier models by working with domain experts to design and build complex, realistic datasets that drive model performance.
Benchmarking &
Evaluation
Build benchmarks that define and advance the AI frontier
Scaling Subject Matter Expertise
Define how subject matter experts encode their knowledge into data
RL, Training, & Data Valuation
Drive dataset development based on feedback from RL and model training
Community and open science
Open benchmarks, conversations, and research for real-world AI performance.
Open Benchmarks Grants
Backed by a $3M commitment, the program funds open-source datasets, benchmarks, and evaluation artifacts that shape how frontier AI systems are built and evaluated.
Bench Talks
Reading Group
DEEP RESEARCH Expertise
Technical advisors and distinguished affiliates
Browse research blogs and academic papers
The surest way to improve foundation models is through more and better data, but Snorkel researchers showed FMs can learn from themselves.
Generative AI can write poems, recite common knowledge, and extract information. GenAI can also help quickly build predictive pipelines.
Users and organizations are generating ever-increasing amounts of private data from a wide range of sources. Incorporating private context is important to personalize open-domain tasks such as question-answering, fact-checking, and personal assistants. State-of-the-art systems for these tasks explicitly retrieve information that is relevant to an input question from a background corpus before producing an answer. While today’s retrieval systems assume…
Getting better performance from foundation models (with less data)
GenAI may be the most transformative technology of the past decade but data is where enterprises are able to realize real value from AI today.
The paper explores the use of pseudolabels, which are heuristic labels for unlabeled data, to enhance the performance of vision-language models like CLIP via prompt tuning. The authors investigate different learning paradigms and prompt modalities and find that iterative prompt-training strategies leveraging CLIP-based pseudolabels lead to significant improvements in CLIP’s image classification performance.
The paper introduces Alfred, a system for programmatic weak supervision (PWS) that creates training data for machine learning by prompting. It enables users to encode their subject matter expertise via natural language prompts for language and vision-language models.
The paper proposes a statistical label model called FABLE that incorporates instance features to improve the accuracy of inferred truth in Programmatic Weak Supervision (PWS). FABLE is built on a mixture of Bayesian label models, where the coefficients of the mixture components are predicted by a Gaussian Process classifier based on instance features.
We used weak supervision to programmatically curate instruction tuning data for open-source LLMs to build a better GenAI.
