Resource library
Evaluating Coding Agent Capabilities with Terminal-Bench: Snorkel’s Role in Building the Next Generation Benchmark
Closing the Evaluation Gap in Agentic AI
Announcing a $3M commitment to launch Open Benchmarks Grants
Benchtalks #1: Alex Shaw (Terminal-Bench, Harbor) – Building the Benchmark Factory
Building FinQA: An Open RL Environment for Financial Reasoning Agents
McKinsey’s Carlo Giovine and David Harvey present “Trends in Enterprise ML and the potential impact of Foundation Models” at Snorkel’s Foundation Model Summit.
Dillon Laird, engineering manager at Landing AI, presents on LandingLens and democratizing AI at Snorkel AI’s 2022 FDCAI Conference.
Jay Alammar, director and engineering fellow at Cohere, presents strategies to enhance the value of Generative AI.
This paper proposes generalizations of CWOLA and SALAD, which exploit multiple reference datasets to improve performance in resonant anomaly detection, and provides finite-sample guarantees to go beyond existing asymptotic analyses.
Stanford researchers concluded that new, larger and more powerful foundation models represent a paradigm shift in AI, providing opportunities and risks that require deep interdisciplinary collaboration to understand and address.
This paper proposes “Ask Me Anything” (AMA), a prompting method that uses weak supervision to combine noisy predictions from multiple prompts generated from an LLM, resulting in an average 10.2% performance lift over the few-shot baseline across a variety of different open-source models.
The authors propose Contrastive Adapting, an efficient adapter training strategy that improves the group robustness of large pretrained foundation models (FMs) without finetuning, leading to up to 56.0 percentage points of increase in accuracy compared to zero-shot.
Zero-shot learning with Common Sense Knowledge Graphs is a general-purpose framework with a novel transformer graph convolutional network for generating class representations from common sense knowledge graphs, which improves over existing WordNet-based methods on zero-shot learning tasks.
This paper demonstrates that WEAPO, a Weak Supervision method for binary classification tasks with only positive labeling sources, is effective and efficient—achieving the highest performance of the tested Weak Supervision approaches in terms of label quality and final classifier accuracy on 10 benchmark datasets.
