Can a learning algorithm recover the underlying rule, rather than merely interpolate the training data? We study program learning, where the learned hypothesis is an executable program. Our propose-and-verify approach uses pretrained language models as structured priors over programs and held-out data to select among candidates, making ERM-style learning over program classes computationally practical without gradient updates or exhaustive enumeration. On algorithmic tasks, the resulting programs can recover exact rules from a few hundred examples and generalize far beyond the training regime, while gradient-trained models can fit vastly more data yet fail to extrapolate.

ICML 2026 · Program Learning · Learning Theory · LLMs

General-purpose algorithms are designed to solve broad classes of inputs, while real computational workloads often consist of repeated draws from much narrower distributions. We introduced distribution-aware programming, where examples from a deployment distribution reveal recurring computational structure that can be compiled into specialized solver code. Across structured optimization problems spanning multiple problem classes, the resulting programs preserve near-baseline solution quality while substantially reducing computation. The broader principle is that algorithms themselves can be learned from the distributions on which they will run.

2026 · Program Synthesis · Algorithm Design · LLM Agents

Scaling the underlying model is not the only way to improve reasoning. We study when generation plus verification can transform weak model calls into substantially stronger systems. Our analysis separates the ability to generate a correct candidate from the ability to identify it, showing that repeated sampling alone is insufficient: reliable amplification requires an independent soundness signal such as execution, tests, proof checking, type checking, or constraint solving. On software-engineering tasks, verifier-backed orchestration substantially improves weak-model performance and approaches the model's best-of-many oracle ceiling. The broader principle is that verification can convert latent capability into reliable performance.

2026 · Reasoning · Verification · Agents · DisCO

Why do pretrained representations support new tasks from only a few examples? Our work develops a geometric theory of transfer spanning supervised and self-supervised learning. We introduced CDNV to quantify class collapse and its connection to few-shot transfer, and later showed that in SSL the key quantity is directional CDNV: variability must shrink along downstream decision directions, not everywhere. This explains both strong few-shot transfer and the emergence of nearly orthogonal semantic directions. Complementary work shows how SSL can acquire this geometry by behaving increasingly like supervised learning in representation space.

ICLR 2022 · NeurIPS 2023 · NeurIPS 2025 · ICLR 2026 · ICML 2026 · JMLR 2026 · Few-Shot Transfer · Self-Supervised Learning · SSL Alignment

Why do deep networks spontaneously develop low-dimensional and highly structured internal representations? Our work studies the optimization mechanisms responsible for this organization. We showed that stochastic gradient descent with weight decay has an intrinsic bias toward low-rank solutions, with the effect strengthening under smaller batches, larger learning rates, and stronger regularization. In complementary work, we introduced the Canonical Representation Hypothesis, which identifies systematic alignment among representations, weights, and gradients and connects phenomena including neural collapse and the neural feature ansatz. The goal is a predictive theory of how optimization produces the structured representations on which generalization and transfer depend.

ICLR 2025 Spotlight · CPAL 2025 · Optimization · Deep Learning Theory