Continual learningMeta-learning
Continual Learning in Diffusion Models
with Christopher MacLellan
We show that the empirical Fisher of a diffusion model is rank-1 in low-SNR regimes.
Using this, we define a rank-1 EWC penalty and show that, combined with replay, the
model forgets less. We then show that this formulation is equivalent to one-step
MAML under local approximations, and observe the effect directly: the model
re-learns previous tasks faster than other methods.
Constraining how much you update is already a form of learning to
learn.
ICLR
2026NeurIPS 2026
Meta-continual learning
Self-Consolidating Language Models
with Christopher MacLellan
The same intuition as the diffusion work, but at LLM scale the Fisher is too
expensive to compute, so SCoL lets the model learn it: meta-RL decides which of
its own layers to change when it absorbs a passage. Trained only on short
contexts, it beats in-context baselines that see the whole passage on LongBench
v2.
Consolidation with autonomy: the model decides where new knowledge
goes.
Preprint
Continual learning
Hierarchical and Incremental Concept Formation
with Christopher MacLellan
Cobweb builds a concept hierarchy incrementally, one example at a time. As a
retrieval index it matches dense encoders and stays robust where kNN collapses;
CobwebTM extends it to lifelong hierarchical topic modeling and matches or beats prior
incremental methods.
The external-memory side: a retrieval structure that is itself
continual.
ACS 2025ACL Findings 2026
Transfer/composition
Multi-Role Reinforcement Learning
with Christopher MacLellan · ongoing
One language model is trained in several study roles over the same examples —
comparing problems, identifying subgoals, explaining. Roles that never produce an
answer still improve answer accuracy.
People study the same material in several ways; so can a model.
Preprint
Transfer/composition
Test-Time Compositional Generalization
with Christopher MacLellan
From a single out-of-distribution query, recover concept prototypes by score-based
mode finding across the noise levels of a pretrained diffusion model, then compose
them into a product-of-experts teacher.
People compose familiar concepts freely; here a diffusion model
does too.
Preprint
Human errors
Simulating Student Misconceptions
with Vijay Ganesh · ongoing
Given a specific misconception, can a model produce the wrong answer a real student
holding it would produce, in a single turn? A counterfactual generation problem.
Human-like errors can benchmark and train models; here, for grading
theoretical computer science, where partial credit is hard.