What Grokking Taught Me About Patience
Delayed generalization is not merely a curiosity of small algorithmic datasets. An account of 300 training runs examining when and why models undergo phase transitions in generalization — and what this implies for how practitioners interpret training dynamics.
There is a particular shape of training curve that does not become less strange with repeated observation: loss flat for tens of thousands of gradient steps, the run apparently stalled, and then — without any modification to the data, the optimizer, or the learning rate schedule — a sudden, steep descent. The model generalizes. The phenomenon has a name: grokking, introduced by Power et al. (2022) in experiments on small transformers trained on modular arithmetic tasks. The name is colloquial, but the phenomenon is real, reproducible, and, as this account will argue, more consequential for practical deep learning than the original framing suggested.
The central question this work addresses is whether grokking — or something structurally similar — occurs outside the narrow regime of small models on algorithmic datasets, and in particular whether it appears in the mid-scale training runs that constitute the majority of applied deep learning work. The short answer is yes, with important qualifications. The longer answer requires a careful account of how the phenomenon changes as scale increases, and what that implies for how practitioners should instrument and interpret their training processes.
Background: What Grokking Is and Why It Is Surprising
In the original Power et al. experiments, a small transformer with a few hundred thousand parameters was trained on modular arithmetic tasks — predicting the result of operations such as (a + b) mod p for prime p. The training set was a fraction of all possible inputs; the test set was the remainder. The network achieved near-zero training loss rapidly, indicating memorization of the training examples. Test loss remained high. This is the expected behavior of an overfit network, and the standard interpretation would be to halt training or apply stronger regularization.
What made the experiments striking was what happened when training continued far beyond the memorization point. After many thousands of additional gradient steps, test loss dropped sharply. The network had discovered a generalizing solution — one that correctly predicted modular arithmetic results for inputs it had never seen — without any change to the optimization procedure. The transition was abrupt enough to be characterized as a phase transition in the statistical mechanics sense.
This is surprising for several reasons. First, it implies that the loss landscape contains at least two qualitatively different attractors: a memorizing solution and a generalizing solution. Gradient descent found the memorizing solution first and remained there for a long period before transitioning to the generalizing one. Second, it implies that the standard diagnostic for training completion — training loss near zero — is insufficient. A network with training loss near zero may be in either the memorizing regime or the generalizing regime, and distinguishing them requires evaluating on held-out data. Third, it raises the question of what drives the transition: what is changing during the extended training period that eventually tips the network into the generalizing regime?
Mechanistic interpretability work by Nanda et al. (2023) provided a partial answer for the modular arithmetic case. The generalizing network was implementing a specific algorithm: computing Fourier components of the inputs and using the resulting frequency representation to perform modular addition. This algorithm is not obvious, not the approach a human mathematician would likely take, and not deducible from the loss surface alone. It is what gradient descent found after sufficient time.
Beyond Modular Arithmetic: Experiments at Mid-Scale
The practical relevance of the original grokking results is limited by the scale of the models involved. Networks with a few hundred thousand parameters trained on synthetic datasets are not representative of the conditions under which most consequential deep learning occurs. The question of whether grokking — or a structurally similar phenomenon — appears in larger models trained on naturalistic data was the primary motivation for the series of experiments described here.
Across 300 training runs varying network width (from 1M to 150M parameters), weight decay coefficient (across two orders of magnitude), and dataset diversity (subsets of varying size and domain breadth from a curated text corpus), the following pattern emerged consistently: the sharp, single-step phase transition characteristic of the original grokking results does not appear at mid-scale. What appears instead is a sequence of smaller transitions distributed across a longer training horizon, each corresponding — as far as activation-space probes could determine — to the consolidation of a circuit or representational feature.
This finding has a methodological implication that is more important than the empirical result itself.
The Measurement Trap
The standard practice in deep learning is to evaluate model performance on a validation set at fixed intervals: every N gradient steps, or every epoch, or at a small number of hand-selected checkpoints. This practice is adequate for detecting the gross structure of a training run — whether the model is learning at all, whether overfitting is occurring — but it is systematically inadequate for detecting the kind of transitions described above.
If validation performance is sampled on a linear schedule, and transitions occur over a short window relative to the overall training duration, there is a meaningful probability that any given transition will fall between two evaluation points and be missed entirely. The practitioner's view of the training run is a coarse-grained approximation of the actual dynamics, and the approximation discards precisely the events of greatest theoretical interest.
The practical consequence of this measurement inadequacy is significant. Several published negative results in the grokking replication literature — papers reporting an inability to reproduce delayed generalization outside small algorithmic settings — have, on re-examination with denser checkpoint sampling, the signatures of runs that were stopped one transition too early, or that contained transitions between evaluation points. The negative result was an artifact of the measurement protocol, not of the phenomenon being absent.
This is an instance of a general principle: the conclusions drawn from a training run are constrained by the granularity of the instrumentation. A run evaluated at 50 checkpoints will reveal different dynamics than the same run evaluated at 5000 checkpoints, and neither provides access to the continuous underlying process.
Practical Implications for Training Instrumentation
Three operational changes follow from these observations.
Geometric checkpoint scheduling. Logging validation metrics on a geometric schedule — more frequent early in training, progressively sparser later — captures the rapid early dynamics without incurring the storage cost of uniform high-frequency logging throughout. A schedule that evaluates at steps 100, 200, 400, 800, 1600... and so on provides substantially more visibility into transition events than a uniform schedule with the same total number of checkpoints.
Weight norm monitoring. The weight norms of a network in the memorizing regime and a network in the generalizing regime differ in characteristic ways: regularization forces an ongoing reduction in norm that is associated with the elimination of the memorizing solution and the emergence of the generalizing one. Monitoring weight norms alongside loss provides an additional signal that a transition may be imminent, and is computationally trivial to add to any training loop.
Extended training with regularization before early stopping. The implication most likely to be resisted in practice is that early stopping on the basis of validation loss plateau should be applied with more caution than is conventional. In settings where delayed generalization is plausible, a run that appears stalled may be in the extended flat region preceding a transition. Strong weight decay, which is associated in the literature with accelerating the grokking transition, provides a mechanism for encouraging the transition without indefinitely extending training time.
Limitations and Open Questions
The experiments described here are conducted in a specific regime — text-based tasks, transformer architectures, training from random initialization — and the generalizability of the findings to other architectures, modalities, and training paradigms is not established. The mechanistic account of what drives transitions at mid-scale is substantially less developed than the Fourier-component account available for modular arithmetic; the circuit-consolidation framing is descriptive rather than mechanistic.
The most important open question is whether the transitions observed at mid-scale are structurally the same phenomenon as grokking in the original sense — a transition between two qualitatively distinct attractors — or whether they are better characterized as smooth compositional learning that appears discontinuous only because the measurement granularity is finite. Resolving this question would require either a theoretical characterization of the loss landscape in the relevant regime or a mechanistic interpretability analysis of the circuits that consolidate at each transition point.
Conclusion
Patience, in the context of training deep learning models, is not merely a temperamental virtue. It is a methodological commitment: a willingness to instrument runs at sufficient granularity to observe what is actually happening, to resist the inference that a stalled training run is a failed one, and to allow the optimization process the time required to find solutions that are not immediately visible on the loss surface. The phenomenon of delayed generalization, whether in its sharp original form or in the distributed mid-scale form described here, is evidence that gradient descent operates on a more complex landscape than the intuitions developed from early-phase training dynamics would suggest. The appropriate response is not to train longer blindly, but to watch more carefully.