Memory Diffusion Blocks explores a language model whose starting state depends on its history. A causal context network addresses routed fast-weight memory, reads a source state, and passes it through correction blocks toward a next-token representation.
The name refers to the project's development history. The current formulation is a learned-source transport model, rather than Gaussian diffusion or a reproduction of the DiffusionBlocks paper.
Questions
The default correction preserves a defined relationship between the primary state and the memory response. Other experiments change the source distribution, construct categorical endpoints, or train blocks independently with explicit source conditioning.
The difficult boundary is the difference between learning a constructed intermediate target and producing useful predictions when the blocks are run together. Some variants learn the training task but degrade during rollout.
The comparisons therefore need both block-level diagnostics and final next-token evaluation. A smaller reconstruction error is useful evidence about a state transformation; it is not sufficient evidence that the complete language model improved.