Persistent Attention Memory explores keeping part of a transformer's key/value memory across inputs. The model combines ordinary attention information with a persistent component and updates that component at detected semantic boundaries.
The experiment includes a GPT-2-based model, boundary detection, and training and comparison scripts. It also investigates organizing memory using questions such as who, what, where, and why.
The intended benefit is continuity across inputs without treating every interaction as independent. Whether that helps depends on what the memory retains, when it changes, and how stale information is handled. Persistence alone is not evidence of understanding or effective continual learning. Those remain questions for controlled comparisons.