Question
What changes when token representations can attend to a shared set of learned prototypes?
Approach
The landscape-attention module compares token queries with learned prototype keys and combines the corresponding values. A transformer block adds this context before its feed-forward network.
The training script uses masked language modeling, with WikiText-2 and a BERT tokenizer as its defaults.
Open question
The prototypes provide an additional source of context and additional parameters. A comparison needs to distinguish the effect of that structure from simply increasing model capacity. No comparative result is claimed here.