Activation Neural Network is a character-level text-generation experiment using Tiny Shakespeare. It combines an MLP expansion network with an attention-based character decoder and generates text autoregressively.
The work includes a separate baseline model and training paths for both approaches. The point of that comparison is to understand what the intermediate activation representation contributes to the next-character prediction task.
Baseline development is still part of the work. The experiment is best understood as a way to examine the representation and decoder together, with model size and training conditions held comparable before drawing conclusions.