AutoTarget Transformer explores learning the target representation used to train a transformer. An encoder produces a representation of the input, while a separate target generator proposes what that representation should move toward.
The experiments include encoder–decoder and decoder-only variants. One version also uses a language model to score generated output, with that score feeding updates to the target generator and decoder.
The central question is how to choose useful internal training targets when a fixed target may be too restrictive. These are experimental training scripts, with several variants still present. They do not establish an improvement over ordinary supervised training or reinforcement learning. Comparing the target generator against a fixed-target baseline remains essential to evaluating the idea.