Learned Error Objectives explores training a language model with an additional error-optimization network. A transformer produces representations, and the error network participates in constructing the training objective.
The experiments use a nanoGPT-style model and WikiText data, with separate optimizers for the two components. Several variants examine how the error network connects to the transformer and how the training signal is formed. The formulation includes likelihood terms alongside error-based terms.
The open question is whether the learned component contributes a useful signal beyond a conventional language-model loss. Evaluating that needs comparable training conditions and a stable objective, while separating changes to the transformer from changes to the network that scores it.