These experiments ask what changes when a language model's output representation or classification objective changes. Two related tracks keep different questions separate.
Hyperspherical losses
A shared decoder-only Transformer compares ordinary softmax with objectives based on normalized representations and output code vectors. Variants change pairwise margins, hard-negative selection, and alignment terms.
Learning rate and seed comparisons are part of the experiment. Raw loss values are not compared across objectives with different units; held-out token prediction and representation diagnostics provide the relevant measurements.
Recorded comparison
The first sweep completed 63 runs: seven variants, three learning rates, and three seeds. Each variant's learning rate was selected on validation accuracy before reporting held-out token accuracy.
Softmax reached 21.57% ± 0.99% held-out top-one accuracy. The hard-negative objective with a budget of 16 reached 20.09% ± 0.87%, compared with 16.00% ± 0.08% for uniform pairwise BCE. These spreads are sample standard deviations across three seeds. Hard-negative selection improved this pairwise objective, but did not outperform softmax in this comparison.
Complex-valued models
A smaller pilot and a small Transformer explore complex softmax, an additional phase objective, and complex Adam updates. Magnitudes determine token probabilities while phases supply a separate prediction target.
This track first checks trainability and numerical behavior. It has not established an advantage over a matched conventional softmax, cross-entropy, and Adam setup. A useful next comparison must hold the data, model capacity, optimization search, and training budget fixed rather than attributing every difference to complex arithmetic.
