Question
Can descriptions of words provide a useful training signal for their embeddings?
Approach
WordScrunch extracts BERT attention scores from word descriptions. It starts with spread-out embeddings and applies contrastive updates, including negative samples.
The work includes description processing, attention-score extraction, and embedding updates. Description generation remains part of the unfinished input pipeline.
Open question
Attention scores provide weights for the update procedure, but do not by themselves establish semantic similarity. The remaining question is whether the resulting embeddings help a downstream task under a controlled comparison.