> For the complete documentation index, see [llms.txt](https://yinqi-h.gitbook.io/ml-models/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://yinqi-h.gitbook.io/ml-models/models/self-attention-model/re-attention.md).

# Re-attention

{% hint style="info" %}
To handle Attention Collapse or learning plateau: re-attention
{% endhint %}

![Difference](https://1490490238-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Ma_JYIRuxx8PCv6UaBM%2F-MbHQ6gxkxngYpSn-08C%2F-MbHS5dwLpHU11lpyYsL%2FScreen%20Shot%202021-06-03%20at%2011.18.32%20AM.png?alt=media\&token=e95bbbe6-27c3-441e-8b7a-49e229e1bece)

$$
Attention(Q, K, V) = softmax(\frac{{Q}\cdot{K}^{T}}{\sqrt{d}})\cdot{V}
$$

$$
Re-attention(Q, K, V) = Norm(\theta^{T}(softmax(\frac{{Q}\cdot{K}^{T}}{\sqrt{d}})))\cdot{V}
$$

[paper](https://arxiv.org/pdf/2103.11886.pdf)