Self-Attention
attention is all you need
self-attention中b1-b4都看过了整个sequence，但可以并行计算(RNN不行）
Embedding to a
embedding从x_{i}得到a_{i}
q: query(to match others)
k: key(to be matched)
v: value(information to be extracted)
a_{n} to b_{n}
put every (query q) to do attention on every (key k)
get alpha
scaled Dot-Product Attention
add softmax layer(normalization) get alpha_head
get b1-b4 in the same way
How self-attention do parallel
In scaled Dot-Product Attention
it is the same
What does self-attention layer does
- calculate Q、K、V
- get Attention A
- use softmax get A_head
- get O
Multi-head self-attention
q, k, v will seperate into two q, k, v
multi-head can do what you like in different head
Positional Encoding
- No position information in self-attention
- sequence don’t have difference
- Original paper: each position has a unique positional vector e_{i}(not learned from data)
- In other words: each x_{i} appends a one-hot vector p_{i}
Seq2Seq with Attention
core: replace all RNN with self-attention
Transformer
If you can use seq2seq, you can use Transformer
