Induction head
A kind of attention head that finds an earlier copy of the current token and looks at the token that followed it.
The previous-token and start-token patterns were scored on 8 short test sentences, and the two copying patterns on a run of random tokens repeated twice: - a previous-token head, which looks at the token just before; - a same-token head, which splits its attention between a token and an earlier copy of it; - an induction head, which finds an earlier copy and looks at the token that came after it.
See it in action
Try induction head hands-on in the lesson Attention, part of the AI fundamentals course.