How Models Learn

Attention sink

A token, often the first one, where heads put attention when nothing else is useful.

Many heads use it as a resting place when nothing else in the sentence is useful to them, which researchers sometimes call an attention sink.
From Attention, step “The whole pattern as a grid”

See it in action

Try attention sink hands-on in the lesson Attention, part of the AI fundamentals course.

Open the lesson