Softmax

Turns a list of scores into positive weights that add up to 1; a score one point higher than another gets about 2.7 times its weight, so large gaps in score become very large gaps in weight.

To turn them into a budget that adds up to 100%, the model uses softmax: it makes every score positive (a token scoring one point higher than another ends up about 2.7 times as likely), then divides by the total.
From Attention, step “From scores to shares”

See it in action

Try softmax hands-on in the lesson Attention, part of the AI fundamentals course.

Open the lesson