How Models Learn

GPT-2

A language model released by OpenAI in 2019. The small version has 124 million parameters, 12 layers and 12 attention heads per layer, and was trained on WebText, about 40 GB of web pages.

Everything in this lesson comes from a real model, GPT-2 small, which OpenAI released in 2019.
From Attention, step “What does “it” mean?”

See it in action

Try gpt-2 hands-on in the lesson Attention, part of the AI fundamentals course.

Open the lesson