Byte-pair encoding
A way to build a tokeniser’s vocabulary: start from single bytes and repeatedly merge the pairs that appear most often in the training text.
Its tokeniser has a vocabulary of 50,257 tokens, learned from its training text by a method called byte-pair encoding: start from single bytes and keep merging the pairs that turn up most often.
See it in action
Try byte-pair encoding hands-on in the lesson Predicting the next word, part of the AI fundamentals course.