What is tokenization in AI? Explained simply

In the first step of text processing, the user input is tokenized. The AI breaks down the input text into smaller units called tokens.

A token can be:

a single character (e.g., “a,” “?”),

a word component (e.g., “pro-” or “-ject”),

or an entire word (e.g., “project management”).

The division chosen depends on the respective tokenization process and, in particular, on the text corpus used for tokenization.

Why is this necessary? Language models work sequentially, generating a response text by building a string of characters step by step. Due to the architecture of the models, a fixed character set is required because the neural networks on which the language models are based have an input and output layer, each containing a specific number of neurons. Each neuron in the output layer is assigned an element from the character set (a token).

The result of a “thinking step” by the neural network is a value distribution at the output layer. The character belonging to the neuron with the highest value is then appended to the character string. The process is repeated until the response text is complete.

Tokenization is more than just a technical preparatory process. It determines the framework conditions for working with language models.

The maximum text size that can be processed simultaneously is measured in tokens. The cost is calculated based on the number of tokens in the question and answer.

Maximum text length in tokens

The models work with a fixed character set. The text size that can be processed simultaneously is measured in tokens, not words or characters.

The so-called context window determines how many tokens a model can “see” at once. GPT-4o, for example, can handle up to 128,000 tokens – enough for entire books – and GPT-5 can handle up to 400,000 tokens. However, each additional token requires memory and computing time.

Costs based on tokens

The costs are also calculated based on the number of tokens processed. That is why it matters how efficiently a text is broken down. A text that requires only 20 tokens in English can quickly have one and a half times as many in German, making it more expensive.

Language differences

English requires the fewest tokens, the larger European languages slightly more, and individual rare languages more than ten times as many. This means that English is cheaper, faster, and more efficient – an important practical factor, as we will show later. Specifically, you can expect English texts to have about 10% more tokens than words, while German texts have about 20-30% more tokens than words. Multilingual tokenizers usually need more tokens than monolingual ones. (Source: https://jina.ai/de/news/a-deep-dive-into-tokenization/)

It would be obvious to use individual letters, numbers, and punctuation marks as tokens. Then the model would need far fewer tokens.

Example:
“Project management” → [“P”, “r”, “o”, “j”, “e”, “c”, “t”, “m”, ...]

This makes the vocabulary very small. However, a single word is broken down into many tokens. The model would therefore have to recognize meanings across very long sequences. This would be inefficient and prone to errors.

At the other end of the spectrum is breaking down words into whole words:

Example:
“Project management is complex.” → [“project", "management”, “is”, “complex”, “.”]

Using only whole words as tokens is not an option, as the vocabulary would become too large. Even for German, millions of words would have to be taken into account—including inflections, dialects, and neologisms.

The solution offered by modern models is a hybrid approach known as subword tokenization: first, all single-character tokens are taken, then character pairs are taken according to frequency, then groups of three and longer chains, until the maximum number of tokens possible for the model is reached. All words that do not correspond to a single token are composed of several tokens. Frequently occurring words or parts of words are given their own tokens, while rarer words are broken down into smaller tokens.

Example:
“Project management” → [“project,” “manage,” “ment”]

A widely used method for subword tokenization of large text corpora is Byte-Pair Encoding (BPE).

Procedure

1. We begin with all characters as individual tokens.
2. Next, we search for the most frequently occurring pairs of characters.
3. We merge these pairs into new tokens.
4. We repeat this process until we reach the desired vocabulary size.

Example (simplified):

Sentence: "With the project management software Projektron BCS, you have projects and business processes under control.”

1. Start: ["W", "i", "t", "h", " ", "t", "h", "e", " ", "p", "r", "o", "j", "e", "c", "t", ...]
2. First step: find the most common pair. “ro” appears most often (6×) → replace it with ↑
3. “With the P↑ject management software P↑jekt↑n BCS, you have P↑jects and business p↑cesses under cont↑l.”
4. Next, ↑j occurs most frequently → replace ↑j with ↔, and also replace ct with ↗
5. “With the P↔e↗ management software P↔e↗↑n BCS, you have P↔e↗e and business p↑cesses under cont↑l.”
6. Now we have three new tokens representing ro, ct, and roj.
7. “_p” (space + “p”) appears three times → replace it with ↘ “With the↘↔e↗ management software↘↔e↗↑n BCS, you have↘↔e↗e and business p↑cesses under cont↑l.” 
8. Then "↘↔”, d, which already corresponds to “_proj”, can be merged into ⇔: With the⇔e↗ management software⇔e↗↑n BCS, you have⇔e↗e and business p↑cesses under cont↑l.”

In a similar fashion, useful units emerge, such as:

• “manage” (from “management”)
• “software”
• “BCS” (as its own token because it appears repeatedly)
• “business processes,” split into “business” and “processes”

Result

Instead of storing each character individually, we now have subwords:

["With", "the", "project", "manage", "ment", "-", "software", "Projektron", "BCS", "you", "have", "projects", "and", "business", "processes", "under", "control"]

This produces tokens that are useful for many kinds of text. The method allows large amounts of text to be represented efficiently using a relatively small number of tokens.

To convert a text into tokens on a trial basis, there is a freely accessible page, the tokenizer from OpenAI. If you would like to view the complete token set from OpenAI's GPT-4, you can do so at https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a09193.

As an example, I had the introduction to this article tokenized once in German and once in English and displayed the result once as text and once as token IDs.

We can see that the English version not only has fewer letters, but is also broken down into 14 fewer tokens than the German version.

As sophisticated as tokenization is, it repeatedly encounters limitations in practice. Even if a particular model does not use an optimal tokenization strategy, it can learn to make the right decisions from imperfect inputs if the network size is sufficient, there is enough data, and training is adequate. As a result, less effort is put into improving tokenization than into other areas. (Source: https://jina.ai/de/news/a-deep-dive-into-tokenization/). The focus of development is currently elsewhere.

The main challenges are:

At first glance, this topic sounds very technical. However, in our experience, it has practical implications for everyday work. It is advantageous to write system prompts in English—they are easier to understand, take up less space in the context window, perform better, and are less expensive.

Tokenization is an inconspicuous but central component in the functioning of modern language models. It determines how efficiently and accurately texts are processed and thus influences the cost, speed, and comprehensibility of AI systems.

For companies that use AI in a targeted manner, it is helpful to understand the mechanisms behind it. At Projektron, a better understanding of tokenization enabled us to make our AI help in BCS more powerful, affordable, and user-friendly.

To learn how we developed the Projektron AI help and what role tokenization plays in the background, read the article “Projektron and AI: Experiences in developing and optimizing BCS help.”