An overview of the basics and development approaches of AI

This article provides insight into the technology behind the current leading AI language models. We will focus in particular on the Transformer architecture, which enables machines to understand language accurately and generate text. These models can be customized to specific requirements through adjustments and training—for example, as an assistant in BCS. In a follow-up article, we will show how we developed the AI assistant and other applications and how they enrich our software.

In the first step, the user input is “tokenized.” This involves breaking down the input text into smaller units, known as tokens. A token can be a word, part of a word, or even a single character, depending on the tokenization method selected.

The attention mechanism is essentially based on vector and matrix calculations. After the text has been divided into tokens, each of these tokens is translated into a vector (“vector embeddings”). This gives the model a numerical representation of the text, which serves as the basis for processing and calculating the relationships.

Which vector a token represents is determined by analyzing large amounts of text. The vector of a word is determined by analyzing the neighboring words. For example, one can imagine that synonyms such as “cat” and “house cat” result in very similar vectors, as they occur largely in the same word environments. “House cat” will occur more frequently in domestic contexts, so the vector will be somewhat more “domestic.” The vectors used by GPT 4 have 512 entries (“dimensions”).

Since the Transformer architecture processes the entire user input in one step, no information about the order of the tokens is available per se. To enable the model to understand which tokens are related to which, positional information is added to each token. This positional encoding ensures that the model can understand not only the meaning of individual tokens, but also their order and relationships to each other. This is important because the meaning of a sentence depends heavily on word order.

The model's output is ultimately generated by repeatedly selecting the most probable tokens until the end-of-sequence token (<EOS>) is reached. This iterative process results in a text sequence that reads as if it were written by a human being.

By applying sampling, the output is no longer deterministic.

AI alignment deals with the question of how AI can be aligned with desired behavior. Risks such as bias or misinformation should be minimized. Language models such as current AI systems are trained with enormous amounts of text, including data from the internet. However, this training data often contains distortions and incorrect information. Language models are able to abstract from the examples available, recognize patterns, and apply them to new situations. At the same time, the biases or false assumptions contained in the training data flow unnoticed into the results.

Two examples illustrate this problem:

1. The issue gained prominence when cases such as Amazon's automatic application evaluation system became known. The system was trained using past decisions – and Amazon had mainly hired men. Amazon's AI then systematically disadvantaged women, even though the application texts did not contain any explicit information about gender. It was enough for the AI to be able to infer gender indirectly from hobbies, sports, or the type of social engagement. Alignment was not successful in this case, and use of the system was largely discontinued.
2. In contrast, Google developers were a little too successful in their alignment efforts to give higher weighting to members of minorities considered disadvantaged. Google AI-generated images of black popes are well known, and some early US presidents were also depicted with dark skin.

Pre-trained language models such as GPT4, Gemma, and Llama come with language comprehension and general training knowledge. In order to use these models for your own application, they must be able to handle company-specific specialist knowledge. The following diagram shows possible ways to customize them. Completely new training from scratch is usually ruled out due to the costs involved. Prompt engineering, fine-tuning, and RAG are practical options for medium-sized companies.

Our attempts at fine-tuning were less successful. Fine-tuning is performed using small, specific data sets from a specific knowledge domain. The aim is to convert a general-purpose model into a special model. The model is “re-trained,” i.e., the internal weights in the neural network are changed. This is the same procedure as in the initial training. A new model is created.

In our experiments to date, fine-tuning has impaired the general capabilities of the models without building sufficient specific capabilities. The models understood that they had learned something. However, they provided useless, generic answers, mainly based on their general prior knowledge from training. General control questions were answered less well after fine-tuning.

We see the cumbersome adaptation to new data sets as a significant disadvantage of fine-tuning: for a help assistant, the model would have to be retrained with each new version. An agile workflow, as established in Projektron's documentation, with one help release per week, is therefore practically impossible to implement. For a RAG-based application, on the other hand, this is not a problem. We have therefore not invested any further effort in fine-tuning for the time being.

Other sources for methods and techniques include: https://www.promptingguide.ai or platform.openai.com/docs/guides/prompt-engineering

Based on our experience with prompts, it works well to have an AI generate the first draft of the prompt: briefly describe what you want the prompt to do as input, and then let a tool such as ChatGPT create the prompt.

It is important to think as precisely, clearly, and concretely as possible about what result you expect. If something else comes out, on closer inspection the AI's response often turns out to be a good and direct implementation of the prompt – it's just that the prompt did not correctly reflect our requirements. You get what you specify, not what you want. That's why testing and gradual improvement are important: prompting is an iterative process.

Prompts in English are better understood. I already mentioned the advantage of the English language in tokenization. “Don't” instructions in the system prompt are less well understood than positive formulations of what is desired.

The documents are split in the first step. There are various methods for doing this, e.g., splits with a fixed length, separation at certain formatting characters such as double line breaks, or according to semantics.

The text splitter recommended in the literature for general text is “recursive splitting by character.” The splitter is parameterized by a list of characters, with the default setting being [“\n\n”, “\n”, “ ‘, ’”]. The splitter divides the text at the separators in the list, i.e., first at double line breaks. If this results in splits that are too long, the splitter divides at single line breaks. This process continues until the pieces are small enough. The aim is to keep paragraphs, then sentences, then words together, as these are generally the semantically related pieces of text.

In the third step, the vector database (e.g., FAISS) takes the vectors and uses them to build a vector index. The vector database specializes in finding a specified number of nearest neighbor vectors for a query vector from a very large number of vectors. This completes the first phase.

The result

The FAQ query provides good and useful answers, even if the questions differ from the wording of the sources, concern only one aspect of the answer, or even several retrieval texts.

The answer fits the question exactly; you don't have to extract the information from the entire text.

The order of the instructions is better than in the original, and the information is presented in a more structured way.

The texts are also better formulated.

GPT4 was used as the language model in this experiment.

Another advantage is that the output can also be in foreign languages without the need for translation. The database can be easilyexpanded with new knowledge. 

Our positive experiences with Retrieval-Augmented Generation (RAG) have convinced us to use this method in our first main application, AI software help. Once the technology is working successfully in one application, it can easily be transferred to other areas such as contract negotiations. The important thing here is to create the right data set and develop a suitable system prompt.

Further details on the development of the software help and the underlying AI frameworks will follow in our next blog article. In addition, we will take a closer look at RAG technology in an upcoming article, as it plays a central role in our applications.