An AI framework for BCS: Requirements and architecture

The following graphic shows the functional view of the framework. The first application entered is the software help. Interfaces exist to data sources, language models, result display in BCS, and via a user interface (UI) to the framework administrator. The administrator defines the applications via the UI. At the core is the system prompt, which determines what the application should do. In addition, the administrator defines the language model (internal/external) and can enter parameters. If, as in the case of the help function, it is a RAG application, it is possible to control how the vector index is generated from the data set, for example, the type and size of the text splits, or how many hits are returned and what their minimum rating must be.

The framework is accessed via an interface from BCS. When a user enters a question in the AI help window, a request is sent to the framework containing the relevant application as a parameter. Depending on the task to be performed (help, summarize ticket), various applications can be requested via the interface.

The framework can contain a large number of applications, as shown in the following graphic. Applications can be chained together. For example, a local model can be used to generate a data set consisting of anonymized summaries from closed tickets. This data set is then available to answer new incoming tickets.

In a later version, customers will be able to define their own individual applications that do not necessarily have anything to do with BCS. This is illustrated in the image on the far right, “Contracts.” This provides support for employees who frequently have to negotiate contracts or explain them, as is the case in a software company where a license agreement is concluded with each new customer. Most of these questions have already been answered in the past. An AI application can draw on this experience to speed up the processing of new cases. The data set consists of documents, each containing a contract clause, the customer question previously asked about it, and the decision made. This can be done simply via a collection of .txt files. These are processed like the help documents, and a vector index is generated that can then be queried.

In the same way, a customer can make company-specific process instructions, security guidelines, and similar data collections queryable by AI.

The technical components are shown in the following graphic. Basically, these are all interchangeable in order to be able to react to the rapidly changing developments in the field, which are difficult to predict. Server-like components such as Ollama and LangServe are more deeply rooted, have a near-standard status, and are also widely used in the industry (as of December 2024). We selected components such as text embedding through testing; more on that later. The language models can be replaced with a simple configuration entry.

To select a local embedding model, we conducted a comparison test with various products. We used OpenAI's text-embedding-ada as a benchmark, but it cannot be operated locally. The test was based on a dataset of approximately 100 texts and 10 questions, each of which was specific to a text, so that it was clear which text should be the top hit for the question. The texts were imported into OpenAI's RAG solution, the questions were asked, and the 4 hits (i.e., TopK=4) were recorded. OpenAI was able to answer the questions well, with the expected documents coming in first place. We compared 6 local embedding models as well as text-embedding-ada-002, which is also available individually but only online. The 10 questions were asked and the 4 hits were compared with the benchmark. A deviation at position 1 was awarded 7 points, followed by 4, 2, and 1 point. We selected the local model with the lowest score. This was BAAI/bge-m3 from the Beijing Academy of Artificial Intelligence.

The results are shown in detail in the following table: the upper part shows the points for each question, and the lower part shows which texts were selected as deviating. We plan to repeat the test at intervals; if there are significant improvements, we could change the embedding model. However, this would require re-indexing the entire document collection, which would involve a certain amount of effort.