AI assistance in BCS: Step by step to precise answer quality

The left side of the images shows the indexing phase, the path from the documents to the vector database. On the right is the inference phase, the path from the user's question to the answer. As a reminder, the question is assigned a semantic vector through text embedding, and then the database is searched for similar vectors—i.e., texts that deal with the same topic as the question. The hits are passed to the language model as context, together with the question and the instruction: “Answer the question based on the texts found.”

Our first attempt at BCS help corresponded to the normal RAG (retrieval-augmented generation) scheme. The following image shows the process flow:

We then implemented a function that, once activated, replaces each split with the entire document, which is then passed on to the language model (“parent document retrieval”). This function also correctly answers the context-free question about BT-115.

There is a conflict of objectives when determining the optimal size of text splits. On the one hand, the splits must be large enough to provide sufficient context to answer the question, but on the other hand, they must be small enough to contain only one topic so that the vector can accurately reflect the meaning. Parent document retrieval can partially resolve this conflict of objectives.

The second version of the process flow, with parent document retrieval, looked like this:

We then proceeded to a new round of testing. The following screenshots show an excerpt from the test documentation with new observations.

When searching a general text corpus, it is a good strategy to give high weighting to very specific terms such as “BCS” or “Projektron” in order to find the presumably few documents that deal with these topics. In our specific application, however, every text in the corpus deals with BCS, so paying too much attention to these terms is misleading. We therefore rewrite the question if it contains certain terms (here is the filter: “BCS” or “Projektron”). If the terms are not included, the question is used unchanged; otherwise, the terms are removed by an AI application. The question should be changed as little as possible. This is a first example of the chaining of AI applications. The process with query rewriting (QRW) now looks like this:

This brings us to a new round of testing. The next finding again has to do with the retrieval stage. We tested whether shorter splits, because they can be semantically vectorized more precisely, deliver better results in combination with parent document retrieval. Surprisingly, this was not necessarily the case, as the example shows. With longer split lengths (250 or 1,000 characters), the test question in the example was answered correctly, but with a length of 100 characters, it was not. Apparently, there is no split that happens to contain all three meaningful terms in the question: “ticket,” “article,” “assign.” The hits mostly revolve around assigning emails to tickets. The question cannot be answered with this contextual material.

As a solution, we have implemented a feature that allows AI-generated additional documents to be added to the data set. In this case, these are short summaries that contain all the keywords from the relevant help page. The summaries are not split further. When the indexer finds one of these summaries, it is replaced by the entire help page, as with text splits, which is then passed on as context. The process looks like this:

The last improvement we would like to introduce also has to do with the splitting and indexing process. We have observed that the splits found by the indexer are often very short. In the default setting of the “Recursive Text Splitter,” splits are first made at double line breaks, then at single line breaks, then at spaces, and finally within words. Since the help documents are highly structured, with many double line breaks, many splits are shorter than the “chunk size.”

The default setting works well for standard texts, but in our special case it delivers less useful results. This is because the semantic search strongly favors these short splits. This often results in a search result as shown in the following figure. Although the split length is set to 1,000, the splits found are only between 16 and 26 characters long.

We suspect that (similar to the terms BCS and Projektron) this often results in splits being identified that contain a highly rated term. These are not necessarily the splits that lead to the optimal original document after parent document replacement. We have therefore made the split parameters configurable via the framework so that you can easily set the fixed length without having to consider structures due to line breaks or semantics. A large overlap ensures that contextual relationships are preserved as much as possible. After testing, it appears that this method delivers slightly better results than standard splitting in our specific case. The overall process now looks like this:

This is the status with which we are delivering the first productive version of the help system. Now we are waiting for the “reality check”: experiences with the first real customers.

Our experience report clearly shows that most of the optimization work focused on retrieval and, in particular, the text splitter. Once the correct context has been determined, the language model also generates a suitable response.