All the Large Language Model (LLM) publishers and suppliers are focusing on the advent of artificial intelligence (AI) Agents and Agentic Ai. These terms are confusing. All the more so as the players do not yet on how to develop and deploy them.
This is much less True for retrieval augmented generation (rag) Architectures where, Since 2023, There has been widespread consensus in the it industry,
Augmented generation through retrieval enables the results of a generative ai model to be anchored in bus. While it does not prevent hallucinations, the method aims to obtain relevant answers, based on a company's internal data or on information from a verified KNOWLEDGE BASE.
It could be summed up as the interaction of generative ai and an enterprise search engine.
What is Rag Architecture?
Broadly speaking, the process of A rag system is simple to undersrstandIt starts with the user sent a prompt – a question or request. This Natural Language Prompt and the Associated Query Are Compared With the Content of the Knowledge Base. The results closest to the request are ranked in order of relevance, and the whole process is then sent to an llm to produce the response sent back to the user.
The companies that have tried to deploy rag have learned the specifics of These components are associateed with the steps required to transform the data, from ingging it into an into a source system to generating a response using an llm.
Data Preparation, A Necessity even with Rag
The first step is to gather the documents you want to search. While it is tempting to engage all the documents available, this is the wrong strategy. Especially as you have to decide whether to update the system in batch or continuously.
“Failures come from the quality of the input. Director of the AI for Business Practice at Sopra Steria Next. “This notion of refinement is often forgotten, even thought it was well undersrstood in the context of machine learning.
An llm is not de facto a data preparation tool. It is advisable to remove duplicates and intermediate versions of documents and to apply strategies for selecting up-to-date items. This pre-selection avoids overloading the system with potentially useless information and avoids performance problems.
Once the documents have been selected, the raw data – html pages, pdf documents, images, doc files, etc.s to be converted into a usable format, such associated metadata, exsp as in a json file, for example. This metadata can not only document the structure of the data, but also its authors, origin, date of creation, and so on. This formatted data is then transformed into tokens and vectors.
Publishers Quickly Realized that with Large Volumes of Documents and Long Texts, it was infective to vectorise the whole document.
Chunking and its strategies
Hence the importance of implementing a “chunking” strategy. This involves breaking down a document into short extracts. A Crucial Step, According to Mistral Ai, which says, “It makes it easy and retrieve the most relevant information during the search process”.
There are two considerations here – the size of the fragments and the way in which they are obtained.
The size of a chunk is often expressed as a number of characters or tokens. A Larger Number of Chunks Improves The Accuracy of the Results, but the Multiplication of Vectors Increases The Amount of Resources and Time Required to Process them.
There are several ways of dividing a text into chunks.
- The first is to SLICE according to fragments of fixed size – Characters, words or tokens. “This method is simple, which makes it a popular choice for the initial phase of data processing where you need to browse the data quickly,” Says ziliz, a vector data editor.
- A Second Approach Consists of a Semantic Breakdown – that is, based on a “natural” breakdown: by sentence, by Section – Defined by an HTML Header For Example – SUBEGECT Or PARAGRAND. Although more complex to implement, this method is more precise. It often depends on a recursive approach, Since it involves using logical separators, such as a space, comma, full stop, heading, and so on.
- The third approach is a combination of the previous two. Hybrid Chunking Combines an Initial Fixed Breakdown with a Semantic Method when a very priority response is required.
In addition to these techniques, it is possible to chain the fragments toge
“Overlap ensures that there is always some margin between segments, which increase the chances of capturing important information even if it is split according to the initial chunking strategy,” Documentation from LLM Platform Coere. “The disadvantage of this method is that it generates redundancy.
The most popular solution seems to be to keep fixed fragments of 100 to 200 words with an overlap of 20% to 25% of the content between chunks.
This splitting is often by using python libraries, such as spacy or ntlk, or with the “text splitters” tools in the language in the language.
The right approach generally depends on the precision required by users. For example, a semantic breakdown seems more approves when the aim is to find specific information, such as the article of a legal text.
The size of the chunks must match the capacities of the embedding model. This is preachisely why chunking is Necessary in the first place. This “Allows you to stay below the input token limit of the embedding model”, explains microsoft in its documentation. “For example, the maximum length of input text for the azure opinai text-mbedding-da-002 model is 8,191 tokens. Given that one token corresponds on average to Around Four! Openai models, this maximum limit is equipped to Around 6,000 words “.
Vectorisation and embedding models
An Embedding Model is responsible for converting chunks or documents into vectors. These vectors are stored in a database.
Here Again, there are several types of embedding model, mainly dense and sparse models. Dense models generally produce vectors of fixed size, expressed in x number of dimensions. The latter generate vectors with size depends on the length of the input text. A third approach combines the two to Vectorise Short Extracts or Comments (Splade, Colbert, IBM Sparse-Ambedding-30M).
The choice of the number of dimensions will determine the accuracy and speed of the results. A vector with many dimensions captures more context and nuance, but may require more resources to create and retrieve. A vector with fewer dimensions will be less rich, but faster to search.
The choice of embedding model also depends on the database in which the vehicles will be stored, the large language language model with it will be associated and the task to be performed. Benchmarks Such as the MTEB ranking are invaluable. It is sometimes possible to use an embedding model
Note that it is sometimes used useful to fin-tune the embeddings model when it does not contain sufficient knowledge of the language related to a specific domain, for example, constitution Engineering.
The Vector Database and its retriever algorithm
Vector databases do more than simply store vectors – they generalize a semantic search algorithm based on the nearest -nearest -newbour technique to index and retrieve information. Question. Most Publishers have implemented the hierchical navigable small worlds (hNSW) algorithm. Microsoft is also influential with diskann, an open source algorithm designed to obtain an ideal performance-cost ratio with large large Volumes of vectors, at the extra of across of across of across of access. Google has chown to develop a proprietary model, scann, also designed for large volumes of data. The search process involves traversing the dimensions of the sector graph in search of the nearest approximate neight neight neighbor, and is based on a cosine or euclidean distance calculation.
The cosine distance is more effective at identifying semantic simlarity, while the euclidean method is simpler, but less demand in terms of computing Resources.
Since most databases are based on an approximate search for nearest neighbors, the system will return several sectors potentially corresponding to the answer. It is possible to limit the number of results (Top-K Cutoff). This is even Necessary, Since we want the user's Query and the information used to create the answer to fit with within the llm context window. However, if the database contains a large number of vectors, precision may suffer or the result we are looking for may be beyond the limit imposed.
Hybrid search and reranking
Combining a tradational search model such as bm25 with an hnsw-type retriever can be useful for obtaining a good cost-peerformance ratio, but it will also get also a reStricted number of Results. All the more so as not all vector databases support the combination of hnsw models with bm25 (also know as hybrid search).
A reranking model can help to find more content deemed useful for the response. This involves increasing the limit of results retried by the “retriever” model. Then, as it name sugges, the reranker reorders the chunks according to their relevance to the question. Examples of Rerankers Include Cohere Rerank, BGE, Janus Ai and Elastic Rerank. On the other hand, such a system can increase the latency of the results returned to the user. It may also be Necessary to re-train this model if the vocabulary used in the document base is specific. However, some consider it useful – relevance scores are useful data for supervising the performance of a rag system.
Reranker or not, it is negassary to send the respons to the llms. Here Again, Not all llms are created equal – The size of their context window, their response speed and their ability to respond factories (even without having access to documents) are all criteria that need to be evaluated. In this respect, Google Deepmind, Openai, Mistral Ai, Meta and Anthropic Have Trained their llms to support this use case.
Assessing and observing
In addition to the reranker, an llm can be used as a judge to evaluate the results and identify potential problems with the llm that is supposed to generate the response. Some Apis relaces to block harmful content or requests for access to confidential documents for certain users. Opinion-Gathering Frameworks can also be used to refine the rag architecture. In this case, users are invited to rate the results in order to identify the positive and negative points of the rag system. Finally, observability of Each of the Building Blocks is Necessary to Avoid Problems of Cost, Security and Performance.