In the early days of commercial LLMs, capabilities were defined by models' inherent architecture and the way they made use of their underlying training data. When you asked a question to a chatbot, it would famously tell you that it only had knowledge going up to 2021. Additionally, chatbots could only process and remember so much information in a prompt. You had to be very specific, clear and consistent in your prompting if you wanted the same in your outputs. Even then, what LLMs could generate was limited by the information they had access to.

In 2023, a new architecture known as Retrieval-Augmented Generation (unflatteringly known as RAG) promised to change that. Retrieval Augmentation refers to the ability of generative AI to augment or enhance its outputs via the retrieval of outside information.

This enabled AI solution-builders to accomplish two primary things:

1. Expand context windows: This means how much information (via tokens) a model could process at once, and essentially enables chatbots to think more broadly and with more "memory".
2. Expand the depth of information: Importantly, AI tools could now pull in new information from outside of the training data to broaden the set of "knowledge" they could process in their responses.

In other words, it allows AI to exist in the context.

By last year, RAG had emerged as a go-to solution for enhancing the natural capabilities of LLMs by incorporating up-to-date, external data sources into their responses. This method addressed a fundamental limitation of traditional models: their reliance on static, outdated training data, which often lead to inaccurate or "hallucinated" responses. In finance and investment practices, where precision and currency of information are paramount, RAG promised to be the key to unlocking more reliable and contextually relevant insights for tasks ranging from due diligence to market analysis.

Since then, however, some data scientists and LLM developers applying generative AI to enterprise use cases such as investing have questioned whether basic RAG architecture is still the best way bring information into those workflows. To better investigate this question, let's start with some foundational knowledge.

Understanding the Basics

RAG works by dynamically fetching information relevant to a query from external databases or knowledge bases, allowing the LLM to generate responses based not only on its pre-trained data but also on the latest available information. This hybrid approach combines the depth of neural networks with the freshness of real-time data, significantly enhancing the model's utility and accuracy.

RAG is not a new AI technology per se, but a way of structuring data and intake pipelines to make existing LLMs more useful. We have shared the basic structure in this newsletter before, but here it is again for reference:

1. Prepare Data: Data from source documents is initially processed for privacy (such as PII handling) and segmented into appropriate lengths to suit the embedding model and the downstream language model application.

2. Index Relevant Data: Document "embeddings" are created and used to populate a Vector Search index, organizing data for effective retrieval in the Vectorstore.

3. Retrieve Relevant Data: A query prompts the program to retrieve text relevant to a user's query from the indexed data, influencing the prompt fed to the language model.

4. Feed into LLM Applications: The final step integrates prompt augmentation and querying into an endpoint, exposed as a REST API for use in applications like Q&A chatbots.

Advantages of RAG:

• Up-to-Date Information: By integrating the latest data, RAG ensures that the outputs are current and reflect the most recent developments, crucial for fast-paced environments like finance.
• Reduced Hallucinations: The accuracy of data retrieval reduces the likelihood of generating incorrect or misleading information as you as the user can more narrowly define its dataset and outputs beyond simple prompt engineering.
• Domain-Specific Responses: RAG enables LLMs to provide answers that are tailored to specific organizational needs, enhancing their applicability in specialized fields such as financial regulations or market trends.

Where It Falls Short:

There aren't many downsides to trying this approach, especially when compared to traditional chatbots, but there are limitations to be aware of.

• Dependence on External Data Quality: RAG architectures are only as good as the data you put into it. Not only does the source data matter, but how well encoded your qualitative data is in your vector database
• "Brittleness": Some developers speak of brittleness, or the idea that the wrong tuning of a RAG powered system can make LLM responses too rigid, or not behave in the way they intended. For instance, if the retrieval component fails to fetch relevant or high-quality information, the entire response generation process can be compromised, leading to erroneous or misleading outputs.
• Complex Implementation: While theoretically appealing, the practical implementation of RAG systems can add time and resource costs to the development process. The integration of multiple data sources, each with unique formats and standards, requires sophisticated preprocessing and data management strategies.

Nonetheless, it is still a good middle ground for many firms shopping for solutions, or looking to build their own, in a way that makes data plug-and-play.

Applications in the Finance Space

In finance, the precision of information can dictate the success of investments and strategies. Many of our readers have told us that data is their primary "moat" or a key driver of investment strategy. RAG leverages this resource to enhance decision-making across several functions:

• Due Diligence: By accessing the most current data from a data room, RAG can provide analysts with comprehensive insights into a company's performance, risk factors, and market conditions.
• Market Analysis: RAG can fetch and integrate the latest market data, news articles, and financial reports, providing a holistic view of market dynamics and enabling more informed predictions and strategies.
• Risks & Red Flags: Financial regulations are continually evolving, and staying compliant requires access to the latest legal texts and regulatory updates. Traditional LLMs, trained on static datasets, often lag behind these changes, posing compliance risks. RAG addresses this by pulling the most recent regulatory information and integrating it into the AI's responses. For example, when analyzing potential compliance issues in cross-border transactions, RAG-enabled systems can fetch the latest international trade regulations to ensure that financial advice is both accurate and up-to-date.

Automating Due Diligence

Due diligence in finance is a complex and data-intensive process, traditionally requiring countless hours of manual review. RAG can automate aspects of this process by retrieving and synthesizing information from various data sources, such as corporate filings, news reports, and legal databases. This not only speeds up the process but also enhances the depth of due diligence, leading to better investment decisions and risk mitigation. For instance, when evaluating a potential acquisition, a RAG-enhanced system could instantly pull up the target company's financial history, litigation records, and market performance, presenting a comprehensive analysis that would typically take days to compile.

Knowledge Management and Operational Efficiency

Financial institutions manage a vast amount of information that needs to be stored, retrieved, and utilized efficiently. RAG can revolutionize knowledge management within these institutions by enabling quick access to required information, reducing time spent by employees searching through documents. For example, a query about past merger strategies can be answered promptly by retrieving and synthesizing information from various internal documents, allowing teams to focus on strategy and execution rather than information retrieval.

Where Next? Building on RAG's Weaknesses

Going from Context-Based Generation to Contextual Reasoning

A primary challenge with RAG is that while it builds outputs off of additional contextual inputs, it isn't actually building new knowledge or intelligence capabilities with that data. For example if I ask the AI, “Who else has invested in early stage consumer companies” it will search its RAG database for "consumer"-related entries, but it won't necessarily lead us to the answer of who the investors are, even if that information is buried somewhere in the database.

One solution experts have proposed to layer in are Knowledge Graphs. These create links between entities to enhance RAG's functionality. The "consumer" tag surely has "invested", "funded", "shareholder", and other connections to different values. After letting the LLM search the vector database, you also push it to search the knowledge graph. Once it connects the associations the "consumer" entity has, it has generally proven to decipher that "consumer" investors are likely those with the most database connections to it.

Going from Searching to Actioning

Another focus area of AI researchers and builders is Agents, especially those that can elevate the field from generating simple text outputs to actually executing on multi-hop tasks. In financial applications, multi-hop agents can be particularly valuable. They can analyze sequences of events and their interdependencies, such as how market shifts influence investment risks or the impact of regulatory changes on asset values. More and more, this will be done with minimal human intervention, other than where it is most valuable.

Testing RAG In Your Company

As innovative as RAG is, there's a growing sentiment in the industry that it might already be reaching an innovation plateau. The initial excitement over its capabilities has given way to a more measured assessment of its long-term potential and limitations. This sentiment is echoed in our discussions with financial technologists who are already looking beyond RAG for the next breakthrough that could offer a more holistic and less error-prone solution.

Still, in combination with powerful LLMs and other architectural techniques, RAG can be an important component of a solution, and a great starting point to understanding whether context can assist in up-leveling your basic chatbot tools into actionable platforms. At Keye, we are researching and actively building ways to combine the many benefits of the various technologies discussed here, including the context of RAG, the security and privacy of native solutions and growing its usefulness with custom knowledge graphs.

If this sounds interesting, please reach out to us at founders@keye.co. We always welcome our readers' questions, input or requests.

‍