The Turing test is where a machine can exhibit intelligence indistinguishable from a human. I have been building chatbots and I can vouch for the complexity but when it comes to understanding your company, your policies and your regulations , they have no clue.
The first chat bot built was called Eliza at MIT in the mid 1960s – it was a simple pattern matching tool, but some people believed they were interacting with a human. The systems developed back in these times were called ‘Chatterboxes’.
Since then many advancements have been made in natural language processing and the natural language understanding domain. Chatbots such as Siri, Alexa and Google assistant have brought AI to the main stream.
LLMs have no understanding of your company data, your enterprise data is complex, there is machine generated data like logs, traces, metrics, human generated data including structured data & unstructured data & semi structured data. You want your Large Language Models (LLM) to be able to answer questions with knowledge about your enterprise data but out of the box, they have no knowledge of any of that.
We can enhance Large Language Models with Retrieval Augmented Generation – RAG. This is a technique where the general capabilities of LLMs are augmented with retrieval – essentially search with a twist. RAG enables the system to respond with context of your data and provide citations.
RAG Overview
In this diagram, we can see that the data is added into the Index, so when a user sends a message query, the index, containing relevant grounded data, can be used in the response. Providing an enriched response that is not limited to web data.
Benefits of RAG
| Improved accuracy | Combines real-time retrieval with generation to produce more precise and contextually correct answers. |
| Reduced hallucinations | Anchors responses in factual content, minimizing fabricated or incorrect information. |
| Business context awareness | Can integrate with specific company or industry data, making the model highly relevant for specialized tasks. |
| Smaller model efficiency | Enables smaller, cheaper models to perform on par with larger ones by pulling in external knowledge. |
| Provides knowledge since Model Training cutoff | LLMs have training cutoff dates making their knowledge set in time |
| Transparency and traceability | Provides users with links or references to source documents, increasing credibility. |
Potential Downfalls with RAG
| Retrieval quality dependency | If the retrieval system pulls irrelevant or low-quality data, the generated answer will suffer. |
| Increased system complexity | Requires integrating search, data pipelines, and model orchestration, adding technical overhead. |
| Slower response time | Retrieval and document ranking add latency compared to direct model responses. |
| Storage and maintenance costs | Managing large knowledge bases or document stores can be resource-intensive. |
| Potential for outdated or biased sources | If the external data isn’t regularly updated or filtered, it may introduce errors or bias. |
Retrieval Augmented Generation with CopilotStudio
A straight forward way to accomplish Retrieval Augmented Generation (RAG) is using Microsoft’s CopilotStudio. In fact, there is a Lite version available through licensed Microsoft 365 Copilot – via the Agent feature. Agents in Microsoft 365 Copilot can be shared to others who also have the extra Copilot 365 license. But if you are looking for a quick win with your own company Chatbot, then think about using CopilotStudio. You can also use this to create interactive chatbots and deliver them to your employees in your intranet or via a web page hosted internally.
Watch my live demo using Copilot Studio
Get in touch if you would like to know more about empowering your employees and improving customer service.