Automate the Analyst: Part 2 - All about context

This is Part 2 of the Automate the Analyst series—a set of thought experiments designed to help you identify the investments necessary for thriving in an AI-enabled world.

In Part 1, we emphasized the importance of a robust data foundation—systems and processes that prevent the "Garbage In, Garbage Out" trap, which is all too common in data platforms. The questions we explored were the kind an analyst might first ask when preparing for the insights that drive future action. Now that we have ensured decent data quality, are capturing the most critical data points, and can confidently trace back to the source, we are ready to dive into retrieving meaningful information.

The Business of Insights: Understanding Context

Data platforms and analysts are in the business of inspiring organizations with insights derived from the information they gather. While machines excel at retrieving raw data points, transforming that data into valuable information is an entirely different challenge. This distinction is at the heart of why data roles exist—it's all about the business context.

Consider this example: Revenue figures of $1 million in Year 1, $1.5 million in Year 2, and $2 million in Year 3. This trajectory might seem impressive if you're in a declining industry, but less so if you were riding the wave of the 2021 crypto boom. Furthermore, understanding the drivers of this growth is key—growth is positive if costs remain stable, but if costs are rising at the same pace, profit margins aren’t improving. Context is everything.

Breaking down how an analyst overlays context provides insight into where AI excels, where it falls short, and where improvements are needed.

Input Style and Formats: The Multimodal Challenge

As you begin your AI journey, you might start with users writing text that converts to SQL queries to retrieve data points and generate information. However, in real-world scenarios—especially when interacting with an analyst—you likely won’t rely on text alone. Communication might involve images, video conferencing, or even complex spreadsheets. This variety makes input "multimodal."

Today's most popular general-purpose Generative AI (GenAI) models are increasingly capable of handling diverse input formats and producing varied outputs. As you assess your AI requirements, it’s crucial to look for models that support multimodal input to allow users to interact in the ways that best suit their needs.

Unlike human analysts, who inherently understand context from prior interactions, GenAI models often require explicit instructions. These models need what’s called instruction tuning, which trains them to respond to specific prompts accurately. For example, asking a GenAI model to retrieve revenue numbers might involve a more complex prompt like: "You are a data analyst working with the following table schemas in a data lake. Create an SQL query that retrieves the revenue data for Years 1, 2, and 3, and provide only the data points."

The Importance of Context: Beyond Simple Prompts

Incorporating context into prompts, as shown above, is essential but often insufficient. Business knowledge involves deeper associations that are difficult to encapsulate in a single prompt. If a Large Language Models (LLM) is like a person with vast general knowledge, a Retrieval-Augmented Generation (RAG) model is like that same person put through a specialized data analyst training program alongside an onboarding into your company.

LLMs often struggle with domain-specific understanding and may not be up-to-date with the latest industry developments. Moreover, LLMs are prone to hallucinations—generating plausible-sounding but incorrect information—when they lack the necessary context. RAGs help mitigate this by augmenting LLMs with external knowledge, such as databases relevant to your company, and training them on business-specific information.

For example, a RAG might take a user query, search for relevant documents, and use embeddings (a method for representing knowledge) to chunk the information into digestible pieces for the LLM. This approach enhances the quality of the AI's response and can significantly improve accuracy in specialized tasks like data analysis. It also allows for more granular control over retrieved information, which is crucial in environments with strict governance protocols.

Improving Retrieval: From Brute Force to Smart Strategies

To improve an AI agent’s retrieval capabilities, we need to move beyond brute-force methods like training models solely on past SQL query logs. This traditional approach performs well on familiar data but struggles with new, unseen queries—a significant limitation in a dynamic business environment. Furthermore, using a corpus of SQL queries by AI will eventually lead to increasing number of SQL query logs becoming part of AI training. Mounting evidence suggests that this can lead to dire consequences.

The next frontier in AI involves models that mimic human thinking, using techniques like knowledge graphs. These graphs represent abstract concepts and their relationships within a business domain, offering a more sophisticated way to reason across entire datasets rather than isolated data points. Tools such as GraphRAG cover interesting approaches that illustrate deployment of knowledge graphs for better retrieval.

Modern BI tools' Semantic layers allow businesses to explicitly define relationships among database entities, which can then be used by RAGs to make retrieval more opinionated in approach. The result? Papers like this illustrate how the use of semantic layers show significant lift in accuracy.

Capturing Feedback from Analysts and End Users: Closing the Loop

Capturing dialogue and feedback from analysts and end-users is crucial for continuous improvement in AI-driven processes.

Analysts often have deep, tacit knowledge about the data they work with, including insights into data quality issues, anomalies, and the subtleties of how different data points relate to each other. From a process standpoint, analysts would often capture additional context by annotating the data, adding comments to explain why certain results are incorrect, or suggesting alternative data sources to consider as part of future development.

End-users, typically business leaders or operational staff, provide a different but equally critical perspective. They rely on, and provide feedback to, the utility and correctness of insights generated from output of an analyst. For instance, a sales manager might receive a report and find that the recommendations are too generic or not actionable enough for the specific needs of their team. They might suggest weighing certain factors more heavily, such as regional sales trends or customer behavior patterns that have a direct impact on sales strategies. This feedback is vital for ensuring that the analytical output is not just accurate but also directly applicable to the business’s strategic goals.

To systematically capture and integrate this feedback, organizations should consider implementing a robust feedback loop through data governance and observability tooling. This would typically be part of metadata platforms like Datahub or Open MetaData. These platforms can serve as a central repository for collecting and organizing feedback from both analysts and end-users. By tagging data with user comments, quality metrics, and access controls, organizations can provide richer context for making decisions. This platform should work in combination with internal messaging apps (like Slack / MS Teams) and centralize conversations about specific data assets.

Conclusion: Building the Future with AI

AI agents will undoubtedly continue to evolve, but their success hinges on foundational investments from you. Incorporating a BI Semantic layer and a metadata platform as context enablers for your AI agent will be essential. These investments, along with the foundational elements discussed in Part 1, will be key to unlocking future success with AI.

Image credits: Freepik