Automate the Analyst: Part 1 - Building a Robust Data Foundation

In quest to being more data-driven, Generative AI (GenAI) stands out as a powerful tool for intelligent information retrieval. Yet despite the machine's superior ability to process numbers, organizations continue to invest heavily in data talent. This paradox highlights the importance of optimizing our information management systems to truly harness AI's potential and maximize the return on investment (ROI).

This series of articles poses the following question: What would it take to automate the analyst?

The thought experiment allows us to not only identify the value generating activities performed by a data team, but also isolate where our information systems fall short. Those gaps then give us a clear roadmap of where investments need to be made.

Part 1

Establishing Strong Foundations: Data Capture, Quality, and Lineage

Data Capture

A prevalent issue within organizations is the unavailability of information. While finding relevant information can be inefficient, the real challenge is when data doesn't exist. Consider the common scenario where an analyst is asked, "Can we know [x] because we want to do [y]?" often responding with, "We don't capture [x], does [z] work?"

Starting with the basics—data capture—is crucial. Recent advancements have revolutionized this area, with tools like Snowplow enabling precise data capture. Effective data capture involves deliberate instrumentation of event origins, robust change management, and an active focus on critical behavioral components of systems and users.

Moreover, adopting an event-based architecture, where each change registers an event for other systems to utilize, is beneficial. This approach is vital for building services and applications that communicate reliably and predictably. Platforms like Apache Kafka can facilitate this process at scale.

Data Quality

The adage "Garbage In, Garbage Out" remains relevant. Analysts frequently express doubt in data quality, saying, "We have that information, but I don't know how much I trust it." This skepticism often stems from issues not only in data capture but also in the subsequent manipulation of data.

Ensuring high data quality begins with accurate capture and a culture that prioritizes clear definitions of data points. Employing data profiling, validation rules, and alerting mechanisms can significantly improve data quality as the organization evolves.

Data Lineage

A common scenario is questioning discrepancies in data, such as, "I thought revenue was [x], why does the dashboard show [y]?" The typical response is, "I don't know, let me check where that data comes from." This highlights the critical importance of data lineage.

Humans tend to doubt information that seems inconsistent. Even with robust data capture and quality checks, new data often needs to fit existing narratives. When it doesn't, the instinct is to question the data.

Tracing the origin of information is fundamental to establishing trust. Understanding the transformations that produce a metric like "revenue" enhances comprehension and trust in that data. Tools that provide data observability and metadata tracking are essential. They help trace how structured data interrelates, surface issues with capture and quality, and ensure data freshness. Additionally, they support transparency, auditability, and regulatory compliance, crucial for business operations.

Conclusion

Building a robust data foundation is essential for maximizing the return on AI investments. By focusing on comprehensive data capture, ensuring data quality, and maintaining clear data lineage, organizations can trust their data and make informed decisions. These elements not only enhance the efficacy of AI tools but also establish a culture of data-driven decision-making, paving the way for sustained success and innovation in an increasingly data-centric world.

Want to chat about where gaps in your organization may be? Let's talk.

P.S. Stay tuned for Part 2, where we'll talk about the synthesizing of business context, and the need to centralize all mission-critical information to keep the organization aligned.

Image Credits: Freepik