Is my data stack appropriate?

Introduction

We previously introduced a range of different stages of the data maturity journey for companies. In this post, we'll take a more technical focus and look at the tooling available in market.  In our experience, the vast majority (probably >80%) of data investments at organisations are done based on what early data-related employees used in their last firm.  This can result in technology choices that don't match the organisation on factors like:

• Centralisation vs decentralisation of data asset creation
• Degree of governance, security and control
• Balance of structured and unstructured data
• Quality and standardisation vs breadth
• Typically predefined analysis vs unstructured exploration
• Priortising human consumption vs feeding data science models
• Low latency or real-time data needs

Unpacking these early investments can be time consuming and involve increasingly larger financial investment as the organisation matures.  Having seen this occur at more oganisations than we can count, we are strong proponents of adding parentheses to the start up adage "Move fast (with purpose) and break things (thoughtfully)".

Early Stage

At this point, the organisation might have survived to-date by querying against production databases or using predefined reports in your ERP or other SaaS tools.  This paradigm is fine to start out with but is faced with challenges when you need to deliver on a specific ask/use case by your leaders in a reliable/stable way and make sure you can answer the inevitable 'what caused this?' question.  There may not be a Data team to speak of; just some inspired engineers/ analysts who double-hat on ad-hoc reporting needs.

Make a rudimentary Data Stack

1. Get data collection right:

   1. You may have already set up Google Analytics or some other free version of event tracking software which stores data elsewhere.
   2. You should customize event tracking to your use case, storing data in your systems as much as possible. Focus on quality, and establish tracking purposefully to enable metrics that would move the needle for you. We like to recommend Snowplow, but RudderStack and PostHog are other options to look into.

2. Set up your analytical warehouse:

   1. You can either go for lightweight OSS (e.g. DuckDB / ClickHouse) or enterprise-ready tools (e.g. Databricks, Bigquery) but be sure to store your data in S3/Google Storage.
   2. What analytical warehouse you choose depends on a variety of factors. Talk to us if you need help here.
   3. Setup replication and change data capture (CDC) into your analytical warehouse
      1. You can either use dlthub if you’re python friendly, or use fivetran / airbyte for low-code solutions

3. Manage data transformations: You also need a reliable way to manage and transform the information without needing to write complex code. SQL is usually best. Hence,

   1. Install DBT locally and connect to your analytical data warehouse
   2. Set up a Github repo that will house your DBT project and create your first few models. You likely have a few SQL files, and are migrating them to DBT.

4. Set up an orchestrator: You want to ensure that you’re running this automatically to keep you focused on “the business” at hand. Setting up a scheduler

   1. For speed: You can run DBT commands on a CRON job with an EC2 instance / Lambda function / Cloud run on the Cloud with daily logs populated for debugging.
   2. For a more scalable setup: Use Airflow, Prefect or Dagster  

5. Adding a BI layer: Finally, you want to allow users a quick ability to consume information from your newly minted Data Warehouse.

   1. For speed: You can setup Metabase, a simple and free Business Intelligence tool that you can self-host and connect to your analytical warehouse for querying.
      1. Note that this requires users to know and write SQL, which is an overhead you may not want to pass on.
   2. For scalability: A modern cost-effective Business Intelligence tool like Looker will make it easy to write models once, while allowing end users to drag and drop information to make actionable dashboards.

Establishing Culture

It’s possible that a lot of manual data entry (in spreadsheets) or data capture isn’t up to the mark in terms of quality. This is the point where enforcing quality of capture will reap dividends for years to come, and you should actively invest in initiatives that prioritize quality of capture.

What success looks like

Reliably fulfilling your primary data use case while not being blocked by your tech stack in exploring implementation of other use cases are your key outcomes in this phase.

Mid Stage

Your company may have established itself in an important market niche and now need to focus on driving fundamentals in line with business priorities.  From a usage standpoint, you probably have <5 power users who look at info daily and <100 users who look at info monthly. You may have 2-3 discrete basic use cases e.g. customer funnel, marketing attribution, revenue reporting, etc. You may or may not have a dedicated data team, but likely have a central data store which ingests information on a daily basis.  Questions start transitioning from 'do we have this data?' to 'why should I trust your data? Mine gives a different conclusion'

Scale your Data Stack

1. Fine-tune your Infrastructure

   1. You should also have a well established Change Data Capture mechanism that reliably mimics what is in your production systems to your Warehouse
   2. You may start looking into internal tooling to optimize data capture e.g. moving users to add and consume information from a CRM rather than from internal spreadsheets
   3. You should have a dedicated orchestrator that handles replication and transformation pipelines instead of EC2 / Cloud runs.

2. Establish Sound Data Ops

   1. Setup appropriate deployment environments; minimally - Development and Production
   2. Institute a PR review policy when deploying models, especially for important models that are already in use by a lot of stakeholders
   3. Start setting up SLAs on data availability, e.g. availability weekly on Monday by 9 am

3. Optimize event capture for Single Customer View

   1. With multiple touchpoints for your customers given multiple frontend properties like a website, app, etc. you should look into creating a unified view of the customer. This starts with establishing a central event tracking platform like Snowplow  

4. Level up your Data Assets and Data Models

   1. This may also be a good time to define hierarchies amongst your datasets: Some are more important than others and efforts to optimize those will get you a larger bang for buck.
   2. With loads of new datasets coming up every day, you may be tempted to model as many into disparate sources. Fight this urge. You should think about how to augment mission critical datasets instead e.g. adding lifetime value and cost of acquisition to your customer master dataset.
   3. Amongst your most critical datasets, it’s time to add data quality checks. Add some tests that must pass, alongside alerting mechanisms for owners in case of failure.
   4. You should have a Semantic model layer in your Stack, either in a downstream BI tool (like Looker) or in a standalone component that integrates with your warehouse (like Malloy) to help map the entities created in your Data Warehouse to final end-user queries

5. Setup fundamental governance structure: If privacy is an issue or if you have multiple legal entities operating under your org, you may need to start looking into differentiated management of your data by establishing account types with.

Establishing Culture

Over and above some rudimentary metrics on data platform usage, you should also roll out a Data Maturity Survey to assess the state of the organization: How much do people trust data? How many decisions are powered by it? And where are your biggest opportunities to invest in for the next 3-6 months?

What success looks like

Consistently hitting high NPS while ensuring data platform usage doesn’t dip is probably the most critical right now.

Late Stage

Your company likely has several million in recurring revenue, has made material data technology investments, and has a data team that is managing towards SLAs.  Automation, operational efficiency, forecasting and data science terms are being talked about with increasing frequency.  Often bloat has started to creep in and there are hundreds if not thousands of transformations, dashboards, reports, pipelines and other data assets running daily.  Noone really knows who should own what, what ownership really means and as a result what to do about spiralling data platform costs.  Process and people investments are required to create an effective, cross functional 'data community' in a matrixed organisation. 

Purposeful Data Investments

1. Metadata visibility and Governance best practices:

   1. PII tagging, ideally as close to the source as possible. You may need to look into isolated warehouse setup and data tenancy as required by your local regulations
   2. Row-level metadata about distinct records to debug and triage issues with source systems e.g. schema changes, erroneous joins, etc.
   3. Investing in a metadata platform (e.g. Datahub / OpenMetadata) that provides data observability (e.g. Elementary Data), lineage, ability to discover relevant datasets with meaningful tags, alerting mechanisms, and integrations to all your data ingress/egress points

2. Delivering against SLAs on Central Data Assets and mechanisms to achieve them

   1. Replication from 10+ types of data sources with dedicated tool for CDC. Ideally, replication pipelines should take a very short amount of time
   2. Running Slim CI at deployment so that you can merge without breaking production pipelines
   3. Differentiated Data Asset treatment through implementing a Medallion architecture, especially for Certified datasets with appropriate dbt tags
   4. Dashboards that monitor Data Freshness and Test Success Rate, alongside alerting mechanisms (on email/Slack etc.). This is especially critical for certified datasets
   5. Create a variety of refresh cadences (hourly, daily, weekly) to keep cost in check. Defer full-refresh until absolutely necessary

3. Enable Niche use cases for your business. E.g.

   1. If you are an online business with large customer base - you might look into A/B testing and experimentation at this stage
   2. If your organization needs just in time decision making, you may need to invest in real-time infrastructure

Conclusion

As you may have gathered, there are loads of really interesting ways to solve fundamental problems associated with building and maintaining the Data Platform.  If you’re starting out, or have been doing this with some major outstanding questions, know that you’re not alone.  You also don’t need to suffer alone, reach out to us here!  We believe that Data Platforms need to be a strategic advantage in your ability to operate, and we’d love to co-develop a solution that helps you succeed.