How to think about Data Platform Cost

Imagine this

You were part of the founding (or foundational) team of a startup that saw rapid growth through the pandemic. You’ve iterated to build a business that solves real pain points for your customers. You’ve made data-driven decisions with all the sources of digital (and non-digital) information you’ve accumulated to get signal from the noise. But as you grow, some of your initial data investments aren’t working as well for you.

With the slowing of business due to the economic environment, you begin to focus on sustainable growth. With the liquidity crunch, cash is in low supply and conserving cost on your data stack is becoming a priority.

You’re not alone

Cost was an oft-forgotten consideration for a lot of companies coming out of their growth stage. Data platforms have gotten cheaper by virtue of moving to the cloud, which partly contributes to why they aren’t monitored as a line-item compared to other bigger costs like salaries.

A note for 3rd party SaaS data platform users

If you are using a third party software to manage your costs, you’re likely not seeing the full breakdown of costs. This also means that your third party SaaS tool is charging you for features that you may not need. We highly recommend you consider switching to a self-owned Data Platform on your existing infrastructure. You’ll have better ability to negotiate for contracts with AWS, GCP, Azure or Alibaba than you will with the SaaS tool. Bulk pricing on these platforms often realizes higher cost savings through your entire lifecycle as a company.

The first step

Cloud providers aren’t incentivised to highlight when you’ve been spending more than you need to on their platform. This means that you can often go months or years paying more than you need to. Awareness of this reality is the first and most important step in getting to a cost-optimized data stack that works with your organization.

Tackling Data Platform Cost

Pricing for Data Platforms can vary depending on which platform and stack your use. There are 2 major components to Data Platform Cost: Storage and Compute and you should log into your platform to see this breakdown. If you are using any of the traditional cloud vendors, chances are that Storage is cheap, what’s really burning a hole in your wallet is Compute. Compute is used to move, transform and read information from your data warehouse.

Get detailed visibility of Compute

You should get a good sense of where Compute is costing you.

Understand the Unit of Measure

Before proceeding, you should have a preliminary understanding of the unit of measure used by each of the platforms you use. For example, the unit used in GCP is slot runs, while in Databricks it’s based on Databricks Units (DBUs). Associating each of the units to your activities below gives you a scientific way to apply and associate costs.

Enable Logging

While some platforms like Databricks collect advanced usage log information out of the box, others like GCP only capture basic information. You should look to collect as fine a grain as possible to ensure you have maximum capability to slice and dice the data.

Evaluate how your data stack operates

Several different paradigms under which teams operate. Some teams use Raw SQL from client app like Metabase, while others use Drag and Drop BI tools with a Semantic layer (e.g. Looker) or without one (e.g. Tableau). Each BI tool is different, and understanding how it operates will be key to your content refactor strategy.

Strategy

The idea is to:

1. Optimize the content that is most used by end users
2. Deprecate / refactor the rest to its level of utility

Identify your most important content

You should be targeted here; focus on the datasets used by 80% of your end users’ queries.

Classify into specific activities

The 3 categories you want to use to classify is:

1. Move: How much compute is used to move data into and out of your data platform?
2. Transform: What is the compute used in re-shaping data into tables that can used for analytical purposes?
3. Read: What compute is used in reading information from your data warehouse?

Each of these entries are non-trivial, and merit further deep-dive.

Moving Operations

The practice of centralising information into a single place is worthwhile. Doing this at scale while minimizing the number of instances that write to your data warehouse is the first step.

Get an understanding of how writes happen. You may have several processes reading data from each of your sources and adding data into your data warehouse.

Some questions to help you tackle compute: Is your refresh cadence appropriate? Are you filtering for the right set of rows from your production DB? Are your API calls to third party software optimized? Are there any in-flight transformations that could be differed?

We are big proponents of effective data capture, so ensure that you don’t compromise on data quality when refactoring your replication pipelines.

Finally, ensure that the compute processes powering your downstream use such as custom audience lists for your ads platform, reverse ETL into content marketing or your CRM tool is copying and refreshing the right set of datapoints at the right cadence.

Transforming Operations

The practice of transforming data into a form that can serve analytical and operational use cases is often a key enabler to centralising definitions for your org. If you are writing SQL to build new tables from existing tables in your Data Warehouse, or using DBT (data build tool) that’s doing this for you, then you are transforming data.

Datasets built by analysts/engineers are likely the most queried ones by your downstream users. Pay close attention to the set of joins happening amongst these datasets from your end users’ queries.

Once you’ve identified these datasets, it’s time to turn to your Logs. Filter logs that involve transformation commands - you want to look out for DDL language such as CREATE TABLE commands if your transformations happen with SQL.

Some questions to help you tackle this: Are you doing transformations in the most efficient way? Is there room to change your incremental refresh strategy? Can you build a DAG structure that minimizes the same set of operations for multiple of these datasets? Are there datasets that are normalized that you’d convert to de-normalized tables to simplify downstream querying from your BI tool?

Querying operations

As mentioned before, how your end users dispatch queries to the data warehouse is dependent on the BI layer you’ve setup. If you have a semantic layer that defines relationships, it’s likely that those are the joins that happen most often.

Wildcard matches on your logs may be an easy way to find the most commonly occuring operations (joins and aggregations) which would in turn tell you how many bytes are scanned. For platforms like GCP, billing happens on bytes processed. Platforms that use Spark typically look at CPU utilization. Understanding the nuance that is specific to your platform here is key to understanding which sets of content may be most costly.

Some questions to help you tackle this: Are you partitioning the data appropriately, and are those partitions being used by users? If not, could you find a way to have default partition filtering behavior? Are you making it easy for the query engine to find the information that is most repeatedly queried (e.g. clustering / Z-ordering)? Is your table architecture most appropriate? Do you need to move any frequently made joins into aggregated or denormalized tables? Is the query profile scanning more rows that it needs to in some intermediate CTEs?

Conclusion

What actions you take after going through an analysis of your logs is highly dependent on your organizational workflow. It’s possible to do some stop-gap measures like changing pricing structure with your data platform (e.g. on-demand vs flat pricing), but you are likely to find more success by pushing content-related changes. Not only does it improve end-user query experience, it also aligns your refactoring efforts towards an information architecture north star.

Interested in Data Platform Costs? Book a free consultation with us here. We’d love to chat about what’s got you worried.