Mapping Business Value to your Cost Optimization Practices

Cost Management so far

Thanks to generous free tier credits from cloud providers, venture-backed startups (especially during the pandemic) often found themselves unburdened to build and deploy systems on the cloud. The increased speed also led to mounting costs, but cost management was often de-prioritized. Components of a data platform in particular (such as data warehouses, datalakes and pipelines) inflated costs substantially. However, as the business landscape shifted, optimizing these costs became critical.

Unfortunately, the burden of reining in expenditures often fell solely on engineering teams. Under pressure to cut costs, they sometimes made decisions detrimental to overall business performance. A common approach involved triaging the largest cost centers and tweaking them. For example, engineers would address data platform costs by looking at the costliest queries run against their data warehouse, and optimizing them.

This wasn’t ideal. Business priorities change when you are building something new, so while costs were optimized for one workload, new business ventures spawned many more “one-off” cost centers inflating the tail of the log distribution. More importantly, this method of addressing biggest-cost-first lacked nuance and a clear mapping of resources to tangible value.

A different approach

The truth is, responsible cost management requires a more holistic framework - one that assesses what is truly value-adding as well as what is strategically critical for an organization's operations and goals. Simply looking at high-level cost breakdowns from cloud vendor dashboards like resource consumption by user, project, table scans, and DML statistics paints an incomplete picture. It doesn't illuminate whether a workload is necessary or superfluous, or to what degree it drives value.

We believe organizations can do better. To bridge this gap, we developed models that classify workloads and resources based on their underlying use cases and value drivers. This additional metadata layer helps organizations determine what is actually useful versus what can potentially be optimized or decommissioned.

Adding Business Context

Our classification models surface attributes like:

• Use case type (reporting, transformation, ad-hoc analysis, operational analytics, etc.)
• Use case details (reporting tooling, transformation mechanisms like DBT vs. stored procedures, optimization techniques)
• Governance characteristics (resource/workload tagging by team or project, builds with complete CI/CD and PR reviews)

The importance of tagging

As you may note, there are some components that need to work in tandem to unlock this value and use case mapping. In particular, tagging of each resource and job is important. For mature organizations this often manifests in the form of medallion architecture, but it doesn’t have to be quite as elaborate.

The tagging mechanism can be built intelligently. For instance,

• Many data platforms use a transformation tool such as dbt or SQLMesh, which consist of pre and post hooks that can tag a resource depending on the folder the resource is in, its creator, or other such environment characteristics.
• BI tools and query engines can also pass metadata upstream to allow for tagging of the end-resource (BI tool name/dashboard ID)

Tagging also helps organizations denote certain workloads as more critical than others. For instance, pipelines tagged to serve embedded dashboards in a client app are more critical than those for internal reporting.

Parsing SQL Text

Having mapped the most valuable resources (and subsequently the most critical workloads), we can get more granular by assessing the SQL that reads and writes from these. This includes things like:

• Use of Partitions and Clustering within workloads
• Cache hits on the most frequent queries
• Split of Merge vs Replace operations (and determining incremental vs full reloads)
• Query stage distributions (query waiting, reading, shuffle, etc.)

The additional layers of information help you fine-tune and build upon existing assets, replacing the more common method amongst data engineers and analysts to audit and re-write assets. Not only does this reduce operational overhead for downstream users (not having to re-learn about new data models) but also helps constantly monitor and fine-tune models post-deployment.

Deployment

These models can be deployed into any data warehouse of choice, or in your BI tool. Model deployment simply requires:

1. Access to data platform metadata and query logs.
2. Appropriate tagging (see section below).

Conclusion

With greater clarity into each workload's purpose and value, organizations can finally align their data platform costs with business priorities instead of taking a blunt, indiscriminate approach. Subsequently, a granular understanding of queries provide a way to optimize queries quickly and thoughfully. As companies strive to be more fiscally responsible with data investments, this level of cost (and value) intelligence is crucial.

With the right cost management strategy and tools, companies can maximize their data platforms' ROI without compromising on innovation or competitive advantage.

Image credits: Freepik