data load tool (dlt): An introduction

An introduction to data load tool

There are always innovations in the data technology landscape, leading to thousands of tools being in existence. See Matt Turck's broad reaching MAD landscape for a scratching of the surface across the ecosystem. Every now and again there is a tool that takes a different approach to the problem space, and we’ve liked what we've seen so far from data load tool. In our experience with clients and their data teams, there is always a reinvention of the wheel when it comes to scripts for loading data from source, dealing with common issues like pagination, rate limiting, parallelisation, schema evolution and so on. Tools like Airbyte and its siblings aim to solve the problem by offering pre-configured connectors from supported sources to supported destinations, commonly tailoring behaviour via config files and low-code solutions. However, often these tools require containerization to handle deep dependency maps, leading to limitations integrating with existing infrastructure and restrictions in where the ELT code can be executed.

dlt is an open core approach to solving this problem in a composable and modular manner, allowing for customisation of logic required for larger enterprises, while providing a smooth enough onboarding path for a junior data analyst with Python knowledge. Importantly, it allows orchestration and infrastructure portability without the need to rewrite ELT logic, meaning that code can be executed in a serverless function or a kubernetes cluster, depending on the use case. By focusing on the foundational architecture of the data processing, and the metadata that this processing creates, we like the scope for dlthub, the company behind dlt, to build value-added features on top that attack the fundamental problems we see in data organizations - inefficient data movements, insufficient instrumentation and wasted spend. As a teaser, early comparisons of dlt running on AWS Lambda vs paid ELT SaaS tooling showed cost savings of two orders of magnitude (>100x), excluding FTE costs to maintain a self-managed setup.

Our initial impressions

dlt was created by data engineers, for data engineers to solve data engineering problems. We felt that the documentation (justifiably) prioritised getting started with a quick walkthrough, and it was easy to see the value, being able to run a simple dlt init github duckdb command in terminal to load gitub data into an in-memory duckDB instance within a few minutes. For users assessing how dlt may fit into their existing setup, the @dlt.xxx function decorators make for a simple enough start to migrating existing scripts, though there were more than a few edge cases in adapting program flow to work nicely with dlt's schema handling and other pipeline parameterization.

The target data engineer persona was clear beyond the quick start, as dlt's documentation quickly dives deeper into technical implementation rather than stepping back to introduce the concepts that are central to the tool--this is by no means a slight on the product, but a reflection of the background from which the tool has originated. For the similarly minded, our take on these is below:

This architecture and approach allows dlt to focus squarely on the tough problems within ELT definition and not get distracted by orchestration. There are valuable nuances not immediately apparent with the simple function decorator implementation including, for example:

• Optional unpacking of nested lists into child tables to a set depth
• dlt's wrapping of the requests library with retry logic to gracefully handle transient errors within a job
• Ability to shard and dynamically create resources (for example, into isolated destinations) based on the payload
• Ability to re-run specific partitions on failure
• Status management handled in dlt minimises the need to introspect the database when determining upsert targets
• Configuration of performance including parallelisation, async execution, memory and file management to fine tune pipeline config to the infrastructure capabilities

The simplicity of switching out datasets and destinations based on environment is welcome, allowing dev work to be done on a subset of data and, for example, an in-memory duckdb instance. Streamlit app integration to inspect destination schema, pipeline statistics and loaded data is a nice touch for improving productivity. This is encapsulated in dlthub's description of dlt making the easy things fast, and the hard things accessible.

There are few prerequisites and caveats to be aware of with the tool

• Code must be executable in Python. We'd love to see future polyglot setup which would enable porting to Rust, a language we're long term bullish on in data pipelines given its comparative performance efficiency
• It requires that input data be in JSON format before dlt normalises and loads it. You can of course, ingest XML and convert ahead of yielding it in a dlt resource. This may hinder fully-featured integration of binary files, but there are workarounds and performance drawbacks of base64 encoding are unlikely to be relevant for the vast majority of organisations
• dlt is both unopinionated and strict in how it manages schema. It supports inference and dynamic destination schema evolution, storing versioned hashes of schema in dlt's metadata tables for pipeline run management. Default behavior for things like table/resource naming is opinionated but can be overridden by users, and data contracts can be enforced with sources if desired. This structure allows for configuration of alerting and pipelines to evolve or fail depending on the use case required.

Overall, dlt is a welcome fresh approach to the ELT landscape and one which we feel fits a more established data engineering team who has felt the pains of pipeline management at scale, and are ready to take advantage of the full feature set it offers. dlt is a more natural fit with some pipeline orchestrators and data environments than others, so drop us a note if you are considering integrating it into your data stack.

Comparison to selected other tools

You may be comparing dlt to other alternatives in the market or migrating from an existing solution - we've highlighted a couple of the major options and their differences below

Singer / Meltano

• Singer was an open standard built in 2016 by Stitch. It showed promising early growth but since Stitch's acquisition by Talend, many of the connectors are not actively developed and quality is hit-and-miss.

• Out of the box, the Singer standard does not enforce controls on the handling of the data, instead focusing on the definition of a tap, the passing of that data to a target, and the definition of the target. This means that common tasks like paginating APIs, dealing with schema versioning and so on are custom to individual taps.

• Meltano has taken over as caretaker of the Singer tap and target ecosystem as it underpins their vision of a open source "dataops" layer as a foundation to teams' data stacks. Its Extractor and Loader plugins began as python virtual environment wrappers around a Singer tap or target.  They started with a broad goal and narrowed it over time to three key components:

  • A Meltano SDK to make it easier to build Singer taps
  • Meltano Hub to discover Singer taps and targets
  • Plugin support for other open source projects (transformations via dbt, data quality via great expectations, orchestration via Airflow, and visualisations via Superset)

• However, the paid Meltano solution appears to be losing focus in favor of Arch.dev, a platform to manage data ingestion for (a consulting service's?) clients, built on top of the Meltano stack. Viewing externally, we read this as another failure to generate enough commercial traction with Meltano itself to sustain the company, and question the foundation on which the system is built.

Airbyte

• Launched in 2020 by ex-Liveramp engineers, and learnt from the experience of data teams using Singer over the prior ~5 years. Airbyte is focused on a open source framework and platform for building connectors. This allows it to manage community contributions to connectors and promote them to the ecosystem if they meet quality standards. Airbyte also provides platform and user interface to configure and run data movement operations, including scheduling, logging, and alerting.
• To deal with the difficulty of managing dependencies in a pipeline, Airbyte is structured as a set of containerised microservices. Each component operates independently and interfaces with the control plane using the Airbyte Protocol. This opinionated approach provides a neat all-in-one solution but at the cost of a heavy installation and limited customizability.
• When it comes to the connector ecosystem, Airbyte touts a thriving catalog of well-maintained connectors, and for simple implementations this is true. The company has gradually moved away from code-first connector creation to a low code approach, recently communicating an intention to stop accepting community contributions made using the CDK. We view this as an acknowledgment that there are valuable standardizations/commodity tasks that were not natively present in the Airbyte platform, and pushing developers to use the low-code CDK or connector builder is required to drive up connector quality, standardization and development speed.

Fivetran

• Fivetran is the default standard in market for paid connectors-as-a-service. Quality is generally high, with uptime >99.9% and the Fivetran team taking the hard work out of schema evolution, type management, incremental updates and more. Similar to Airbyte, Fivetran adds on transformation support via dbt as well as a UI layer to manage and monitor pipelines.
• The main pains we hear from clients are around the steep increases in pricing on contract renewal / volume increases. At rack rates, inserting, updating or deleting 100m rows in a month will cost ~USD10k. For 10m rows, costs are ~USD3k.  At time of writing, we have heard reports of 30-60% price increases on renewal with limited engagement from sales staff in finding a win-win path forward, and a similar theme emerging with customer support when connector issues arise.
• Note that caution must be taken when comparing with open source solutions given the hosting, monitoring and human resource cost is often overlooked. As a rule of thumb, spending north of USD5k/mo often triggers our clients to re-evaluate their data ingestion setup.
• Talk to us about our data spend benchmarking and cloud cost optimisations if you are in this category, as we have helped clients save more than 60% on their data platform cost through intelligent optimization