Course Overview
DBT For Data Engineers

Incremental Views

Lesson #11

In this lesson we will:

  • Learn about DBT incremental update feature, which allows us to incrementally keep our destination tables up to date as new data is entered into our data sources.

What Are Incremental Updates

As we have seen in previous lessons, DBT will build new tables or views from existing tables and views, potentially in a chain of dependencies.

By default, each time DBT creates the new objects, it drops and then rebuilds the table from scratch.

This makes DBT very simple to reason about, and each time we can be sure we are getting fresh up-to-date data.

There are however a number of downsides of this approach:

  • It can be slow, as we are recomputing all of the queries and rebuilding all of the tables;
  • It can create significant load on the database;
  • This database load can in turn result in additional costs if you are paying for compute;

Incremental views enable us to update our views based only on deltas.

This is quite a simple ask, but does result in the data engineer taking on additional complexity in ensuring that their tables remain up to date and containing accurate data.

Next Lesson:

Ethemeral Views

In this lesson we will learn about DBTs ethemeral view feature to improve your pipeline readability.

0h 15m

Continuous Delivery For Data Engineers

This site has been developed by the team behind Timeflow, an Open Source CI/CD platform designed for Data Engineers who use dbt as part of the Modern Data Stack. Our platform helps Data Engineers improve the quality, reliability and speed of their data transformation pipelines.

Join our mailing list for our latest insights on Data Engineering:

Timeflow Academy is the leading online, hands-on platform for learning about Data Engineering using the Modern Data Stack. Bought to you by Timeflow CI

© 2023 Timeflow Academy. All rights reserved