In Data Engineer's Lunch #54, we will discuss the data build tool, a tool for managing data transformations with config files rather than code. We will be connecting it to Apache Spark and using it to perform transformations. Accompanying YouTube: https://youtu.be/dwZlYG6RCSY Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Data Engineer’s Lunch Weekly at 12 PM EST Every Monday: https://www.meetup.com/Data-Wranglers-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ Join The Anant Team: https://www.careers.anant.us

1. Version 1.0
Dbt and Spark
In Data Engineer's Lunch #54, we will discuss the data build
tool, a tool for managing data transformations with config files
rather than code. We will be connecting it to Apache Spark and
using it to perform transformations.
Obioma Anomnachi
Engineer @ Anant

2. Dbt
● dbt is a transformation workflow that lets teams quickly and collaboratively deploy analytics code
following software engineering best practices like modularity, portability, CI/CD, and
documentation
○ Essentially, dbt performs in place data transformations within the data warehouse you are already using to
store your data
○ It uses projects composed of .yml and .sql files to build a dag of simple transformations that combine to
create a full transformation pipeline

3. Dbt Adapters
● Dbt Labs, some software vendors, and some user communities maintain adapters for connecting
dbt to some of the most used data warehouse technologies and BI tools

4. Dbt Profiles
● Dbt connects to your data warehouse through data stored in a profiles.yml file
● Each adapter has different requirements for its profile
● Multiple profiles can be stored in the profiles.yml but only one can be used at a time

5. Dbt Projects
● Projects contain the yml files and sql files that define transformations and dependencies
○ Must contain
■ A project file: A dbt_project.yml file tells dbt that a particular directory is a dbt project, and also
contains configurations for your project.
■ Models: A model is a single .sql file. Each model contains a single select statement that either
transforms raw data into a dataset that is ready for analytics, or, more often, is an intermediate step
in such a transformation.
● Created via dbt init commands

6. Dbt Models
● A model is a select statement. Models are defined in .sql files (typically in your models directory):
○ Each .sql file contains one model / select statement
○ The name of the file is used as the model name
○ Models can be nested in subdirectories within the models directory

7. Dbt Apache Spark Adapter
● The dbt Spark adapter connects to persisted Spark SQL tables and performs transformations,
saving the results into new intermediate and final tables
● There are 3 possible connection methods depending on what type of Spark deployment you have
○ odbc is the preferred method when connecting to Databricks. It supports connecting to a SQL Endpoint or an
all-purpose interactive cluster.
○ http is a more generic method for connecting to a managed service that provides an HTTP endpoint.
Currently, this includes connections to a Databricks interactive cluster.
○ thrift connects directly to (a thrift server on) the lead node of a cluster, either locally hosted / on premise or
in the cloud (e.g. Amazon EMR).

8. Demo
● https://github.com/Anant/dbt_demo
● https://pastebin.com/6qVk6uzx

9. Resources
● https://github.com/dbt-labs/jaffle_shop
● https://docs.getdbt.com/tutorial/setting-up

10. Strategy: Scalable Fast Data
Architecture: Cassandra, Spark, Kafka
Engineering: Node, Python, JVM,CLR
Operations: Cloud, Container
Rescue: Downtime!! I need help.
www.anant.us | solutions@anant.us | (855) 262-6826
3 Washington Circle, NW | Suite 301 | Washington, DC 20037