My experience

While I only tried out the one AI tool, my experience with it was positive. It took maybe 20 minutes to create that diagram in total. I gave it a prompt, it generated a decent diagram on the first shot, then I followed it up with 2 rounds of edits via prompts and a final round of manual edits.

At the end of the day, while I do quite prefer how my manually created diagrams come out in terms of style and feel (and the fact that I can reuse components I’ve made before for new diagrams), I would definitely try AI diagramming again.

Ingestion

Starting with ingestion, you have a couple different options depending on the type of data source you want to ingest. If you are ingesting from some sort of database (such as an on-prem DB, an AWS Relational Database Service DB, or an AWS Dynamo DB), you will most likely use AWS Database Migration Service (DMS). As the name implies, you can migrate (copy over) a database from one database engine to another. You can also use DMS to set up Change Data Capture (CDC) between your source and target. The target for ingestion is usually a raw / landing zone bucket in S3. Some more options from ingesting from a DB is Glue and EMR.

Another popular ingestion source are SaaS tools (such as from your crm, finance, and operations tools), for which you will use AWS AppFlow for. Finally, when it comes to streaming data sources (like IoT, sensors, and data for real time analytics), you have options ranging from micro-batch (AWS Firehose) to true streaming (AWS Data Streams) to managed kafka (AWS MSK), just to name a few.

Lake & Processing

The data lake is where most of your data ends up. Typically all of your raw, unprocessed data will get ingested into a bucket designated as the landing zone. From there you will have one or more buckets as designated “processing zones”, which store different stages of processed data. To keep your data catalog up to date, you’ll have AWS Glue Crawlers crawl the final processed zone on a recurring basis.

For the actual data processing / transformation steps, you have three main options. The simplest one is AWS Lambda, which is a stupid simple serverless compute service. You define a function in a language such as python and set up rules regarding when the function gets triggered and what happens downstream. For data engineering, lambda is commonly used for setting up data validation checks or light data transformations. You can have tons of lambdas triggering per second, but the main limitation is the amount of data it can process at once. For bigger workloads, reach for AWS Glue or EMR (Elastic Map Reduce). Glue is used for building ELT pipelines and is built on top of apache spark while EMR is used to run big data frameworks (such as spark, trino, and pig) for processing data. Glue is serverless and simpler to setup than EMR, which provides more customizability and better performance at scale on large workloads. EMR is also used for data ingestion due to its highly customizable and powerful data processing.

Consumption

This layer is quite straightforward as there is only one option for each consumption-use case. If you want to query your data catalog (or warehouse), use AWS Athena. If you want to build visualizations, dashboards, or reports, use AWS QuickSight. Finally, if you want to do machine learning, you will most likely use AWS SageMaker (I know there are other options here, but I personally haven’t looked into them extensively).

One small nuance worth mentioning is that these consumption tools can be used on either your data catalog (your S3 lake) or your AWS RedShift data warehouse (if you chose to set that up). Usually the warehouse will host a subset of the lake’s data for higher performance querying at the tradeoff of higher storage cost.

Orchestration

The last main class of services data engineers should know in AWS relate to workflow orchestration. A popular option is Glue, but its limited to orchestrating Glue tasks (Glue jobs, crawlers, and triggers). If you aren’t using Glue it makes more sense to use AWS Step Functions, which has tight integration with most AWS services. You can define your pipelines using amazon state language or interactively with a UI.

Finally, my favorite choice for orchestration is AWS Managed Workflows for Apache Airflow (MWAA). Since most data teams are intimately familiar with Airflow or one of its popular descendants (such as Prefect and Dagster), this option is the most portable. Defining your orchestration workflows in python code is better from a reproducibility and engineering perspective than the other options which are only available in AWS.

What is Apache Superset?

Analysts use Business Intelligence (BI) tools to build data visualizations, reports, and dashboards that tell a story on how the business performed, is performing, or will perform. Compared to its older siblings data science and machine learning, BI is the most accessible method by which organizations can turn data into insights.

Apache Superset is one such BI software. An open source one at that! As someone that wants to learn at least one open-source tool per component of the data engineering lifecycle, Superset seemed like a great match for my projects.

Setup issues and how I fixed them

The main issue I had with Superset initially was finding a minimal and simple way of running it with docker compose. It had to have a small footprint and easily integrate with the other services I would be running (such as airflow). I looked to the official documentation and LLMs for help, but they weren’t very helpful. That’s when I stumbled upon this video by a data engineer named Calvin Yoon.

He stripped down the docker compose template provided by superset down to its essentials (sure, you could probably strip it down a lot more if you wanted to, but its already a big improvement from following superset’s official instructions of cloning the whole repository and working with that).

All you need is less than 100 lines added to your docker compose and a folder containing some bash scripts and configs. Here’s a look at what the docker compose would look like if you were only running superset and no other services:

We’re basically just setting up the: superset database (postgres), caching (redis), a container to handle initialization, and a container for the superset app itself.

You may also notice the environment variable on line 68 that appears to turn off examples. If you change that to “yes” and run docker compose up, superset should launch with some visual examples you can explore and learn from.

Conclusion

That’s basically it!

If you want to try running this for yourself, take a gander at the repo accompanying this article. It has all the code and startup instructions. Here is the link for that again: sammyhansali/superset-docker-quickstart

The dbt data quality toolkit

dbt provides two main data quality features, tests and model contracts. Contracts enforce your data’s *shape*: field names, types, and constraints (Not all constraints are enforceable, however. This depends on your dbt model’s materialization strategy and your specific data warehousing platform). Tests are exactly what they sound like: lines of code that test your data’s integrity, logical structure, and more.

Let’s talk more about tests because there is a lot going on here. Generally there are two types of tests, data tests and unit tests. The latter should sound familiar to those with a background in software. Unit tests in dbt work by taking a sample provided input and output and checking if the model produces the expected output.

Within data tests, there is yet another bifurcation into generic and singular data tests. Singular tests are any kind of test written for a single scenario. Some common uses of singular tests include checking assumptions (such as, order_amount_dollars >= 0) or other business logic (such as, revenue - expenses = profit). Generic tests, as they sound, are data tests that are reusable for multiple scenarios. Unlike singular tests which can be written to take multiple dbt models and multiple columns into account, generic tests can only be applied to one model and column.

Finally, there are two types of generic tests: built-in and custom. dbt comes with four built-in generic tests: not_null, unique, accepted_values, and relationships. Use them to test if columns respect the corresponding SQL constraints. Other “built-in” generic tests can be imported from dbt packages, such as dbt-expecations. If you want to make your own generic test, such as a test that checks if a column is positive, that is called a custom generic test.

Brief notes on implementation

Data tests, specifically singular and custom generic tests, are defined in .sql files in your dbt project’s ./tests and ./tests/generic directories, respectively. Singular tests are written as a sql query that returns rows that violate the test. Custom generic tests are written in a similar fashion, but require additional jinja templating.

Data tests are activated once written under the data_tests or tests attribute for a model as defined in a yaml file in the project directory (typically in the same directory as the model). Unit tests are defined and activated once written under the unit_tests attribute for a model, in the same type of yaml file.

~

Hopefully that served as a useful introduction to the different tools you have available to you in dbt’s data quality tool kit! Below are some helpful links that helped me understand everything:

• https://www.metaplane.dev/blog/dbt-test-examples-best-practices#types-of-tests-in-dbt

• https://docs.getdbt.com/blog/announcing-unit-testing

• https://docs.getdbt.com/reference/resource-configs/contract

• https://docs.getdbt.com/docs/mesh/govern/model-contracts#how-are-contracts-different-from-tests

• https://docs.getdbt.com/docs/build/data-tests

I’ve recently finished reading Itzik Ben-Gan's book "T-SQL Fundamentals, Fourth Edition", and thought it would be helpful to distill some of the more conceptual topics from the first chapter. This was difficult to write and there are some oversimplifications at certain points, but overall it helped me improve my understanding.

Theoretical background to T-SQL querying and programming

Structured data is stored in tables, with rows and columns, within relational databases. Relational database management system (RDBMS) software is used to query and manipulate these tables. These operations are performed using the SQL (structured query language) programming language.

Each RDBMS implements a different "dialect" of standard SQL. Each dialect extends a subset of the SQL standard and adds vendor-specific extensions. For SQL Server (Microsoft's on premises RDBMS), a dialect called T-SQL (Transact SQL) is supported. It adds richer metadata, error-handling, control flow, and more.

The R in RDBMS stands for "relational". This is because RDBMS's are based on the "relational model". This model enables consistency, completeness, and integrity for data management. It is built upon a branch of mathematics called relational algebra, which itself is based on two other branches of mathematics: set theory and predicate logic. Set theory is concerned with a mathematical object called a set, which is a collection of distinct objects in a collection. Sets that are non-distinct are called multisets (or bags). Predicate logic is concerned with predicates. In two-valued predicate logic, expressions are either true or false. In three-valued predicate logic, which T-SQL supports, an expression can also evaluate as unknown.

Relational algebra itself is concerned with operations performed on a mathematical object called a relation. Mathematically, a relation is a representation of a set. In the relational model, a relation is a table. In a fully normalized database, there is one relation for each entity (such as Customer, Order, or Order Details). A de-normalized database introduces redundancy. Each relation has a set of attributes (or columns) with a name, data type, and (optional) constraint. Relations also have a set of tuples (or rows) which comprise the table. Predicates can be used to define constraints on attributes to ensure data integrity.

Types of database workloads

An OLTP database is a database where you store transactional data - data that adds, modifies, or deletes data on a regular basis. OLTP stands for "online transactional processing". OLTP databases are usually highly normalized, following the relational model, which leads to poor performance on analytical/reporting queries since they require many joins.

A data warehouse is a database that stores analytical data - data that is more de-normalized and requires minimal processing time for analytical/reporting queries. Here, the data is modeled into a schema design such as a star (one fact table and several dimensions tables that relate to it) or a snowflake (an expanded version of the star). Data warehouses can also have data marts, which are like mini-data warehouses designed for particular entities (such as for the Sales department, Marketing department, etc).

In many data pipelines, an ETL/ELT software regularly extracts data from an OLTP database into a data staging area (DSA), transforms it, and loads it into the data warehouse.

SQL Server Architecture

How is data managed in on-premises SQL Server? There are two aspects to consider: logical and physical.

Logical Architecture

In Microsoft SQL Server, there are multiple instances of the server stored on the same computer. One instance must be default, the others are named. These instances serve as contained environments. You specify the instance you want to access during server connection time.

Each instance is composed of databases, which are containers of database objects (such as tables, views, and stored procedures). There are two types of databases: user-defined databases and system databases. There are five system databases: master, model, tempdb, msdb, and Resource.

• master holds system-wide metadata, server configuration data, instance-specific data, and initialization information.

• model is used as a template for creating user-defined databases. Each new database is created as a copy of the `model` database.

• tempdb is used to store temporary data, such as: temporary tables, sort and hash table data (when needed to persist those), row versioning, and more. This database is destroyed and recreated whenever the SQL Server instance is restarted.

• msdb is used by the SQL Server Agent service to store its data. Also holds info related to other SQL Server features (dbmail, service broker, backups, and more).

• Resource is a hidden, read-only db holding definitions of all system objects. All system objects in a database appear to be in that database's `sys` schema, but actually their definitions reside in the `Resource` database.

Each database is composed of schemas, and each schema contains database objects.

Physical Architecture

Physically, databases are made up of at least one data file, at least one transaction log file, and (optionally) checkpoint files holding memory-optimized data. Data files are organized into logical groups called filegroups, which are targets for creating a database object. When you create an object, its data is physically distributed across files in that filegroup. You must have at least one primary file group, which contains the primary data file (a .mdf extension). Non-primary data files have a .ndf extension and log files have a .ldf extension.