Data Quality: How to Test Your Data Pipelines

Last Updated: July 2026 | 14 min read

Quick Answer: Data quality testing means running automated assertions inside your pipeline so broken or missing data is caught at load time instead of surfacing later as a wrong number on a dashboard. You test across six dimensions — completeness, accuracy, uniqueness, validity, consistency, and timeliness — at three points: at ingestion, after transformation, and before serving. Use dbt tests (not_null, unique, accepted_values, custom SQL) for warehouse models and Great Expectations for ingestion-time validation. When a check fails, either block the pipeline (for invariants like a null primary key) or quarantine the bad rows and keep going. The goal isn't perfect data — it's known data, where every failure is caught and named before a human is.

There's a specific kind of dread in data engineering: an executive messages you that a number "looks off," and you realize a silent data problem has been quietly poisoning a dashboard for two weeks. No error, no failed job — just wrong data that everyone trusted. After years of building pipelines on Spark, dbt, and Airflow, I can tell you the teams that avoid this aren't smarter; they just test their data like they test their code. This guide shows you exactly how.

You'll learn the six data quality dimensions, where checks belong, the quarantine pattern, and how to choose between dbt tests and Great Expectations — with code you can copy.

What Is Data Quality?

Data quality is a measure of how well data fits its intended use — and in engineering terms, it's a set of automated tests that assert your data is correct before anyone depends on it. It's the difference between a pipeline that runs successfully and one that produces correct results. Those are not the same thing.

A job can finish green while delivering garbage: a joined table that silently dropped half its rows, a currency column that flipped sign, a daily load that never arrived. Code tests catch bugs in your logic; data quality tests catch problems in the data itself — bad inputs, broken assumptions, upstream changes you didn't know about.

The mindset shift that fixes this: treat data as a tested artifact, not a byproduct. Every table that feeds a decision should carry assertions about what "correct" means, and those assertions should run automatically on every load.

The Six Dimensions of Data Quality

Almost every real data quality problem maps to one of six dimensions. Knowing them turns "is this data good?" from a vague worry into a concrete checklist.

How to use this table: for every important table, walk the six dimensions and write at least one test per relevant dimension. Most tables need completeness, uniqueness, and validity checks at minimum; fact tables that feed metrics also need consistency and timeliness. You don't need hundreds of tests — you need the right handful that guard what actually matters.

Where to Put Checks: The Three Gates

Data quality checks belong at three gates in every pipeline — at ingestion, after transformation, and before serving — because a failure caught early is exponentially cheaper than one caught late.

1. Ingestion gate (validate the raw input). Test data the moment it enters your platform, before it touches anything downstream. Schema conforms? Required fields present? Row count in a sane range? Catching a malformed source file here stops it from corrupting everything after it.
2. Transformation gate (validate the logic). After your dbt models or Spark jobs run, test the output. Did the join keep the rows you expected? Are the aggregates in a believable range? This is where you catch bugs in your own transformation logic.
3. Serving gate (validate before consumers see it). The last line of defense before dashboards, APIs, and ML models read the data. Final invariants: no nulls in key columns, metrics within expected bounds, freshness confirmed.

The cost curve is the whole argument. A bad row rejected at ingestion costs seconds. The same bad row discovered as a wrong KPI in a board meeting costs your credibility and a week of forensic SQL. Push checks left.

Writing Data Quality Tests with dbt

dbt tests are SQL assertions that live beside your models and run with dbt build — the fastest way to add data quality to any warehouse-based pipeline. If your transformations are already in dbt, this is where you start.

dbt ships four built-in generic tests you declare in YAML — no SQL required:

# models/schema.yml
models:
  - name: orders
    columns:
      - name: order_id
        tests:
          - not_null          # completeness
          - unique            # uniqueness
      - name: status
        tests:
          - accepted_values:  # validity
              values: ['pending', 'paid', 'shipped', 'cancelled']
      - name: customer_id
        tests:
          - not_null
          - relationships:    # consistency (referential integrity)
              to: ref('customers')
              field: customer_id

For anything the built-ins can't express, write a singular test — a SQL query that returns the bad rows. If it returns any rows, the test fails:

-- tests/assert_order_total_matches_line_items.sql
-- accuracy: order total must equal the sum of its line items
select o.order_id, o.total, sum(li.amount) as line_total
from {{ ref('orders') }} o
join {{ ref('line_items') }} li using (order_id)
group by o.order_id, o.total
having o.total != sum(li.amount)

Run everything with dbt build, which executes models and their tests together and stops the run if a blocking test fails. Add dbt to your Airflow DAG and these checks run on every scheduled load automatically.

Validating at Ingestion with Great Expectations

Great Expectations is a standalone Python framework for validating data before it reaches your warehouse, with a large library of ready-made checks and human-readable validation reports. Reach for it at the ingestion gate, where dbt can't help because the data isn't in the warehouse yet.

You declare "expectations" against a batch of data — in a PySpark or pandas pipeline — and get a pass/fail result plus a detailed report:

import great_expectations as gx

context = gx.get_context()
batch = context.sources.add_spark("raw").read_dataframe(df)

batch.expect_column_values_to_not_be_null("customer_id")           # completeness
batch.expect_column_values_to_be_unique("order_id")                # uniqueness
batch.expect_column_values_to_be_between("total", 0, 1_000_000)    # validity
batch.expect_column_values_to_be_in_set("status",
        ["pending", "paid", "shipped", "cancelled"])               # validity
batch.expect_table_row_count_to_be_between(1000, 5_000_000)        # completeness

results = batch.validate()
if not results.success:
    raise ValueError("Raw batch failed validation — see Data Docs")

The payoff over hand-rolled assert statements is the library and the reporting: dozens of prebuilt expectations, automatic data profiling to suggest checks, and shareable "Data Docs" that show non-engineers exactly what passed and failed.

dbt tests vs Great Expectations — which to use

The verdict: they're complementary, not rivals. Use Great Expectations to guard the front door (raw ingestion) and dbt tests to guard your models and serving layer. If you only have bandwidth for one and you're already on dbt, start with dbt tests — they're nearly free to add.

The Quarantine Pattern: Don't Drop, Don't Crash

When a batch contains bad rows, the quarantine pattern routes the good rows onward and the bad rows to a separate table with their failure reason — instead of failing the whole job or silently dropping data. It's the single most useful data quality pattern in production.

The naive options are both bad: crash the pipeline on one bad row and you halt everything for a single typo; silently WHERE-filter bad rows and you've now hidden a data problem. Quarantine gives you a third path:

from pyspark.sql import functions as F

# tag each row with why it's invalid (null = valid)
checked = raw.withColumn("dq_error",
    F.when(F.col("customer_id").isNull(), "null_customer_id")
     .when(F.col("total") < 0, "negative_total")
     .when(~F.col("status").isin("pending","paid","shipped","cancelled"), "bad_status")
     .otherwise(None))

valid   = checked.filter("dq_error IS NULL").drop("dq_error")
invalid = checked.filter("dq_error IS NOT NULL")

valid.writeTo("lake.orders").append()               # clean data flows on
invalid.writeTo("lake.orders_quarantine").append()  # bad rows preserved + reason

Now the pipeline keeps running, no data is lost, and every rejected row is inspectable. You alert on quarantine volume, fix the upstream source, and reprocess the quarantined rows once it's healthy. Pair this with idempotent writes so reprocessing never creates duplicates.

Blocking vs Warning: Not Every Check Should Page You

Tag every data quality test as blocking or warning so critical failures stop the pipeline while minor anomalies just alert — otherwise you either corrupt data silently or drown in false alarms.

• Blocking checks guard invariants that must never break: a null primary key, a duplicated id, revenue that dropped to zero overnight, a missing daily partition. These stop the pipeline before bad data is published. A false negative here is a corrupted dashboard.
• Warning checks flag things worth a human look but not worth halting production: a 15% row-count drop, a brand-new category value, a slightly stale feed. These fire an alert and let the pipeline continue.

Getting this split right is what makes data quality sustainable. Make everything blocking and one flaky source pages you at 3 a.m. for nothing until you start ignoring alerts. Make nothing blocking and corruption sails through. The invariants block; the anomalies warn.

Common Data Quality Mistakes to Avoid

• Testing code but never data. Green pipeline ≠ correct data. Assert on the data itself.
• Only checking at the end. A bad row caught at serving already cost you everything upstream. Test at ingestion first.
• Silently filtering bad rows. WHERE is_valid hides the problem. Quarantine and count instead.
• Making every check blocking. Alert fatigue kills data quality programs. Split blocking vs warning.
• No freshness check. The most common silent failure is data that simply stopped arriving. Always test timeliness.
• Hundreds of trivial tests, zero on what matters. Guard the invariants that feed real decisions, not every column for the sake of it.

See how SolutionGigs can help → Retrofitting data quality onto a pipeline — dimensions, gates, quarantine, alerting — is exactly what our vetted data engineers do. Get matched on solutiongigs.in.

Frequently Asked Questions

What is data quality in data engineering?

Data quality is how well data fits its intended use, measured across dimensions like completeness, accuracy, uniqueness, validity, consistency, and timeliness. In practice it means running automated checks inside the pipeline — asserting a key is never null, an amount is never negative, yesterday's data actually arrived — so broken data is caught at load time instead of surfacing later as a wrong number on a dashboard.

What are the six dimensions of data quality?

The six dimensions are completeness (no missing values or rows), accuracy (values match reality), uniqueness (no unintended duplicates), validity (values fit a format or range), consistency (the same fact agrees across tables), and timeliness (data is fresh and on schedule). Most real-world tests map to one of these six, which makes them a practical checklist when deciding what to assert about a table.

Where should data quality checks go in a pipeline?

Put checks at three gates: at ingestion (validate raw input before it enters the lake), after transformation (confirm your logic produced correct results), and before serving (final assertions before dashboards and ML read the data). The earlier a check runs, the cheaper the failure — catching a bad row at ingestion costs seconds; finding it as a wrong KPI later costs credibility and days.

What is the difference between dbt tests and Great Expectations?

dbt tests are SQL assertions that live beside your models and run with dbt build — ideal for in-warehouse checks like not_null, unique, and accepted_values. Great Expectations is a standalone Python framework with a large library of expectations, data profiling, and rich reports — better for validating data before it reaches the warehouse or across non-SQL sources. Most teams use both: Great Expectations at ingestion, dbt tests for warehouse models.

What is the quarantine pattern in data quality?

The quarantine pattern splits a batch into rows that pass and rows that fail: valid rows continue into the clean table while invalid rows go to a separate quarantine table with their failure reason, instead of crashing the job or silently dropping data. It keeps the pipeline running, preserves bad records for investigation, and lets you reprocess them once the source is fixed.

Should a failed data quality check stop the pipeline?

It depends on severity. Blocking checks guard invariants that must never break — a null primary key, a duplicated id, revenue dropping to zero — and should stop the pipeline before bad data is published. Warning checks flag anomalies worth investigating — a 20% row-count drop, a new category value — and should alert without halting. Tagging every test with a severity avoids both silent corruption and needless 3 a.m. pages.

How do you measure data quality?

Measure it with a small set of scored metrics per table: the percentage of rows passing each dimension's checks (e.g. 99.98% completeness on customer_id), the number of rows quarantined per run, and freshness lag (how old the newest row is). Track these over time so you can see quality trends and set thresholds that trigger blocking or warning behavior.

Conclusion

The teams whose numbers people trust aren't the ones with the fanciest stack — they're the ones who test their data like they test their code. Data quality isn't a tool you buy; it's a habit you build into every pipeline: walk the six dimensions, put checks at the ingestion, transformation, and serving gates, quarantine bad rows instead of dropping or crashing, and split your checks into blocking invariants and warning anomalies.

Start small. Pick your most important table, add a not_null and unique test on its key and a freshness check today, then expand outward. The goal was never perfect data — it's known data, where every failure is caught and named before a human is. That's what turns "the number looks off" from a recurring nightmare into a problem you caught yesterday, automatically.

Building or fixing a pipeline and want data quality done right from the start? Get matched with a vetted data engineer on solutiongigs.in today — it's free to post. New to the field? Start with What Is Data Engineering? and The Modern Data Stack in 2026.

Mohammed Yaseen

Founder, SolutionGigs

Mohammed builds data pipelines on Spark, dbt, and Airflow, and writes about testing data as rigorously as code. LinkedIn →