AI and Data Quality: The Real Reason AI Projects Fail

Quick Answer: AI data quality is how accurate, complete, consistent and correctly labelled a dataset is for the specific decision a model has to make. It is the dominant cause of AI failure — Gartner expects organisations to abandon 60% of AI projects unsupported by AI-ready data through 2026. The reason is mechanical, not cultural: a bad row in a dashboard is visible, but a bad label in a training set silently becomes part of what the model believes. In the lab further down this page, a real classifier falls from 85.7% to 60.7% accuracy and starts routing genuine production outages to the low-priority queue when one in five of its labels is wrong. Not one line of the model code changes.

Every team I have worked with wanted to talk about which model to use. Almost none of them wanted to talk about who labelled the training data, or what the label actually meant.

That ordering is the single most reliable predictor of whether an AI project ships. The model is a weekend decision — you can swap it in an afternoon. The data is a quarter of work, and it is the part that decides the outcome.

This article does something the other results for this query do not: it lets you prove it to yourself. There is a real machine learning model embedded below. You control its training data with three sliders. You will watch it fail, and — more unsettling — you will watch it fail while still reporting a healthy number.

---

What "AI data quality" actually means

AI data quality is the fitness of a dataset for the decision a model must make — accuracy, completeness, consistency, representativeness, timeliness, and above all label correctness. It overlaps with classic data quality but is not the same discipline, and treating them as identical is where most teams lose a quarter.

The difference is what happens to an error. In business intelligence, a wrong value produces a wrong number on a dashboard, and someone eventually notices the revenue line looks odd. In machine learning, a wrong value produces a model that has learned the error and applies it confidently to every future case. One is a bug report. The other is a belief.

	Traditional data quality	AI-ready data
Unit of failure	A wrong row	A wrong belief, applied at scale
Detection	Reconciliation, dashboards look wrong	Often none — the model is confident either way
Labels	Usually not applicable	The primary quality axis; ambiguous labels cap accuracy
Representativeness	Nice to have	Decisive — the model can only learn what the data contains
Duplicates	Inflate a count	Leak between train and test, inflating your score
Fix latency	Correct the row	Correct the row, then retrain and re-validate
Who owns it	Data platform team	Nobody, usually — which is the real finding

That last row is the one worth sitting with. Almost every organisation has an owner for pipeline uptime and no owner at all for whether "urgent" means the same thing this quarter as it did last quarter.

The evidence: what actually kills AI projects

Across every credible survey of the last two years, the top cause of AI failure is data, not modelling. Three sources are worth citing precisely, because the numbers circulating on this topic are frequently misquoted.

• Gartner predicts that through 2026, organisations will abandon 60% of AI projects that are unsupported by AI-ready data. The same February 2025 release reports that, in a Q3 2024 survey of 248 data management leaders, 63% either did not have the right data management practices for AI or were unsure whether they did. (Gartner newsroom)
• MIT's Project NANDA, in The GenAI Divide: State of AI in Business 2025, found that around 95% of enterprise generative AI pilots produced no measurable P&L return, drawn from 300+ public initiatives, 52 organisation interviews and 153 leader surveys. Treat it as a strong signal rather than settled evidence — it is a single, non-peer-reviewed report, and the headline number is widely quoted without that caveat.
• Google researchers interviewed 53 AI practitioners across India, East and West Africa and the USA and found 92% had experienced at least one "data cascade" — a compounding downstream failure caused by upstream data problems — with 45.3% hitting two or more in a single project. The paper's title is the entire industry in one line: "Everyone wants to do the model work, not the data work." (CHI 2021)

Notice what none of those say. None of them say the models were not good enough.

Four real failures, and the data problem inside each

Abstract statistics do not change behaviour. Specific disasters do. Each of these is a well-documented public case, and in each one the model did exactly what its data told it to.

Amazon's recruiting engine taught itself to prefer men

Amazon built an experimental tool to score CVs, trained on a decade of the company's own hiring history. Because that history skewed heavily male, the system learned that male-associated patterns predicted "hired" and penalised CVs containing the word "women's" — as in "women's chess club captain". The project was scrapped. The model was fine. It faithfully reproduced a decade of the input distribution.

Zillow lost hundreds of millions to a forecasting model

Zillow Offers bought homes based on algorithmic price forecasts. In Q3 2021 Zillow recorded an inventory write-down of roughly $304 million after buying homes above its own estimates of future selling prices, guided to a further $240–265 million of Q4 losses, announced it would wind the business down, and cut about 25% of its workforce. The full-year 2021 inventory write-down came to $407.9 million. CEO Rich Barton's explanation was, in effect, a data statement: the unpredictability in forecasting home prices "far exceeds what we anticipated." (Zillow Q3 2021 results)

Air Canada was held liable for a policy its chatbot invented

In Moffatt v. Air Canada (2024), British Columbia's Civil Resolution Tribunal ordered the airline to pay damages after its website chatbot told a grieving passenger he could buy a full-price ticket and claim a bereavement rate retroactively — which was not the airline's policy. Air Canada argued the chatbot was effectively a separate entity responsible for its own statements. The tribunal disagreed and awarded CA$812.02. This is the modern shape of a data quality failure: no metric moved, no alert fired, and the organisation acquired a legal liability.

The Dutch childcare benefits scandal

The Dutch tax authority used risk-scoring to flag childcare benefit fraud. Tens of thousands of families — disproportionately those with dual nationality — were wrongly accused and forced to repay benefits they were entitled to, in a scandal severe enough to bring down the government in January 2021. The features fed to the system encoded a discriminatory assumption. The data was the policy.

The pattern is identical in all four: the failure was fully present in the data before a single model was trained.

How bad data reaches your model — the four gates

There are only four places data quality breaks on the way to a model, and each one fails with a different signature. Knowing the signature is how you diagnose an AI failure in an afternoon instead of a month.

1. Collection. The data does not represent the world the model will meet. Signature: the model performs well in evaluation and badly in production, and the gap does not close with more training.
2. Labelling. The ground truth is wrong or ambiguous. Signature: accuracy plateaus below what the problem should allow, and two annotators disagree on the cases the model gets wrong.
3. Training. Test information leaks into training, or the split is not truly held out. Signature: an excellent offline score that production never reproduces.
4. Serving. The features computed at inference differ from those computed in training — a different join, a different timezone, a stale table. Signature: the model degrades slowly and nobody can point to a change.

Gates 2 and 3 are the ones you are about to operate by hand.

Predict before you play

Before you touch the controls, commit to an answer. This matters — you will remember the result far better if you have staked a position first, and the honest answer surprises most experienced engineers.

What the lab just showed you

Three findings come out of that widget, and all three contradict something people believe about AI.

Finding 1: label noise is not linear. It is a cliff.

Here is the full sweep, computed from the same code running in your browser:

Mislabelled training tickets	Accuracy	Change	Real outages missed
0% (clean)	85.7%	—	3
5%	82.1%	−3.6 pts	4
10%	82.1%	−3.6 pts	2
15%	67.9%	−17.9 pts	5
20%	60.7%	−25.0 pts	7
30%	53.6%	−32.1 pts	7
40%	53.6%	−32.1 pts	6

Ten percent of your labels can be wrong and the model still looks broadly fine — it loses 3.6 points, which any team would attribute to noise, a bad week, or "we need more data". Double that to 20% and it loses 25 points and drops to barely above a coin flip by 30%.

This is why label quality problems are discovered late. The damage is invisible exactly as long as it is cheap to fix, and becomes obvious only once it is expensive.

Finding 2: the model changes its mind about a real outage, and never tells you

Type the api is returning 500 for every checkout into the probe box and move the noise slider. The model's log-odds on that ticket go:

Mislabelled	Verdict	Confidence (log-odds)
0%	URGENT	+3.04
10%	URGENT	+2.91
15%	URGENT	+1.42
20%	ROUTINE	−0.99
30%	ROUTINE	−2.04

Somewhere between 15% and 20% mislabelled tickets, a total checkout outage stops being an incident and becomes a support question. The system produces no error, no warning, no degraded-mode banner. It answers with the same interface and the same confidence as before. A model has no way to tell you that it is now wrong, and that is the whole problem with putting one in a decision path.

Finding 3: leakage lies to you, and it lies by more than the model improves

Turn on "leak 12 test tickets into training" at 20% noise:

• Reported accuracy: 60.7% → 78.6% (+17.9 points)
• Honest accuracy, on the 16 rows leakage never touched: 56.3% → 62.5% (+6.2 points)

The reported number nearly tripled its improvement over the real one. Twelve of the 28 "held-out" tickets were sitting in the training set, so the score was partly measuring memorisation. This is the most common way an AI demo beats a production system: someone deduplicated after splitting instead of before, and the duplicates did the rest.

There is a fourth, quieter lesson in there. Drag "labelled tickets available" from 10 to 60 and you will see accuracy wobble — 92.9% at 30 rows, 85.7% at 60. That is not the model getting worse. With a 28-row evaluation set, every single ticket is worth 3.6 accuracy points, so the measurement itself is coarse. If your eval set is small, your accuracy number is mostly a rumour. Treat differences smaller than a few points as noise until you have the rows to say otherwise.

One honest caveat about the lab. This is a 60-row toy trained in your browser, not a production system, and 85.7% is a modest baseline. The magnitudes would differ with a real dataset and a deep model. The shapes — the non-linear cliff, the silent flip, the inflated metric — are the parts that generalise, and they are the parts that cost money.

The 2026 twist: LLMs made data quality invisible

Everything above assumed you could see a metric fall. With retrieval-augmented generation and agents, you usually cannot. This is the genuinely new risk of the current wave, and it is why "we're not training models, we just call an API" is not the exemption teams think it is.

A classifier with bad data gives you a number that goes down. A RAG system with bad data gives you a fluent, well-formatted, confidently wrong paragraph — assembled from the wrong document, or from a document that was correct eighteen months ago. Nothing in the pipeline is aware that anything went wrong. The Air Canada ruling above is exactly this failure mode with a legal bill attached.

Three data problems now hit systems that never train anything:

• Stale or duplicated source documents. If your knowledge base has three versions of the refund policy, retrieval will confidently pick one. Which one is essentially arbitrary. Retrieval quality is a data engineering problem before it is an embedding problem — see our guides on RAG vs fine-tuning and how vector databases actually work.
• Chunking that severs meaning. A table split across two chunks retrieves as two half-facts. The model will happily combine them.
• Too much retrieved context. More retrieved documents is not better. Irrelevant passages actively degrade answers, which is the counterintuitive core of context engineering — the discipline of deciding what not to put in front of a model.

The old rule was garbage in, garbage out. The 2026 rule is worse: garbage in, confident and well-formatted garbage out, at scale, with citations.

Model collapse: what happens when AI eats its own output

Train a generative model on its own output for enough generations and it degrades irreversibly. Shumailov and colleagues demonstrated this in Nature in 2024: indiscriminate training on model-generated content causes the tails of the original distribution to disappear — first as early collapse, where distributional errors accumulate and the model drifts, then as late collapse, where low-frequency events vanish permanently. The effect appears in LLMs, variational autoencoders and simple Gaussian mixture models alike. Their mitigation is blunt: fresh human data must be periodically injected. (Nature 631, 755–759)

Why this matters to you and not just to frontier labs: your internal corpus is being quietly filled with model output right now. AI-drafted tickets, AI-summarised meeting notes, AI-written documentation. If you fine-tune on last year's support tickets in two years' time, a meaningful share of those tickets will have been written by a model that was itself summarising a model.

The practical response is unglamorous and takes ten minutes: record provenance. One column that says whether a row was human-authored, AI-assisted or AI-generated is the cheapest insurance in modern data engineering, and it is nearly impossible to reconstruct after the fact.

Even the benchmarks are mislabelled

Here is the finding that reframes the whole field. In Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, Northcutt, Athalye and Mueller audited the test sets of ten of the most-used ML datasets and estimated an average of 3.4% label errors, including roughly 6% of the ImageNet validation set — errors confirmed by human review, not just flagged by an algorithm. (arXiv:2103.14749)

Be precise about the consequence, because this result is routinely overstated: the authors found that relative model rankings were largely unaffected by correcting the labels. What changed was which model you should actually deploy. On corrected ImageNet labels, the smaller ResNet-18 outperforms the larger ResNet-50 once the prevalence of originally mislabelled examples rises by just 6%.

The lesson is not "benchmarks are worthless". It is sharper than that: the higher-capacity model was partly learning the annotator's mistakes. If your test set is noisy, you will systematically select the model that best reproduces your noise.

How to make your data AI-ready: seven things that actually work

In order of return on effort, from a decade of shipping data platforms.

1. Write the label definition down, then test two humans against it. Before any modelling, take 50 examples and have two people label them independently. If they agree less than ~90% of the time, your accuracy ceiling is already set and no model will break it. Measuring inter-annotator agreement is the highest-leverage hour in the whole project and almost nobody spends it.
2. Deduplicate before you split, never after. This one line prevents the leakage the lab demonstrated. Split on a stable entity key — customer, document, session — not on rows.
3. Freeze an evaluation set no training job can reach. Store it separately, with its own access path. If a pipeline can read it, one day it will.
4. Put automated tests at all three gates — ingestion, transformation and serving — with blocking versus warning severity and a quarantine lane for bad rows. The mechanics, with real dbt and PySpark, are in our guide to testing data pipelines.
5. Put a contract on every producer feed. Schema, semantics, nullability and an owner, enforced in CI so an upstream change fails the producer's build rather than your model's Tuesday. See data contracts in practice.
6. Audit the labels of everything the model gets wrong. Not the model — the labels. On most real projects a meaningful share of "model errors" turn out to be annotation errors, and each one you find is worth more than a hyperparameter sweep.
7. Record provenance on every row. Source, timestamp, and whether a human or a model produced it. This is your only defence against the collapse problem above.

If your organisation is earlier than that — if the honest answer is "we don't really have pipelines yet" — start with what data engineering actually is and the modern data stack, then come back to this list.

Common mistakes that quietly cap your accuracy

Mistake	What it looks like	The fix
Chasing model architecture first	Three frameworks evaluated, zero labels reviewed	Spend the first week on 50 labelled examples
Splitting before deduplicating	Great offline score, disappointing production	Split on entity key, dedupe first
One annotator, no guidelines	Accuracy plateaus for no visible reason	Measure inter-annotator agreement
Averaging away a minority class	95% accuracy, useless on the cases that matter	Report per-class recall, not just accuracy
Trusting a small eval set	Every retrain "improves" or "regresses"	Compute how many points one row is worth
Treating RAG as exempt	"We don't train, so data quality is not our problem"	Retrieval quality is data quality
No provenance column	Cannot tell human data from model output	Add it now; it cannot be backfilled

Frequently Asked Questions

What is AI data quality?

AI data quality is the degree to which a dataset is accurate, complete, consistent, representative and correctly labelled for the specific decision a model must make. It differs from traditional data quality because a model learns errors instead of reporting them: a wrong value in a dashboard is visible, while a wrong label in a training set silently becomes part of what the model believes.

How much does bad labelling actually hurt accuracy?

Not proportionally, which is what makes it dangerous. In the lab on this page, mislabelling 10% of training tickets costs 3.6 accuracy points — easy to dismiss as noise. Twenty percent costs 25 points and flips a genuine production outage from urgent to routine. Label noise hides until it suddenly does not.

Why do most AI projects fail?

Because the data is not ready, not because the model is wrong. Gartner predicts organisations will abandon 60% of AI projects unsupported by AI-ready data through 2026, and found 63% of surveyed data management leaders either lacked the right practices for AI or did not know whether they had them. Model selection takes an afternoon; data readiness takes a quarter.

Does data quality still matter if I only call an LLM API?

It matters more, because the failure becomes invisible. A classifier with bad data shows a falling metric. A retrieval-augmented LLM with bad data returns a fluent, confident answer built on the wrong document, and nothing goes red in your monitoring. Air Canada was held liable in 2024 for a bereavement policy its chatbot invented.

What is data leakage?

Data leakage is when information from the evaluation set reaches the model during training, so the score measures memorisation rather than learning. In the lab above, copying 12 test tickets into training adds 17.9 points to reported accuracy while the honest score on never-seen rows gains only 6.2. The usual cause is deduplicating after splitting instead of before.

What is model collapse and should I worry about it?

Model collapse is the irreversible degradation that occurs when generative models are trained on their own output across generations — distributional drift first, then the permanent loss of rare events (Shumailov et al., Nature, 2024). You should worry about it indirectly: your internal corpus is filling with AI-generated text right now, so add a provenance column before you fine-tune on it later.

How much labelled data do I actually need?

Fewer rows than most teams assume, and cleaner ones than most teams have. In the lab, 30 well-labelled tickets outperform 60 with 20% noise — by a wide margin. Start with a few hundred carefully labelled examples and a defensible label definition, measure, and only then decide whether the bottleneck is volume or quality. Usually it is quality.

Conclusion: do the data work

The uncomfortable thing about the lab on this page is not that the model broke. It is how quietly it broke — full confidence, same interface, same clean output format, routing real outages to a queue nobody watches.

That is the shape of nearly every AI failure worth knowing about. Amazon's recruiter, Zillow's forecaster, Air Canada's chatbot, the Dutch benefits system: in each case the software worked exactly as built, on data that encoded something nobody had checked. Gartner's 60% and MIT's 95% are not measuring bad engineers. They are measuring how many teams did the model work before the data work.

So do the boring things first. Write down what your label means. Get two people to agree on 50 examples. Deduplicate before you split. Freeze an eval set nobody can touch. Add the provenance column. None of it will impress anyone in a demo, and all of it is what separates the pilots that ship from the 60%.

At SolutionGigs we build data platforms and AI systems for teams who have hit exactly this wall — the pilot that demoed beautifully and will not survive contact with production. If that is where you are, tell us what you are building and we will tell you honestly whether the problem is your model or your data. It is almost always your data.

Now go back and drag that slider to 20% one more time. Then ask who labels the data your company is about to bet on.