We've taken over failed AI projects. The pattern in each case is the same: the project didn't fail because of a bad LLM or a bad team. It failed because of a specific, preventable mistake that was visible in retrospect and often visible before the project started — if anyone had been looking.
These are the seven patterns we see most often.
1. Solving the AI problem before the data problem
The most common failure mode. A team decides to build a RAG chatbot, spins up a vector database, connects to an LLM, and starts building. Three months later, the system gives inconsistent answers. The problem isn't the retrieval algorithm or the model — it's that the underlying data is a mess.
What it looks like: documentation scattered across three platforms with conflicting versions, Confluence pages that haven't been updated since 2021, PDFs with encoding issues, knowledge that exists only in employees' heads.
The warning sign: in the discovery phase, when you ask "can we see a sample of the data?", the answer is "we'll get that to you." If the data isn't immediately accessible and well-organized, you have a data problem before you have an AI project.
How to fix it: data audit before any AI build. Not a review of where data lives — an actual inspection of what's in it. Quality, structure, completeness, freshness. If the data isn't ready, the AI project isn't ready.
2. Building for the demo, not the edge cases
The demo works. The team shows it to stakeholders and it handles the prepared questions gracefully. Everyone is impressed. Then real users get access and ask questions the demo didn't anticipate, and the system either hallucinates, refuses to answer, or produces responses that are technically correct but practically useless.
What it looks like: an evaluation set of 15 curated examples that the team uses to show the system working. No systematic testing of edge cases, unusual queries, adversarial inputs, or the long tail of real user behavior.
The warning sign: when asked "how are you testing this?", the answer is "we run queries and see if they look right." Eyeball testing on a small sample is not an evaluation strategy for a production system.
How to fix it: build the evaluation harness before building the system. Define what "working" means with specific metrics and a diverse test set. The test set should include normal cases, edge cases, and intentionally adversarial inputs. See our LLM evaluation framework post for specifics.
3. No eval harness means no production confidence
Closely related to failure mode 2, but broader: the absence of a systematic way to measure quality means you can't detect when quality degrades. This is particularly dangerous because AI systems degrade for reasons that have nothing to do with code changes:
- The LLM provider updates their underlying model
- Your knowledge base becomes stale
- The distribution of user queries drifts from what you optimized for
- Prompt behavior changes subtly as context accumulates
Without an eval harness running on a schedule, you learn about quality degradation from user complaints — the worst possible feedback mechanism.
The warning sign: the team has no automated tests for AI behavior, no quality dashboards, and no defined process for reviewing system output post-launch.
How to fix it: treat eval as infrastructure, not a feature. A scheduled eval run that tests the system against your test set weekly costs almost nothing to run. Catching a quality regression early costs much less than a production incident.
4. Prompt engineering without versioning
Prompts change. Teams iterate, improve, try different approaches, and roll back when something doesn't work. Without versioning, this iteration produces a system where no one knows what the current prompt is, why specific language was added, what earlier versions said, or whether recent changes caused behavior to change.
What it looks like: prompts stored in environment variables or configuration files with no history. "We changed the prompt last week but I'm not sure exactly what we changed" is a common explanation when trying to debug unexpected behavior.
The warning sign: asking for the prompt version history and getting a shrug.
How to fix it: store prompts in version control. Treat prompt changes like code changes: commit with a message explaining why the change was made, run evals before and after, document what changed. Tools like LangSmith, Langfuse, or PromptLayer make this easier at scale, but a git repo with a prompts/ directory is sufficient to start.
5. Confusing a prototype with a production system
A prototype built to demonstrate feasibility has different requirements than a production system that handles real customer data, runs 24/7, and needs to recover gracefully from failures. Teams launch prototypes into production, often under time pressure.
What it looks like: no error handling in the agent loop (a tool failure crashes the whole conversation), no rate limiting on the API (a spike in traffic causes LLM API rate limit errors that propagate to users), no monitoring, no alerting, no retry logic, no graceful degradation.
The warning sign: the codebase has no tests, no error handling beyond happy paths, and was built to show a demo rather than handle a production failure.
How to fix it: before production launch, systematically inject failures: what happens when the LLM API returns a 429? When a database query times out? When the user sends an empty message? When the conversation history exceeds the context window? Each of these needs a tested, graceful response.
6. No ownership post-launch
An AI system launched with no one responsible for its ongoing health will degrade. This is not a risk — it is a certainty. Knowledge bases go stale. Model providers update their APIs. User behavior evolves in ways that weren't anticipated. The company's products change in ways that the AI's training data doesn't reflect.
What it looks like: the project team disperses after launch. There's no designated owner. The system runs unmonitored. Quality complaints surface months later when the degradation has compounded.
The warning sign: "we don't have anyone to maintain it after launch." This is a valid constraint, but it should inform the build decision. A system that no one will maintain is either very simple (easy to maintain) or expensive to fix when it breaks.
How to fix it: assign an owner before launch, not after the first incident. Define what ownership means: reviewing logs weekly, re-indexing knowledge when documents change, responding to quality alerts. If internal ownership isn't feasible, a maintenance retainer with the development team provides continuity.
7. Solving the wrong problem to begin with
The most expensive failure mode — the one that produces a technically successful project that doesn't move the business forward.
A company invests $40K in an AI agent to automate customer support inquiries. The agent works. It handles 60% of inquiries without human intervention. But the support team headcount stays the same, the 40% of escalated cases take longer to resolve because the agent interaction added friction, and customer satisfaction scores drop slightly.
The AI "worked" but the business problem wasn't solved because the actual problem wasn't the number of inquiries — it was the resolution quality and time.
The warning sign: the project is scoped around "implementing AI" rather than "solving problem X." The discovery phase focused on the AI solution (what model to use, what to connect it to) rather than the business problem (what customers actually need, where the workflow breaks down, what metric we're trying to move).
How to fix it: start with the business outcome, work backward to the technical solution. Define the metric you're trying to move before you scope the technology. Be willing to say "AI isn't the right solution here" if the discovery process leads there.
The common thread
These seven failure modes are not independent. They cluster: teams that skip discovery (failure mode 7) also skip data audits (failure mode 1) and build for demos (failure mode 2). Teams that rush to launch without evals (failure mode 3) also don't set up monitoring (failure mode 6).
The projects we've taken over after failure consistently had two or three of these patterns, not just one. Addressing any one of them in isolation helps; addressing them systematically is what makes AI projects work in production.