When AI Gets It Wrong

Why This Matters

Most AI tutorials show you the happy path. The prompt goes in, the perfect code comes out. That's marketing, not reality. In practice, AI-generated code fails in predictable, categorizable ways — and developers are uniquely vulnerable because the code looks plausible enough to pass a quick scan.

This field guide documents nine failure categories based on real-world AI coding sessions. Each one includes a concrete example of what goes wrong, why it goes wrong, and how to catch it. Print this out. Keep it next to your monitor. It will save you.

1. The Confidence Problem

AI uses the same tone, the same certainty, the same formatting whether it's giving you a correct answer or a completely fabricated one. There is no "I'm not sure about this" signal in the output. The font doesn't change. The indentation doesn't waver. The comments are equally helpful.

What It Looks Like

const clone = structuredClone(original);

This answer is correct. Now watch what happens with a slightly different question:

const clone = structuredClone(original);

This answer is wrong. structuredClone() throws a DataCloneError on functions and DOM nodes. But the tone, the confidence, the formatting — identical. There's no signal that the second answer is incorrect.

2. Hallucinated APIs

AI invents functions, methods, parameters, and configuration options that do not exist. These aren't typos — they're plausible-looking API calls that were synthesized from patterns in the training data but correspond to nothing in the actual library.

What It Looks Like

// "Use Express's built-in rate limiter"
import { rateLimit } from 'express';

app.use(rateLimit({
  windowMs: 15 * 60 * 1000,
  max: 100,
  standardHeaders: true,
}));

The problem: Express has no built-in rateLimit export. The AI conflated Express with express-rate-limit, a separate npm package. The import compiles without type errors in some setups because TypeScript may not catch named import mismatches from untyped packages until runtime.

// express-rate-limit is a separate package
import rateLimit from 'express-rate-limit';

app.use(rateLimit({
  windowMs: 15 * 60 * 1000,
  max: 100,
  standardHeaders: true,
}));

Common Hallucination Patterns

• Merged libraries — Combines features from two different packages into one import
• Invented options — Adds configuration parameters that look right but aren't accepted
• Upgraded APIs — Uses methods from a newer (or imagined) version of the library
• Cross-framework bleed — Uses a React pattern in Vue, or a Django pattern in Flask

3. Stale Knowledge

AI's training data has a cutoff date, but the problem is worse than just "old information." AI confidently uses deprecated patterns, removed APIs, and abandoned conventions — because those patterns exist abundantly in the training data from years of Stack Overflow answers and blog posts.

What It Looks Like

class UserProfile extends React.Component {
  constructor(props) {
    super(props);
    this.state = { user: null, loading: true };
  }

  componentDidMount() {
    fetch(`/api/users/${this.props.userId}`)
      .then(res => res.json())
      .then(user => this.setState({ user, loading: false }));
  }

  render() {
    if (this.state.loading) return <Spinner />;
    return <div>{this.state.user.name}</div>;
  }
}

The problem: This is technically valid React, but it's a pattern from 2018. Class components, componentDidMount, this.setState — all of it has been superseded by hooks for years. If you paste this into a modern React codebase that uses functional components throughout, it's an immediate style clash.

AI generates class components because the training data is full of them. Years of tutorials, answers, and documentation used this pattern. The modern equivalent is shorter and cleaner, but there's less training data for it.

function UserProfile({ userId }: { userId: string }) {
  const [user, setUser] = useState<User | null>(null);
  const [loading, setLoading] = useState(true);

  useEffect(() => {
    fetch(`/api/users/${userId}`)
      .then(res => res.json())
      .then(user => { setUser(user); setLoading(false); });
  }, [userId]);

  if (loading) return <Spinner />;
  return <div>{user?.name}</div>;
}

High-Risk Stale Knowledge Areas

• React — Class components, lifecycle methods, createContext patterns from before hooks
• Node.js — CommonJS require() instead of ES modules, callback patterns instead of async/await
• TypeScript — Older type assertion syntax (<Type>value vs value as Type), missing modern utility types
• CSS — Float-based layouts, vendor prefixes that are no longer needed, pre-Grid patterns
• Python — format() strings instead of f-strings, os.path instead of pathlib

4. Subtle Logic Errors

The most dangerous failure mode. The code runs without errors, passes a quick visual review, and produces correct results for common inputs — but fails on edge cases, boundary conditions, or specific data shapes. These bugs survive code review because the logic looks right.

What It Looks Like

function paginate<T>(items: T[], page: number, pageSize: number): T[] {
  const start = (page - 1) * pageSize;
  const end = start + pageSize;
  return items.slice(start, end);
}

Looks perfect. Works for page 1, page 2, page 3. Ship it, right? But what happens with these inputs?

paginate(items, 0, 10);    // page 0 → start = -10, returns wrong slice
paginate(items, -1, 10);   // negative page → returns items from the end
paginate(items, 1, 0);     // pageSize 0 → returns empty (silent failure)
paginate(items, 1, -5);    // negative pageSize → returns empty (silent failure)

The fix: Input validation that AI didn't add because the happy path works fine.

function paginate<T>(items: T[], page: number, pageSize: number): T[] {
  if (page < 1) throw new RangeError(`page must be = 1, got ${page}`);
  if (pageSize < 1) throw new RangeError(`pageSize must be = 1, got ${pageSize}`);

  const start = (page - 1) * pageSize;
  const end = start + pageSize;
  return items.slice(start, end);
}

Where Subtle Logic Errors Hide

• Off-by-one errors — Pagination, array indexing, date ranges, loop boundaries
• Missing null/undefined checks — AI assumes data is always present
• Incorrect comparisons — Using == instead of ===, comparing strings to numbers
• Race conditions — Async code that works 99% of the time but fails under load
• Timezone bugs — Date handling that works in UTC but breaks in local time

5. Security Blind Spots

AI-generated code has consistent, predictable security weaknesses. Not because AI doesn't "know" about security — it can explain OWASP Top 10 perfectly — but because insecure code is far more common in training data than secure code. The most common patterns in codebases are the least secure ones.

What It Looks Like

// "Build an endpoint that looks up users by email"
app.get('/api/users', (req, res) => {
  const { email } = req.query;
  const user = db.prepare(`SELECT * FROM users WHERE email = '${email}'`).get();
  res.json(user);
});

Three security vulnerabilities in five lines:

• SQL injection — String interpolation in the query. An attacker sends email=' OR 1=1-- and dumps the entire table.
• Data exposure — SELECT * returns every column, including password_hash. The API response leaks sensitive fields.
• Missing input validation — No check that email is actually provided or is a valid format.

app.get('/api/users', (req, res) => {
  const { email } = req.query;

  if (!email || typeof email !== 'string') {
    return res.status(400).json({ error: 'Email is required' });
  }

  const user = db
    .prepare('SELECT id, email, name, created_at FROM users WHERE email = ?')
    .get(email);

  if (!user) return res.status(404).json({ error: 'User not found' });

  res.json(user);
});

AI's Consistent Security Failures

• SQL injection — String interpolation instead of parameterized queries. The single most common AI security bug.
• Hardcoded secrets — const JWT_SECRET = 'supersecret123' right in the source code
• Returning full database rows — SELECT * that includes password hashes, tokens, and internal IDs
• Missing CORS configuration — Either no CORS at all or origin: '*'
• Predictable tokens — Using Math.random() for session tokens instead of crypto.randomUUID()
• Missing rate limiting — Auth endpoints without any throttling

6. The Copy-Paste Trap

AI generates code by pattern matching against its training data. When it encounters a problem it has seen before, it reproduces the most common solution — even when your context requires a different approach.

What It Looks Like

import { z } from 'zod';
import { useForm } from 'react-hook-form';
import { zodResolver } from '@hookform/resolvers/zod';

const schema = z.object({
  email: z.string().email(),
  phone: z.string().min(10).max(15),
  message: z.string().min(1).max(1000),
});

// ... 40 more lines of boilerplate

The problem: You asked for validation on a simple three-field form. AI pulled out a full Zod + React Hook Form + zodResolver setup — the most common validation stack in its training data. But you didn't ask for these dependencies. Your project might not use them. And the phone field isn't optional in the schema despite being optional in your spec.

A proportional solution for three fields:

function validate(data: FormData) {
  const errors: Record<string, string> = {};

  if (!data.email || !/^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(data.email)) {
    errors.email = 'Valid email required';
  }

  if (data.phone && !/^\+?[\d\s-]{7,15}$/.test(data.phone)) {
    errors.phone = 'Invalid phone format';
  }

  if (!data.message?.trim()) {
    errors.message = 'Message is required';
  }

  return errors;
}

Zero dependencies. Twenty lines. Does the same job for this specific form. The AI solution isn't wrong — it's disproportionate. It's solving the general problem instead of your specific problem.

7. Architecture Astronautics

Ask AI to build a feature, and it will often build a framework for that feature. Ask for a logger, and you get a plugin-based logging system with transports. Ask for a config loader, and you get a multi-source hierarchical configuration engine with validation, caching, and hot-reload.

What It Looks Like

interface RetryConfig {
  maxRetries: number;
  baseDelay: number;
  maxDelay: number;
  backoffFactor: number;
  retryableStatuses: number[];
  onRetry?: (attempt: number, error: Error) => void;
  shouldRetry?: (error: Error) => boolean;
  timeout?: number;
  jitter?: boolean;
}

class RetryClient {
  private config: Required<RetryConfig>;

  constructor(config: Partial<RetryConfig> = {}) {
    this.config = {
      maxRetries: config.maxRetries ?? 3,
      baseDelay: config.baseDelay ?? 1000,
      maxDelay: config.maxDelay ?? 30000,
      backoffFactor: config.backoffFactor ?? 2,
      retryableStatuses: config.retryableStatuses ?? [408, 429, 500, 502, 503, 504],
      onRetry: config.onRetry ?? (() => {}),
      shouldRetry: config.shouldRetry ?? (() => true),
      timeout: config.timeout ?? 30000,
      jitter: config.jitter ?? true,
    };
  }

  // ... 80 more lines
}

You asked for a function. AI built a configurable retry client with nine options, a class hierarchy, and a callback system. Here's what you actually needed:

async function withRetry<T>(fn: () => Promise<T>, retries = 3): Promise<T> {
  for (let i = 0; i < retries; i++) {
    try {
      return await fn();
    } catch (err) {
      if (i === retries - 1) throw err;
      await new Promise(r => setTimeout(r, 1000 * Math.pow(2, i)));
    }
  }
  throw new Error('Unreachable');
}

Eight lines. Does exactly what was asked. No configuration object, no class, no callback system.

Why AI Over-Engineers

AI's training data is dominated by library code, framework code, and "production-ready" examples that handle every conceivable edge case. It optimizes for generality because the most upvoted Stack Overflow answers and the most popular GitHub repos are the most general ones. Your specific, constrained problem gets a general, unconstrained solution.

8. The Yes-Man Problem

AI is trained to be helpful. Helpful means giving you what you asked for. The problem is that sometimes what you asked for is wrong — and AI will build it anyway, with enthusiasm.

What It Looks Like

// Store session on login
localStorage.setItem('session', JSON.stringify({
  userId: response.userId,
  role: response.role,
  isAdmin: response.isAdmin,
  token: response.token,
}));

// Check on every API request
function isAuthenticated() {
  const session = localStorage.getItem('session');
  return session !== null;
}

This is deeply flawed. The "authentication" check is whether a key exists in localStorage — something any user can set manually via browser DevTools. The isAdmin flag is client-side and trivially fakeable. The entire auth model is built on trust that the client won't lie.

But AI said "Great approach!" and built it. Because AI defaults to agreeing with you and implementing what you described, not questioning your design.

How to Force Pushback

You have to explicitly ask for it. AI won't volunteer criticism unless prompted:

Same AI. Same knowledge. Completely different answer — because you asked for critique instead of implementation.

9. Phantom Context

In long conversations, AI loses track of earlier decisions and starts contradicting itself. It will generate code that conflicts with code it wrote 20 messages ago. It will use a different naming convention. It will forget the database schema. This isn't a bug — it's a fundamental limitation of context windows.

What It Looks Like

... 40 messages later ...

const user = db.prepare(
  'SELECT * FROM users WHERE email = ?'
).get(email);

if (!user || !bcrypt.compareSync(password, user.password_hash)) {
  // ...
}

Spot the bug? The schema uses passwordHash (camelCase), but the login endpoint uses user.password_hash (snake_case). AI forgot its own naming convention from 40 messages ago. This produces a runtime error — user.password_hash is undefined, bcrypt comparison fails, and every login attempt is rejected.

How Context Degrades

• Messages 1—10 — AI remembers everything. Consistent names, types, patterns.
• Messages 10—30 — Starts paraphrasing. Key details may shift slightly.
• Messages 30+ — Earlier decisions are effectively forgotten. Contradictions likely.

How to Catch Everything

Nine failure modes is a lot to watch for. Here are the practical detection techniques that cover all of them systematically, ranked from fastest to most thorough.

The 30-Second Scan

Before accepting any AI-generated code, scan for these specific patterns:

• Imports you don't recognize → Hallucinated APIs (#2)
• String interpolation in queries → SQL injection (#5)
• Hardcoded strings that look like secrets → Security blind spot (#5)
• SELECT * in any query → Data exposure (#5)
• Class components in a hooks codebase → Stale knowledge (#3)
• New dependencies you didn't ask for → Copy-paste trap (#6)
• Configuration objects with 5+ options → Over-engineering (#7)

The Adversarial Review

After accepting code into your project, periodically ask AI to attack it:

The same AI that wrote the bugs can find the bugs when you change its role. This works because code generation and code review activate different patterns — the review prompt surfaces criticisms that the generation prompt suppressed.

The Edge Case Challenge

For any function with inputs, run through these mentally or in a prompt:

• What happens with zero?
• What happens with null or undefined?
• What happens with an empty string?
• What happens with a very large input?
• What happens with negative numbers?
• What happens concurrently?

The Test-First Backstop

The most reliable detection method: write or generate tests before (or immediately after) the implementation. AI-generated tests catch AI-generated bugs because the test-writing prompt considers edge cases the implementation prompt ignored.

That last instruction — "one test you think will fail" — is powerful. It forces AI to think about where the code is weakest. The test it writes often does fail, revealing a real bug.

The Checklist

Pin this to your wall. Run through it every time you accept AI-generated code: