Write Prompts That Survive Model Updates

Your prompt broke on a model upgrade and none of your code changed. The usual cause is not model drift arriving like weather — it is the text you added months ago to compensate for the old model's weaknesses. When the weakness gets fixed, that text becomes an active bug. Split every prompt into contract text, which is model-independent, and compensation text, which has a shelf life. Then delete the compensation and test what is left.

Why prompts break when nothing in your code changed#

A prompt is not code. It is a request to a statistical system whose behavior changes underneath you, so the same input can produce a different answer next quarter with no diff on your side.

The failure is rarely dramatic. A 2023 study of five GPT-3.5-family models found that in 87.9% of updates where overall accuracy improved, at least one previously correct prediction still regressed — real damage sitting underneath a headline number that moved the right way. The same study found the update from gpt-3.5-turbo-0301 to gpt-3.5-turbo-0613 cut accuracy 9.6% for one prompt while raising it 5.1% for another on the same dataset.

Those numbers come from one toxicity-detection task across API versions spanning March 2022 to September 2023. Treat the mechanism as general and the percentages as specific to that setup.

The mechanism matters more than the numbers. If one update can help prompt A and hurt prompt B on the same task, a single spot-check after an upgrade tells you almost nothing.

Contract vs compensation: the audit that predicts which prompts break#

Open your production prompt. Highlight every sentence that exists only because a model once got something wrong. That highlighted text is your upgrade debt.

Anthropic says this outright in its prompting guide: prompts written to stop an older model undertriggering on tools or skills may now overtrigger, and "the fix is to dial back any aggressive language." Its worked example is turning "CRITICAL: You MUST use this tool when..." back into "Use this tool when...". Read that again — the vendor is telling you that your own prompt is the liability.

That last row is a third category worth naming. Mechanism text leans on a provider feature rather than on model behavior, and it breaks at feature boundaries rather than capability boundaries. Anthropic's documented replacement for prefill-based formatting is structured outputs, or for classification, a tool with an enum field holding your valid labels.

Make the output format a contract, not a preference#

Here is a real support-triage prompt written the way most of them start.

You are a helpful support triage assistant. Please read the customer
email carefully and think step by step. Be thorough — do not skip any
details. Categorize it sensibly and rate how urgent it is. Don't use
markdown. Return JSON.

Almost every line here is compensation or vibe. "Sensibly" is not a spec, "think step by step" is anti-laziness text, and "Return JSON" says nothing about which keys.

Now the contract version.

Classify the support email into exactly one category.

<categories>billing, bug, feature_request, account_access, other</categories>

Return a single JSON object with exactly these three keys:
  category  one of the five values above, lowercase, nothing else
  urgency   integer 1-4, where 4 means the customer cannot use the product
  evidence  a verbatim quote of 15 words or fewer from the email

If the email fits more than one category, choose the one the customer
asked you to act on. If it fits none, use "other".

The second version is shorter and says more. Anthropic recommends wrapping each content type in its own XML tag to reduce misinterpretation, and advises being specific about the desired output format and constraints — that is what the tag and the key list are doing.

Constrain instead of describe#

Descriptions leave room. Constraints do not.

• Describe: "rate how urgent it is." Constrain: "integer 1-4, where 4 means the customer cannot use the product."
• Describe: "keep it brief." Constrain: "a verbatim quote of 15 words or fewer."
• Describe: "categorize it sensibly." Constrain: an enum of five lowercase values.

The same rule applies to negatives. Anthropic's first recommendation for steering output format is "Tell Claude what to do instead of what not to do," with the worked example of replacing a ban on markdown with a request for flowing prose paragraphs. A ban tells the model what the space of wrong answers looks like; a positive spec tells it where to land.

Write the tie-break rule before the ambiguous case finds you#

Most silent failures live in inputs that genuinely fit two answers. An email that says "I was charged twice and the invoice will not download" is both billing and bug. Without a rule, the model picks one, and which one it picks is a model-version detail you have quietly built a business process on.

So write the rule down. "Choose the one the customer asked you to act on" is a product decision, it is stable across every model you will ever use, and it makes disagreement between two model versions a bug in your spec rather than a mystery.

Every enum also needs an explicit escape hatch. If nothing fits, the model must have a legal answer — other — or it will invent a category that your downstream switch statement has never heard of.

Validate the shape in code — schema adherence is not correctness#

Structured outputs are worth using. OpenAI describes the feature as ensuring the model always generates responses adhering to your supplied JSON Schema, and that removes a whole class of parsing bugs.

It does not remove validation. The same OpenAI guide states that "Structured Outputs can still contain mistakes," and suggests adjusting instructions or splitting the task when you see them. The schema constrains shape. Nothing constrains truth.

There are practical edges too. OpenAI's guide lists unsupported schema keywords including allOf, not, if, then and else; it requires every field to be marked required, with optional fields emulated as a union with null; and it requires additionalProperties: false on objects. A refusal arrives in a separate refusal field, so your parser has to handle a perfectly valid response that carries no schema-shaped payload at all.

This validator runs on Node 18 or newer with no dependencies. Save it as validate.mjs and run node validate.mjs.

// validate.mjs — run with: node validate.mjs
const CATEGORIES = ['billing', 'bug', 'feature_request', 'account_access', 'other']
const KEYS = ['category', 'urgency', 'evidence']

export function validateTriage(raw, email) {
  const problems = []

  let data
  try {
    data = JSON.parse(raw)
  } catch {
    return { ok: false, problems: ['response is not valid JSON'] }
  }
  if (data === null || typeof data !== 'object' || Array.isArray(data)) {
    return { ok: false, problems: ['top level is not a JSON object'] }
  }

  const got = Object.keys(data)
  for (const key of KEYS) if (!got.includes(key)) problems.push(`missing key: ${key}`)
  for (const key of got) if (!KEYS.includes(key)) problems.push(`unexpected key: ${key}`)

  if (!CATEGORIES.includes(data.category)) {
    problems.push(`category outside the allowed set: ${JSON.stringify(data.category)}`)
  }
  if (!Number.isInteger(data.urgency) || data.urgency < 1 || data.urgency > 4) {
    problems.push(`urgency is not an integer 1-4: ${JSON.stringify(data.urgency)}`)
  }

  // The check no schema can make: the quote has to actually be in the email.
  const quote = typeof data.evidence === 'string' ? data.evidence.trim() : ''
  if (!quote) problems.push('evidence is missing or empty')
  else if (!email.includes(quote)) problems.push('evidence is not a verbatim quote from the email')

  return { ok: problems.length === 0, problems }
}

/* --- regression cases: one per failure you have actually seen in production --- */

const EMAIL = 'I was charged twice for July and the invoice PDF will not download.'

const CASES = [
  { name: 'happy path', raw: '{"category":"billing","urgency":3,"evidence":"charged twice for July"}', expect: true },
  { name: 'invented category', raw: '{"category":"payments","urgency":3,"evidence":"charged twice for July"}', expect: false },
  { name: 'urgency as string', raw: '{"category":"billing","urgency":"high","evidence":"charged twice for July"}', expect: false },
  { name: 'extra commentary key', raw: '{"category":"billing","urgency":3,"evidence":"charged twice for July","note":"hi!"}', expect: false },
  { name: 'paraphrased evidence', raw: '{"category":"billing","urgency":3,"evidence":"the customer was billed two times"}', expect: false },
]

let failed = 0
for (const c of CASES) {
  const result = validateTriage(c.raw, EMAIL)
  const pass = result.ok === c.expect
  if (!pass) failed += 1
  console.log(`${pass ? 'PASS' : 'FAIL'}  ${c.name}${result.problems.length ? `  — ${result.problems.join('; ')}` : ''}`)
}
console.log(`\n${CASES.length - failed}/${CASES.length} regression cases behaved as expected`)
process.exitCode = failed > 0 ? 1 : 0

Running it prints five PASS lines. The interesting check is the last one: evidence is well-formed, on-schema, plausible English, and completely made up. No JSON Schema in the world catches that. A four-line substring test does.

Grounding checks like this one are the highest-value validation you can write, because they catch the failure mode that survives every structural guarantee. If the model must quote the source, you can verify the quote.

Keep prompts short enough to audit in one sitting#

Every sentence you add is a sentence someone has to re-evaluate on every upgrade. A 900-word prompt is not more reliable than a 200-word one — it is a 900-word audit you will not do.

Anthropic offers a good clarity test: show your prompt to a colleague who has minimal context on the task and ask them to follow it. If they would be confused, the model will be too. Add a second pass for upgrade debt: ask them which sentences would still make sense if the model were twice as capable.

Pin the model version, and know exactly what the pin buys you#

Use a dated model version in production, not a floating alias. Google states it plainly for Gemini: "Most production apps should use a specific stable model," and notes that its latest alias gets hot-swapped with every new release of a model variation, which may be stable, preview or experimental.

A pin buys you time, not immunity. How much time depends entirely on the provider, and this is where secondhand summaries mix vendors up.

Read those as minimums, not promises. OpenAI states that if safety or compliance concerns require retiring a model sooner, it will give as much notice as reasonably possible.

Two more things the vendor pages tell you that a summary will not. Anthropic defines four lifecycle states — Active, Legacy, Deprecated and Retired — warns that deprecated models are likely to be less reliable than active ones, and says flatly that "Requests to retired models will fail." And Anthropic's retirement dates apply to its own platforms; partner-operated platforms such as Amazon Bedrock and Google Cloud set their own schedules, so the same model can have two different end dates depending on where you call it.

To find out what you are actually exposed to, audit what your keys call rather than what your config says. Anthropic lets you export a usage CSV from the Usage page in the Claude Console for exactly this. Budgeting for the switch is a separate exercise — the guide to estimating LLM API costs before you build covers the arithmetic.

Build a regression set small enough to run on every change#

Thirty cases you run on every prompt edit will catch more than three hundred you run twice a year. The 2026 Commey paper used expanded 30-case suites and still surfaced a regression from 26 of 30 down to 9 of 30, which tells you a small suite is enough to see real damage.

1. Freeze twenty real inputs

   Pull them from production logs, not your imagination. Include the boring ones — a regression set of only hard cases will not notice when easy cases break.

2. Add every ambiguous case you have argued about

   Any input where two people on your team picked different labels belongs here, with the answer your tie-break rule produces. These are the cases that flip silently between model versions.

3. Add one case per bug you have already shipped

   This is the highest-signal source you have. Each past incident becomes a permanent test, which is the only way a regression set stays honest as it grows.

4. Assert on shape first, meaning second

   Run the validator above on every response. Shape failures are deterministic and cheap; only compare semantic labels once the structure passes, or you will spend your review time reading JSON parse errors.

5. Run the whole set against both model versions before switching

   Compare per-case, not just totals. A stable aggregate score can hide a batch of correct answers turning wrong and a different batch turning right.

6. Log the prompt version alongside every production call

   Without it you cannot tell whether last Tuesday looked different because of the model, the prompt, or the input mix. Applying the same discipline to the surrounding pipeline is covered in the guide to error handling in automated workflows.

Who this is not for#

If you are drafting emails in a chat window, ignore all of it. You see every output before it leaves your hands, so re-reading is the whole quality process and the overhead here buys you nothing.

Skip it too for one-off analysis where nothing is scheduled and nothing is stored. Contracts, tie-break rules and regression sets earn their keep when the output feeds another system without a human between — routing, extraction, classification, ranking. Below roughly a hundred automated calls a week, a pinned model plus the shape validator alone is a fair place to stop.

Your model-upgrade checklist#

Do these six things, in this order, the next time you move models.

1. Read your prompt with a highlighter and delete every line that only exists to patch the old model.
2. Rewrite the remaining vague lines as constraints — enums, integer ranges, word limits.
3. Add the tie-break rule and the other escape hatch, then check nothing else about your spec depends on undocumented behavior.
4. Run the shape validator on every call in production, including a grounding check that the quoted evidence really appears in the input.
5. Move from the alias to a dated version, and put the provider's deprecation page in your calendar.
6. Run your 30 cases against old and new side by side, and compare per-case.

Do the first step even if you do nothing else. Deleting compensation text is the only item on this list that costs nothing, takes twenty minutes, and removes bugs you have not hit yet. Judging the tools you point this discipline at is a related question — start with the checklist for evaluating AI tools before buying, and the rest of the tutorials section for the surrounding workflow.