Engineering

7 min read

Why your prompts need a CI pipeline

Treat every prompt change like a code change: reviewed, tested and reversible. Here is the pipeline we run on 40,000 prompt edits a month.

Mara Kessler

Every team we talk to has the same story: a prompt tweak that looked harmless in the playground quietly broke a feature in production. The fix is not more careful humans. It is a pipeline.

The problem with doing it by hand

Manual review does not scale past a handful of prompts. Changes pile up, context gets lost in chat threads and nobody can say which version is actually running in production. Worse, the feedback loop is measured in days, not seconds.

What we do instead

We treat prompts as code. Each change is a versioned commit with a diff, an owner and an automatic evaluation run. Nothing merges unless it holds up against real traffic.

Three rules we never break

  • Every prompt change runs against a golden dataset before merge.

  • Every production call is traced, with cost and latency attached.

  • Every deploy can be rolled back in one click, without a code release.

The best prompt is the one you can prove is better than the last one.

If you want to try this workflow on your own prompts, the free plan includes everything in this post. Start at zero, ship today.

Next post →

Routing across 40 models without losing your mind

Create a free website with Framer, the website builder loved by startups, designers and agencies.