Capstone: Prompt a Real Task End-to-End

Before you build, take one minute to name what you now have.

From “How LLMs Work” — you know the model is a next-token prediction engine. It sees tokens (not words, not characters), maintains state only inside the context window of a single call, and hallucination is a structural property of the mechanism. That’s why every decision in a prompt is really a decision about the probability distribution over the next token.

From “Prompt Engineering: The Core Moves” — you have four concrete tools: role prompting (reshaping the prior), few-shot examples (locking format structurally), chain-of-thought (using intermediate tokens as scaffolding), and output format constraints (suppressing the acknowledgment prior and enforcing parseable shape).

From “Temperature, Top-p, and the Knobs” — you can distinguish the job of each parameter. Temperature controls how peaked or flat the distribution is. Top-p cuts the long tail. Stop sequences enforce structural fences. For structured output (JSON, code) the right starting point is T=0, top-k=1, and a stop sequence on the closing delimiter.

From “Evaluating and Iterating” — you have a diagnostic before the rewrite: is this a prompt failure (missing signal) or a model limitation (capability gap)? And you have the one-at-a-time iteration discipline that builds transferable knowledge instead of one-off fixes.

From “Calling LLMs from Code” — you know the messages array is your context window in code form. The API is stateless. Always check stop_reason. Streaming matters for UX. Three failure modes hit almost every programmer on day one: max_tokens truncation, HTTP 429, and context overflow.

That is the full toolkit. The capstone gives you one real task to use all of it on.

Your task: Build a commit-message generator.

A user pastes a git diff into your program. Your program calls an LLM and gets back a well-formed, conventional commit message — subject line + body — that a professional engineer would be proud to ship.

Acceptance criteria:

1. The subject line follows the Conventional Commits format: <type>(<scope>): <summary> where type is one of feat, fix, refactor, test, docs, chore, perf.
2. Subject line is ≤ 72 characters.
3. Body (if present) explains why, not what — the diff already shows what changed.
4. The response is pure text — no preamble, no “Certainly! Here is your commit message:”.
5. The function handles at least three edge cases: an empty diff, a diff that is obviously test-only, and a diff that touches multiple unrelated areas (multi-scope).
6. The code calls the API, handles stop_reason: "max_tokens" and HTTP 429 (with retry), and is runnable with a real API key.

Stretch goals (attempt after the base version works):

• Accept the diff on stdin so it can be used as a Git hook.
• Add a few-shot example for a fix type and a separate one for refactor.
• Write a 3-case regression harness and run it after every prompt change.

This is not a toy task. Real git diff output is messy — renamed files, binary-file markers, large deletions. Your prompt needs to be robust enough that a colleague could adopt it.

Start from this skeleton. It is minimal but real: it connects to the Anthropic API, passes a diff, and returns text. Your job is to fill in the system prompt and tune the parameters.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic(); // reads ANTHROPIC_API_KEY from env

// ── PART 1: Write your system prompt here ──────────────────────────────────
// Hint: Start with a role. Add output format constraints. Add a few-shot
// example only after you've confirmed the role+format works on its own.
const SYSTEM_PROMPT = `
YOU WRITE THE SYSTEM PROMPT HERE.
`;

// ── PART 2: Set your parameters ────────────────────────────────────────────
// Temperature? Top-p? Stop sequences? Max tokens?
// Start with what the course taught you for structured output.
const PARAMS = {
  temperature: 0,   // start here; justify any change in your comments
  max_tokens: 256,  // commit messages are short — justify if you change this
  stop_sequences: [] as string[],
};

// ── PART 3: The function ────────────────────────────────────────────────────
export async function generateCommitMessage(
  diff: string,
  maxAttempts = 4
): Promise<string> {
  if (!diff.trim()) {
    return "chore: empty changeset";
  }

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      const response = await client.messages.create({
        model: "claude-opus-4-5",
        system: SYSTEM_PROMPT,
        max_tokens: PARAMS.max_tokens,
        temperature: PARAMS.temperature,
        stop_sequences: PARAMS.stop_sequences,
        messages: [{ role: "user", content: `Diff:\n${diff}` }],
      });

      // ── Guard: truncated output means malformed commit message ──────────
      // TODO: fill in the guard from the API lesson here

      return response.content[0].type === "text"
        ? response.content[0].text.trim()
        : "";
    } catch (err: unknown) {
      const status = (err as { status?: number }).status;
      if (status === 429 && attempt < maxAttempts - 1) {
        // TODO: add exponential backoff here (from the API lesson)
        continue;
      }
      throw err;
    }
  }

  throw new Error("Max retry attempts reached");
}

// ── PART 4: Quick smoke test ────────────────────────────────────────────────
const SAMPLE_DIFF = `
diff --git a/src/auth.ts b/src/auth.ts
index 3e4f1a2..8b2c7d3 100644
--- a/src/auth.ts
+++ b/src/auth.ts
@@ -12,6 +12,10 @@ export async function login(email: string, password: string) {
+  // Validate email format before hitting the database
+  if (!email.includes("@")) {
+    throw new Error("Invalid email format");
+  }
   const user = await db.findByEmail(email);
`;

generateCommitMessage(SAMPLE_DIFF)
  .then((msg) => console.log("Generated:\n" + msg))
  .catch(console.error);

The scaffold gives you the structure. It does not give you the system prompt or the parameter choices — those are the lesson. Work through Section 4 (the walk-through) before filling them in.

Here is one path from the starter scaffold to a prompt that meets all six acceptance criteria. Each step is annotated with why the change was made — the diagnosis that motivated it.

Step 0: First attempt (no system prompt, no examples)

Leave SYSTEM_PROMPT empty (or put only "You are a helpful assistant.") and run the scaffold on the sample diff. The model will almost certainly produce a prose paragraph, or a subject line without Conventional Commits format, or start with “Certainly!”.

Note the exact output. This is your baseline. Do not change anything yet.

Step 1: Add the role (one change)

You are a senior software engineer writing commit messages for a professional engineering team.

Run again. The preamble may disappear; the format may improve slightly. Note what changed and what didn’t. Do not add the format constraint yet.

Step 2: Add the output format constraint (one change)

You are a senior software engineer writing commit messages for a professional engineering team.

Always respond with only the commit message. No preamble, no explanation, no quotation marks.

Format:
<type>(<scope>): <subject>

<body>

Where:
- type is one of: feat, fix, refactor, test, docs, chore, perf
- scope is the affected module or file (optional but encouraged)
- subject is imperative mood, ≤ 72 characters total for the first line
- body explains WHY the change was made, not WHAT changed (the diff shows what)
- body is optional for trivial changes

Run again. Check: is the preamble gone? Does the subject line parse as type(scope): summary? Is the total subject length ≤ 72 chars?

If the format is still wrong on the sample diff, check whether the model is following the format description or producing something adjacent. If it’s producing adjacent-but-not-right output, this is still a prompt failure — add a few-shot example (Step 3). If it’s producing something that doesn’t even approximate the format, re-read the format instruction for ambiguity.

Step 3: Add a few-shot example (one change)

The format constraint is verbal. Make it structural by adding one example:

You are a senior software engineer writing commit messages for a professional engineering team.

Always respond with only the commit message. No preamble, no explanation, no quotation marks.

Format:
<type>(<scope>): <subject>

<body>

Where:
- type is one of: feat, fix, refactor, test, docs, chore, perf
- scope is the affected module or file (optional but encouraged)
- subject is imperative mood, ≤ 72 characters total for the first line
- body explains WHY the change was made, not WHAT changed (the diff shows what)
- body is optional for trivial changes

Example:
Diff: `+  const MAX_RETRIES = 3;` in `src/fetch.ts`
Output:
fix(fetch): cap retry attempts to prevent infinite loop

Unbounded retries caused production incidents when upstream APIs were
degraded. Cap at 3 attempts with exponential backoff.

Run again. The format should now be locked in structurally — the few-shot example taught the exact shape. ^{[Language Models are Few-Shot Learners]}

Step 4: Test the three edge cases

Now break it deliberately:

1. Empty diff — the scaffold already handles this (if (!diff.trim())). Good. But what about a diff that is just whitespace changes? Try: diff: - const x = 1;\n+ const x = 1; (only whitespace). Does the model produce something sensible?

2. Test-only diff — pass a diff that only touches *.test.ts files. Does the type come back as test? Or does the model use chore or feat?

3. Multi-scope diff — pass a diff touching both src/auth.ts and src/payments.ts. What does the model put in the scope? * is acceptable; auth, payments is also reasonable. auth alone would be wrong.

For each failure: diagnose first (prompt failure or model limitation?), then change exactly one thing, then re-run all three edge cases plus the original sample.

Step 5: Verify and tune parameters

For this task the right starting parameters are:

• Temperature: 0 — commit messages are not creative; you want the highest-probability conventional form.
• Max tokens: 256 — a well-formed commit message with a full body is around 80–120 tokens. Budget 256 to be safe; check that stop_reason is end_turn on your test cases, not max_tokens.
• Stop sequences: ["\n\n\n"] — three consecutive newlines after the body signals the model is generating padding. Add this as a safety fence.

If stop_reason comes back as max_tokens, something went wrong — either the diff is extremely large (truncate it upstream at ~2 000 tokens) or your format instruction is accidentally asking for verbose output.

Step 6: Fill in the two TODOs from the scaffold

// Guard 1: truncated output
if (response.stop_reason === "max_tokens") {
  throw new Error(
    `Commit message truncated after ${response.usage.output_tokens} tokens. ` +
    `The diff may be too large — truncate input to 2000 tokens or fewer.`
  );
}

// Guard 2 (inside the catch block): exponential backoff for 429
if (status === 429 && attempt < maxAttempts - 1) {
  const delay = Math.pow(2, attempt) * 1000 + Math.random() * 500;
  await new Promise((r) => setTimeout(r, delay));
  continue;
}

At this point, your artifact satisfies all six acceptance criteria. If it doesn’t, run the diagnostic:

• Is the subject line still over 72 chars? → Prompt failure: add “subject line must be ≤ 72 characters, truncate if needed.”
• Is the type wrong on test-only diffs? → Prompt failure: add an explicit rule.
• Does the body explain what instead of why? → Prompt failure: strengthen the “explains WHY” instruction, or add a negative few-shot example.

Before calling this done, run through this checklist. Be honest — the items you skip are the ones that will fail in production.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

// ── System prompt ─────────────────────────────────────────────────────────
// Role: sets the probability prior toward "professional engineering" register.
// Format constraint: suppresses the RLHF acknowledgment prior (no "Certainly!").
// Body rule: distinguishes *why* from *what* — the most common failure mode.
// Scope rule: handles multi-module diffs, which the format spec doesn't cover.
// Example: locks the format structurally so description alone isn't needed.
const SYSTEM_PROMPT = `
You are a senior software engineer writing commit messages for a professional engineering team.

Always respond with only the commit message. No preamble, no explanation, no quotation marks.
Your response must start directly with the commit type.

Format:
<type>(<scope>): <subject>

<body>

Rules:
- type is one of: feat, fix, refactor, test, docs, chore, perf
- scope is the affected module or file (omit parentheses if not helpful)
- subject is imperative mood, ≤ 72 characters for the entire first line (including type and scope)
- body explains WHY the change was made, not WHAT changed — the diff already shows what
- body is optional for trivial changes
- if the diff spans multiple modules, use * as the scope
- if the diff touches only test files, use the test type

Example:
Diff: added MAX_RETRIES constant to src/fetch.ts
Output:
fix(fetch): cap retry attempts to prevent infinite loop

Unbounded retries caused production incidents when upstream APIs were degraded.
Cap at 3 with exponential backoff.
`.trim();

// ── Parameters ────────────────────────────────────────────────────────────
// temperature: 0 — commit messages require conventional form, not creativity.
//   Using greedy decoding for maximum structural consistency.
// max_tokens: 256 — a full commit message with body is ~80–120 tokens;
//   256 is the 99th-percentile budget, not a target.
// stop_sequences: triple newline — signals the model has finished the body
//   and is generating padding. Structural fence from the parameters lesson.
const PARAMS = {
  temperature: 0,
  max_tokens: 256,
  stop_sequences: ["\n\n\n"],
};

export async function generateCommitMessage(
  diff: string,
  maxAttempts = 4
): Promise<string> {
  if (!diff.trim()) {
    return "chore: empty changeset";
  }

  // Truncate very large diffs to stay well within the context window.
  // At ~4 chars/token, 8000 chars ≈ 2000 tokens. The system prompt and
  // response budget consume ~400 tokens, so this leaves ample headroom.
  const truncatedDiff = diff.length > 8000 ? diff.slice(0, 8000) + "\n…[truncated]" : diff;

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      const response = await client.messages.create({
        model: "claude-opus-4-5",
        system: SYSTEM_PROMPT,
        max_tokens: PARAMS.max_tokens,
        temperature: PARAMS.temperature,
        stop_sequences: PARAMS.stop_sequences,
        messages: [{ role: "user", content: `Diff:\n${truncatedDiff}` }],
      });

      // Guard: max_tokens means we hit the ceiling before finishing.
      // For a commit message this is almost always a diff-too-large problem.
      if (response.stop_reason === "max_tokens") {
        throw new Error(
          `Commit message truncated after ${response.usage.output_tokens} tokens. ` +
          `Reduce diff size or increase max_tokens.`
        );
      }

      const text = response.content[0].type === "text"
        ? response.content[0].text.trim()
        : "";

      return text;
    } catch (err: unknown) {
      const status = (err as { status?: number }).status;
      // 429 = rate limit — back off exponentially, do not hammer the endpoint.
      if (status === 429 && attempt < maxAttempts - 1) {
        const delay = Math.pow(2, attempt) * 1000 + Math.random() * 500;
        await new Promise((r) => setTimeout(r, delay));
        continue;
      }
      throw err;
    }
  }

  throw new Error("Max retry attempts reached");
}

// ── Smoke test ────────────────────────────────────────────────────────────
const SAMPLE_DIFF = `
diff --git a/src/auth.ts b/src/auth.ts
--- a/src/auth.ts
+++ b/src/auth.ts
@@ -12,6 +12,10 @@ export async function login(email: string, password: string) {
+  if (!email.includes("@")) {
+    throw new Error("Invalid email format");
+  }
   const user = await db.findByEmail(email);
`;

generateCommitMessage(SAMPLE_DIFF)
  .then((msg) => console.log("Generated:\n" + msg))
  .catch(console.error);

The base version works. If you want to push further, here are three directions ordered by difficulty.

Stretch 1: Git hook integration

Add a small shell wrapper that reads git diff --staged on stdin and calls your function. Drop it in .git/hooks/prepare-commit-msg and your commit message is pre-filled every time you run git commit. This turns the function from a demo into a daily tool.

Stretch 2: Richer few-shot examples

Your system prompt currently has one example (a fix). Add a second example for a refactor — a case where no behavior changes but the code is reorganized. Then add a third for a docs change. Run your three edge-case test inputs against each version.

The pedagogical test: does adding more examples make the model more reliable on the original sample diff, or does it introduce new inconsistency? This is the diminishing-returns question the parameters lesson raised — few-shot examples have ~4–8 examples as the practical ceiling before context cost outweighs quality gain. ^{[Language Models are Few-Shot Learners]}

Stretch 3: Regression harness

Build the minimal harness from the evaluation lesson:

type TestCase = {
  name: string;
  diff: string;
  expectType: string;    // the first word before the colon in the output
  maxSubjectLen: number;
};

const cases: TestCase[] = [
  {
    name: "feature add",
    diff: `+export function validateEmail(email: string): boolean { ... }`,
    expectType: "feat",
    maxSubjectLen: 72,
  },
  {
    name: "test-only change",
    diff: `+it('should reject invalid emails', () => { ... }) // in auth.test.ts`,
    expectType: "test",
    maxSubjectLen: 72,
  },
  {
    name: "multi-scope change",
    diff: `+// changes in src/auth.ts AND src/payments.ts`,
    expectType: "refactor",
    maxSubjectLen: 72,
  },
];

async function runSuite() {
  let pass = 0;
  for (const tc of cases) {
    const msg = await generateCommitMessage(tc.diff);
    const firstLine = msg.split("\n")[0];
    const actualType = firstLine.split(/[(: ]/)[0];
    const lenOk = firstLine.length <= tc.maxSubjectLen;
    const typeOk = actualType === tc.expectType;
    const ok = lenOk && typeOk;
    console.log(`${ok ? "✓" : "✗"} [${tc.name}] type=${actualType} len=${firstLine.length}`);
    if (ok) pass++;
  }
  console.log(`\n${pass}/${cases.length} passing`);
}

runSuite().catch(console.error);

Run this harness after every prompt change. If fixing the multi-scope case breaks the test-only case, you need to diagnose why — it’s almost always a scope instruction that was too prescriptive.