AI Prompts for Code Review Summaries When Everything Breaks Again

I’ve rebuilt my AI-powered code review flow four times now. Not even exaggerating. First it started when the LLM outputs got longer than some build logs, then the webhook fired twice and overwrote the summary, and once the VS Code extension completely froze. So yeah—this is what actually works (as of this week lol).

So if your AI tooling is summarizing too vaguely, skipping important diffs, or just refusing to inline comments like it’s gaslighting you… here’s what I fight with and how I fixed it 🙂

1. Prompt structure affects attention to line-level changes

When I first set this up, my base template was something like:

“Summarize this pull request and explain significant code changes in plain English.”

Yeah… that gave me:

>”The developer made numerous changes across multiple files. These updates improve functionality and performance.”

¯\_(ツ)_/¯ Thanks for the vibes.

Turns out if you don’t explicitly mention structure, the model treats every file equally, even if one just updates a README. I had to rephrase it to something closer to:

“For each file changed, list the filename, describe the purpose of the change, and note any downstream impact on other services. Also flag any hardcoded values or environment-specific logic.”

That long-winded structure worked better. Still, sometimes it would skip entire files if the diff was too noisy. There’s some kind of token compression going on behind the scenes—I can’t prove it, but the model consistently dropped mid-sized files that had only a few edits within big blocks of unchanged code. I wound up using git diff –unified=0 to minimize that surrounding noise.

2. When GitHub summaries skip context from large pull requests

I tested both GitHub’s built-in Copilot-generated PR summaries and my own automation using Claude from an internal dev tool. There’s a pattern: If the PR was over a certain number of lines modified (somewhere between 400–800), GitHub’s default summary missed the reasoning and just restated the filenames. Literally:

>”Updated HomeController and Dashboard models.”

Tell me why!!

So I started chunking the pull request into parts: high-level description from the PR body, then sorted commits by scope (e.g., middleware, frontend templates, etc.), then passed these buckets one at a time into my prompt pipeline. I used the title of the PR as a guiding anchor inside the prompt so the AI could keep it contextually focused:

“Assume the goal is to migrate dashboard cache logic to Redis. Within these changes, what specific issues are being addressed within the models folder?”

That shift let it actually follow intent.

Oh, and small detail—GitHub Copilot’s summary ignores deleted files completely. So if you remove some ancient validation helper, the summary acts like it never existed. Really messes with context for reviewers who only read the AI’s TLDR.

3. Managing hallucinations in multi-file change summaries

You ever ask the model to describe a file update and it just… invents a behavior? Like, I edited the logging utils to disable debug logs in prod, and the AI said:

>”This change introduces a more comprehensive tracing infrastructure.”

No it didn’t!  I checked the diff. It was a single if-statement toggle.

If you give the AI full diffs for 10 files, and just say “summarize” — it tries to connect the dots, sometimes creating clever but false narratives. A surprisingly simple fix was to prefix every chunk with line annotations (like patched line numbers and filenames) and then add flags like:

“Only describe observable code behavior changes. Do not infer implementation design unless explicitly modified.”

Weirdly, telling the AI NOT to be smart made it smarter. It started doing:

– “main.py: modified line 41 to include fallback timeout for external request”
– “handlers/logger.py: removed verbose trace output in production environment”

Way more accurate than the fantasy novel it was writing before 😛

4. Review reactions change based on prompt phrasing

This one surprised me. My first few AI summaries read like dev blog intros. Lots of “This update improves frontend responsiveness…” kind of language. And it made reviewers skeptical. I had two teammates ask if I was just fluffing up my own PRs 🤷

Once I switched to a more neutral structure:

– bullet points per file
– no adjectives
– passive tone (e.g., “route modified to include auth check”)

…people started actually trusting the summaries more. One coworker even commented: “Oh cool, this helped me skip the DB migration details.”

So yeah—boring is better here. The more objective the tone, the more the team considered it a productivity tool instead of a marketing gimmick.

Also: people ignore the AI writeup completely if it starts with “This innovative refactor…”

5. Using AI to flag potential test impact or missing coverage

I wasn’t initially using the AI to focus on tests. Big miss. One day I realized more than half my rollbacks were due to test gaps that should’ve been obvious. So I started running a post-summary check through the same prompt engine but asked:

“Given the code changes, is there any modified logic not covered in test files?”

And yeah, it caught stuff I missed. Especially things like adding optional parameters to helper methods—those don’t register as errors, but quietly slide through untested logic paths.

One snag: the AI sometimes assumed tests existed if the filenames were similar. So if it saw a change to `user_cleanup.py`, it would say “Test coverage exists in `test_user.py`” even if that file had no relevant cases. I added this fragment to the prompt:

“Review test file contents for actual assertions that invoke the modified functions.”

Fixed the false positives. It may still be brittle if test folders aren’t well-structured, but for monorepo-style setups with a clear naming scheme, it saves me from over-trusting my own memory. Honestly, it’s like a test-sniffing intern that doesn’t ask for snacks 🙂

6. Selectively summarizing diff types for maintainers

A senior backend dev on my team told me the AI summaries were too chatty. He just wanted to know what changed in API behavior, not that someone added comments to a utility file or cleaned up tests.

So now I use this setup:

1. Analyze the diff per file
2. Classify as: logic-impacting, cosmetic, test-related, or dependency
3. Only generate a final summary if classified as logic-impacting

I do that classification step using a tiny local GPT model (via ollama) before calling the cloud model for final phrasing. It’s fast. Plus, I don’t waste prompt tokens on “removed extra newline” diffs 🙂

Eventually I want to make it toggleable per reviewer. Some people want full traceability. Others, just the high-level deltas. Right now, it’s one-size-fits-no-one unless you pre-filter by that type.

7. Environmental inconsistencies affect AI responses mid-pipeline

Last week, I noticed the same prompts were generating way different summaries depending on the local dev environment. No obvious differences in the diffs. Same git hashes. Same tooling version. But still… wildly inconsistent outputs.

After ~an hour of diffing logs and setting debug flags, I figured it out: the Node version behind my bun.sh dev server was outputting newlines differently when generating staging PR diffs. Literally, the newline characters changed—`\r\n` vs `\n`. That changed token behavior in the AI prompt, which then altered how the shots were constructed (especially in OpenAI models).

It’s a level of fragility none of us anticipated. I now normalize diff inputs with a line ending sanitizer before prompting. I also log token counts per file chunk, because if a 100-line file suddenly consumes 4000 tokens, something broke.

This sounds excessive, I know—but AI prompt pipelines are just as brittle as any CI system. You don’t notice the edge cases until they write a confusing summary that makes someone ship the wrong change to prod.

8. Preventing prompt drift as PR complexity scales

One last ongoing issue: the longer the prompt, the blurrier the model output gets. At under 200 lines changed, my summaries are great. Around the 600-line mark, though, the AI starts compressing or skipping sequences entirely.

I learned this the hard way when I got a “Looks good to me” on a PR that quietly changed our token expiration from 15 minutes to 4 hours. It was in the summary field… but got glossed over.

Now I have a hard-coded token break at around 3200 total tokens (based on average context size for my target model). Anything past that, I paginate and color-code keylines in the summary.

Also worth noting: the model degrades predictably. First it skips small deletions, then folds sections together, then drops anything nested. Once you learn its collapse order, you can slice your prompts strategically.

So yeah. The summary worked great—until it didn’t. Then I had to rebuild from scratch, again. 🙂