Using AI to Draft Legal Docs Without Breaking Zapier Again

Using AI to Draft Legal Docs Without Breaking Zapier Again

ByUS Stacked

1. Why static prompt blocks fail for real legal workflows

The first time I tried to build a prompt-driven contract generator in Zapier, I dropped the entire clause logic into a single multi-line field in OpenAI’s action step. Worked perfectly during my test. Didn’t work at all when my paralegal triggered it on a real client. Turns out, conditional phrasing and invisible variables don’t play nice live — OpenAI silently drops missing data without warning. No error, just a blank paragraph.

Contracts aren’t blog posts — you can’t just drop a templated prompt and hope for the best. Legal documents rely on specific phrasing based on if–then logic, jurisdictional terms, and contextual phrasing that doesn’t generalize well. If one clause says “the Employee agrees,” you can’t let the next clause say “the party responsible” unless you want a confused client or worse, an invalid NDA.

Static blocks feel safe because they preview well. But in production, the cracks show up fast. Reusable prompt fragments, dynamic injection, and proper function-style outputs are non-negotiable once you’re serving more than three document types.

2. Structuring reusable prompt fragments for clause-level injection

After banging my head against malformed PDFs for a week, I broke out the generic intro block (“This contract defines the following terms…”) into a standalone field. Then the compensation clause got its own, then liability, then governing law. I realized this wasn’t just a cleanup — it finally let me swap logic in real-time.

I now keep a Notion database with all the clause fragments, tagged by jurisdiction and agreement type. Each row is a JSON-ready text block, usually under 700 characters. That feeds into a step in Zapier that pulls the right fragment based on a dropdown field from a pre-intake Airtable form.

If you want to set this up sanely, here’s what helped:

  • Keep fragments under 800 characters, or OpenAI sometimes skips them entirely
  • Use triple quotes or XML-style tags or both to isolate each clause
  • Ensure newline characters don’t break when passed through to OpenAI — parse them in JavaScript first
  • Use predictable labels like [CLAUSE_COMPENSATION] so you can swap easily
  • Add a testing fallback clause: something like “TEST-ONLY-FRAGMENT” to confirm the loop is injecting

The weird thing is, Zapier’s test mode doesn’t always pull live Notion content due to caching — so in actual runs you might get different fragments than your preview. That wasn’t obvious until I looked at a generated contract and saw a confidentiality clause from a test client in Iowa inside a document for a New York designer. Not helpful.

3. Handling OpenAI prompt length limits without losing entire sections

I hit the token limit without realizing it the first time I added five dynamic clauses. The result? The model silently cut off at clause three, generated a partial contract, and the next step emailed it. Nobody — not even OpenAI’s error handling — flagged this truncation. Only caught it when a user replied “Uh, is this half a contract?”

OpenAI’s gpt-3.5-turbo has a cap around 4k tokens (roughly 10–12k characters), but Zapier doesn’t show a counter. I now run a preflight character count using a Code step before passing anything to OpenAI. Anything over around 9000 characters triggers an alert and skips generation entirely.

One odd thing I found: newlines and indentation count more than you’d think. A 700-word body with proper formatting ballooned past the cutoff, but a squished no-indentation version got through. So I wrote a Code step that squeezes the prompt before generation and adds formatting back after — ugly, but it works.

“length = JSON.stringify(prompt).length; if(length > 9500){throw Error(‘Prompt too long’)}”

You can find tools like OpenAI’s tokenizer, but none of them reflect Zapier’s API behavior exactly. Best bet is to stay under, test live, and don’t trust preview mode.

4. Fixing double injections when Zapier test data backfills improperly

One of the nastiest bugs I ran into looked like a token duplication. Took me two days to trace: when testing with the “Insert Data” menu in Zapier, I had accidentally added the same field twice into the prompt (“{{Field: Role}}” appeared in two spots inline). Totally fine if the field was a short label. Not fine if it was a rich-text NDA clause. The result was a prompt that said the same thing twice — differently — and confused the hell out of the model.

The UX didn’t help. Zapier doesn’t always show the full inserted field in preview mode, and when JSON payloads get long, the error console just truncates the log. You think it’s a model hallucination. It’s not. It’s your own prompt looping back on itself.

Since then, I wrap all prompt injections with visibly unique markers, like:

<<START_CLAUSE_COMP>>{{FIELD_CompensationClause}}<<END_CLAUSE_COMP>>

If the model ever repeats or drops a label, you’ll see it clearly in the output.

5. Building fallback prompts that still legally make sense

It finally happened: Notion API throttled me. I guess triggering six document generations in two seconds was pushing it. The OpenAI call still fired, but without clause data, it generated a skeleton contract filled with generic placeholders. Looked clean. Made no legal sense.

I now include a fallback prompt for every clause, kept in a separate static Zapier field per block. If the dynamic content fails or times out, the fallback language gets inserted instead. It’s basic, but at least keeps the doc legally coherent.

Why it matters for humans reading this stuff

Lawyers (or procurement managers, or whoever gets these things) check wording. If your contract says “The agreement shall commence on” and the next section says “[start_date]” with brackets still in, you lose trust. You have one shot to look competent. Fallback fragments might be repetitive, but they don’t break alignment. I’d take boring over broken any day.

Zapier’s filter logic *does* let you pre-check clause fields for null or empty strings, but only if they actually resolve by the time the Filter step runs. If the upstream Notion call times out but doesn’t error — which it sometimes does — you won’t catch it unless you inspect the raw input.

6. Adding QA layers without turning document review into a second job

I was doing manual checks until I missed a “Governing Law: UNITED STARES” typo that made it straight into a signed agreement. Not even a model typo — that was a prompt fragment typo in my Notion clause text that had been reused forty-two times. AI just copied it verbatim. My bad.

Now I route the output through Grammarly’s API via a webhook, which only flags real grammar issues. (Spelling is weird — AI corrections can introduce errors if you let them). Anything grammar-flagged gets held for human review via Slack. If it passes, it gets sent as a formatted PDF via DocuSign. If not, the Slack message lets someone click a Review button made in Zapier’s webhook + Slack integration.

That eats maybe 30 seconds per doc now. Better than ending up in a client call about constitutional typos. Also added a daily report that shows how many docs got flagged vs passed — the numbers help win arguments when someone suggests turning off the QA layer because “everything’s fine now.”

7. The obscure toggle in OpenAI that made everything finally click

This was buried under Debug -> Advanced Settings in the OpenAI Plugin for Zapier: “Enable function calling JSON output.” I had skipped it for weeks because I thought it was meant for API developers only. It’s not.

With function calling on, you define strict output formats like this:

{"type": "agreement", "clauses": [{"title": "Confidentiality", "body": "..."}...]}

The model, weirdly, adheres to the structure better — but only if your prompt also enforces it AND you use GPT-4. With GPT-3.5, it randomly drops closing brackets or includes apology paragraphs like “I’m an AI language model and cannot…” which defeats the purpose.

But with GPT-4’s upgraded context and the JSON schema enforcing clause boundaries, something just worked. The final file parsed flawlessly, no post-cleanup required. Don’t ask me why. I tried it five more times and it hasn’t broken yet.