When pi's output cap eats the file: a maxTokens truncation, caught in the wild

TL;DR — pi's OpenAI-compatible provider defaults to maxTokens: 4096, and a local model asked to write one large file will hit that cap mid-file. The write still lands: the tool reports "Successfully wrote 32643 bytes" and the file is silently half-written. Below is a real session where the cap (8192 at the time) cost 7 minutes and ~49k output tokens on a single ~33 KB file before the model escaped by shrinking the file. The fix is one line of config, and it is why every models.json here carries an explicit maxTokens.

• Evidence: evidence/2026-07-04-ntfy-cron-maxtokens-8192.jsonl — 223 assistant turns, tool payloads stripped (see Provenance below).
• Date: 2026-07-04, evening. Model: qwen3.6-35b-a3b-vision via llama-swap.
• Not a benchmark run: this is an ordinary coding session on a personal project (ntfy_cron), which is exactly why it is worth keeping — the benchmark harness never ran with a low cap, so the failure only ever showed up in real use.

The config that caused it

maxTokens is per-model in models.json; when it is absent, pi's OpenAI-compatible path falls back to 4096. History of the shared client config (llama_cpp_on_my_desktop/clients/pi/models.json):

The session below ran on the evening of 2026-07-04 — the day before 3692e27. It is the reason that commit exists.

What the cap looks like from the inside

The model has just found a bug in frontend/index.html (a ~33 KB single-file frontend) and decides to rewrite the whole file. Every attempt is one write tool call whose content argument is the entire file, so the generation runs into the cap inside the tool call:

Its own narration, turn by turn:

404  "I see the bug — `once-field` has class="hidden" and setSchedule() is only called when…"
406  "The file got truncated. Let me write the complete file properly:"
408  "The file keeps getting truncated. Let me use a different approach to write the complete file:"
410  "Let me write the file using a Python script to avoid heredoc issues:"
422  "The file is truncated. Let me write a Python script to create the complete file:"
424  "Let me write the file in parts using the edit tool on a complete but minimal file:"

Three things are worth noticing.

The write is not rejected — it is executed. pi hands the truncated tool call straight to the write tool, which happily reports success. Nothing in the loop says "your generation was cut off"; the model only learns something is wrong when it re-reads the file. That is the "silently truncates large file writes mid-file" behaviour in one screenshot.

Routing around the write tool does not help, and gets worse. Switching to bash heredocs moves the truncation into the shell: the terminator (HTMLEOF, PYEOF) never gets emitted, so bash swallows the rest of the script and Python dies on an unterminated string. The diagnosis the model reads back ("heredoc delimited by end-of-file", "SyntaxError") points at quoting, not at a token limit — so it keeps trying different quoting tricks. The real cause is invisible from where it stands.

It escapes by accident, not by insight. Turn 424 writes a smaller rewrite — 31478 bytes of arguments, output: 8101, ninety-one tokens under the cap — and it completes. wc -l confirms 425 lines ending in </html>. The model then does what would have worked all along: targeted edit calls instead of full-file rewrites.

Cost: 431 s (~7 min) and 49,152 output tokens across the six cut turns, on a task the model had already solved correctly in its head.

Why the number to raise is maxTokens, not the context window

The cap here is the output budget for one generation. The session had ~87 k tokens of context in cache and was nowhere near the server's context limit — usage on the first cut turn reads input: 8791, cacheRead: 86671, output: 8192. A model can therefore be comfortable on context and still be unable to emit one large file, which is what makes this failure so easy to misread as "the model is bad at long files".

The rule of thumb this leaves us with: maxTokens must exceed the largest single file you expect the model to write, in tokens. A 33 KB HTML file costs roughly 8–9 k output tokens (~3.8 bytes/token here), so 8192 was just barely too small, and 4096 is not enough for anything past a couple hundred lines. Every model in models.json now declares its own value (the local Qwens at 32768; Bielik stays at 8192 because its whole context is 32768).

Ways this failure is worth designing against

Raising the cap removes the common case, but a big enough file will always find it again. The cheap mitigations, roughly in order of value:

1. Tell the model the truth. A generation cut by length inside a tool call should not be executed as if it were complete — block it and return "your write was truncated by the token limit; write it in chunks" as the tool result. The model recovers immediately when it is told the actual cause; it flailed for 7 minutes because it was not.
2. Prefer edit over full-file write. The escape at turn 424 and everything after it is edit-based. A standing instruction to chunk writes over ~10 KB costs nothing.
3. Watch for stopReason: length at all. It is a one-line check on any transcript and it is the only unambiguous signal that the cap bit.

Points 1 and 2 are what the terminal-bench harness (terminal_bench_on_rtx_3090) ended up shipping as its write guard and scaffolding preamble, and what the reusable llama-local-hooks pi extension in llama_cpp_on_my_desktop/clients/pi/ carries into day-to-day use.

Provenance

The artifact is a trimmed copy of a real pi session; the original lives outside this repo at ~/.pi/agent/sessions/--home-lzieniew-Documents-ntfy_cron--/2026-07-04T20-58-59-485Z_019f2eed-….jsonl.

Line 1 of the .jsonl is a provenance record; every following line is one assistant turn:

{"line": 404, "ts": 1783200347043, "stopReason": "length", "truncated": true,
 "usage": {"input": 8791, "output": 8192, "cacheRead": 86671, "totalTokens": 103654},
 "text": "I see the bug — `once-field` has `class=\"hidden\"` …", "thinkingChars": 523,
 "toolCalls": [{"name": "write", "argBytes": 33846, "path": "frontend/index.html"}]}

Trim rules: assistant messages only; thinking text replaced by a character count; tool arguments replaced by {name, argBytes, path} so no file contents or shell commands are committed; user messages dropped; visible text capped at 400 characters; absolute host paths redacted. line is the line number in the original session file, so anything here can be traced back to the source.