Translation pipeline can write invalid/corrupted output when updating against an existing translation (duplicate frontmatter, leaked LLM preamble, broken YAML)

[vdauwera]

Summary

The "Update translations (manual)" workflow run that produced PR #984 introduced real content bugs into 24 translated files across 8 languages — not just cosmetic drift. Root-caused and fixed on the PR branch (translate-ef75b806, commit 030461d28), but the underlying pipeline issue should be fixed so it doesn't recur on the next translation run.

What went wrong

Three distinct failure modes, all traced to the incremental "translate against an existing translation + diff" code path:

1. Duplicated frontmatter blocks (6 files)

For index_page-templated files (seqera_scale/index.md, nf4_science/genomics/index.md, nf4_science/_template/index.md across de/hi/ko/tr), the original English frontmatter block was left in place and a second, translated frontmatter block was appended immediately after it:

---
title: Scale with Seqera
...
---

---
title: Seqera ile Ölçeklendirme
...
---

Since MkDocs only parses the first ---/--- block, these pages built with English metadata and rendered the translated frontmatter as literal garbled text in the page body.

2. Leaked LLM response artifacts (14 files)

The model's own reasoning preamble (e.g. "Looking at the diff, I need to update: ...", "I'll analyze the diff and update only...") and/or the raw %%% delimiter used to wrap the existing-translation input in the prompt got saved directly into the output file, sometimes at the top, sometimes as a trailing line at the very end (independent occurrences in the same file in a couple of cases).

3. Invalid YAML produced by translated word order (3 files)

Translated sentences sometimes reordered words such that a frontmatter list item began with a character that's special in a YAML plain scalar but wasn't in the English source: a literal { (from {DOMAIN}-style placeholders), a backtick (tw CLI...), or a stray * escape (valid in Markdown, not YAML). These broke yaml.safe_load outright.

Where this lives

• _scripts/translate/prompts.py:60,78,87,92,132,144 — the %%% delimiter wrapping existing/original content in the prompt. The model appears to echo this token pattern back in some responses.
• _scripts/translate/core.py:157 — result.text.strip() is the only post-processing applied to the raw model response before it's written and verified; no detection of leaked preamble text or a duplicated frontmatter block.
• _scripts/translate/verify.py:107 (verify_translation_structural) — already has a real check for "wrapped in code fence" (line 120-125), but:
  • it only inspects trans_lines[0] — a fence or preamble anywhere else (including the trailing-line cases found here) isn't caught
  • there's no check for a second ---/--- block appearing after a valid first one (the duplicate-frontmatter case)
  • there's no check for leaked reasoning prose that isn't a code fence
• _scripts/translate/core.py:271-296 — verification does retry up to MAX_VERIFY_RETRIES on structural issues, so tightening verify_translation_structural should be sufficient to catch these before they're ever written to a PR, without needing new retry plumbing.

Suggested fix

In verify_translation_structural:

1. Check the last non-blank line too, not just the first, for a stray fence or %%%.
2. Flag a second --- line appearing anywhere after the first properly-closed frontmatter block (duplicate frontmatter).
3. Flag any line matching a "reasoning preamble" pattern before the first heading/frontmatter (e.g. starts with "Looking at the diff", "I'll analyze", "I need to update" — or more robustly, flag any non-blank content before frontmatter/first heading that isn't the expected structure at all).
4. Optionally, validate frontmatter with yaml.safe_load directly as part of the structural check, rather than relying on it never breaking — this would have caught all 3 of the invalid-YAML cases for free.

Reference

See commit 030461d on branch translate-ef75b806 (PR #984) for the concrete before/after diff across all 24 affected files.