Everything in this talk starts from OLMo‑3‑7B‑Instruct‑SFT, reproduced from Dolci‑Instruct‑SFT. It's a solid instruction-following model — just an overwhelmingly English one.

Every result in this deck is "on this checkpoint" — we haven't tried these recipes anywhere else yet.

We keep training that checkpoint — same instruction-tuning recipe, new mixtures of English + EU-language examples. The open question every run has to answer: does English survive?

Full numbers for Phases 0 & 1 are in the appendix.

Each prompt draws a topic (11 topics, weighted — creative writing, QA, translation, math, summarization, and more), then — with a per-run probability — a persona (8 of them, e.g. "a non-native speaker still learning the language") and a constraint (15 templates, e.g. "respond in exactly N sentences," "must include the word X"). One call turns that combination into an instruction, a second generates the response.

Generator: google/gemma-4-31b-it (bf16), vLLM, TP=2 on Leonardo A100s. Back-translation ran on a pilot subset only — roughly doubling the cost of every example it touches, so it wasn't repeated at full production scale. Full axis tables and the back-translation methodology are in the appendix.

10 SFT runs — 5 English/EU mixtures × 2 scales — graded the same way as the earlier phases. Reading the 1M-scale sweep against the baseline:

1M-scale runs, winrate vs. baseline (Qwen3-30B-A3B judge). Overall Elo peaks at 767 (50% English, 1M, Qwen3.5-27B judge) — higher than the English-only run (725). Full 10-run table, 3-judge robustness check, and training hyperparameters are in the appendix.

Winrates climbed, but reading the actual generated pairs, the recipe wasn't there yet — a topic, a persona and a constraint are each sampled independently, so nothing stops them from landing on a combination nobody would really write.

Sampling becomes hierarchical: first a domain family (local — needs country-specific knowledge — or general), then a specific domain within it, then a role and an intent that belong together. One call writes the instruction and response as a pair; an LLM judge replaces the mechanical checker.

Goal: a dataset and a technique — hierarchical topic sampling plus an LLM-judge quality filter — for training multilingual instruction models, not just one more mixture.

State lives on disk at every stage, so a killed run just resumes: unfinished generate/judge calls pick back up by id, and the filter/dedupe steps recompute cheaply over everything accumulated so far.

You are a native speaker and expert writer of
{lang}. Create ONE instruction + response.

Domain: {domain}
Diversity seed: {salt}
Suggested role: {role}
Suggested intent: {intent}

Instruction: native-fluent, realistic & self-
contained, genuinely non-trivial; any constraint
must be NATURAL and motivated (a real user would
actually ask for it).

Response: factually correct, no fabricated
claims; clearly explained, no padding.

Silently re-check before answering: does role /
intent / domain fit together naturally? Fix
anything that doesn't hold.

Output JSON only:
{{"instruction":"...","response":"...",
 "role_used":"...","intent_used":"..."}}

You are a strict quality filter. Give ONE
holistic score 0-10, weighing whether the
TASK is worth training on AND how well the
RESPONSE executes it -- the WEAKER decides.

DISQUALIFYING (score 0-2 regardless of the
response): trivial trivia, a pointless /
unmotivated constraint, not a genuine request.

Otherwise, score the response by its single
worst flaw, not the sum of every small one:
9-10 excellent  7-8 good  5-6 fair
3-4 weak        1-2 poor  0 unusable

Output JSON only:
{{"score": <0-10>, "reason": "..."}}

Domain sampled by weight (e.g. estimated from LMArena traffic), not uniformly at random. Abridged — both prompts are longer in full. Source: synthgen/localized/prompts.py.  Open question: the "diversity seed" is just an integer in the prompt text, not a true sampling seed — an LLM call has no real notion of one. Whether it nudges the model toward a less-obvious angle or does nothing is untested, worth discussing.

Several well-known open recipes share the "generate, then filter" shape ours does — but differ in what they filter with, and none target multilingual grounding as a first-class axis.

As documented in each project's public papers/repos — not independently re-verified by us, so treat this as directional. The throughline: none of these use a holistic per-example LLM-judge score as the primary keep/discard signal the way our v2 does, and none is built around multilingual grounding.

Same judge (Qwen3.5-27B), same five English/EU ratios, both v1 scales — only the EU data source differs: a translated EU-language mix (paired with English replay) vs. v1's native synthetic data.

v1 matches translation data at 100% English (no EU data yet, as expected) and pulls ahead at every other ratio, at both scales — most dramatically at 0% English, where v1 holds at 750–759 and translation data collapses to 677. 1M sits at or above 500k almost everywhere, confirming more data helps rather than just adding noise. Source: appendix.

The 0%-English point makes v1 look like an outright win. But translation data's own declared sweet spot was 75/25 — so that's the fairest single ratio to compare at, and the picture there is more honest: v1 buys a lot of EU winrate, at some cost to English winrate. Overall Elo comes out close to a wash.

Overall Elo: Qwen3.5-27B, LMArena Bradley-Terry. Winrate: Qwen3-30B-A3B, vs. each track's own baseline. The Overall-Elo edge (752–755 vs. 747) is within bootstrap noise (±9–12) — at matched ratio, call it a tie on the blended metric and a clear tradeoff on the split one, not a clean win for either source.

v1 already beats the translation-data baseline, and — checked under three independent judges — beats the Dolci-Translated models too (next slide). v2's job is to beat v1 itself: the same grounded, judge-filtered idea, minus the axis-independence problem.

What we want in the end isn't just a higher judge score — it's language that reads fluent and localized to a native speaker, which is why the judge and an eventual human read-through both matter.

To make sure the ranking isn't one judge's quirk, we re-scored baseline, two v1 configs — the more balanced 50en-1M and the most-multilingual 0en-1M — and the Dolci-Translated models under three independent judges.

Labels shown at the rightmost judge only, for clarity — full numbers in the appendix. Gemma judges score everything ~40–50 points lower than Qwen3.5 in absolute terms (an offset, not a re-ranking); compare within a judge, not across. Not evaluated under Qwen3.5-27B: Dolci-Translated. On every judge where both exist, v1 beats Dolci-Translated — and both v1 configs beat baseline under all three.

We handed the judge examples we know are good, and examples we deliberately made bad — wrong answers, mixed-up languages, nonsense constraints — across 11 EU languages, and checked whether its scores actually separate the two, language by language.

Overall: 9.06 good vs. 2.22 bad (n≈108 each). Czech (cs) separates least cleanly of the eleven; wrong-answer bad examples are the hardest defect type — both worth another look.

Everything so far starts from one checkpoint. Once v2's core experiment is settled, the next question is whether the same recipe — hierarchical sampling plus a judge filter — helps on a different base model too.

v1 already matches or beats the translation-data baseline at every English/EU ratio — and by a wide margin at 0% English (750 vs. 677 overall Elo). Checked under three independent judges, it beats the Dolci-Translated models too. The open question is whether v2's grounding beats v1 itself — and whether the whole recipe holds up on a checkpoint that isn't OLMo-3-7B.

• 01Finish topping up v2 to production scale across all 11 languages.
• 02Train and grade a v2 sweep, and compare it against all three baselines — sourced data, translated English, and v1 — at matched ratio and scale.
• 03Check fluency and localization directly, not just the judge score, on the winning mix.
• 04Repeat the recipe on another base model — Prelude once it's ready, or Qwen3-8B-base / Apertus-8B-base sooner — to see if it generalizes.
• 05Scale language coverage from the current 11 EU languages up to the full 36 OpenEuroLLM target languages.

Elo

Bradley‑Terry rating fit over pairwise LLM-judge battles. Win probability = 1 / (1 + 10^((R_b−R_a)/400)); a 100-point gap ≈ 64% expected win rate, 200 points ≈ 76%.

Winrate

The share of head-to-head battles a model wins against a fixed baseline. 50% = tied with baseline.

Judge model

A separate model used to grade outputs — either picking a winner between two answers, or scoring one answer on its own.

Checkpoint

A saved snapshot of a model at some point in training, that can be picked back up or evaluated.

LUMI

A European public supercomputer (in Finland) with AMD GPUs, used for the Track G training sweep.

Replay

Mixing a checkpoint's own original training data back in during continued training, to protect what it already knows.

Hierarchical sampling

Sampling a broad category first (e.g. a domain family), then a specific value within it, then dependent axes — rather than sampling every axis independently.

synthgen

The tool built for v2 that generates and quality-filters the grounded multilingual training examples.

The main slides avoid these labels and just describe what each source actually is (real/existing data, translation data, or our own synthetic data), but the underlying lab notebook names each experiment a Track, A through G. For anyone cross-referencing the source docs:

Tracks A–F predate this work (team effort); Track G is v1's training sweep, described throughout the main slides simply as "v1."

Elo: Qwen3.5‑27B judge, LMArena Bradley‑Terry, 100 bootstraps, 200 battles/lang · internal labels A–F refer to the team's six Phase‑0 experiment tracks · source: multilingual_eu_results.md

Axes defined in synthgen/config.py; topics can also be sampled empirically from WildChat via synthgen topics.

Back-translation QC: a sample of non-English generations was translated back to English by five independent reader models — DeepSeek-V3.2 (222 samples), Claude Sonnet 4.6 (250), Qwen-2.5-72B-Instruct (250), GPT-OSS-120B (243), Gemma-4-26B-it (800) — and read for meaning, register, and whether the constraint survived the round trip. This qualitative pass was the bar for approving the scale-up.

source: Taskboard #345 comments

LMArena-140k, 2204 battles, Qwen3.5-27B judge (Elo) & Qwen3-30B-A3B judge (WR). "Track F ref" is the matching-ratio Track F number from multilingual_eu_results.md — same ratio, different EU data source, so it's the direct comparison. Track G matches or beats it at every ratio, most dramatically at 0% English.

source: trackg-eval/REPORT.md; Taskboard #186 comments, 2026-07-13 & 2026-07-21

Switched training hardware from Nvidia H200 (HoreKa) to AMD MI250X (LUMI) for this sweep.

• ·Re-judged every cached generation with two more judges — google/gemma-4-31b-it and gemma-4-26B-A4B-it — on top of the original Qwen3.5-27B. Only the judge re-ran on GPU; generations were reused from cache.
• ·Rank agreement is high: Spearman ρ on overall Elo — Gemma-31B↔Gemma-26B = 0.96, Qwen↔Gemma-31B = 0.86, Qwen↔Gemma-26B = 0.75 (the lower figure is dragged by a baseline/G1 cluster within ~4 Elo of each other — noise, not real disagreement).
• ·Judge severity is an offset, not a re-ranking: both Gemma judges score every model ~40–50 Elo below Qwen3.5 in absolute terms, but preserve the same ordering.
• ·Under the independent per-language method (Gemma judges only): English barely moves across the whole English-ratio sweep (~910–950), while the EU-language mean climbs +75 to +90 Elo — described in the source notes as "the multilingual data is ~free on the EU side," and it survives the judge swap.
• ·The two Dolci-Translated models were LMArena-only runs (no separate English/EU winrate split was ever launched for them), so their winrate column is the single pooled crowd winrate, not a split.
• ·Which v1 config is "best" depends on the judge. Under Qwen3.5-27B alone, 50en-1M scores highest (767) — just ahead of 0en-1M (759, within noise) — and keeps a closer-to-balanced English/EU winrate (49.5% EN vs. 0en-1M's 44.1%). Under both Gemma judges, 0en-1M scores highest instead. So the main slide plots both: 50en-1M as the more balanced pick, 0en-1M as the most-multilingual extreme.

source: trackg-eval/REPORT.md · scripts/collect_three_judge.py, scripts/make_plots_threejudge.py

Every panel shows the same shape: English (x) stays in a narrow band once the model has any multilingual training, while EU-mean (y) fans out as more multilingual data is added — the "free lunch" holds under all three judges. Dolci-Translated (diamonds, Gemma judges only) sits on a similar English level but noticeably lower on EU than the strongest Track G points. Not shown: Qwen3.5-27B never scored the Dolci models. Self-computed from raw per-language JSON (simple mean of the 11 EU languages, full-anchor method) for consistency across judges — a different convention from the "independent" numbers quoted elsewhere in this appendix.

source: trackg-eval/results_perlang/TABLE_with_errorbars.txt (Qwen3.5); trackg-eval/results_gemma31b/, results_gemma26a4b/ raw result JSON (elo_per_language)

synthgen build-localized-prompts
synthgen generate --backend vllm \
  --mode localized
synthgen generate --backend vllm \
  --mode judge
synthgen decide-thresholds
synthgen quality-filter
synthgen topup --target N \
  --endpoints-dir endpoints/

$SYNTHGEN_ROOT/
├── prompts/prompts.jsonl
├── outputs/
│   ├── loc_<model>.jsonl
│   ├── judged_<model>.jsonl
│   ├── loc_full.kept.jsonl
│   └── loc_full.dedup.jsonl
├── review/
└── logs/

Grounding axes: 20 "local" domains that require country-specific knowledge (bureaucracy, healthcare, holidays, slang…) plus 10 general domains (work, creative writing), each crossed with a role and an intent. Generation runs against local models served on LUMI (vLLM) or a hosted API, through one shared backend.

11 EU languages tested, good mean 9.06 vs bad mean 2.22 overall. Wrong-answer examples are the one type that occasionally scored high — a concrete lead for tightening the judge prompt. Judge: Qwen/Qwen3.6-27B; generator: google/gemma-4-31b-it; both served via vLLM.

source: studies/localized_bootstrap/outputs/summary.json

Porting this from the original (Nvidia) supercomputer to LUMI's AMD chips needed genuine fixes — hardcoded paths, model-compatibility gaps, and a self-contained training image so every run reproduces the same environment. Validated end-to-end on real hardware before the full sweep launched.

• →Taskboard #186 (this investigation, incl. Track G + 3-judge results) — github.com/OpenEuroLLM/Taskboard/issues/186
• →Taskboard #345 (v1 synthetic data recipe) — github.com/OpenEuroLLM/Taskboard/issues/345
• →v1 dataset — openeurollm/EU-Instruct-Synthetic on Hugging Face
• →Track G evaluation campaign — trackg-eval/REPORT.md (3-judge scoring, ComparIA, per-language plots)
• →v2 tool (synthgen, this repo) — synthgen/localized/
• →v2 judge validation study — studies/localized_bootstrap/
• →LUMI training pipeline — oellm/experiments/dolci_synthetic/, oellm/pipelines/ in fabio‑open‑instruct/open‑instruct
• →Full early-experiment results — oellm/experiments/multilingual_eu/results/multilingual_eu_results.md