Which Qwen 3.5 fits sophisticated document work?

Eval of qwen3.5 0.8b / 2b / 4b / 9b / 27b · test material: Did Your AI Just Sigh? by Uli Hitzel (17,286 words, 76 chapters) · 2026-07-05

Result matrix

Percent of checks passed; rows marked v2 use the hardened second-round tests, which supersede the four first-round rows that turned out too easy.

What each size step buys

• 0.8b → 2b: multi-turn stability (drift at turn 6 → none), faithful summaries, copyediting, freedom from repetition collapse in creative output. The biggest single jump in usefulness.
• 2b → 4b: two capability cliffs. Paraphrased retrieval jumps 33 → 100 — below 4b, models find facts only when the question reuses the document's own words. Reasoning jumps 57 → 86.
• 4b → 9b: reasoning completes the climb (86 → 100) and unlabeled context switching goes clean (12/12). Not a uniform win: on whole-book synthesis both get the structure fully right (including an epilogue every smaller model and our own first-pass parser missed), but 4b recalled all 12 probed narrators where 9b recalled 5 — bigger models abstract more and retain fewer specifics.
• 9b → 27b: the surprise of the eval. We predicted a plateau; instead 27b caught the planted logical contradiction that all four smaller models missed — with a correct explanation and fix — and delivered the only 100 on judged rewrites. Semantic proofreading is a scale-bought capability that switches on here. Interpretive classification also nudges up (60 → 70).
• What no size fixes: interpretive classification never gets good (70 at 27b — though the remaining misses are genuinely ambiguous placements), and whole-book narrator recall declines past 4b as bigger models favor abstraction over detail.

Cheat sheet

The challenge

Two things made this non-trivial. Material: a real 17k-word book, not benchmark snippets — long-context behavior, multi-turn drift, and satire/irony comprehension all matter, and public benchmark scores don't predict them. Method: everything scripted, logged, resumable, and runnable unattended, so results are reproducible rather than anecdotal — and hardened over two rounds when first-round checks proved too easy (see iterations).

Approach

Twelve scenarios spanning classification, extraction, summarization (chapter / part / whole book), rewriting, copyediting with injected errors, reasoning, needle-in-haystack retrieval, and three multi-turn stress tests (sustained Q&A, document switching, task switching). Scoring is deterministic where possible — JSON validity, schema keys, fact presence, error-catch counts, per-turn pass marks — plus manual transcript judging for prose quality. Each model runs warm through all scenarios, then unloads; a memory guard checks pressure between items; every request/response/timing lands in a JSONL log.

Setup

• Five model sizes: qwen3.5 0.8b / 2b / 4b / 9b / 27b · temperature 0.2 · thinking disabled — without it Qwen 3.5 burns 300+ tokens reasoning about "say OK"
• Book converted from RTF and split deterministically into 76 chapters / 4 parts (prepare_book.py)
• Pipeline: build_scenarios.py → run_all.sh → aggregate.py → analyze_turns.py, v2 mirror for round two · ~580 scored generations across the five models

Decisions

• One warm model at a time, sequential scenarios — clean isolation between models, no contention effects in the measurements.
• Deterministic checks over LLM judging — a local judge model would bias results toward its own family; prose quality judged by reading transcripts instead, as a separate layer.
• Chat scenarios carry full history per turn — measures real multi-turn degradation, not stateless per-question accuracy.
• Partial credit on coverage checks — "found 9/10 errors" scores 0.9, not 0.
• Same prompts, same seeds of material, for every size — differences in the matrix are model differences, nothing else.

Iterations

Round 1 → skepticism

Four tests returned suspiciously clean 100s. A score that doesn't discriminate is a test that's too easy: the needle questions shared words with the planted facts (string matching, not comprehension); rewrite checks measured fact survival, not prose (0.8b "won" while looping the same sentence eight times); context-switch questions named their document; summary checks were keyword-level.

Round 2 (v2) — hardened tests, same models

Paraphrased needle questions with zero lexical overlap; degeneration detection + a from-scratch creative task for rewrites; unlabeled context-switch questions with overlapping numbers planted in both documents; summaries probed by follow-up who-did-what questions. Result: the fake 100s collapsed exactly where suspected — most dramatically retrieval: 0.8b/2b fell 100→33 while 4b held 100, the clearest size-buys-capability result in the project.

Round 3 — adding the 9b

The 9b ran the full v1+v2 suite. It confirmed the size gradient where one existed (reasoning 43 → 57 → 86 → 100) and appeared to confirm plateaus elsewhere: interpretive classification stuck at 60 with the same borderline misses as 4b, the logical contradiction still uncaught. At this point the working hypothesis was that the remaining failures were class limits rather than size limits.

Round 4 — the 27b falsifies the plateau hypothesis

Before committing to a full 27b round, a single cheap probe: the copyedit test, chosen because its planted logical contradiction had defeated all four models identically. The stated decision rule: if 27b misses it too, skip the round. It didn't miss. It found all 10 errors including the contradiction, with a correct explanation and fix — so the full suite ran, and the pattern held: 100 on judged rewrites (vs the 82 plateau), 70 on interpretive classification (vs 60). The "class limit" hypothesis was wrong for semantic error detection; it survives, weakened, only for interpretive classification. A good reminder that a plateau over two points is thin evidence.

Corrections against ourselves

• v1 claimed few-shot examples would fix classification. Tested: it didn't (scores unchanged; 2b still answered one label for everything). A control with plain topic labels showed ordinary classification is fine from 2b up — the failure is specific to interpretive labels (narrative-arc placement).
• v1 accused 2b of hallucinating an "Epilogue: Manual Override" in the whole-book test. 2b was right — the epilogue exists; our chapter splitter had glued it onto the last chapter. The model read the book more carefully than the parser. (The 9b later named that epilogue unprompted.)

Multi-turn drift, turn by turn