Proofreading on a Mac: Qwen3-8B vs Qwen3-4B

Can a model running on a laptop proofread English well enough that you no longer need to paste your text into ChatGPT or another cloud service? We measured Qwen3-8B and Qwen3-4B on a MacBook Pro against human corrections: fixing grammar, leaving correct text alone, and rewriting tone.

Results

Qwen3-8B is the better proofreader: 0.718 grammar-correction F0.5 against 0.675 for Qwen3-4B on llama.cpp (+0.043, 95% interval +0.023 to +0.062), at 1.17 s vs 0.70 s per sentence. Both are usable as a suggestion tool, neither as blind auto-correct: both change about a third of sentences that were already correct.

Use Qwen3-8B when correction quality matters. Use Qwen3-4B for faster interactive suggestions that a person accepts or rejects one by one.

Each configuration saw the same 748 learner sentences, 150 already-correct sentences and 40 tone rewrites.

Examples

Both models on llama.cpp, same prompt, greedy decoding. Highlighted: words the model added or changed; struck through: words it removed.

Grammar prompt: You are a careful proofreader. Correct grammar, spelling and punctuation errors in the user's text and make it read fluently. Keep the meaning and the author's wording; change as little as necessary. If the text is already correct, return it unchanged. Reply with only the corrected text, without quotes or comments.

Tone prompts: Rewrite the user's message in a relaxed, friendly, casual tone, as a short chat message to a colleague. Keep every fact exactly: names, dates, times, numbers, amounts and places. Reply with only the rewritten message. and Rewrite the user's message in a formal, polite business tone. Keep every fact exactly: names, dates, times, numbers, amounts and places. Do not use contractions, slang or emoji. Reply with only the rewritten message.

Learner sentences and human corrections are quoted from JFLEG (Napoles et al., 2017), CC BY-NC-SA 4.0. The tone messages are synthetic.

Applied observations

What happens to a learner sentence

• Most output needs review, not rejection. The largest share of sentences is partly fixed: some human edits made, some missed or done differently. Only 5–7% of sentences got nothing but wrong edits.
• 8B leaves fewer errors in place. Qwen3-4B on llama.cpp returned 6.6% of flawed sentences unchanged, Qwen3-8B 2.8%. Recall is where 8B gains most (0.61 vs 0.54).
• Precision is high for both. About three of every four edits either model makes match an edit a human corrector made.
• The score is strict. A fix that differs from all four human corrections counts as wrong even when it reads fine, so the numbers understate both models somewhat (see the notes in the examples above).

Correct text gets edited

Of 150 sentences that needed no changes, every configuration changed 52–60. The edits are small: 63% of those changed sentences carry a single edit. Some are real fixes, because a few human corrections still contain errors. In a tool that shows suggestions, the rest is noise a person dismisses quickly; in an auto-correct that applies them silently, it rewrites a third of what you wrote.

Tone rewrites keep facts, with different failure habits

All configurations pass 88–100% of the 40 rewrites.

• Qwen3-4B drops detail: "the Berlin office" became "Berlin" in 3 casual rewrites on llama.cpp, which changes where the meeting is.
• Qwen3-8B reformats: it kept facts in an equivalent but different form ("4 PM" for 16:00, "June 3rd" for June 3) in 40% of rewrites, against 20% for Qwen3-4B. The check accepts equivalent forms; a reader who expects the original format may not.
• Qwen3-8B on MLX-LM kept slang: "awesome" survived in 5 formal rewrites.

Which differences are real

An interval that excludes zero means the difference is unlikely to be chance. The 8B grammar gain holds on both runtimes, but on MLX-LM it only just clears zero. The two 4B builds use different 4-bit schemes, and this is the only place the weight format showed a measurable effect.

How it is scored

A model's edits are compared with the edits human correctors made, using ERRANT, the standard scorer for grammatical error correction.

1. Data: the JFLEG test set (CC BY-NC-SA 4.0), 748 sentences written by English learners, each corrected independently by 4 people. Pinned revision 8b9fb0e6b3; the dataset is downloaded, not redistributed.
2. Prompt: the model gets the sentence with an instruction to fix grammar, spelling and punctuation, change as little as needed, and return only the corrected text.
3. Edits: ERRANT aligns the original with the model's output and lists the edits (words replaced, inserted or deleted). It does the same for each human correction. All texts are tokenized the same way, so spacing around punctuation never counts as an edit.
4. Matching: a model edit counts as correct (TP) when a human made the same edit at the same position. Extra edits are false positives (FP); human edits the model missed are false negatives (FN). Each sentence is scored against whichever of its 4 human corrections fits the model best.
5. Totals: TP, FP and FN are summed over all 748 sentences, then turned into one score:

F0.5 weights precision twice as much as recall: for a proofreader, a wrong change is worse than a missed one.

Correct text untouched is the share of the 150 already-correct sentences (JFLEG human corrections fed back as input) that come back with zero ERRANT edits. Tone rewrites are 40 synthetic messages rewritten casual → formal or formal → casual; one passes only if every fact is kept (names, times, numbers, places; equivalent forms such as 14:45 = 2:45 PM count), the text actually changed, and a formal rewrite has no slang, emoji or contractions.

Intervals come from a paired bootstrap: draw the items at random with replacement, recompute both sides' score on that same sample, record the difference, and repeat 3,000 times. The middle 95% of those differences is the interval. Pairing removes the noise of which items happen to be easy.

Caveats

• Learner English: JFLEG is essays by English learners. Your own messages may contain fewer and different errors.
• One prompt: all numbers use one minimal-edit prompt. A stricter prompt may raise the share of correct text left untouched.
• Small style set: 40 synthetic rewrites, checked by rules rather than people, can only reveal large differences.
• One machine, under load: an 18 GB M3 Pro with other apps holding memory. Speeds are pessimistic; quality scores are not affected.

Reproduce: python scripts/prepare_proofread.py, then llm-sheet evaluate --config configs/experiments/proofread.yaml --download. Scores recompute from the saved model outputs with llm-sheet evaluate --runs results/proofread.