LLM Peer Reviewers Favor Confident Writing in Study | Zeniteq
Loading
Loading article
LLM Peer Reviewers Favor Confident Writing in Study
An arXiv preprint reports that changing a paper’s rhetorical confidence can influence LLM assessments, even when the rewrites are designed to preserve the underlying science.
A recent arXiv preprint titled Style over Substance? Linguistic Bias in LLM-based Peer Review reveals that LLM reviewers systematically favor confident rhetoric, even when scientific content remains unchanged. By using controlled rewrites to manipulate linguistic confidence while preserving experimental data, researchers demonstrated that models often conflate assertive tone with scientific validity. This bias poses significant risks for AI-assisted academic workflows, as model-generated scores or rankings could be skewed by stylistic presentation rather than research quality. Consequently, developers must rigorously test whether their systems distinguish between justified scientific certainty and mere rhetorical confidence before deploying them for critical evaluation.
LLM peer reviewers systematically favored confident writing in a controlled experiment that changed how research papers were written while aiming to keep their scientific content intact. The finding raises a practical concern for AI-assisted reviewing: a model’s assessment may respond to how strongly authors present their work, alongside the evidence they provide.
The study, Style over Substance? Linguistic Bias in LLM-based Peer Review, was posted to arXiv on October 26, 2026, by Sai Suresh Macharla Vasu and eight co-authors. The arXiv record identifies it as a preprint submitted to ICLR 2027. Its results should therefore be treated as preliminary research.
The authors used LLMs to rewrite papers along two linguistic dimensions, then had different LLM reviewers evaluate the resulting versions. Their central finding concerns confidence: the reviewers rewarded assertive rhetoric despite interventions intended to leave the research unchanged.
The useful contribution is the comparison itself. Comparing differently written versions of the same research offers a more direct test of style sensitivity than comparing unrelated papers, where differences in methods, experiments and results could explain different assessments.
How the Controlled Rewrites Work
The researchers built a framework to alter specific linguistic characteristics of scientific papers. Instead of relying on a broad request to improve or simplify the writing, they developed fine-grained rewriting instructions grounded in linguistic literature and applied those instructions using LLMs.
Native-associated versus non-native-associated English
Linguistic features associated with those writing styles
Whether presentation linked to language background affects model assessments
High-confidence versus low-confidence language
How confidently the research is expressed
Whether rhetorical certainty affects assessments of the same underlying work
These are manipulated properties of the text. They should not be read as verified descriptions of the original authors’ language backgrounds.
The distinction matters because an ordinary collection of papers cannot cleanly isolate a writing-style effect. A highly polished paper might also contain stronger experiments. A cautiously written paper might investigate a more uncertain problem. If those papers receive different reviews, the writing alone cannot explain the difference.
Controlled rewriting tries to reduce that ambiguity. The research stays anchored to the same underlying paper while the presentation changes. If a reviewer then responds differently, style becomes a plausible explanation, subject to the quality of the content-preservation controls.
I’d put more weight on that design choice than on a broad claim that AI prefers polished prose. The consequential question is whether a reviewer’s assessment remains stable when the evidence it is supposed to judge remains stable.
Preserving the Science Is the Crucial Condition
The authors explicitly say their interventions are designed to preserve underlying scientific content. That is the foundation of their interpretation: if the science changes during rewriting, a changed assessment may be justified.
Content preservation is particularly delicate when the manipulated feature is confidence.
A hedge can be a stylistic habit, but it can also communicate a real scientific limitation. Removing uncertainty from a sentence might make it sound more direct. It might also turn a tentative interpretation into a stronger claim than the evidence supports.
That creates an important distinction between rhetorical confidence and scientific certainty. The first concerns presentation. The second concerns what the results warrant.
For this experiment’s logic to hold, rewrites need to retain such details as:
The scope of the research claims.
The methods and experimental conditions.
The reported results and comparisons.
Limitations and qualifications that affect interpretation.
Those are analytical requirements of a content-preserving comparison, not a description of additional checks verified here. The authors’ abstract states the preservation goal, but that statement alone is not independent confirmation that every rewrite retained every meaningful qualification. The strength of the conclusion depends on the authors’ controls and measurements. Source: study abstract and framework description
My view is that this is the most important detail for readers evaluating the work. A compelling demonstration needs both parts: evidence that the style changed in the intended direction, and evidence that the scientific meaning stayed sufficiently stable. Otherwise, a reviewer could be reacting to an altered claim rather than merely a more confident voice.
What the Reviewer Comparison Shows
The researchers evaluated the rewritten papers using different LLM reviewers. Their reported conclusion is that the reviewers:
“systematically reward confident rhetoric independent of scientific content”
The word systematically describes the authors’ finding across their experiments. It should not be interpreted as a claim that every review favored every confident rewrite.
Using multiple reviewers makes the result more informative than a demonstration involving one model response. It examines whether the effect appears across different LLM evaluators within the study’s setup. It does not make the result universal.
The relevant unit is the configured reviewing system: the model, its version, the instructions it receives, the review rubric and the papers it evaluates. A finding under one set of conditions cannot simply be carried over to another model or discipline.
The two style dimensions also need to remain separate in reporting. The framework investigates both native-associated language features and confidence, but the abstract’s headline finding is specifically about confident rhetoric. It would be an overreach to turn that into an equally strong claim about every language-background intervention.
The abstract reports the direction of the confidence effect without providing a numerical effect size or a model-by-model breakdown. That supports a warning about assessment sensitivity, but not a precise claim about how many rating points a confident rewrite gains or which reviewer is most susceptible.
Assessment Changes and Decision Changes Are Different
A style-sensitive assessment can matter even when it does not visibly reverse an acceptance recommendation.
Consider a hypothetical review workflow that uses model scores to prioritize papers for closer inspection. A style-driven score change could alter that ordering. In a workflow with a cutoff, the same change could affect which papers cross it.
Those are practical implications of the reported effect, not measured conference outcomes established by the abstract.
Three distinct questions help keep the stakes clear:
Does the assessment change? A reviewer responds differently to versions intended to contain the same science.
Does the ordering change? Those differences affect how papers compare with one another.
Does a decision change? A recommendation, threshold or human decision is affected.
Evidence for the first creates a reason to investigate the other two. It does not supply their frequency.
This distinction is especially important near a decision boundary. A modest assessment difference could matter for a borderline paper while leaving a clearly strong or clearly weak submission’s outcome unchanged. The practical impact depends on the review system’s rules and on how people use its output.
The same reasoning applies to summarization tools, although the study’s headline result concerns reviewing. If a venue uses model-written summaries, it would be sensible to check whether those summaries preserve uncertainty accurately. Whether confidence bias changes summaries in a particular system still needs to be tested.
What Venues and Tool Builders Should Check
The study suggests a useful robustness question for AI reviewing tools: does the system’s judgment survive a controlled change in presentation?
That question is more demanding than checking whether a generated review sounds plausible. A fluent review can still be sensitive to irrelevant features of its input.
For teams considering LLM-assisted scientific assessment, the experimental design points toward several concrete checks:
Compare matched versions of a paper. Use style interventions with a defensible content-preservation process, then inspect changes in the model’s assessment.
Require evidence-linked judgments. Ask the reviewer to connect claims about novelty, soundness and experimental support to specific material in the paper.
Measure consequences separately. Track changes in written assessments, scores, rankings and recommendations instead of treating them as interchangeable.
Record the full reviewing setup. Preserve the model version, prompt and rubric so that a later model update can be checked under comparable conditions.
Inspect uncertainty handling. Check whether the system distinguishes justified caution from weak evidence and confident presentation from strong evidence.
These are implications drawn from the study’s design, not mitigations the cited abstract demonstrates will eliminate the effect.
I would prioritize the uncertainty check. Scientific writing often needs to express exactly where a result stops being reliable. A reviewing tool that treats careful qualification as a weakness could create pressure to remove information readers need.
Human oversight also needs a concrete role. A person who simply accepts a polished model review may carry its assumptions forward. A more useful review step would examine whether the model’s favorable or unfavorable judgment follows from the paper’s evidence, especially when its language emphasizes author confidence.
A Warning About What the Model Is Measuring
The preprint identifies a potential mismatch between the intended task and the signal a model uses. Scientific reviewing asks whether claims are supported, methods are sound and contributions are meaningful. Confidence in the prose may be easier to recognize than those deeper properties.
The experiment does not establish that LLMs are unusable for every reviewing task. It gives a narrower, actionable warning: the tested reviewers’ assessments were sensitive to confidence-oriented rewrites intended to preserve scientific content.
For venues and tool builders, the practical standard should be equally specific. Before relying on a model’s scores or recommendations, check whether its judgment changes when the research stays the same but the writing becomes more assertive. A reliable reviewing aid needs to distinguish evidence of quality from language that merely sounds authoritative.
Frequently Asked Questions
4 questions
1
What Did the Study Find?
The authors report that the tested LLM reviewers systematically favored confident rhetoric in rewritten scientific papers, even though the interventions were designed to preserve the underlying research.
2
Is the Paper Peer-Reviewed?
The arXiv record labels it a preprint submitted to ICLR 2027. That record does not establish completed peer review or conference acceptance.
3
Did Confident Writing Change Acceptance Decisions?
The abstract reports an effect on reviewer assessments. It does not quantify actual conference acceptance changes or provide a verified decision-flip rate.
4
Does This Apply to Every LLM?
No. The finding should remain tied to the models, papers and experimental setup tested. Other model versions, prompts, rubrics and disciplines require their own evaluation.
