Jessica Hullman: AI Science Needs More Than Verification | Zeniteq
Loading
Loading article
Zeniteq is now available on mobile for iPhone, iPad and Apple Watch.Download nowLearn more
Jessica Hullman: AI Science Needs More Than Verification
Verified AI answers can be useful, but Jessica Hullman argues that science also needs people and institutions capable of questioning, defending and repairing them.
Jessica Hullman, a Northwestern University professor, argues in her October 30, 2026 Cosmos Institute essay "Verification Is Not Knowledge" that AI-generated scientific results cannot sustain knowledge through technical correctness alone — accountability must accompany verification.
Her argument, extended in a November 2 follow-up, draws on philosopher Helen Longino's work on criticism and Robert Brandom's reasoning framework to contend that AI systems need persistent claim records, stable responsibility, and durable criticism processes so that challenges can actually change the research record rather than merely receive generated replies.
Jessica Hullman argues that AI-generated results cannot sustain scientific knowledge through technical correctness alone. In Verification Is Not Knowledge, published by the Cosmos Institute on October 30, 2026, the Northwestern University professor makes accountability central: someone must be able to explain what a claim rests on and respond when it is challenged.
Her argument gives AI-for-science teams a design problem beyond producing correct outputs. Systems also need ways to preserve commitments, process criticism and revise results, while keeping human researchers capable of interrogating the work.
The essay is a philosophical argument, not an experimental finding or a benchmark of current AI systems. Hullman’s November 2 follow-up makes its practical implications clearer, particularly around persistent state, reputation and the risk of eroding scientific judgment.
The useful question for developers is therefore broader than whether an AI result passes a check: what happens when someone later finds a reason to challenge it?
What Verification Can and Cannot Settle
Verification can provide a strong reason to trust a result. Hullman explicitly acknowledges this in her follow-up: AI verification of AI-generated work supplies a rationale for believing that work is reliable. Her objection concerns treating that assurance as the entire machinery of science.
A formal proof illustrates the distinction. Checking that a conclusion follows from specified assumptions settles an important technical question. Choosing those assumptions, connecting the result to a scientific problem and deciding whether it supports a particular application remain additional tasks.
That distinction becomes consequential when the problem changes. A valid result may depend on an assumption that no longer fits the intended use. The original verification can remain sound while the application needs revision.
Hullman develops the point through a thought experiment she attributes to mathematician Terry Tao: an AI-discovered cancer treatment comes with a formal correctness certificate and a passed phase 3 trial, but no human understands how it was found. The treatment is hypothetical, not a reported medical breakthrough. In her account, the concern is whether the AI could have exploited a weakness in the trial itself, and whether anyone could investigate that possibility. Her essay uses the scenario to shift attention toward what happens when something goes wrong.
A patient could reasonably value the treatment without resolving those institutional questions. Accepting a useful output and maintaining the capacity to investigate its failures are compatible choices.
My view is that this is the strongest practical reading of Hullman’s argument. It avoids making complete human comprehension a prerequisite for using every result. Instead, it asks whether the surrounding research process can still detect trouble, locate the relevant assumptions and organize a response.
Why Criticism and Values Remain Part of Science
Hullman connects enthusiasm for automated science to the value-free ideal: the view that scientific judgments should be insulated from ethical and social values, relying on epistemic criteria alone.
Her criticism is not an invitation to ignore evidence. It concerns the belief that procedures can remove the need for judgment about evidence, research priorities and acceptable uncertainty.
In her follow-up essay, Hullman draws on philosopher Helen Longino to explain why criticism matters. Scientific communities negotiate and refine standards for what counts as objective evidence. Challenges must be capable of changing beliefs and practices for that process to function.
The practical consequence is easy to miss in an automated workflow. A system can accept comments, generate replies and continue producing papers without any criticism altering its commitments. That would provide the appearance of scientific exchange while leaving its corrective function weak.
Hullman also invokes philosopher Robert Brandom’s thought experiment about a community whose beliefs arise reliably but whose members cannot give reasons for them. In her account, the missing ability to give and ask for reasons prevents that community from possessing knowledge, despite holding true beliefs.
These are philosophical arguments, presented through Hullman’s interpretation. Their relevance to AI engineering is specific: reliable output generation and a functioning process of justification are different capabilities.
For a research team, this means a criticism channel needs a destination. An objection should be able to trigger a changed claim, a revised method, a narrower scope or an explicit defense. Generating another plausible explanation is insufficient if the system never records what the challenge changed.
When Named Authors Cannot Answer for a Claim
Hullman defines accountability as:
“being answerable for a claim, able to describe what it rests on and respond when it’s challenged”
That definition, from the original essay, makes authorship a continuing responsibility.
She cites computer scientist Tom Dietterich’s concerns about arXiv papers submitted by people who appear unlikely to understand their claims. The example illustrates an accountability gap; it is not a quantified audit of how often this happens.
A name on a paper ordinarily gives readers somewhere to direct questions. If the listed author cannot explain the argument, identify its dependencies or help investigate a failure, that connection becomes much less useful.
AI assistance does not automatically create this problem. Researchers already depend on collaborators, specialized instruments and methods they do not individually understand in full. The relevant distinction is whether responsibility remains workable across that division of expertise.
I would judge an AI-assisted authorship process by its response to a substantive objection. Can the responsible team trace the claim, explain the evidence and authorize a correction? That is a more useful standard than the percentage of prose or code generated by a model.
Turning Accountability Into a System Requirement
Hullman’s November follow-up identifies a mismatch between scientific accountability and how frontier models are developed. She argues that agents are not trained to maintain the persistent state needed to hold commitments and update beliefs consistently as challenges arrive. She also argues that reputation should constrain what gets claimed. Her discussion leaves open how far present systems can satisfy those requirements.
A practical implementation would need more than a long conversation history. The following framework is an engineering interpretation of her argument, not an architecture she tested or a checklist validated by the essays.
Design requirement
What it would preserve
Why it matters after verification
Persistent claim records
The claim, assumptions, evidence and relevant system versions
Later criticism can address the actual result and its dependencies
Stable responsibility
An identifiable researcher, team or institution responsible for responses
Questions and corrections have an owner after the generating session ends
A durable criticism process
Objections, replies, unresolved issues and resulting revisions
Challenges can change the research record
Traceable updates
Connections between an original claim and its corrected or restricted versions
People using an earlier result can understand what changed
These requirements separate memory from accountability. A database can preserve every output without creating any obligation to answer for it. A named agent can retain an identity without giving anyone authority to withdraw its claims or notify affected users.
For example, imagine an AI research agent produces a verified mathematical result that depends on an assumption later challenged by a collaborator. An accountability-oriented system would retain the objection, identify which claims depend on that assumption and record whether those claims were defended, narrowed or withdrawn.
A fresh session that simply generates a different answer would leave those commitments unresolved. The issue is continuity: researchers need to know which version remains supported and why.
Reputation also requires care. A score attached to an agent is not automatically the constraint Hullman describes. My inference is that the score matters only if it changes something consequential, such as whether the agent’s outputs receive additional scrutiny or can enter a trusted workflow.
The clearest implementation priority is ownership. Before adding more elaborate agent identities, I would establish who receives a challenge, who can change the record and who remains responsible when the model or service is replaced.
Preserve the Ability to Challenge AI Results
Hullman’s longer-term warning concerns human capability. If researchers progressively lose the ability to engage with AI-generated results, they may also lose the practices that develop their judgment. Her follow-up presents this as a possible trajectory, not a claim that science has already reached that point.
That creates a design tension. Automating difficult work can expand what a team accomplishes. Automating every opportunity to examine assumptions and defend conclusions could also remove the work through which researchers learn to evaluate future results.
A generic human-approval step offers little protection if the reviewer lacks the information, time or authority to intervene. The useful safeguard is a review process that makes a challenge possible.
For an AI-for-science team, that suggests three concrete questions:
Can a researcher inspect the basis of a result? Preserve assumptions, evidence and verification records in forms that support investigation.
Can an objection change the workflow? Give reviewers a way to pause downstream use, request further checks or narrow a claim.
Can a correction reach dependent work? Track where a result has been reused so revision does not stop at the original document.
These are proposed consequences of Hullman’s argument. They connect her philosophical concern to decisions a team can actually make.
The underlying tradeoff should be explicit: verification checks outputs, while accountable inquiry maintains the ability to respond when those outputs are questioned. Scientific infrastructure needs both. A system that produces more checked results but leaves nobody able to repair their interpretation has expanded output without securing the process that makes those results usable over time.
Frequently Asked Questions
3 questions
1
Does Hullman Argue Against Using AI in Science?
Her argument targets the idea that verified outputs alone are enough. It focuses on preserving criticism, accountability and human judgment as AI takes on more research work.
2
Does Accountability Require Everyone to Understand Everything?
The practical standard is distributed responsibility: the relevant people or institutions must be able to investigate a claim, explain its basis and organize a correction.
3
Is Persistent Memory Enough for an Accountable Agent?
Memory helps preserve commitments, but it does not create responsibility by itself. A functioning process also needs ownership, a response to challenges and consequences when claims require revision.
