Zeniteq is now available on mobile for iPhone, iPad and Apple Watch.Download nowLearn more
VulnCheck Challenges Anthropic Glasswing Results
A dispute over vulnerability counts exposes a practical question for AI security: how much discovery turns into useful fixes, and who does the validation work?
VulnCheck researcher Patrick Garrity recently challenged the efficacy of Anthropic’s Project Glasswing, highlighting significant discrepancies between Anthropic’s reported vulnerability discoveries and actual remediation outcomes. While Anthropic claims a 94.2 percent true-positive rate for its 37,966 findings, Garrity notes that this metric relies on a subset of externally reviewed candidates. Crucially, over 63 percent of disclosed findings bypass this independent validation, potentially burdening maintainers with raw, unactionable data. This conflict underscores a vital tension in AI-driven security: high-volume discovery often creates a bottleneck, as the speed of automated reporting frequently outpaces the capacity for meaningful, verified remediation.
VulnCheck researcher Patrick Garrity challenged Anthropic’s Project Glasswing results in a November 9, 2026 analysis, arguing that duplicate findings and slow remediation undermine its vulnerability-discovery claims. His comparison says Anthropic’s ledger added 11,813 findings between September 8 and November 2 while recording only 15 additional fixes. VulnCheck’s analysis also questions how much validation work reaches open-source maintainers.
Anthropic’s own dashboard complicates that account. Its November 2 summary lists 629 findings patched upstream, substantially above the 217 marked fixed in Garrity’s analysis. It also reports a 94.2% true-positive rate, with qualifications about what that number measures.
The strongest conclusion supported by these sources concerns measurement: finding volume, report validity and completed remediation are different outcomes. Glasswing’s headline figures need those distinctions to be useful.
What VulnCheck’s Counts Actually Show
Garrity says his tracking identified 442 CVEs attributable to Anthropic over seven months. His collection draws on CVE records, GitHub Security Advisories, Linux kernel commits and Anthropic’s ledger, supplemented by manual advisory checks. It is therefore broader than a count of CVEs displayed on Anthropic’s dashboard.
He also reports that two of those 442 CVEs appeared in VulnCheck’s catalog of vulnerabilities known to be exploited. That measures exploitation identified by VulnCheck, not the potential severity or future usefulness of the remaining vulnerabilities. Treating a low observed-exploitation count as proof of low security value would go beyond the evidence. VulnCheck
The central comparisons look like this:
Measure
VulnCheck’s account
Anthropic’s November 2 dashboard
How to interpret it
Findings discovered
37,966
37,966
Both use the same headline total.
CVEs
442 attributed to Anthropic
230 CVE records listed
Broader attribution tracking and dashboard records have different coverage.
Remediation
217 marked fixed
629 patched upstream
These counts cannot safely be treated as interchangeable.
VulnCheck’s post has an internal discrepancy: its prose gives 1,654 duplicates, while its comparison table gives 1,670. Both support its broader point about duplicate volume, but the exact count needs that qualification.
More consequentially, the available pages do not reconcile 217 fixed with 629 patched upstream. Anthropic also lists 503 GitHub Security Advisories alongside its 230 CVE records, explaining that one finding can carry both identifiers. Adding advisory records is therefore not a way to count unique vulnerabilities.
I would read Garrity’s “15 additional fixes” figure as a change in the ledger category he tracked, not as a demonstrated count of every fix produced during those two months. The dashboard’s larger patch total makes that distinction essential.
The 94.2% Rate Applies to a Reviewed Subset
Anthropic’s dashboard describes a workflow involving six external security research firms. Those firms reproduce candidate issues, assess validity and severity, and prepare reports for confirmed bugs.
As of November 2, the dashboard showed:
37,966 discovered findings.
6,376 candidates reviewed by external firms.
6,009 confirmed valid, yielding the reported 94.2% true-positive rate.
5,087 findings disclosed to maintainers across 680 projects.
629 findings known to have been patched upstream.
These are Anthropic’s published figures, not an independently audited measurement of Glasswing’s effectiveness. Anthropic’s dashboard
The company explicitly includes real bugs later found to be duplicates in its validity rate. It also includes issues maintainers will not fix, such as bugs outside a project’s threat model or affecting code that is not typically reachable. Anthropic acknowledges that external reviewers can make classification errors and describes the rate as “one proxy for impact.”
That definition explains how a high true-positive rate can coexist with duplicates and relatively few patches. A finding can describe a real bug without giving a maintainer a new, actionable security problem.
VulnCheck questions whether duplicates and withdrawals are feeding back into the reported rate appropriately. But duplicate status alone cannot disprove Anthropic’s rate under its stated definition. A withdrawn report is also difficult to interpret without knowing why it was withdrawn.
My view is that the missing measurement is maintainer usefulness: the share of reports that identify unique, relevant issues worth addressing. Validity matters, but it answers a narrower question.
Most Disclosed Reports Took the Direct Route
Anthropic’s dashboard contains a particularly consequential detail: not every disclosed report receives the external review described above.
It separates its 5,087 disclosures into:
1,862 findings reported following external review.
3,225 findings reported directly by Anthropic, without the same independent check.
The direct route can include cases where maintainers specifically request untriaged findings. Anthropic warns that this route may contain false positives. Anthropic’s dashboard
Dividing 3,225 by 5,087 shows that approximately 63.4% of disclosed findings used the direct route. That is a calculation from Anthropic’s published counts, not a separate benchmark.
This is the clearest reason readers should not apply the 94.2% rate to every report arriving in a maintainer’s inbox. The rate describes externally reviewed candidates; most disclosed findings followed a different path.
Direct disclosure is not inherently a bad arrangement. A maintainer may prefer early access and have the expertise to assess a report quickly. But it changes where validation happens. An opt-in stream of raw candidates is a different service from a stream of reproduced, externally assessed vulnerabilities.
Anthropic says human triage is a rate-limiting step and that confirmed bugs can remain undisclosed because of capacity limits. Faster discovery therefore does not automatically produce faster remediation.
Maintainer Accounts Point in Different Directions
VulnCheck cites a curl sample of 34 published reports: 19 rejected as vulnerabilities, six confirmed as low severity and nine awaiting assessment. That sample supports scrutiny of report usefulness, but it was incomplete at publication and cannot establish results across other projects. The counts are presented in Garrity’s analysis.
The backlash also attracted substantial discussion. The opened Hacker News thread showed 391 points and 180 comments, a snapshot of attention rather than a representative survey.
One participant, posting as pizlonator, described receiving nine Anthropic reports concerning Fil-C and wrote: “All of them were real bugs.” The commenter praised the reports and test cases, while describing the proposed fixes as useful to varying degrees.
Other commenters cited complaints about false positives. Those accounts should remain attributed experiences, not a pooled success or failure rate.
The contrast matters. One project can receive valuable reports while another spends substantial effort rejecting findings. Differences in code, threat models, report quality and maintainer expectations can all affect the outcome. Neither account supplies a program-wide verdict.
The sources reviewed here contain no Anthropic response specifically addressing Garrity’s November 9 critique. Its dashboard provides relevant methodological explanations, but it should not be presented as a direct rebuttal.
What Maintainers Should Request Before Accepting Bulk Reports
The practical issue is whether reports save maintainers more work than they create. Based on the documented workflow, I would ask for these details before agreeing to a large submission stream:
The review route. Was each finding reproduced by an external reviewer, checked internally or forwarded without the same validation?
A minimal reproduction. Request affected versions, build options, triggering input, expected behavior and observed behavior.
Security prerequisites. Identify what an attacker controls, what access is required and which security boundary the bug crosses.
Duplicate checks. Ask whether the sender searched existing issues, advisories and recent fixes, and grouped related reports.
Separate assessments. Keep model-assigned, external-reviewer and maintainer severity ratings distinct. Anthropic’s dashboard supports these assessment categories.
Outcome and effort tracking. Record unique accepted issues, rejected reports, duplicates, patches and approximate review time. Use report-date cohorts so recent submissions are not judged against older ones that have had months to mature.
A maintainer response should not automatically count as acceptance. Anthropic says its acknowledged category includes reports maintainers have responded to; a response can begin a clarification or rejection process.
Likewise, patches are a more concrete outcome than discoveries, but they lag. Comparing newly discovered findings with all-time patches can expose a growing workload without revealing the eventual fix rate.
The useful operating measure is a combination: unique accepted issues, time to remediation and maintainer effort per accepted report. Those figures would make the tradeoff between discovery speed and validation cost much easier to judge.
Frequently Asked Questions
3 questions
1
Do Duplicate Reports Count as False Positives?
Not necessarily. A duplicate can describe a real vulnerability that was already reported. Anthropic includes such bugs in its true-positive rate, although they may add little value for maintainers.
2
Does Anthropic Review Every Report Externally?
No. Its November 2 dashboard lists 3,225 direct disclosures without the same external check, alongside 1,862 reports disclosed after external review.
3
Does VulnCheck’s Analysis Prove Glasswing Failed?
No. It identifies discrepancies and workload concerns, but the sources use different counting categories. The public evidence supports scrutiny of usefulness and remediation, not a settled verdict on Glasswing’s overall effectiveness.
