GitHub Replaces LLM Secret Checks With ModernBERT

GitHub says a fine-tuned ModernBERT classifier reduced serving costs by more than 98% compared with its previous remote-LLM pipeline for generic-secret detection. The smaller model also improved detection quality and cut median post-push scan execution time by 86%, according to the company’s implementation write-up.

Regex rules still find strings that could be credentials. The classifier examines each candidate’s context to decide whether it is likely a real secret or a harmless lookalike.

The post, published October 13, 2026, and updated October 14, describes a shipped security pipeline, not a new foundation-model launch. It offers an implementation lesson: for a narrow, repeated classification decision, a purpose-trained encoder can outperform remote generative calls while retaining the rules that already work.

All performance figures are GitHub-reported results from this particular pipeline. They are not independent evidence that smaller models universally beat LLMs.

Regex Finds Candidates; ModernBERT Judges the Context

Secret detection becomes difficult when credentials lack a distinctive format. A provider-specific token may have a recognizable prefix or checksum. A generic password or credential-like string can resemble a build identifier, asset hash, documentation placeholder, or test value.

Adding exceptions to regex rules can suppress those false positives, but exceptions can also hide genuine credentials. A string appearing in a test fixture, for example, is not automatically safe.

GitHub’s scanner separates discovery from judgment:

• Candidate generation: Regex patterns scan content broadly for possible secrets, prioritizing recall.
• Context classification: A fine-tuned ModernBERT model evaluates the candidate value alongside its file path, secret type, variable names, and nearby code or text.
• Alert decision: The scanner uses the classification score to advance likely secrets and filter likely false positives.

The model does not search the repository independently; it evaluates candidates that the deterministic stage has already surfaced. Engineers can therefore inspect whether a missed secret failed to match a candidate rule or was incorrectly suppressed by the classifier. Each failure requires a different fix.

The classifier also cannot recover a secret that candidate generation never finds. Keeping the first stage broad remains essential, even as the second stage becomes more accurate.

Context supplies evidence, not certainty. Words such as “dummy” or “replace-me” may suggest a placeholder, while an assignment beside a production database URL may suggest a genuine credential. Neither observation should become an unconditional exemption. The classifier’s job is to learn that distinction across examples without accumulating another brittle exception list.

The Task Needed a Score, Not Generated Text

GitHub chose ModernBERT for a specific output: a classification score over a bounded candidate window. A generative model can assess the same context, but natural-language output is unnecessary here. The scanner needs to decide whether to pass a finding onward. It does not need to compose an explanation or conduct an open-ended investigation.

The ModernBERT research paper describes an encoder-only transformer designed for efficient classification and retrieval, with code included in its pretraining mixture. An encoder processes the supplied input to build contextual representations, which a classification head can use to score a candidate.

GitHub says this serving approach fits its workload better than its previous LLM-based pipeline: short contextual inputs, high throughput, and low latency. Fine-tuning lets the team train directly on the distinction the product needs to make, then evaluate it using precision and recall.

The company also reports exporting the model to quantized ONNX. Quantization uses lower-precision numerical representations, which can reduce memory and computational demands. The reported savings cannot be attributed to quantization alone, however. GitHub changed the model and serving approach together, and the disclosed comparison does not isolate how much each change contributed.

The result shows that a specialized classifier was a better fit for this particular decision. It does not establish that ModernBERT has broader reasoning capabilities than the previous LLM.

Offline Gains and Production Results Measure Different Things

GitHub reports improvements in both controlled evaluation and production:

The reported precision and recall gains need to be considered together. Precision measures how many flagged findings are genuine secrets; recall measures how many genuine secrets the system catches.

A filter can improve apparent alert quality simply by suppressing more findings. Higher recall alongside higher precision suggests that GitHub’s classifier improved the boundary between the two classes on its offline evaluation, rather than merely becoming more conservative.

Production A/B testing provides a different measure. GitHub reports an 11.5% increase in repository-owner-weighted acceptance, a product-level signal about the findings reaching developers. This is not interchangeable with offline precision. Owner weighting also makes it different from simply counting accepted alerts across all repositories, and the reported increase is a percentage improvement, not a stated 11.5-percentage-point change.

The latency improvement affected where the scanner could operate. GitHub says the efficiency gain made it practical to extend generic-secret detection into push protection, where findings can be returned before a push is accepted instead of only after code has entered a repository.

The 86% figure specifically describes median post-push scan execution time. It is not a published guarantee about every push-protection interaction, end-to-end latency, or worst-case response time.

Security teams evaluating the result should keep these measures separate. Offline detection quality, developer acceptance, scan duration, and serving cost answer different questions. A credible replacement needs to perform across all four without concealing a regression elsewhere.

The Data Split Is as Important as the Model Choice

GitHub’s training strategy combines product-derived examples, high-recall regex candidates, and synthetic examples targeting sparse or difficult cases.

The company describes human validation and annotation, weak labels to expand coverage, and synthetic examples that pass deterministic validation and model-based quality review. Training emphasizes hard negatives: realistic strings that match credential patterns but are not usable secrets. Random non-secret text would make an easier, less useful training problem. The classifier must distinguish genuine credentials from the particular lookalikes that survive regex filtering.

A more consequential safeguard is how GitHub separates training and evaluation data. Before splitting, it groups examples that share a repository, candidate value, or example identity.

Without that control, a model could encounter a repeated placeholder during training and then receive credit for recognizing a near-duplicate during evaluation. Repository overlap could also expose it to related configuration conventions or copied fixtures.

GitHub’s grouped splitting approach reduces this route to inflated results, testing something closer to generalization than a random row-level split would. It does not prove coverage of every unseen credential type or repository convention.

For another security team, the actionable step is to identify its own sources of related examples before creating a holdout set. Depending on the corpus, those relationships might include shared templates, copied configurations, or repeated candidate values.

Evaluation should also check the artifact actually deployed, including quantization. A training-time model’s scores should not be assumed to transfer unchanged after optimization.

Treat the Replacement as a Pipeline Decision

GitHub had a bounded decision, contextual inputs, labeled examples, and metrics tied directly to the security product. Those conditions explain the choice; the lesson is not to replace every remote LLM call with ModernBERT.

Teams considering a similar change should first establish what the deterministic stage already does well. If regex can cheaply generate a broad candidate set, keeping it avoids asking a model to rediscover recognizable patterns. Model capacity can then focus on the ambiguous cases.

Threshold selection deserves its own evaluation. GitHub says it selects thresholds offline and validates behavior against production-like traffic before broader rollout. For another deployment, a false-positive reduction is useful only if the chosen threshold preserves the required recall, including difficult credential categories that may be uncommon in aggregate results.

A controlled rollout should separate model quality from developer behavior. Acceptance is useful feedback, but it should accompany labeled evaluation. Likewise, median execution time should sit beside tail-latency measurements when a scanner moves into a blocking workflow.

The architectural result is more persuasive than any single percentage: GitHub reports better detection while making a previously post-push capability practical before acceptance of a push. Lower serving cost helped change where the security check could operate.

A smaller model earns its place when a team can define the decision, protect the evaluation from leakage, and demonstrate that the deployed system improves the workflow. GitHub’s write-up offers a concrete example of that process, with deterministic scanning still doing the first job.