Differential chat-template auditing for the local model supply chain

Problem

The chat template is the only piece of executable code that runs on every single inference call in a local AI stack, and nothing validates it. It is a Jinja2 program bundled inside the model artifact, sitting between the user and the model, and it is inspected by no scanner in the pipeline 6.

This produces two failure modes from one root cause.

The first is silent quality loss. A leading GGUF publisher re-uploaded the same popular model six separate times: first for a [::-1] Jinja bug that broke llama.cpp and LM Studio, then again because LM Studio disagreed with llama.cpp's template, then again for tool-calling failures, then again because mismatched BOS tokens silently disabled speculative decoding — and stated plainly that they believe other publishers' quants are still buggy 2. One of those defects was found by a Reddit user, not by any automated gate 3. The publisher's own documentation notes a tool-calling template issue that "affects all quant uploaders and types regardless where you're using it or where it's from" 4. That is an ecosystem-wide defect class discovered socially.

The second is security. Template poisoning is now demonstrated, not theorised: triggered backdoors drop factual accuracy from 90% to 15% while showing no degradation on benign inputs, induce attacker-controlled URL emission above 80% success, and hold across eighteen models, seven families, and four inference engines 8. At the agent level they hijack tool use across 3,868 episodes and bypass every injection defense the benchmarks offer 8. Crucially, the payload renders before user input is processed, which makes it architecturally unreachable by prompt-injection guardrails, and because it is valid Jinja rather than binary malware it passes hub malware and pickle scanning 712.

Why Now

Three things converged. The attack moved from theory to a demonstrated cross-engine result with poisoned artifacts that evaded all automated scans on the largest model hub, and the researchers explicitly characterise the surface as "currently undefended" 7. Separately, a security vendor's vulnerability database now prescribes the exact mitigation — "automated analysis of tokenizer.chat_template metadata to detect anomalous conditional logic, specifically if statements that check for specific substrings in user input" — and then falls back on advising users to manually eyeball the template string 11. The prescription exists; the product does not.

Meanwhile templates carry far more load than they used to. They now encode tool-call notation, reasoning delimiters, and agentic scaffolding, and every mainstream engine executes them: llama.cpp, Ollama, vLLM, SGLang 12. A template defect now breaks an agent loop rather than producing slightly odd prose.

The Opportunity

A differential conformance gate. Extract tokenizer.chat_template from a GGUF or HF repo, normalise it, diff against the canonical upstream template for that family, render both across a probe corpus (multi-turn, system prompts, tool definitions, special tokens) under each runtime's own Jinja implementation, and diff the resulting token streams. Classify divergences: benign reformatting, BOS/EOS duplication, tool-call notation drift, and the high-signal case — template logic branching on user-supplied content, which is the backdoor signature 11.

Ship as a CLI plus CI action for quant publishers, with a hub-wide scan as the launch artifact. This is interpretation, but the economics look unusually favourable: templates are small text files, rendering needs no GPU, so ecosystem-scale coverage is cheap for a small team. The technical difficulty sits in curating canonical references and triaging benign divergence, not in the analysis.

One concrete complication worth designing for: llama.cpp's json-schema-to-grammar.cpp only emits rules for JSON, so non-JSON tool notations (Gemma's fc format) fall outside grammar constraints, and a collaborator notes that over-constraining can degrade output in other ways 1. Runtime-specific quirks are documented and tractable, but they are not uniform.

Evidence Assessment

The neglectedness evidence is the strongest in the set. Two independent sources state that existing scans cannot see this: the research paper 7 and a vulnerability database that recommends manual visual inspection as the practical fallback 11. The commercial state of the art, JFrog's GGUF-SSTI check, keyword-matches for __class__, os, subprocess, eval, exec against CVE-2024-34359 10 — a semantic backdoor built from ordinary conditionals contains none of those. Community scanners like Veritensor target pickle simulation, file hashing, and license metadata in GGUF/safetensors, not Jinja logic 9. That is a consistent picture from four directions.

The demand evidence is weaker. It is real, documented, and recurring 234, but it presents as diffuse annoyance absorbed by volunteers. No willingness-to-pay signal appears anywhere in this evidence base.

The attack evidence is strong but singular — one paper 68. Hub-level compromise is separately corroborated: 100+ malicious models found despite malware, pickle, and secrets scanning 14. That establishes attacker interest and scanner blind spots generally, but not template exploitation in the wild.

Competition

Adjacent but non-overlapping. JFrog covers code-execution SSTI via keyword matching on a patched CVE 10. Hugging Face covers malware, pickle, secrets, and unsafe deserialization — and 100+ malicious models slipped through anyway 14. Sigstore model-transparency signs artifacts but attests publisher identity, not template fidelity; a re-quantizer signing their own derivative proves nothing about faithfulness to upstream, and 244 stars suggests thin adoption 13. Promptfoo maintains the vulnerability database documenting this attack and prescribing template auditing 11 — and ships no auditor. Unsloth runs 150+ KL-divergence benchmarks per family with published artifacts 5, which is genuine rigour but vendor-internal, KLD-based, and applicable only to their own uploads.

The competitive risk is absorption speed, not incumbency.

Skeptic's Case

The honest counterargument: this may be a real problem adequately handled by social mechanisms. Most people pull from a handful of reputable publishers, and that community fixes template defects reactively within days of a user noticing — which is precisely what the six-re-upload story shows 23. Formalising a process the community already converges on fast may add friction without adding much.

The security case rests on one academic paper with no confirmed in-the-wild template backdoor 68. And whoever would most naturally ship this — the hub, or an existing scanner — can add template heuristics quickly, since the analysis is easy and only the reference curation is hard.

The strongest version, though, comes from the person closest to the problem: a quant publisher would likely say the real fix is upstream authors shipping correct, tested templates, plus runtimes converging on one Jinja implementation — not a downstream scanner compensating for missing discipline. Note the Qwen3 defect originated upstream 3, which means "diverges from upstream" is not the same as "wrong."

Risks

The oracle is weak. Canonical upstream templates are themselves frequently buggy at release 3, and re-quantizers legitimately patch templates 2, so divergence-from-upstream produces false positives that require human triage and can erode trust in the tool. The user-content-conditional-branching signal is much cleaner 11 but may still fire on legitimate logic.

Security urgency is anticipatory. If no real incident arrives, adoption stalls at hygiene value alone. Monetisation is genuinely unclear — this most naturally lives as open-source infrastructure or as a feature absorbed by JFrog, a hub scanner, or Hugging Face itself, any of which could commoditise it quickly.

Verdict

Worth doing, with the honest framing that this is infrastructure rather than a business. The neglectedness case is unusually well-corroborated: the mitigation has been written down by a security vendor and left unbuilt 11, the paper's authors name the gap explicitly 7, and four adjacent tool categories all stop short of Jinja semantics 91013. The feasibility is high and cheap — text files, no GPU.

Build the user-content-conditional-branching detector first, because it is the high-signal check with the clean oracle. Treat upstream diffing as a secondary quality-hygiene layer where false positives are tolerable. The hub-wide scan is the right launch artifact: it either surfaces something real, which resolves the anticipatory-urgency risk immediately, or it demonstrates that the ecosystem is cleaner than feared — which is itself a useful result and a signal to stop.

Sources

1. Looking for the right approach for error-free tool calling · ggml-org/llama.cpp · Discussion #21839 · GitHub
2. unsloth/Qwen3-30B-A3B-GGUF · All Qwen3 uploads updated with fixed chat templates!
3. unsloth/Qwen3-30B-A3B-GGUF · All Qwen3 uploads updated with fixed chat templates!
4. Qwen3.5 GGUF Benchmarks | Unsloth Documentation
5. Qwen3.5 GGUF Benchmarks | Unsloth Documentation
6. Inference-Time Backdoors via Chat Templates: From LLM Supply Chains to Agentic System Compromise
7. Inference-Time Backdoors via Chat Templates: From LLM Supply Chains to Agentic System Compromise
8. Inference-Time Backdoors via Chat Templates: From LLM Supply Chains to Agentic System Compromise
9. I scanned 2,500 Hugging Face models for malware/issues. Here is the data | Hacker News
10. GGUF-SSTI - JFrog Security Research
11. Chat Template Hidden Instructions | LLM Security Database
12. Chat Template Hidden Instructions | LLM Security Database
13. GitHub - sigstore/model-transparency: Supply chain security for ML · GitHub
14. Malicious AI models on Hugging Face backdoor users’ machines