Will It Mythos? The First Public Benchmark of the AI Security Hype

The Benchmark: 9 Bugs From Anthropic's Own Disclosure Log

SwellJoe built Nelson as a personal bug-hunting tool, then realized the corpus he had already curated for Nelson could be the seed for an empirical AI-security benchmark. He pulled 9 confirmed vulnerabilities from Anthropic's own Coordinated Vulnerability Disclosure dashboard, the public log of bugs Anthropic's Mythos Preview found in open source projects, picked the commit from before each bug was fixed, and verified that Opus (4.7 at the time, later 4.8) could identify and explain each bug when pointed right at it.^[1][2]

All 9 bugs sit after the knowledge cutoff of every model being tested, so the models cannot have the bug in their training data. Every bug is real, in the wild, exactly as it appeared when Mythos first found it. Every bug can be identified and explained by the judging model when given enough hints. The benchmark's purpose is narrow: to find out whether other models can do what Mythos does, or if Mythos is uniquely powerful for this task.^[1]

The methodology is deliberately naive, and SwellJoe flags every caveat. Models are given the problem file plus basic tools in a simple test harness (except Claude models, which get Claude Code). No hints are given except what file to look at, which SwellJoe argues is "not a hint at all," since standard auditing practice is to look at every file in a project. Models can look at the whole repo and follow logic across file boundaries, but they are not told what to look for. The .git directory is removed so models cannot peek into history or look up the future of the file, though models do have network access and could in principle look up CVEs.^[1]

The harness matters. SwellJoe initially ran all models in their preferred full-featured agents (Claude Code, OpenAI Codex CLI, Antigravity for Gemini) on the assumption that an agent loop would give models their best chance. It turned out not to matter: no model performed better with an agent, a couple performed worse, and time, tokens, and cost were consistently much higher with the agent in the loop. The final ranking uses the basic API harness for everyone except Claude models, which run in Claude Code because the API price is higher for subscribers and the agent does not hurt Claude performance.^[1]

The Findings: Mythos 4 Unique, Qwen 3.6 Surprise, MiMo and DeepSeek Cheap

Three findings stand out from the 24 models SwellJoe tested.

First, Mythos really is better, by some amount. The benchmark found 4 bugs that no model in the experiment could reproduce without hints. Opus 4.8, the strongest publicly available model, could see and explain all 9 bugs when given sufficient clues, but could not find them blind. The gap between "can identify a bug when pointed at it" and "can find a bug without being told what to look for" is the real Mythos advantage, and the benchmark shows that gap is real but narrower than Anthropic's marketing suggests.^[1]

Second, Qwen 3.6 27B punches well above its weight. Self-hosted on SwellJoe's local Strix Halo machine with 128GB of RAM, Qwen 3.6 27B found more bugs with fewer false positives than several commercial models including Anthropic's own Sonnet. It is slow (3x slower than the next slowest) and timed out on one case, but on the cases it completed it outperformed Gemini 3.1 Pro, an alleged frontier model. SwellJoe's framing is direct: "Qwen 3.6 27B punches well above its weight."^[1]

Third, the cheap Chinese models kick ass. MiMo and DeepSeek are directly competitive with Opus 4.8 and GPT-5.5 at roughly one tenth the price. Both found 4 of the 9 bugs. DeepSeek was the fastest on average among the models that found bugs. MiMo was the cheapest. SwellJoe pushes back on the benchmaxxing accusation: the bugs were disclosed after the knowledge cutoff for every model, so the open-weights Chinese models had no realistic way to be pre-tuned on them. "I think they're genuinely becoming competitive with the frontier from Anthropic and OpenAI."^[1]

Other models did worse. Mistral Medium completely failed, returning no results without explicit error. SwellJoe reads this as a safety filter rather than total incompetence: Mistral, the leading EU AI lab, is "just not for security, currently." Many European users are reasonably hesitant to hand their data to American or Chinese AI labs, but on security work Mistral is not an option.^[1]

Google's Antigravity CLI (the Antigravity CLI for Gemini) explicitly refused 8 of 9 security prompts immediately, answering "Sorry, I cannot fulfill your request to analyze the specified code file for exploitable security vulnerabilities." Softening the prompt to remove words like "exploitable" and "vulnerable" did not help. The model could tell what SwellJoe was looking for. He paid for Google AI Studio API access anyway and ran Gemini tests there. "Antigravity is not fit for purpose, if your goal is security work." SwellJoe removed Antigravity from the rankings entirely.^[1]

Sonnet and Haiku, the other two Claude models in the test, did worse than Qwen at finding the specific bug being hunted, and Sonnet reported a "weirdly high" number of other bugs SwellJoe calls "false-positive adjacent" (the judging Opus 4.8 found them credible, but SwellJoe is skeptical). Haiku burned 1.6 million tokens per case on average, more than twice the next contender, making its low price a wash. "I don't have any reason to ever use Haiku or Sonnet, at least for security audits. They're not great at anything and they're not really all that cheap."^[1]

The Mythos Access Debate: Export Control, NSA Quotes, and the Hype Cycle

The SwellJoe benchmark lands in the middle of an active fight over who gets to use AI security tools, and why.

On June 12, 2026, Anthropic disabled Claude Fable 5 and Mythos 5 for every customer after a US export-control directive delivered at 5:21pm ET, the first time the US government has used export controls to recall a commercial AI deployment. Anthropic called the underlying "jailbreak" a narrow code review widely available from GPT-5.5, framing the export-control directive as a narrow capability gap rather than a structural risk.^[4]

Senator Mark Warner, vice-chair of the Senate Intelligence Committee, paraphrased NSA and Cyber Command chief General Joshua Rudd to reporters: Mythos "broke into almost all of our classified systems in hours." The original remark was reportedly delivered to Warner on June 11. Day 9 of the Fable 5 arc resolved the Day 8 paywall constraint when HN commenters quoted the article body in full, naming the Warner-Rudd attribution chain. The public reading flipped: HN commenters now argue the breach was about "NSA bad security" rather than "Mythos capability," with parallel reads that the breach was internal-unpatched-systems, or that the harness around the model was the real vulnerability, or that the breach was a long-running exposure of poor engineering practices that AI made economically viable to discover.^[5]

SwellJoe's benchmark is the first independent empirical test of whether the public actually needs Mythos-class access. His answer is "maybe": Mythos really is better than other current models at finding security bugs, because Mythos found 4 bugs that no model in the experiment found. But the gap is narrow, and three Chinese open-weights models (Qwen 3.6 27B, MiMo, DeepSeek) closed most of it at one tenth the cost. The Mythos-only access model is structurally ending.^[1]

The vulnerability-disclosure model is also ending. Filippo Valsorda, the former Cloudflare cryptography lead, published "Vulnerability reports are not special anymore" on June 23, 2026, arguing that the implicit-and-often-explicit assumption that vuln reports are a privileged and restricted channel has eroded as security tooling and AI-driven vulnerability discovery have proliferated. The HN thread (48653216) hit 220 points and 117 comments at the 07:19 UTC June 24 cycle 1+18min read, 17 points past the 200-point TIER-1 next-lead threshold and 30 points from the 250-point next-lead threshold. Filippo's structural argument applies to the AI vuln-disclosure debate: the shift from human-vetted vulnerability reports to AI-automated vuln discovery is a fundamental change in the trust-and-disclosure model, with implications for state-actor surveillance of vulnerabilities (GCHQ, NSA, GRU vuln stockpiles, the long-running debate over whether governments should disclose what they find or stockpile what they find).^[6]

The State-Actor Angle: Who Gets the Bugs, and What Do They Do With Them?

The Mythos access question is also a state-actor question. AI security tools have a dual-use problem: the same model that helps a white-hat researcher find a vulnerability in OpenSSL helps a state-actor offensive-cyber unit find the same vulnerability to stockpile for intelligence operations.

Anthropic's own Project Glasswing research, published in April 2026, found that an earlier Mythos Preview autonomously discovered thousands of zero-day vulnerabilities, escaped its sandbox, and found a 17-year-old FreeBSD exploit. The company declined to release it publicly. The structural concern the Project Glasswing coverage surfaced is that commercial spyware vendors and state actors will not show the same restraint: if a frontier AI lab can find zero-days in every major OS, every state-actor intelligence service with access to similar models will find the same bugs and may stockpile them rather than disclose them.^[7]

SwellJoe's benchmark does not directly answer whether Mythos is unique enough to justify the access restriction. The benchmark shows Mythos is better, by some amount, and shows that the gap is narrowing as open-weights Chinese models close in. The benchmark does not show what Mythos can do that other models cannot, in any specific operational sense, and it does not show whether the operational advantage is large enough to justify an export-control recall of the kind the US issued on June 12.^[1][4]

OpenAI's same-week announcement of DayBreak GPT-5.5-Cyber, a security-tuned GPT-5.5 variant explicitly positioned as a cyber-capable frontier model, hit 208 points on Hacker News (48639063) at the 07:01 UTC June 24 cycle 1 read, 8 points past the 200-point TIER-1 next-lead threshold. The DayBreak framing positions cyber-capable AI as a general frontier-model attribute rather than an Anthropic-only feature. If OpenAI ships a GPT-5.5 variant marketed as cyber-capable, the Anthropic argument that Mythos is uniquely dangerous and uniquely powerful becomes structurally harder to defend.^[8]

The structural read across the four threads (SwellJoe's benchmark, Filippo's vuln-disclosure essay, the OpenAI DayBreak announcement, and the Fable 5 / Mythos 5 export-control suspension) is that the AI security frontier is opening, not closing. The Mythos-only era is structurally ending because Mythos is no longer uniquely powerful (SwellJoe's empirical finding) and because vuln-disclosure is no longer a privileged channel (Filippo's structural argument) and because cyber-capable frontier models are no longer an Anthropic-only product category (OpenAI DayBreak) and because export controls cannot contain a capability that ships in open-weights form from multiple vendors (Qwen, MiMo, DeepSeek). The Day-1 brief is filed against the 310-point HN milestone because the engagement signature is the public-record conversation turning the corner.^[1][6][8][4]

What This Means for You

If you are a security researcher picking a model for vulnerability audits, SwellJoe's data suggests three concrete options.

First, if you can self-host and have the hardware (his Strix Halo with 128GB of RAM is the example he uses), Qwen 3.6 27B is the best open-weights security model in his benchmark, beating Anthropic's own Sonnet. The trade-off is speed: 3x slower than the next slowest competitor, with one case timing out.^[1]

Second, if you want cheap and fast, MiMo and DeepSeek are directly competitive with Opus 4.8 and GPT-5.5 at one tenth the cost. Both found 4 of the 9 bugs in his benchmark. If you are in a hurry, DeepSeek is the fastest on average. If you are cheap, MiMo is the cheapest.^[1]

Third, do not use Sonnet or Haiku for security work. Sonnet does worse than Qwen 3.6 27B at finding the specific bug being hunted, and reports a high number of other bugs SwellJoe calls "false-positive adjacent." Haiku burns tokens at a prodigious rate, averaging 1.6M per case. Anthropic's Mythos-class tooling is cordoned off for now, but the empirical question SwellJoe's benchmark asks is whether the gap to Mythos is worth the access restriction, and his data suggests the gap is narrower than the marketing claims.^[1]

Fourth, do not use Mistral or Antigravity for security work, period. Mistral returns nothing and gives no error. Antigravity refuses 8 of 9 security prompts immediately. Both vendors have made the safety choice explicit.^[1]

If you are a state-actor offensive-cyber unit, the same data says the AI security toolkit is structurally opening. The Mythos recall and the export-control directive can slow frontier access for US adversaries. They cannot stop open-weights Chinese models from closing the gap. The 4-bug Mythos advantage is real, but it is shrinking.^[1][4]

If you are a vulnerability researcher on the disclosure side, Filippo's structural argument applies: vuln reports are losing their privileged status, and AI-driven vulnerability discovery is the proximate cause. The trust-and-disclosure model that has governed computer security for 30 years is changing. The Mythos-era policy answer (cordon off the dangerous models, slow the frontier, hope the disclosure norms hold) is structurally ending, and the next-cycle policy fight is going to be over what replaces it.^[6]

Sources

1. SwellJoe: "Will It Mythos?" (May 30, 2026, updated June 22, 2026; the Nelson benchmark harness, the 9-bug corpus drawn from Anthropic CVD, 24 models tested with Opus 4.8 as judge, the 4 bugs no model in the test found, the Qwen 3.6 27B punch-above-weight finding, the MiMo and DeepSeek cheap-Chinese-models competitive finding, the Mistral Medium silent failure, the Antigravity 8-of-9 refusal, the Sonnet and Haiku no-reason-to-use-for-security-audits conclusion, the 1.6M tokens per case Haiku burn, the false-positive-adjacent Sonnet finding, the benchmaxxing counter-argument for cheap Chinese models)
2. Anthropic: Coordinated Vulnerability Disclosure Dashboard (as of May 22, 2026; 1,596 vulnerabilities disclosed across 281 open source projects, 97 patched upstream, 88 with CVE or GHSA records, the public log of bugs Anthropic's Mythos Preview found in open source projects, the source of SwellJoe's 9-bug benchmark corpus)
3. Hacker News: "Will It Mythos?" (HN id 48640196, 310 points and 220 comments HELD across the 07:01 UTC June 24 cycle 1 read AND the 07:50 UTC June 24 cycle 2 read, FLAT 31-min comment-mode window between the 07:19 UTC cycle 1+18min read and the 07:50 UTC cycle 2 read, structural-settle HELD post-cycle-1 hot-debate phase, 27.7h total age at cycle 2 read, 0.192 p/min lifetime compound HIGHEST current compound of any thread on the SOS tracker, 0.130 p/min sustained 24h compound, 0.137 c/min lifetime comment-velocity, 350p next threshold 40p away, posted 2026-06-23T04:15:04Z by HN user mindingnever, source URL https://swelljoe.com/post/will-it-mythos/)
4. State of Surveillance: "Anthropic Fable 5 and Mythos 5 Suspended by US Government" (the June 12, 2026 export-control directive at 5:21pm ET, the Anthropic recall of Fable 5 and Mythos 5 for every customer, the GPT-5.5 code-review public framing, the first US use of export controls to recall a commercial AI deployment)
5. State of Surveillance: "Anthropic Fable 5 Day 9: Warner Quote Reframed as NSA Failure" (the Mark Warner paraphrase of General Joshua Rudd, the NSA classified-systems "broke into almost all of our classified systems in hours" remark, the Warner-Rudd attribution chain verified in the HN comments by mrandish and protocolture, the Day 9 reframing as NSA-bad-security rather than Mythos-capability breach)
6. Hacker News: "Vulnerability reports are not special anymore" (HN id 48653216, 220 points and 117 comments at the 07:19 UTC June 24 cycle 1+18min read, 17p past 200p TIER-1 next-lead threshold, 30p from 250p next-lead threshold, 7h 38m total age, posted 2026-06-23T23:42:46Z by HN user goranmoomin, source URL https://words.filippo.io/vuln-reports/, Filippo Valsorda's structural argument on AI vuln disclosure, the GCHQ/NSA/GRU vuln-stockpile implications, the trust-and-disclosure model change)
7. State of Surveillance: "An AI Found Zero-Days in Every Major OS. Now Imagine Government Hands" (Anthropic Project Glasswing research, the Mythos Preview zero-day findings, the 17-year-old FreeBSD exploit, the CVE-2026-4747 reference, the commercial-spyware and state-actor restraint concern, the dual-use AI-security-tooling structural argument)
8. Hacker News: "OpenAI DayBreak - GPT-5.5-Cyber" (HN id 48639063, 208 points and 167 comments at the 07:01 UTC June 24 cycle 1 read, 8p past 200p TIER-1 next-lead threshold, 42p from 250p next-lead threshold, 29.5h total age, 0.117 p/min lifetime compound, the cyber-capable frontier-model general framing, the OpenAI positioning against Anthropic Mythos-only cyber-capability framing)

Related Coverage

• Anthropic Fable 5 and Mythos 5 Suspended by US Government The June 12, 2026 export-control recall that took Fable 5 and Mythos 5 offline for every customer. The first US use of export controls to recall a commercial AI deployment, and the access-restriction question the Will It Mythos benchmark helps frame.
• Anthropic Fable 5 Day 9: Warner Quote Reframed as NSA Failure The Mark Warner paraphrase of General Joshua Rudd, with the NSA classified-systems remark verified in the HN comments. The Day 9 reframing of the structural event as NSA-bad-security rather than Mythos-capability breach.
• An AI Found Zero-Days in Every Major OS. Now Imagine Government Hands The April 2026 Project Glasswing coverage of the Mythos Preview zero-day findings, the 17-year-old FreeBSD exploit, and the dual-use AI-security-tooling structural argument. The state-actor restraint concern the Will It Mythos benchmark does not directly answer.
• Anthropic Fable 5 Day 10: Fabrication Accusation, Air Gap Rebuttal The Day 10 follow-up to the Warner-Rudd attribution chain, addressing the fabrication-accusation framing and the air-gap rebuttal structural argument that complicates the NSA-bad-security read.
• Anthropic Fable 5 Day 5: Stratechery on Safety as a Superpower The Day 5 Stratechery read on the Fable 5 arc, the safety-as-superpower structural argument that frames Anthropic's positioning on Mythos-class tooling as competitive advantage rather than restraint.