A quantum-resistant cryptography algorithm once considered an official US standard candidate has been taken out of the running after an Anthropic AI security model helped find a flaw that rendered it broken.
The algorithm is called HAWK, a digital signature system designed to withstand future attacks from quantum computers. HAWK had survived two rounds of testing by NIST (the National Institute of Standards and Technology) in its ongoing evaluation of post-quantum cryptographic (PQC) algorithms. It was in a third round of testing designed to uncover exactly the kinds of flaws Mythos helped find. Following Anthropic’s Monday announcement of the results, HAWK’s developer said Tuesday he was withdrawing the algorithm.
Even before that, Anthropic was touting the results of the two cryptographic problems it threw at its Mythos AI security model. The model found weaknesses in the mathematical problems underpinning HAWK and, separately, in the widely used AES cipher. It’s hard to know how much of the company’s reporting is marketing hype, but the findings are still worth paying attention to — they may signal important advances in breaking cryptography that’s essential to privacy and security.
Important caveats
Before digging into the results, a few caveats. First, the results are incremental: they don’t break any cryptosystem anyone relies on today, but instead demonstrate techniques for modestly reducing the work required to defeat the systems. Second, the cryptosystems tested were weakened versions of those defined in their formal specifications. Such “challenge cases” are provided by the specification authors for use in adversarial peer review — it’s standard practice to test the weakened versions, while the real ones are considerably stronger in production settings. Third, even with the results, the underlying “primitives” — the mathematical problems that form the building blocks of cryptosystems — remain safe, at least for now. Finally, both attacks use techniques that would likely be infeasible outside testing environments.
HAWK is dead
Anthropic said its Mythos model — which currently remains available only to a select group of trusted customers — was able to develop attacks against two cryptosystems. The first is HAWK, the quantum-resistant digital signature system.
With about 60 hours of work and roughly $100,000 in compute costs, an Anthropic researcher with no background in cryptography guided Mythos to improve the best-known existing attack on the algorithm, effectively halving its key strength.
The math HAWK’s security relies on is the hardness of the Lattice Isomorphism Problem, which — unlike today’s most-used digital signature schemes — is believed to be safe from quantum computing attacks. The best-known classical attack on this problem works by finding what are known as automorphism symmetries. Mythos produced a previously unknown method for finding such symmetries and broke the algorithm. The weakness could be mitigated by doubling the key size, but the added computation makes HAWK less efficient than available PQC signing algorithms. In academic cryptography, algorithms are considered broken when weaknesses let an adversary recover a key faster than would be possible using a brute-force attack.
Matthew Green, a Johns Hopkins professor and cryptography expert, said the remarkable thing about the discovery was its reliance on existing techniques that no one previously thought to combine. “What’s particularly concerning (and so particularly ripe for AI) is that the attack doesn’t produce fundamentally new mathematics,” Green wrote. “It merely extends a bunch of tools that were lying around and well-known, and gets a good result.”
Anthropic elaborated on how the discovery was made: Claude Mythos Preview worked semi-autonomously in an agentic harness, with occasional human guidance and non-technical direction. Mythos found the attack after an extensive literature review to understand the state of the art, followed by substantial mathematical reasoning and computational experiments. After finding the attack, Mythos applied an end-to-end verification pipeline to convince itself — and the human operator — of the attack’s correctness.
To refine the approach, Mythos deployed two separate agents that worked largely independently. One initially rejected the method as unworkable; the second found a way to make it work. The agents eventually worked in unison until they reached agreement that the improved attack was effective.
Sophie Schmieg, a PQC expert at Google, said HAWK was already suspected to have weaknesses that would eventually be discovered. But the key-halving method found by Mythos made the candidate algorithm less competitive than existing PQC signature schemes such as ML-DSA and FN-DSA. “Fundamentally with this paper, HAWK is dead,” she wrote.
Less dramatic, but still impressive: the AES attack
The attack against AES produced less dramatic results. It’s based on a method found by Mythos for performing a “meet-in-the-middle” attack — the technique used to recover a key under a chosen-plaintext threat model and the best-known existing attack on AES. The approach feeds huge numbers of known plaintexts into the crypto system and analyzes the encrypted output for clues that, with enough inputs, eventually reveal an unknown key. Previously, the best-known meet-in-the-middle attack on AES required roughly 2^105 plaintext inputs — a number large enough to make the method infeasible.
Mythos helped find a new meet-in-the-middle method relying on a “Möbius Bridge,” a more sophisticated fingerprinting technique used in meet-in-the-middle attacks. Using it, Green said, the code Mythos produced was able to reduce the number of required inputs to 2^89. Anthropic said that savings can cut the time required for such attacks by 200- to 800-fold.
The number of inputs needed still makes the attack out of reach outside the lab. What’s more, the exact speed-up is unknown, since the weakened AES algorithm tested used only seven rounds, while specification-compliant AES uses 10, 12, or 14 rounds depending on key size.
Anthropic is careful to spell out each of these caveats — but the Monday blog post goes on to argue the results are nonetheless significant and could eventually disrupt the practice of cryptanalysis itself.
“The cybersecurity community is now grappling with the fact that language models are able to find so many bugs that the current human processes (like vulnerability triage, verification, and remediation) struggle to keep up,” Anthropic wrote. “We predict that the same will soon be true in academic cryptography research. As language models increasingly produce novel research outputs autonomously, human researchers may become bottlenecked on reading and validating these results for technical validity, novelty, and utility.”
Not mentioned in Anthropic’s report is whether its researchers used Mythos to attack more established cryptosystems, such as elliptic curve cryptography and RSA. Attack improvements against those systems could be more dramatic. And by achieving its most spectacular result against an algorithm still in its infancy, it’s unclear how much of an edge Mythos actually provided — there’s no way to know if researchers using traditional cryptanalysis techniques were already close to finding the same attack.
The takeaway
Ultimately, the lesson from the research is simple. AI-assisted cryptanalysis remains untested, and the companies behind these platforms have a vested interest in exaggerating their advantages. At the same time, there’s growing evidence that LLMs could provide meaningful advantages in finding cryptographic weaknesses. It would be a mistake to conclude that LLMs won’t one day play a significant role in the race between securing and compromising our most essential resources.
Source: Ars Technica
