The most consequential near-term use of AI in offensive security may not be a fully autonomous hacker. Research presented at Black Hat suggests that a skilled human paired with a model can discover attack concepts that neither could reach alone. That conclusion is less cinematic than an independent cyber agent, but it may be more immediately relevant to defenders.
Security researcher James Kettle tested models from Anthropic and OpenAI over several months, beginning in September 2025. He found them extremely limited at developing genuinely new attack paths without assistance. When he supplied methodology, context and judgment at key moments, however, the systems became powerful partners in exploring unusual hypotheses.
Autonomous research still breaks down
Kettle initially wanted to test whether models could perform theoretical vulnerability research. A recurring problem was false novelty: systems returned obscure existing findings as if they were original, making verification difficult. Fluency could disguise the difference between discovery, retrieval and confident error.
That limitation matters because security research depends on precise claims. A plausible explanation is not a vulnerability until someone can reproduce the behavior and demonstrate impact. Models also struggle to decide which oddity deserves further investigation when the search space becomes large.
False novelty creates a practical cost as well as an academic one. Researchers can spend hours rediscovering prior work or investigating a persuasive dead end. Retrieval systems, citations and automated checks may reduce that burden, but expert review remains necessary.
Kettle responded by encoding parts of his own research process and using models to extend it. This made the human contribution substantial. He selected the problem, shaped the reasoning and evaluated results, while the AI helped traverse possibilities faster.
One new attack surface emerged from collaboration
The work led to a concept Kettle calls Shared-Parser Confusion. It concerns web servers that use shared code to process both requests and responses, potentially creating a broad attack surface. Kettle said the AI generated the hypothesis after noticing a line in technical documentation, while he confirmed and developed the finding.
He is explicit that the model could not have completed the discovery alone. He also believes he would not have recognized the implication by himself, even if shown the same documentation. That makes the case a genuine collaboration rather than an automated replacement story.
The claim is based on Kettle’s research and conference presentation. Its wider impact will depend on technical disclosure, vendor response and whether other researchers reproduce related vulnerabilities.
Defenders should prepare for augmented attackers
Security teams often frame AI risk as a race toward autonomous exploitation. Human-guided systems lower the threshold sooner. An experienced attacker can use models to summarize code, generate hypotheses and explore variants, then intervene where judgment is required. Less capable users may also become more productive, although expertise remains important for novel work.
Defenders can use the same pattern. Models can help review unusual parser behavior, connect observations across systems and suggest tests. The operational challenge is verification: every proposed vulnerability needs evidence, and every generated fix needs review.
Organizations should therefore measure AI security tools by validated findings, not the volume of possible issues they produce. Workflows need provenance, reproducible test cases and a clear record of human decisions. Otherwise automation can bury teams beneath convincing but low-quality alerts.
Access controls remain essential during research. A model exploring attack ideas should operate against authorized targets, with clear limits on network access and disclosure procedures for any genuine flaw. Productivity does not excuse unsafe experimentation.
Kettle’s result avoids two exaggerated conclusions. It does not show that AI is incapable of original contribution, and it does not show that human researchers are obsolete. It shows that capability depends on interaction design. For now, the strongest system may be a loop in which a model notices an unexpected connection and an expert knows whether it matters.
