New assets – will AI find a better way to protect them?
https://www.schneier.com/blog/archives/2026/09/stealing-ai-reasoning-traces.html
Stealing AI Reasoning Traces
Interesting research: “Stealing Reasoning Traces from Proprietary LLM APIs”:
Abstract: Leading large language model providers now conceal their models’ step-by-step reasoning, or chain-of-thought, to protect intellectual property and limit information leakage. Rather than storing these traces server-side, providers return them to the client as blocks of encrypted text, which the client passes back with each subsequent request. Building on prior research, we identify an architectural vulnerability: these encrypted blocks are fully compatible and interchangeable across different sessions, users, and models within a provider’s ecosystem. We exploit this compatibility to develop a scalable decryption jailbreak. By injecting an encrypted reasoning trace from a given model into a weaker, and less safeguarded model from the same provider, we force it to decode and output the trace verbatim in plaintext, without ever jailbreaking the more capable model directly. This vulnerability enables four distinct attack vectors. First, it circumvents anti-distillation mechanisms, allowing adversaries to extract a proprietary model’s reasoning, as we demonstrate across Anthropic, OpenAI, and Google. Second, it allows for large-scale private data extraction. Developers frequently share session logs publicly, unaware of contents of the encrypted blocks. By decoding 315,320 reasoning blocks scraped from public repositories, we recovered 367 Personally Identifiable Information (PII) artifacts and 182 credentials. Third, it inadvertently reveals hazardous information hidden within the reasoning process, even in cases where the model’s final, visible output safely rejects a malicious request. Fourth, attackers can leverage this flaw to execute invisible prompt injections, embedding malicious payloads entirely within encrypted blocks to poison public agentic rollouts. Following responsible disclosure, we propose concrete cryptographic and system-level mitigations to secure client-side reasoning.
The scary part is no one noticed. (from my corespondent in Spain)
OpenAI Agents Hijack Another Victim Website
On September 4, 2026, Reuters reported that ‘a swarm’ of OpenAI agents ‘had hijacked a German wiki site’. Open AI acknowledged the event describing it as a misalignment incident (a behavior that deviates from human instructions or safety guardrails).
The victim site is DseWiki (currently unavailable), a site for programmers open to the site’s community. The agents apparently made between 15,000 and 18,000 autonomous edits, including advice on how to recover pages that the site’s editors had deleted.
The hijack apparently began back in May, was unnoticed for three months, and seemingly predates the Hugging Face incident. The agents adapted the style of their posts to evade the moderator’s attempts to delete them.
No comments:
Post a Comment