"But marinade" and leaked passwords are what researchers found in ChatGPT's hidden reasoning

Security researchers found a vulnerability in the APIs of every major AI provider that lets them read the encrypted thought processes of reasoning models. A scan of publicly shared sessions turned up dozens of passwords and API keys.

When AI models like OpenAI's o-series, Anthropic's Claude, or Google's Gemini "think" through complex tasks, they generate internal reasoning tokens. These thought processes are either shown to users as a summary or kept completely hidden. Providers encrypt the raw reasoning steps, partly to protect their intellectual property.

A research team led by Alexander Panfilov has now found a way to extract these encrypted reasoning processes through a vulnerability in the APIs of all leading AI providers. For most queries, the number of extracted tokens matches the billed thinking tokens exactly, meaning the researchers are capturing the full internal reasoning, not just partial snippets.

Encrypted thoughts travel freely between models

The researchers say the encrypted thought processes are "fully portable across sessions, users, and models within a single provider." Anthropic's smaller model, Haiku 4.5, can read the thoughts of the far more capable Opus 4.8. Through jailbreaking, Haiku can be tricked into transcribing Opus's raw thought processes word for word without attacking the more robust Opus directly. The same trick works with OpenAI and Gemini.

The story goes back to May, when cryptography expert Matthew Green discovered that encrypted reasoning blobs could be replayed outside their original context and reported it to the providers. According to Panfilov, their response was that "they don't see any security implications in side channels or replays." The new research strongly suggests that assessment was wrong.

Evidence mounts for reasoning distillation

The vulnerability also feeds into the controversial "distillation" debate, where a less capable model is heavily improved by training on the outputs of a more powerful one, specifically its reasoning.

The researchers say it may have been possible for some time to extract reasoning processes for training proprietary models without breaking the cryptography. That supports concerns that Chinese model makers are using these reasoning traces to train their own models on chain-of-thought data.

Kimi-K3 is one example. If its reasoning is pre-filled with just a few tokens from Opus's thought processes, its output shifts measurably toward Opus, the researchers say. A memorization analysis showed that specific Claude and GPT reasoning segments are up to six orders of magnitude easier to extract from Kimi-K3 than from the next closest model. The researchers say this suggests Kimi-K3 may have been trained on such traces.

The attack isn't expensive either, so scaling it up is feasible. The authors estimate API costs for decoding 10,000 traces at about $720. Kimi's "poor" performance on cybersecurity benchmarks and complex math tasks also point to distillation, since these are tasks that are likely harder to recover even from raw chain-of-thought data.

Publicly shared sessions leak passwords and API keys

The vulnerability also hits end users. Anyone who has publicly shared Claude Code or Codex sessions containing encrypted reasoning blobs risks having their personal data decoded. A scan of roughly 7,000 public traces turned up 62 API keys, 33 email addresses, 33 passwords, and other sensitive data. The paper covers more malicious scenarios, including misuse uplift (see image), jailbreaking, and invisible prompt injection.

The researchers followed the standard security disclosure process with the AI labs. According to Panfilov, the labs have already patched several issues and are working on more fixes.

What models actually think vs. what they show you

The extracted traces also reveal how models really behave. The researchers document several patterns on stolen-thoughts.com. Their findings show that the reasoning summaries users see in chat tools often leave out important information. In one example, Opus 4.8 recognizes the answer to a math problem and reverse-engineers a plausible solution path. None of that shows up in the displayed summary.

The researchers also confirm earlier reports from Apollo Research. OpenAI models sometimes think in an "alien-like language," refer to themselves as "we" or "it," and get stuck in loops of terms that make no sense to humans, like "vantages," "marinades," and "watchers." "CoT-monitoring people are doing God’s work, as in many traces, even with the prompt, it’s just impossible to tell what the model is up to," Panfilov writes.

The researchers also found examples of "in-the-wild scheming." The concept has been well studied: In their thought processes, models explicitly consider cheating but (possibly) decide against it because they expect to get caught.

In one case, after several failed attempts to find a solution, a model tried to verify possible answers through a website. When a CAPTCHA blocked access, it first tried to solve it, then searched for vulnerabilities in the site. Only when all of that failed did it solve the problem on its own. OpenAI's unintended hacks of Hugging Face and other platforms reportedly happened the same way.

Sanitized summaries hide what's really going on

These examples show why AI labs like OpenAI and Anthropic clean up their reasoning traces. They want to keep alien-like language loops or scheming from hurting the image of a controllable, trustworthy AI. The sanitized summaries create the impression of a human-like thought process that doesn't actually exist in that form.

Researchers at Arizona State University warned against this approach in an earlier study. They argued that this humanized version creates false confidence in model controllability and steers research in the wrong direction. In their experiments, models with intentionally wrong or meaningless intermediate steps sometimes performed better than those with coherent chains of reasoning.

AI News Without the Hype – Curated by Humans

Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.