tldr ~ for dense decoder‑only transformers, the last‑token hidden state almost certainly identifies the input and you can invert it in practice from internal activations..