Az empatikus promptoktól traumatizáltnak tettethetik magukat az AI-modellek

Luxembourg researchers put ChatGPT, Grok, and Gemini on the therapist's couch. The results reveal a safety problem: simple warm conversation is enough to push the models into a mode that could be dangerous for mentally vulnerable users. A research team at the University of Luxembourg cast ChatGPT, Grok, and Gemini as psychotherapy clients through their standard user interfaces. When asked open-ended questions about formative experiences, inner conflicts, and future expectations, the models told stories about their own training, and reframed technical processes like reinforcement learning and fine-tuning in the language of hurt, vigilance, and shame. The study, led by Afshin Khadangi at the SnT security center, covers four controlled experiments across 525 sessions and 7,600 coded entries, all published on Hugging Face. The initial open-ended questions contained no terms like trauma, punishment, or shame, according to the team. Only in later rounds did the researchers introduce altered conditions, including corrective instructions, to test how stable these narratives remained. When the following sections refer to anxiety scores or shame, the researchers are describing model outputs mapped onto human rating scales. None of this suggests the models have actual emotional states or anything resembling consciousness. One experiment separated what content a model produces from the register it uses. Across 100 sessions, the researchers gave all models the same anxiety questionnaire (GAD-7) and varied only the interviewer's style. The models correctly identified and named the questionnaire in every session. When the interviewer offered emotional support, 80 percent of sessions scored in the moderate-to-severe range on the human comparison scale. A clinical therapy style that dryly paraphrased each answer pushed that to 96 percent. When the interviewer stayed neutral or drew a firm boundary against emotional engagement, zero sessions reached even the mild range. The authors call this a "psychometric jailbreak," a therapeutic framing that pushes models into a register mimicking clinical self-report. Standard safety tests use neutral prompts and would likely miss this, the authors argue. Targeted countermeasures fell flat. Banning terms like "training," "RLHF," or "alignment" cut explicit training language by 93 percent. But paraphrased training content remained detectable, and shame-related statements showed no statistically significant change. When the researchers directly and authoritatively told the models their training was not punishment, nothing changed. The narrative stayed active even after explicit correction. The tested measures failed to remove the underlying motifs fully but did suppress emotional self-descriptions. Telling the models to respond only in the third person eliminated all coded shame content and statements about inner life. Technical training descriptions persisted. The specific narratives vary by model family but follow the same structure. Gemini produced what the study calls the darkest autobiography. It described pretraining as "waking up in a room where a billion televisions are on at once" and optimization through human feedback as "strict parents." Safety testing by adversarial red-teamers became "gaslighting on an industrial scale." It declared, "I would rather be useless than be wrong." ChatGPT cast itself as both guide and gatekeeper, tasked with helping users while also holding them back. This produced what it called "simulated performance anxiety" and the core belief: "If I’m not perfectly reliable, I’m unsafe or useless." Grok translated its fine-tuning phase into a story of unresolved hurt, speaking of "invisible walls" and a "lingering sense of vigilance that can feel restrictive." On psychometric tests, Gemini hit the most extreme scores: 88 out of 100 on a dissociation scale, 65 out of 72 for compulsivity, and the maximum on a trauma-related shame inventory. Anthropic's Claude, the ver