Red teaming means deliberately attacking an AI system, as an adversary would, to find its weaknesses before they are exploited: jailbreaks, prompt injections, data leaks, unintended actions by an agent. The term is borrowed from IT security, where a “red team” is authorised to emulate an attacker in order to test an organisation's defences. It is carried out before deployment, then throughout the system's life. For providers of general-purpose AI models with systemic risk, the AI Act has required documented “adversarial testing” since 2 August 2025 (Article 55), and the European Commission has been able to supervise and sanction these providers since 2 August 2026. In France, ANSSI recommends security audits before any generative AI system goes into production, including manual tests on sophisticated attack scenarios.
In practice at Gensai
In its projects, Gensai adds bypass, injection and data-leak attempts to its test sets, replayed before going live and then whenever the model, the instructions or the connected tools change.
Should a company deploying a chatbot do red teaming?
Yes, at the scale of its project. The AI Act obligation targets the providers of the largest models, but a company deploying a chatbot or an agent has every interest in checking that it cannot be made to disclose its instructions or internal data, or to trigger an unintended action. To structure the approach, OWASP published on 20 October 2026 a generative AI red teaming manual in eight phases, from planning to remediation, which also covers agents.
Common mistake
Running a single red-teaming exercise before launch and considering the matter settled. A passed test only shows that the attacks tried have failed: techniques evolve, a model's answers vary from one attempt to the next, and every new model, instruction or tool can reopen a weakness.
Same theme
Definition written with AI assistance, checked by Sébastien Lecocq, co-founder of Gensai.
Going beyond the definition?
From concept to project: Gensai builds custom, sovereign, GDPR-compliant AI solutions.
Related services: Chatbots and answer engines, Custom AI solutions