Chatbots answering customers, assistants analysing contracts, agents booking appointments or updating a CRM: generative AI is moving into the tools of SMEs, mid-sized companies and large groups. It brings a new attack surface with it.
An AI is driven by language. So an attacker can talk to it too.
What risks does a chatbot or an AI agent create for a company, how can they be guarded against, and which tests should be run before going live?
No protection makes an AI system invulnerable. AI cybersecurity is about reducing the risk: fewer exploitable flaws, limited damage when one is found, faster detection.
Why does AI create new security risks?
AI creates new risks because a language model does not always tell a legitimate instruction from a booby-trapped text. A classic application executes code. A model interprets sentences, including those it reads in an e-mail, a PDF or a web page.
The risk grows with autonomy. A chatbot that makes a mistake gives a wrong answer. An agent connected to an inbox, a calendar or a customer database can send, change or delete.
The safeguards built into the models are not enough. In February 2026, the UK AI Security Institute presented a fully automated method that defeated Anthropic's and OpenAI's safety filters, which have been strengthened since. Providers patch, attackers adapt.
What are the main AI-related risks?
The main risks are listed by OWASP, a reference foundation in software security, in its Top 10 for LLM applications. The 2026 edition puts prompt injection first and moves excessive agency up to third place.
- Prompt injection: malicious instructions, typed directly or hidden in a document the AI reads, hijack its behaviour.
- Jailbreak: role play, fictional scenarios or split requests to get round the model's guardrails.
- Data leaks: the AI reveals confidential information, another user's data or its own instructions.
- Data poisoning: a booby-trapped document slipped into a knowledge base distorts the answers of a RAG system (Retrieval-Augmented Generation), the technique that makes an AI answer from the company's documents.
- Booby-trapped tools: an MCP server (Model Context Protocol), which connects an agent to tools, can hide instructions in a tool's description.
- Excessive agency: an agent with too many rights or too much autonomy turns a mistake into an incident.
- Unbounded consumption: bursts of requests drive up the bill or make the service unavailable.
In practice, a customer service chatbot can disclose another customer's history. A legal assistant can obey an instruction hidden in a contract sent by the other party. An HR agent can send a file to the wrong address. A voice agent can be manipulated by a caller posing as an executive.
Add to this a more diffuse risk, shadow AI: employees using consumer tools outside any framework.
How to protect yourself and limit the risks?
The most effective protection lies in the architecture, even before filters. Several barriers are better than one.
- Least privilege: the AI only accesses the data and actions its task requires. An agent that summarises e-mails does not need to be able to send them.
- Human validation of sensitive actions: payment, sending, deletion, contractual commitment.
- Distrust of outside content: documents, web pages and tool responses are treated as data, never as orders.
- Controlled sources for the knowledge base: who can add a document, and where does it come from?
- Logging of exchanges and actions, to understand an incident and detect it quickly.
- Caps on usage and volume.
The ANSSI, France's cybersecurity agency, takes the same line in its security recommendations for a generative AI system (in French), published in April 2024: control privileged access, log all processing, ban the automated use of AI for critical actions on the information system.
These measures do not remove the risk. They limit its consequences: an agent that has been manipulated, but has no right to send e-mails, cannot leak data through that channel.
Which security tests should be run on a chatbot or an AI agent?
The reference test is the penetration test, or pentest: deliberately attacking a system, with its owner's agreement, to find its flaws before an attacker does. The ANSSI recommends these audits before any generative AI system goes into production and distinguishes four families:
- classic penetration tests on technical components, such as web servers, the orchestrator or the database
- security tests of the code developed, with automated analysis tools
- automated tests targeting vulnerabilities specific to the models
- manual tests by auditors, on more sophisticated attack scenarios
For an AI application, a pentest therefore combines several angles:
- Direct and indirect injections: making the model ignore its instructions, or carry out an instruction hidden in a document.
- Jailbreak: putting the guardrails to the test with known techniques.
- Data leaks: checking that a user can access neither another user's data nor the system instructions.
- Knowledge base: measuring the effect of a booby-trapped document and checking write access.
- Agents and tools: testing what a manipulated agent can actually do with its rights and its MCP connectors.
- Application and infrastructure: web interface, API, authentication, servers.
The test can be run as a black box, knowing nothing about the system, like an outside attacker, or as a grey box, with a user account and part of the documentation. It is repeated after every significant change: new model, new instructions, new connected tool. Red teaming extends this logic over time, with broader attack scenarios.
How much does a pentest cost?
A pentest generally costs from a few thousand to several tens of thousands of euros, depending on the scope. There is no official price list. According to a price survey published by Yeeply in September 2026 for the French market:
- a basic web application pentest costs €3,000 to €8,000, over 3 to 5 days
- a full web pentest, across several applications, €8,000 to €25,000
- a red team exercise, €15,000 to €70,000
- a day of a senior consultant, €800 to €2,200
For an AI application, the price mainly follows the number of functions to test: a chatbot alone, a RAG system with several access levels, an agent connected to business tools. Some organisations, such as operators of vital importance or public administrations, must also use an audit provider qualified by the ANSSI, known as PASSI.
How does Gensai approach AI security?
Gensai makes sure each of its solutions is secured and compliant from the design stage, in line with the GDPR and the AI Act: minimal rights, human validation, logging. When the stakes call for it, the studio carries out a deeper analysis, up to a pentest, on its own solutions as well as on AI applications developed in-house or by another provider.
Sébastien Lecocq, co-founder of Gensai and a specialist in cybersecurity applied to AI, has obtained OpenAI's Daybreak Blue accreditation, reserved for vetted cybersecurity professionals. An equivalent accreditation with Anthropic is under way, expected in early 2027.
A well-protected AI remains exposed. It is designed so that a successful manipulation has no serious consequence.
When it comes to AI security, the useful question becomes: if this AI were fooled tomorrow, what could it do?
A chatbot or an AI agent to secure? Gensai analyses the risks of your system and tells you which tests to run. Discover the AI cybersecurity offer.
