AI data leaks: the hidden mistakes most businesses make

In the past month alone, OpenAI and Anthropic have committed data breaches, from uninstructed cyber-attacks to ‘private’ conversations becoming indexed on Google. According to AvePoint’s State of AI 2026 report, 88.4% of companies have experienced an AI agent-related security breach.

 

AI data leaks: the two most common mistakes

Permissions. Often, an AI assistant is granted access to email, files or calendars without an employee registering it. Its access then outlives the task it was installed for and puts company data at risk. This mistake usually stems from assuming the app connected to an AI agent doesn’t have sensitive data or files in it and not having asked the owner of those systems. 

 

Shadow AI. Shadow AI is a real risk, and the difficult part is that it’s invisible. As new tools appear daily, tracking what employees are experimenting with becomes difficult. A business leader may have set up AI-related controls when first introducing a tool, but it doesn’t always incorporate ways in which employees may go on to use it.

 

The big breaches are almost never one dramatic failure. They’re many small things that seem unimportant in isolation, but together let an attacker gather intelligence or hop from one system to another. Every unsanctioned AI tool holding fragments of company data is another one of those small things, as is giving an AI agent permission on a system. One might be nothing. A hundred nothings can become something.

 

Signs of an AI data breach to look out for

Here are four things to look out for that may signal the existence of a data breach: 

• Unfamiliar tools showing up in network traffic or expense reports
• Integrations or API keys no one remembers approving
• AI tools holding broader permissions than a task requires
• Company information surfacing somewhere it shouldn’t be 

How to prevent AI data leaks

Teams need to invest in monitoring capabilities and include as many important systems as possible. Monitoring technology is important, but ensuring humans are a part of that plan is essential. Employees should be trained to spot and encouraged to report any signs of unusual activity.

 

Social engineering. The vast majority of attacks involve some level of social engineering, which no technology can fully fix. AI is making impersonation cheaper and more convincing, so verification habits need to be built into the company culture. 

 

Empower employees to hang up and call back through a known channel to check whether the person on the phone really is their supplier or CEO. Ensure employees also have a clear understanding of the level of risk associated with their accounts and systems by highlighting the most important asset the organisation wants to protect.

 

Phishing. The phishing threat has industrialised. Passwords were conceived in an era when phishing was largely manual, but that era is over. The answer isn’t stronger passwords – it’s fewer passwords and in-depth defence, with authentication built on cryptographic proof rather than human memory. Keeping permissions minimal will also reduce the attack surface. 

 

Shadow AI. Providing a sanctioned AI tool that meets employees’ needs and is convenient will limit shadow AI use. Configure it to align with company policies as much as possible, and add additional security controls. Employees shouldn’t need any particular knowledge of security or encryption to be protected; it should come with the product. 

 

Oversharing. The danger with AI tools is convenience. Employees can slip into a vicious cycle of feeding AI agents incremental amounts of information until they’re sharing data they wouldn’t have shared on day one. 

 

Leaders need to interrupt this drift with clear, concrete norms about what goes into which tool, and with sanctioned tools that are secure by default, so employees don’t need a security mindset just to do their jobs safely.

 

Opportunities for things to go wrong

The more AI is used, the more opportunity there is for something to go wrong. Everyone deserves confidentiality, but some organisations have extra reasons to worry.

 

For high-risk sectors, the advice gets stricter: separate private and professional devices as much as possible, keep professional systems monitored by dedicated professionals, don’t use official email for unrelated third-party apps, and use end-to-end encrypted tools.

 

The same logic extends to any organisation holding data that a capable, motivated adversary wants. The more attractive the target, the less room there is for shadow AI and improvisation. 

 

---

 

Patricia Egger is co-founder of Women in Cyber Switzerland and Head of Security at encrypted business cloud storage solutions Proton

 

Main image courtesy of iStockPhoto.com and style-photography