When AI in Customer Experience Fails, it Fails in Public

A customer contacts your AI chatbot at 11pm about a refund. The agent responds confidently,  with the wrong policy, the wrong amount, completely fabricated. By Monday morning, the interaction has been screenshotted and shared. A brand reputation that took a decade to build has been undone before your team even saw it.

 

This is no longer a hypothetical situation. AI agents are now customer-facing, acting on real-time data, often without a human in the loop. And most organisations haven’t caught up to what that means for their business.

 

 

The hidden cost of “good enough” AI

Many organisations are approaching AI deployment in CX as they would any other software rollout. They’re focused on speed and efficiency, but this mindset underestimates the unique risks of AI systems.

 

Traditional software is deterministic, but Generative AI is non-deterministic at heart. It generates outputs based on patterns in data, meaning errors can be replicated at scale. The hidden cost of a poorly tested AI delivering thousands of incorrect responses before detection is rarely captured in ROI calculations.

 

When AI mishandles complaints or mismanages loyalty schemes, the impact compounds quickly. Customers lose trust, become less willing to share data, and can disengage from the brand. This also has an impact on the effectiveness of AI, creating a negative feedback loop that can directly affect revenue.

 

 

From technical risk to business risk

Part of the challenge for organisations lies in how much these platforms now connect to, and influence, the business. Modern CX systems are integrated with CRM platforms, HR systems and operational workflows. They’ve moved on from just collecting feedback to helping determine pricing strategies, informing product development, and even influencing compensation decisions. 

 

At the same time, the data they process is uniquely sensitive. It may not always fit neatly into traditional definitions of personally identifiable information, but it often includes candid customer opinions, employee sentiment, and behavioural insights that are critical to business performance.

 

For these reasons, one of the most important shifts organisations need to make is moving from a purely technical view of risk to a business-centric one. Most companies have a clear understanding of their technical blast radius. For example, which systems are connected, where data flows and which APIs are exposed. But fewer have mapped their business blast radius – the real-world consequences if the data feeding their AI is wrong, biased or manipulated.

 

This can lead to operational risks with direct commercial implications, so there’s a need for organisations to implement continuous monitoring to detect anomalies in real time, intervene quickly, and prevent issues from scaling.

 

The regulatory exposure compounds this. With the EU AI Act now in force, organisations need to think beyond data protection frameworks alone. AI-specific compliance obligations are layered on top, and disclosure requirements are still evolving. 

 

 

Trust as the new competitive advantage

Despite the risks, the answer is not to slow down AI adoption. CX leaders are under competitive pressure to innovate and deliver more personalised, efficient experiences. This is where greater collaboration between security, CX and business teams can ensure that guardrails are in place without becoming barriers to innovation.

 

Consumers are willing to engage with AI-driven interactions, but only if they believe their data is being handled responsibly. Working together under one mission, organisations can ensure their CX platforms are building trust with customers.

 

The organisations that win will be those that treat data input integrity as a core KPI. It will be monitored continuously, not just audited at deployment. That means clear visibility into data access and permissions, robust monitoring of user activity, and automated enforcement of security policies. But it also means asking a harder question: how will we know when something is going wrong, and how fast can we intervene? The answer to that question is where competitive advantage now lives.

 

---

 

Assaf Keren is Chief Security Officer at Qualtrics

 

Main image courtesy of iStockPhoto.com and Dragon Claws