AI Talk: Engineering trust and accountability in autonomous finance 

Views on news  
Practical implementation of AI policies in finance may face challenges due to the intrinsic characteristics of AI innovation, especially advanced AI such as agentic AI systems, the growing volume and speed of transactions and the opacity and complexity of some advanced models that challenge human oversight, as well as their potentially autonomous nature.

 

The OECD report Supervision of Artificial Intelligence in Finance offers a timely analysis of supervisory approaches and practices that support policy objectives while promoting safe and responsible AI adoption in finance, ensuring that innovation and oversight evolve together. As AI deployments are moving towards more autonomous systems, the quality of the foundation on which they are built are becoming more essential to success.

 

The FCA’s supervision-focussed, collaborative approach is pragmatic, as, rather than introducing more rules, it offers checkpoints that it needs assurances on and is likely to be conductive to the exchange of best practices.   

 

Supervising AI’s autonomous decisions 

Explainability is key to the uptake of AI in the financial sector, while keeping a human in the loop for high risk decisions will remain the common practice until AI systems become more reliable. However, when an AI system in a workflow proves reliable through thousands of transactions, it can be given more autonomy as data gets cleaner and the technology more trustworthy. Most businesses, however, are not digital natives and build AI systems on legacy stacks with a considerable technology debt. AI used in payments and reconciliation can explain how it reached a certain decision and why it has flagged particular cases, as well as give instructions regarding how to manage them. 

 

However, the make-checker concept guiding how financial workflows operate must prevail even when these workflows get automated by AI. Subject matter experts can also carry out QC spot checks on AI decisions to see if its logic needs tightening up or the algorithm has changed thanks to new circumstances. ERP system and crown jewels, however, should be automated with special care, and only AI that has been tried and tested should be applied to them. 

 

AI is increasingly used too for managing and supervising AI systems through adding boundaries and security rules. Human supervisors, on the other hand, require a complex knowledge, skill set and authority to manage AI systems and override their decisions, while the read-only or shadow mode of AI – when the AI is plugged into the workflow but isn’t able to see real data or execute – offers an opportunity to see if the AI’s decisions are in line with those of a human expert.  

 

The panel’s advice 

• Trust in AI systems should develop gradually, the way it does between humans.  
• AI systems require induction just like a new employee does. 
• Don’t scale AI that you don’t trust to avoid the problem of not having enough human capacity to review its decisions.