A human in the loop is not AI governance

If you ask a wealth manager how they keep their AI tools in check, the answer is usually some version of the same thing. Nothing goes to a client until a person has read it, so an adviser approves the review letter and someone in compliance looks over the onboarding file before the account is opened. It is a reassuring answer, and in most rooms it ends the conversation.

IBM published a short talk this week, Human in the Loop Is Not a Governance Strategy for Agentic AI, which questions that reassurance in a way we found useful. Its argument is that having a person review what an AI tool produces is not the same thing as having the tool under control. It runs to about four minutes and is worth watching, and what follows is how we think it applies to the firms we work with.

What the reviewer is actually looking at

When an adviser approves a review letter that an AI tool has drafted, what they actually have in front of them is the letter. They cannot see which documents the tool relied on, whether it picked up last quarter’s risk questionnaire, or why it described the client as having a “balanced” attitude to risk when the adviser clearly remembers the word “cautious”. The letter is well written and in the firm’s style, and there are usually several more waiting to be approved before the end of the day.

So what the adviser is really approving is the wording. That tells you very little about whether the tool’s reasoning was sound, and a well-written letter can make a mistake harder to spot rather than easier.

Why we trust a colleague, and what that means for a model

It helps to think about why we trust a good colleague’s judgement in the first place. We know what evidence they tend to work from, we can usually follow how they reached a view, and we know whose interests they have in mind. We also trust that they are the same careful person today that they were last week.

An AI tool needs to earn trust in much the same way. Before relying on what it says, a firm should be able to answer a few straightforward questions about it: what information it draws on, why it gave this answer for this client, whether it is still working as well as it did when it was set up, what it has been designed to prioritise, and whether anyone could have altered it without the firm noticing. Most firms have not yet asked these questions about the tools they are already using, largely because a person checking the output felt like enough.

Why this gets harder when the tools start acting

At the moment, most AI use in wealth management follows the same pattern, where the tool drafts something and a person decides whether to send it. That is starting to change. The newer tools arriving in Microsoft 365, and in the CRM and portfolio systems the sector relies on, can file documents, update client records and send emails without waiting for anyone to approve each step.

Once a tool is acting on its own, there is no longer a person checking its work before it happens. The best a firm can usually do is look back at a record of what it did, assuming that record exists and that someone takes the time to read it.

This is the point at which AI governance becomes a security question as well as a policy one. These tools generally work with the same access as the person who set them up, and in a small firm that can easily include a partner’s mailbox and every client folder on the system. It is worth knowing who is able to change the tool’s instructions, and whether there is a record of its actions kept somewhere the tool itself cannot change. These are exactly the questions we would ask about any account with that level of access, and an AI tool acting on the firm’s behalf deserves the same scrutiny.

What a person in the loop needs to have

The talk closes with a useful test. Having a person in the loop is a real safeguard only when that person has four things.

They can see how the answer was reached. The reviewer needs to see the sources and reasoning the tool relied on, rather than only the finished document.

They can stop it. The reviewer needs the authority to halt the process, in a firm where pausing to question something is treated as a normal part of the job rather than an inconvenience.

Someone checks whether the reviews are working. If an adviser has approved four hundred AI-drafted letters without changing a single one, it is fair to ask whether they are still reviewing or simply signing.

They bring what the model cannot. This is the one this sector understands better than most, which is the judgement that comes from knowing the client. An adviser who has looked after a family for fifteen years knows things that will never appear in the file, such as a son going through a difficult divorce, or a widow who finds any talk of drawing down capital distressing. That understanding is a large part of why clients value a personal adviser, and it goes to waste if the adviser’s role has been reduced to checking the spelling.

Where this sits for a regulated firm

None of this changes who is responsible. A registered adviser’s fiduciary duty does not pass to the software that drafted the letter, and an FCA-authorised firm cannot explain a poor outcome by saying the model made the decision. The person approving the output was always meant to be the evidence that the firm had taken proper care, and that evidence only holds up if they could see what they were approving and were in a position to stop it.

The question to ask on Monday

A good place to start is to choose one area where your firm already uses AI, whether that is meeting notes, review letters or onboarding checks, and speak to the person who signs off the results. Ask them what they can actually see when they approve something, and when they last decided not to.

If the honest answers are “just the finished document” and “I can’t remember”, the firm has made a reasonable start but does not yet have governance in place. That is not a bad position to be in, provided everyone understands that it is a starting point.