AI in Banking and Insurance: When Failure Gets Really Expensive

Introduction

Day two after go-live. Someone in marketing types an innocent question about a budget into the new AI assistant and gets the management payroll in return. Day three: an intern asks for a customer overview and receives the papers on a live acquisition as a bonus. Nobody broke in. Those files had been open for years, and these people always had technical access to them — they simply never knew, because without the assistant they would never have found the documents.

Think of a key ring that has grown for twenty years. The spare key to the archive, handed out for a project that ended in 2019. The master key someone received while still in a department they left long ago. The key to a storage room that now holds entirely different things. None of these keys was ever a problem, because nobody remembered which lock they belonged to. That changes the moment an AI assistant arrives: it does not just answer questions, it hands out the floor plan with them.

Microsoft coined a name for this — oversharing — and now builds tooling against it. The term covers exactly the case where an assistant surfaces files a person can technically open but was never meant to see. The assistant reads everything that person has access to, across file shares, mailboxes and chat channels. What used to sit in the depths of a network drive is now one query away.

In an ordinary company that is embarrassing. In a bank, an insurer or a law firm it is something else: a reportable incident, with a deadline, a regulator, awkward questions and a possible fine. The uncomfortable part is that the institutions with the strictest requirements often hold the messiest data — decades of legacy systems, mergers, duplicates and permission structures nobody fully understands any more. The place where pointing a large language model at everything is most tempting is also the place where failure costs most.

Hence four questions before any rollout. Not one of them is about the model.

Key quotes

The AI didn't break anything. In a single sentence of plain language, it simply made visible what had been going wrong for years. (translated from German)
— 00:00:45

That is the whole mechanism in one sentence. The risk was already there; it was merely unfindable — and unfindability is not a security concept.

The AI inherits your permission landscape one to one, holes included.
— 00:06:13

This is not a bug in the implementation, it is the requirement: a properly built application has to pass through the permissions of whoever is asking. That pass-through is worth exactly as much as the permissions themselves.

I can't just tip thirty years of non-anonymised claims files into the model simply because it is technically possible.
— 00:09:57

The second question is the one most often skipped, because it sounds dull. Under the GDPR, personal data may only be used for the purpose it was collected for — and an AI project is rarely that purpose.

An auditor will never ask whether your AI is clever or impressive. They will ask: this one decision — which documents is it based on, on what principles, who signed it off?
— 00:13:52

There are two possible answers to that question: a clean audit trail, or a shrug. The shrug is the expensive one.

The very first decision our client made was an organisational decision, not a model decision.
— 00:20:05

The client in question sells compliance software into regulated industries. Before anyone talked about models, the rules were set: customer data does not leave the European cloud environment, access runs through a roles and permissions concept, everything encrypted, everything traceable. The system runs in production — precisely because of that.

What you'll learn in this episode

Why the same mishap carries a different price tag under regulation — An over-broad share is an annoyance in an ordinary company. In a bank it is automatically a data protection incident that has to be reported to the supervisory authority within days. Wrong information given to a customer is unpleasant anywhere, but in a regulated setting it has legal consequences. Same technology, same error rate, entirely different cost.

How to answer question one: who is allowed to see what? — The episode walks through the usual holes: shares granted "just for this one project" and never revoked; permissions left standing after someone changed departments, which now also cover every file created since; unclear inheritance between folders. The fix is not a better model but a clean-up — validated against reality rather than against the concept document.

How to answer question two: may this data be processed this way? — Health information at insurers and pharmaceutical companies, credit data at banks, privileged client material at law firms: these holdings carry the strictest rules that exist. The episode explains why the answer is rarely a plain yes or no, but usually "yes, pseudonymised" or "yes, as an extract".

What the right to human intervention means in practice — Where an AI application makes decisions that significantly affect people — a loan refused, a policy cancelled, a claim denied — the person affected is entitled to a human stepping in at the end. Settled up front, that is sound craft. Settled afterwards, it is a court case.

Why traceability is an architectural decision — An answer without a source is not fit for approval, however confident it sounds. What is required: logging, documentation of how decisions are reached, human oversight, transparency towards those affected, and a right to an explanation in the individual case. You can build that in; you cannot bolt it on afterwards without touching the implementation in many places.

What mistakes cost under the EU AI Act — Prohibited practices: up to 35 million euros or 7 % of global annual turnover. Neglected obligations for high-risk applications: up to 15 million or 3 %. False information to the supervisory authority: 7.5 million. Plus the Air Canada case, where a customer sued for a discount the chatbot had invented — and won it.

The four-point self-test — Can anyone state with certainty who has access to which documents? May the data be used for this purpose? For a single answer, can you show without gaps what it rests on and who approved it? Is it settled how responsibility and liability are distributed? Hesitate on any of these and the problem is not the AI, it is the organisation.

Why the foundation pays off more than once — A clean permissions concept, a settled purpose limitation and a traceable architecture are not costs attributable to this one project. They carry the next application too. The homework is done once and used repeatedly.

The comfortable route is the expensive one here, and that is arithmetic rather than morality. Every hour spent up front on permissions, purpose limitation and logging is cheaper than a single reportable incident. The awkward question is not whether that work gets done, but whether it gets done before or after the first audit. And for anyone convinced their own house is different: when did you last check which keys are still out there?