Risk Appetite for AI

A board approves an AI initiative. Someone asks whether it is consistent with the organisation’s risk appetite. The answer is reassuring: the firm has a low appetite for AI risk, human oversight will remain in place, and the usual controls will apply. The minutes record the approval and the meeting moves on.

I wonder what has actually been agreed. One director is worried about dependence on a handful of model providers. Another is thinking about agents acting without permission. Management hears an endorsement of its plans, provided someone checks the outputs. The same sentence accommodates all of them because it leaves the difficult questions unanswered. What is missing is an agreement about behaviour: which exposures the organisation will accept, what must remain under control, and when a fresh decision is required.

In 2011 I was principal author of the Institute of Risk Management’s guidance on risk appetite and tolerance. We wrote it because the UK Corporate Governance Code had just made boards responsible for determining “the nature and extent of the significant risks it is willing to take” in achieving their strategic objectives.

Fifteen years on, AI is the sternest test I have seen of whether boards can do that in a way that changes decisions rather than merely decorates them.

A single word is not an appetite

One distinction from the guidance is still routinely blurred. Risk appetite is the amount of risk an organisation is willing to seek or accept in pursuit of its long-term objectives. Risk tolerance sets the boundaries beyond which it is not prepared to venture. Tolerance can be expressed as absolutes, lines in the sand not crossed without the board’s approval, or as tripwires that warn of an impending breach. Appetite is more demanding: it describes what the organisation actively wants to do and how it intends to go about it. For most organisations, the appetite sits well inside the tolerance.

The guidance showed this as a fan of possible outcomes over time. The risk universe runs from where you might end up if bad things happen to where you might end up if good things do. Tolerance sets the outer limits on both sides, because some upside comes with risks the organisation would not take on. Appetite is the narrower band it actively wants to pursue.

“A low appetite for AI risk” does neither job. It is not a tolerance, because nothing in it tells anyone where to stop. It is not an appetite, because it says nothing about what the organisation is trying to achieve or which exposures are worth accepting to achieve it.

The first principle we set out was that risk appetite can be complex, and that excessive simplicity, however attractive, leads into dangerous waters. Labels such as “averse”, “cautious” or “hungry” have a place, but they cannot describe an organisation’s appetite as a whole. An organisation that is genuinely averse to risk has little future; one that is reckless can expect a rapid exit. AI gives boards every incentive to reach for the comfortable word, because the alternative means admitting how much they do not yet know.

A useful tolerance for AI looks quite different. It might say that no agent will commit funds, change a customer’s account or communicate externally beyond explicitly approved limits without a named person’s approval; that customer personal data will not be placed in tools the organisation has not approved; or that no critical service will depend on a single model provider without a tested route out. Those are statements people can act on and auditors can test.

There is more than one appetite

In all but the simplest businesses there is more than one risk appetite, and the appetites must align within a coherent framework, with a holistic view at the top. AI makes this unavoidable.

An organisation might be willing to concentrate its AI spending with one supplier because that supplier’s capabilities are particularly valuable. It might invest in an experimental agent while tightly restricting the authority delegated to it. Those are different decisions about different risks.

Concentration risk raises questions about access to models and the cost of changing suppliers. Circular financing concerns whether money moving between investors, infrastructure providers and customers supports businesses that can sustain themselves; it may matter to a firm because of the continuity of a critical service, even if it holds no investment in the supplier. Agentic AI raises questions about delegated action. Shadow AI concerns uses developing beyond the organisation’s approved arrangements.

The appetites also interact. Encouraging experimentation without providing suitable tools invites shadow AI. Restricting staff to one approved platform may improve oversight while increasing concentration. A board that sets each appetite in isolation can end up with an overall exposure nobody chose.

Why are we taking this risk?

In earlier work I argued that organisations engage with a risk for four reasons: to take more managed risk in pursuit of reward, to avoid pitfalls, because of their performance culture, and because of their ethics and behaviours. Each poses a different management challenge. Many of the large banks before 2008 leaned heavily on the first, driven by the third, and gave too little weight to the other two.

A good deal of AI adoption today is driven by performance culture: the fear of being left behind, productivity targets, competitors’ announcements. That is not a disreputable motive, but it is a different one from a considered judgement that a particular exposure is worth taking. Directors should ask which of the four reasons is actually doing the work in a given proposal. If the honest answer is “because everyone else is”, the appetite needs more thought.

A cautionary tale from before AI

While writing the guidance, we heard about an organisation that decided it was “hungry” for IT risk. It took this to mean it could relax many of the normal controls around systems development. At least two major implementations failed because basic disciplines were not followed. Most of the board resigned or were removed.

The lesson we drew was that risk appetite has two components, risk and control, and that considering either in isolation leads to poor decisions. Replace “IT” with “AI” and the story could be written next year.

A board that declares a large appetite for AI has not thereby reduced the need for control. It has changed the kind of control required. There is a useful contrast between empowering controls (high delegation, light supervision, reporting by exception) and harsh ones (detailed sign-off, re-performance, authorisation before and after the event). A substantial appetite for experimentation calls for empowering controls that still work. A low appetite for delegated action calls for tightly bounded permissions and reliable intervention. The failure, in either direction, is to set the appetite and forget the control.

What kind of knowledge are we relying on?

John Adams distinguishes between risks that are directly discernible, risks visible through science, and virtual risks, where knowledge is thin and experts cannot agree. Two points matter for AI. First, the classification depends on the organisation’s own experience: what is routine in one industry may be virtual in another. Second, organisations differ in their appetite for each kind. Some stick to what they know and avoid virtual risks; others try to bring virtual risks under managerial control.

A single AI exposure can involve all three kinds of knowledge at once.

An interruption to a supplier’s service makes dependence directly apparent. Mapping the organisation’s dependencies and testing a migration can establish more. Whether credible alternatives would remain available during a widespread disruption may still be unresolved. The board is making one decision on three kinds of knowledge.

The same applies to agentic AI. An agent taking an unauthorised action is an experience from which to learn. Testing permissions provides evidence about defined circumstances. It cannot establish how the agent will behave in every unfamiliar situation or interaction with other systems. For circular financing, analysis can show who funds whom and whether revenues depend on continuing investment; how those relationships would unravel together under stress remains a judgement. With shadow AI, a discovered disclosure shows a concrete problem and investigation can establish its extent, but the organisation may still know little about how undiscovered uses are changing the way people work.

The distinction stops an observed incident from standing in for the whole risk, and stops a successful test from being treated as the resolution of uncertainty. Where evidence cannot settle the issue, the board should be explicit about the assumptions it is prepared to act on. And where it chooses to engage with a virtual risk, it should be able to say how it intends to bring that risk under control, not merely that it hopes nothing goes wrong.

Appetite has a context

Figure 6 of the guidance places risk appetite within what we called risk capability: the combination of risk capacity, the ability to carry risk, and risk management maturity, the ability to manage it. Capacity covers financial and reputational resources, people and knowledge, and infrastructure. Maturity covers the business context, the risk management culture, processes and systems. Our conclusion was blunt: setting a risk appetite is only worthwhile if the organisation can manage risk to the level at which it is set.

For AI, financial capacity is only part of the question. People, knowledge and infrastructure may be the binding constraints. A firm may be able to afford a pilot but lack anyone able to challenge its results. It may have the budget to change suppliers but not the infrastructure to make the change. Consider GEC’s transformation into Marconi, a move into telecoms and internet businesses about which parts of the management team knew little. It is not hard to imagine boards approving AI strategies on a similar footing.

Maturity raises questions about how the organisation actually works. Does its culture allow a promising project to be challenged? Can its processes absorb an unexpected outcome? Do its systems preserve the evidence needed to investigate? Risk data deserves the same governance as accounting data. For AI, that means being able to reconstruct which model version produced an output, what instructions an agent received and what it did. Without that evidence, an organisation cannot demonstrate that it is operating within its agreed appetite.

Buying access to a capable model does not give the buyer the capability to govern it. Nor does an ambitious appetite create that capability. The board may need to invest in it before expanding the exposure.

From the boardroom to the front line

Appetite operates at three levels. At the strategic level it is mainly about the risks in which the organisation has a comparative advantage, and those from which it must protect itself. For AI, that means asking where the advantage really lies: in using models that others can also buy, or in data, customer relationships and judgement that others lack. A decision to concentrate a critical capability on one supplier is strategic, whoever makes it.

The tactical level is the cusp between strategic vision and implementation, where many strategies quietly fail. Here, choices about suppliers, architecture and delegated authority determine whether the appetite survives contact with delivery, and existing controls may need to be refined.

At the operational level the emphasis shifts towards control. One organisation we spoke to wanted front-line supervisors to respond to new risks “as though a member of the executive management team were standing at their shoulder.” That is the purpose of an operational appetite: people know how to act and when to escalate. AI raises a pointed version of the question. When the front line is an agent, what stands at its shoulder? The answer has to be permissions, escalation thresholds and stop conditions that the system actually enforces, not a statement of intent in a policy.

Risk-taking and control develop together

The propensity to take risk and the propensity to exercise control have to be considered together, and their relative weight shifts between strategy and operations.

Imagine a firm deploying an agent to handle routine customer requests. At first, staff examine every proposed action. As confidence grows, the agent receives more authority and handles more cases. Review becomes selective. That progression may be justified, but it changes the exposure. If the original approval simply required human oversight, everyone may believe the agreement still holds while understanding it differently.

Workload can change the agreement too. A reviewer who once examined each recommendation may begin accepting them routinely. Formal authority remains with the person; practical authority moves to the system. That is behavioural drift, and it increases exposure without any fresh decision about appetite.

Keith Smith’s idea of risk management clockspeed is useful here. Slow clockspeed risks, which mature over long periods, suit traditional controls. Fast clockspeed risks, which move from first sign to impact faster than internal processes are designed to respond, have to be managed through culture and well-rehearsed rules of thumb as well as process. Agentic AI can operate at a clockspeed that ordinary governance processes cannot match. An agent can take consequential actions before an ordinary escalation reaches the responsible executive. A human somewhere in the process provides little reassurance unless that person can intervene in time.

An agreement that survives use

This is why I think of risk appetite as a behavioural contract: the agreement about behaviour that the opening scene lacked. It translates the board’s intentions into decisions that people, and now systems, can make: which opportunities they may pursue, what evidence they need before expanding an activity, and when another decision is required.

Appetite must be measurable, or it becomes empty; too much risk management operates in data-light or data-free zones. At the strategic level, that means linking risks to the drivers of value, such as margin, growth and the period over which competitive advantage lasts. Concentration on a single model provider, for example, bears on margin if prices rise and on competitive advantage if a rival can buy the same capability. At tactical and operational levels, it means key risk and control indicators drawn from properly governed data.

For supplier concentration, that might mean evidence that a tested alternative remains usable. For agentic activity, it means examining what interventions achieve as well as how often they occur. Few interventions could indicate reliable operation, or oversight that no longer works.

The board governs appetite at four points: approve, measure, monitor and learn. In 2011 we were deliberately provocative about monitoring, suggesting that an appetite producing no breaches and no tensions had probably not been properly developed. The point is about challenge, not failure. An AI appetite that never exposes a tension or changes a decision may be too comfortably drawn. Breaches, when they occur, should prompt examination of both the controls and the appetite itself.

Appetite also needs consequences. A board that declares zero tolerance for customer data in unapproved tools must see that reflected in policy, in the tools actually provided, and in what happens when the line is crossed. Otherwise the statement teaches staff that the board does not mean it.

Finally, appetite changes over time. It cannot be set once and left alone for the rest of the year. An appetite appropriate to a bounded experiment may be unsuitable once the business depends on it, and a breach may signal that the appetite itself needs revisiting part-way through the cycle.

The borrowed nervous system

The guidance suggested that risk appetite is better understood through fight or flight than through the language of hunger. Organisations have no brains or nervous systems of their own; they borrow them from their boards and their people, connected by data and management information. AI is becoming part of that borrowed nervous system, sensing, deciding and sometimes acting on the organisation’s behalf. That is precisely why the board’s appetite has to be specific enough for it to follow.

A board needs a coherent set of appetites for the risks AI brings into its strategy, each reflecting what is known and what remains uncertain, and each grounded in the organisation’s capability to take and manage the exposure.

The question I would leave on the board table is this:

If you’d like to test whether your board agrees on its AI appetite, my Seven Questions take about two minutes.

Sources: Richard Anderson, principal author, Risk Appetite & Tolerance, IRM, September 2011; John Adams, Risk Management and the Limitations of Measurement. The guidance draws in turn on Keith Smith’s work on risk management clockspeed and on Black, Wright and Bachman’s shareholder value model. The AI examples are applications of these frameworks.