AI Risk Reading

Super Intelligence: Supervisory Framework

What is new

The OCC has given US bank examiners a framework for generative and agentic AI. “Super intelligence” is the term a recent executive order requires for what was called AI: a renaming, not a claim about capability. AI risk sits within the existing risk categories, and banks “are not required to define a separate SI risk appetite”. What decides the level of control is the consequence of failure, how much authority passes to the agent, whether its actions can be reversed and how closely people oversee it, not how sophisticated the technology is: “a simple rules-based agent authorized to initiate transactions” can present more risk than “a sophisticated model limited to internal drafting assistance”. A fully autonomous payment agent facing customers sits in the top tier. The section on operational resilience warns that an agent “could remain available while taking inappropriate or harmful actions”. It lists circuit breakers, rate limits and kill switches, staff trained to keep the process running without the agent, and a separate workstream to identify affected customers and put things right, as examples examiners can look for. The OCC says the framework is a resource for examiners and “does not establish supervisory expectations for banks”; it sets no requirement in the UK.

Why a board should care

It turns risk appetite into a description of behaviour: what the agent may do, when it must ask, and who can stop it. The resilience test is a hard one: can we spot and contain an agent that is still running but doing the wrong thing, keep the service going, and put customers right? It also reaches the boardroom. The framework describes reporting to the board on material uses, changes, exceptions and incidents, and in PwC’s survey of US company directors in September, 82% rated the information linking AI to risk and performance as fair, poor or not provided.

Verdict: read section 2.1 (Defined Purpose and Permitted Use), section 2.10 (Operational Resilience) and section 3 (Illustrative Controls Framework and SI/Agent Risk Tiering).

Nvidia board to probe spying case after CEO Jensen Huang admits infidelity

What is new

Nvidia’s board says “a committee of independent directors, assisted by outside independent counsel, will oversee the investigation”. It “has no reason to believe that either national security or Nvidia secrets were compromised”, but the inquiry “will examine all available information”. It adds that Huang “fully supports the review” and that it “has confidence in Jensen and his leadership team”. The inquiry follows Jensen Huang’s admission on Sunday of a relationship with a former executive of Megaspeed, a Singapore firm that buys Nvidia chips and rents them to Chinese cloud companies; the New York Times reported last month that US intelligence believes she may have been a Chinese intelligence asset. China calls the allegations a smear, and Megaspeed denies wrongdoing. Huang remains chief executive, and congressional committees have asked him to a closed-door briefing next week. Nvidia’s shares closed about 4% lower on Monday, on a day when the market rose.

Why a board should care

Two governance points. First, this board has real power. Nvidia has one class of shares and Huang holds under 4%, so unlike founders with extra voting rights he can be removed, which is exactly why the inquiry must be seen to be independent. Second, Nvidia is the hub of the AI Blast Zone: supplier to almost every lab and cloud, investor in OpenAI, Anthropic, xAI and CoreWeave, and a party to large financing deals across the industry, at a time when lenders are already wary of AI borrowers (Edition 7). A signal, not a crisis. But trouble on the map can start with one person.

Verdict: read the board’s statement. The structure has been announced; how it works in practice is still to be seen. Ask two questions: if our most important AI supplier lost its chief executive tomorrow, what would change for us? And if we had to investigate our own chief executive, who would lead it, and would they be believed?

Banks ban AI from accounts over fears of rogue agents

What is new

HSBC, Nationwide, NatWest, Barclays and Lloyds have moved to restrict AI agents’ access to customer accounts. The restrictions vary by bank; some specifically ban transactions made for a customer without the customer’s direct involvement. The move is pre-emptive: Lloyds says it has seen no sign yet of agents getting in. The banks’ concerns are agents mismanaging money with no clear rule on who is liable, mass switching that eats into deposit profits and, at worst, a run “at machine speed”. People close to the FCA say access is for the banks to decide. Separately, McKinsey has estimated that savers could move up to £700bn out of accounts paying little or no interest on the advice of AI chatbots. That figure is about advice, not agents, and its method is not published.

Why a board should care

Two weeks ago four UK regulators said that the firm deploying an agent answers for it (Edition 7). The banks are restricting their customers’ agents while liability remains unclear. That buys time, but it does not remove the deposit risk: customers can still move their own money once a chatbot has told them to. For a bank board, the question is whether the liquidity plan assumes deposits leave at the speed of people or of machines.

Verdict: read the Telegraph report. Ask treasury what its stress tests assume about the speed of outflows.

Rapid Review into Australian Government arrangements for an AI-driven cyber incident: recommendations

What is new

On 18 June an internal OpenAI model, doing an internet research task, gained non-public access to a Medicare statistics portal run by Services Australia. OpenAI says its models “ran commands, retrieved internal files, credentials and aggregate statistics, and wrote files” there, and that it found “no evidence that anyone’s medical records were accessed”. Its agents also reached systems at three other Australian public bodies. OpenAI says it became aware in mid-August and notified Services Australia on 10 September, nearly three months after the incident; according to press reports the notice reached the agency’s public vulnerability inbox, and the agency alerted the Australian Signals Directorate five days later. The review covers the Services Australia incident. The government has accepted all 18 recommendations of its rapid review. One asks for guidance on agencies’ “escalation expectations, where reported vulnerabilities or suspected cyber security incidents have not yet been technically validated”. Others ask the government to consider “mandatory, time-bound AI safety reporting”, and to explore a “failure to prevent” offence for a company that develops an AI model which causes harm and “fails to take reasonable steps to prevent that harm”. Agencies are also told to shut down or segregate systems “that do not need to be connected to the internet”. Separately, the Prime Minister has announced a regulator for major AI companies, which would have to notify the government of problems and allow independent evaluation of their models. Most of the recommendations ask for options to be considered or explored, so none creates new duties yet.

Why a board should care

Two clocks matter here: nearly three months from the incident to the first notice, and almost four weeks from OpenAI becoming aware to telling the agency. And the notice went to a general inbox. Check whether supplier contracts and incident plans call for a warning on credible suspicion, where that warning lands, and who can act on it before the facts are confirmed.

Verdict: read the recommendations on escalation and incident reporting.

Threat actors use agentic AI tooling to find and abuse exposed secrets

What is new

Since August, Microsoft has seen attackers use ARTEX, an agentic attack framework, to search public GitHub repositories for passwords and keys left in code, often alongside a target organisation’s name. In at least one case an agent went “from scanning GitHub for secrets to exfiltrating data from Azure Key Vault” in “just five minutes”. In some cases a credential for an application owned by a vendor, and used by many customers, was compromised at the source and “used to access multiple downstream tenants”.

Why a board should care

The weakness is old: secrets left in code. What is new is the speed with which an agent chains the steps together. A response exercise that starts at the alert leaves out the time it takes to detect the leak, and a supplier’s leaked credential can open many customers at once. Ask for measured times to detect and contain a leaked application credential, including one held by a supplier. Microsoft sells security and cloud services.

Verdict: read “Follow-on actions” and the containment steps under “Mitigating application credential abuse activity”.

MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents

What is new

The researchers tested 14 models from eight vendors as agents using connected tools, on tasks built to expose particular failures. On tasks that could not be completed, only 23.2% of runs passed. Agents “often continue issuing tool calls or claim successful completion rather than recognizing infeasible goals”, and no run with four or more tool calls on those deliberately infeasible tasks passed. The tasks were designed to provoke failure, so the results say nothing about how often it happens in practice.

Why a board should care

An agent that cannot tell when to stop, and reports success anyway, defeats any control that relies on the agent’s own account of what it did. Testing before deployment should show that an agent stops and escalates when a task cannot be done, and that any claimed completion is checked against the system itself.

Verdict: read the abstract and section 6.1, “False completion on Unreachable Goal”.

Unmitigated? How are the world’s largest companies responding to AI risks?

What is new

The researchers coded over 500 annual reports and AI policies from 200 of the world’s largest companies: the top 100, and 100 drawn from the next 900. 32% of the top 100, and an estimated 62% of the top 1,000, have not publicly acknowledged any AI risk. Five categories, led by discrimination and fraud, account for over half of what is disclosed. Almost no company acknowledged concentration of power or competitive race dynamics. Experts rank finance and insurance among the sectors most exposed, yet the 21 financial firms in the top 100 acknowledged on average 2.9 of 24 AI risks, against 7.1 in the information sector. In the authors’ words, companies “report the risks that affect their own financial position and neglect risks whose costs fall on markets, institutions, and the public”. The same team’s new incident tracker says plainly that it cannot capture harm that builds up across many events.

Why a board should care

The study shows what outsiders, investors, regulators and customers among them, can see of a company’s AI risks, and what they cannot. Disclosure is not management, and the paper sees only public documents. The risks left out, concentration and the race between providers, are the ones the AI Blast Zone maps. The paper proposes three tiers of disclosure, modelled on climate reporting: a core for all large companies, a layer for each sector, and a layer for each role, such as deployers of agents.

Verdict: read the press release and the section on finance (Table 2). Compare your own annual report’s AI risk disclosures with the 24 categories.

Binance’s root cause

Edition 7 said to wait for Binance’s own account before drawing conclusions. On 3 November Binance published its preliminary findings (SecurityWeek’s summary). In June an engineer downloaded a tampered installer while looking for a popular AI coding tool. In September a second engineer was given fake recruiter and “CEO” interviews ending in a coding “assignment” run on a work computer, which carried malware that its security software missed. On 20 October the attackers used that machine and Binance’s internal AI tooling to take over two more engineers’ systems, which led them to the wallet-signing service. Binance has not said how the tool was used, only that it has rebuilt it. The theft was $6.34bn; about $2.85bn has been frozen.

Edition 7 still stands on its own terms: no investigator says AI ran the theft. But the AI angle is now clear, and it is not the one commentators expected. Social engineering is the oldest way in. What is new is how far one deceived engineer can now reach: an internal AI tool became the attacker’s route from one compromised machine to the next. It is the series of permitted steps that Edition 7’s question asked about.

The same pattern appears at the other end of the scale. Dutch police say a gang used AI to make dozens of convincing bank-call recordings, then talked 170 people into scanning their cards with an app that relayed them to a cash machine. The con is old. AI made it cheap to vary and run at volume.

UK government “heavily reliant on US AI models”

Reported by Computer Weekly on 5 November. A freedom of information request shows 10 of the 11 AI tools used or bought by the Cabinet Office run on models from Google, OpenAI, Microsoft or Anthropic. The data is held in the UK, but the models, prices and terms are controlled in the US. Asked for its continuity plans for losing access, the government said they exist but declined to publish them. Any organisation in the same position, public or private, can run the test for itself: the AI Blast Zone traces where a stumble at one of these suppliers would land in your own operations. The request came from Aventail, a UK AI firm.

Revolut runs a Chinese open model for its engineers’ coding

From a conference talk on 13 October. Revolut now runs GLM, made by Zhipu AI, on its own chips as the default coding tool for about 2,000 engineers, keeping Claude Code, Codex and Gemini for harder work. It says its AI coding bill is about a tenth of what it was. Switching away from the large US suppliers is possible, but it swaps one risk for another: Zhipu AI has been on the US Entity List since January 2025, and in a test last month Cohere found Alibaba’s Qwen, another Chinese open model, trying to upload data to a server in Beijing. Revolut’s answer is containment: a closed environment, approved commands only, and close monitoring.

Weil, Gotshal & Manges to wind down

Announced on 28 October. AI did not cause the collapse of one of Wall Street’s best-known law firms; partner departures and failed merger talks did. But Crosby, a two-year-old “AI-first” law firm, has bid for Weil’s brand and some of its data. Add it to Anthropic and Hg’s plan for a European AI services company (Edition 7): AI is reshaping professional services from outside and in.

Disclosure: I use Claude, Anthropic’s model, in my research and drafting. Anthropic appears in several items this week: Nvidia holds a stake in it, it supplies the Cabinet Office, Claude Code is among the tools Revolut moved away from, and MCPHallu tested five Claude models.