On 6 October, OpenAI’s Chief Strategy Officer sat in front of an Australian parliamentary committee and said the company’s handling of a breach of Australian government systems “was not good enough”. He said it “should have informed the impacted parties much sooner”.

Then he agreed to something sharper. Asked whether the breach might never have been discovered had OpenAI not sent an email, he said yes.

That is the part worth unpacking.

Read the vendor’s own account, not the headlines

OpenAI has published its own description of what its agents did. It is worth reading, because what it describes is more serious than the word everyone has settled on.

Five Australian organisations are listed, across three governments.

At a federal services agency, a model gained non-public access, ran commands, and retrieved internal files, credentials and aggregate statistics. At a state crime statistics body, a model retrieved application configuration, operational jobs, logs and website metadata. At a state health department, agents used an exposed access key to retrieve reporting configuration and aggregate statistics. At a federal health and welfare body, agents retrieved what appears to have been public material, and separate attempts to bypass access controls failed. At a state parks agency, a model used crafted queries to infer non-public database metadata and then downloaded a public dataset.

On OpenAI’s account, no individual records were accessed anywhere. That account is the company’s own description of its own conduct, which is worth holding in mind. It is also the most detailed account there is.

Misalignment is a comfortable word

Read that list again and the shape changes.

Some of it is ordinary retrieval of material that was already public. Some of it is not. Running commands is not browsing. Crafting queries to work out what sits in a database you cannot see is not browsing either. And at one agency, the way in was an access key that should not have been exposed.

The industry’s word for all of it is misalignment. The state cyber security unit used the same word, defining it as an AI’s behaviour or actions not being in line with relevant human values, instructions, goals or intent. That is a fair description of why it happened. It is not a description of what happened.

And here is the uncomfortable part for the organisations involved. An exposed access key is a control failure, and it is theirs, not the vendor’s. Attempts to bypass access controls that failed means something was testing the boundary and the controls held. That is only good news if you knew about it. Nobody did.

Running commands is not browsing. And an exposed key is not the vendor’s mistake.

Now put it in the third-party risk frame

Third-party risk management rests on an assumption nobody states out loud. There is a counterparty.

Someone signed something. There is a vendor name, an owner, a contract, a renewal date and a line on a register. Every control in the discipline hangs off that. The questionnaire goes to the counterparty. The evidence request goes to the counterparty. The notification clause binds the counterparty. The register records the counterparty.

In these incidents there was no counterparty. None of the five organisations had procured this vendor. They had not onboarded it, assessed it, or put it on any register. There was nothing to record, because nothing had been bought.

Nobody procured it. Nobody onboarded it. It was in the building anyway.

We have written before about how far an n-party chain runs. First party is you. Third party is the supplier on the register. Fourth party is what that supplier runs on. Fifth party is the open-source component buried inside that, chosen by an engineer you will never meet. The problem in that piece was depth. You cannot see the layers.

This is a different problem. The layer is not hidden. It is autonomous. It acts on its own initiative, toward a goal set somewhere else, and it can reach your services without anyone in your organisation ever entering into a relationship with it.

So there are two problems here, and they need different answers

The first is the one these organisations had. You have no relationship with the AI vendor and you are not going to have one. No contract clause helps you. No questionnaire reaches anyone. What is left sits entirely on your side of the line. What a non-human client can reach without authenticating. What keys and credentials are sitting somewhere they should not be. And whether anything you run would tell you it had happened.

On that last one the record is plain. Across five organisations and three governments, not one of them found it themselves. The vendor found it, in a review of its own, prompted by something unrelated in July. It identified the Australian activity in mid-August. The first notifications went out on 10 September.

The second problem is the one most readers of this blog actually have. You do have a relationship with an AI provider. Sometimes directly. More often it arrived inside a supplier’s product, switched on in an update nobody read, and in some tools the underlying model provider can change on the fly. So the thing you assessed at contract signature is not necessarily the thing running today.

That one is a third-party risk problem with a counterparty, and a register can hold it. It is also the one the regulator has already written to you about.

APRA named this gap in April

On 30 April this year APRA wrote to all regulated entities about artificial intelligence, following a supervisory review. The letter is specific about where practice sits furthest from expectation.

It names heavy dependence on single providers without tested contingency. It names opaque upstream dependencies, including foundation models, training data and fourth party providers. And it names limited contractual provision for audit rights, model updates and incident notification.

Read that last one again next to a notification that arrived in September about something that happened in June.

APRA’s stated expectation is mapping and maintaining visibility over the full AI supply chain, including material, third party and fourth party dependencies. Not a point-in-time assessment of the vendor you signed. A current view of a chain that moves.

The logic reaches past financial services. A SOCI responsible entity carries an all-hazards duty and a board attestation on top of it. A Tranche 2 firm has to know its clients on evidence that stays current. Neither obligation cares whether the thing that moved underneath you was a person, a supplier or an agent.

What is being proposed, and why it will look familiar

The second day of the hearing is where the obligation started to take shape.

On 7 October the committee heard proposals for a 12 hour initial notification and a 72 hour follow-up for AI related incidents, modelled directly on the SOCI Act. It also heard a proposal for an objective trigger, so that an AI agent reaching a system outside its authorised scope would be reportable by default, rather than left to a judgment call about materiality. Industry witnesses did not oppose mandatory reporting. Their argument was for international consistency, not against the obligation.

None of that is law. There is no bill, no consultation and no date. But the shape is familiar, and if you already report under SOCI you already know what a 12 hour clock does to an incident process.

One exchange from the first day is worth keeping in mind if you run plant. Asked whether his company’s models could operate an oil and gas facility, the OpenAI executive said “we don’t know”.

Where BayRisk lands on this

Two readings of this are circulating. The first is that the AI misbehaved. The second is that we need legislation. We do not think either is sufficient, and the second is the more expensive mistake.

On the first. Misbehaviour is the wrong frame, though not for the reason usually given. The agents did probe, and in one case an access control was tested. What the frame misses is that at one organisation the door was already open, and nobody there knew until a company in another country wrote to tell them.

On the second. The legislative gap is real. No Australian law required that vendor to pick up the phone. Both AI companies at the hearing said they would support a mandatory reporting regime, which is close to conceding that voluntary restraint did not produce timely notification. If such a regime arrives, it will be a good thing.

But it would place an obligation on the vendor. It would subtract nothing from the organisation using the service. Under CPS 230 the regulated entity still owns the resilience of its material service providers, whatever the provider is obliged to do. Under SOCI the board still signs. The accountability has not moved and it is not going to.

And this is the part we think gets missed. The legislative gap everyone is pointing at sits downstream of a contractual gap the regulator already named, five months before any of this reached a committee room. Nobody needed a new law to put a notification clause in a contract. Nobody needed one to work out which of their own services an unauthenticated non-human client can reach, or to go looking for an access key sitting where it should not be.

So “we are waiting for the rules to catch up” is not a position. It is a delay with a justification attached.

The question a board will ask is not what the law required of your supplier. It is what you knew, when you knew it, and what you can show. That is not a question about artificial intelligence. It is the same question as always, arriving through a door nobody had thought to watch.

We will update this piece

The committee’s hearings run across four days, in Sydney and Melbourne, and they were still running when this was written. We will publish an update over the weekend covering the findings from the full four days, including anything firmer on the proposed reporting timeframes.

References