APP 11 and AI — the security obligation nobody's applying to their own AI infrastructure

Most of the AI-privacy conversation focuses on APP 8 — the cross-border disclosure question, and the reason on-premise AI is architecturally interesting in the first place. Once a firm has moved to an on-premise system, the natural assumption is that the privacy analysis is done.

It is not. APP 11 — the security obligation — now applies to the AI system itself, and firms that move AI in-house without thinking about it can end up with a materially worse security posture than they had before. This article is a plain walk-through of what APP 11 actually requires for an on-premise AI system, and what “reasonable steps” looks like in practice.

What APP 11 says

APP 11.1 requires an APP entity that holds personal information to take reasonable steps to protect it from:

• Misuse, interference and loss; and
• Unauthorised access, modification or disclosure.

APP 11.2 requires destruction or de-identification of personal information no longer needed for any purpose for which it may lawfully be used.

The obligation is scaled — “reasonable steps” depends on the sensitivity of the information, the nature of the harm that could result, and the practicability of the safeguards. A solo GP practice handling health information faces a different specific “reasonable” than a listed corporate handling employee data. Both are subject to the same principle.

Where AI infrastructure sits in this analysis

An on-premise AI system, by construction, holds or has access to personal information — including sensitive personal information like health records, financial detail, or legal matter facts. It is a system holding personal information for APP 11 purposes.

That means the reasonable-steps analysis attaches to it in the same way it attaches to your practice management system, your document management system, and your email server.

Firms that would never leave those systems poorly secured sometimes leave an on-premise AI system in a state that would not withstand any competent security review, on the assumption that “it’s just an AI box in the cupboard”.

What “reasonable steps” looks like for on-premise AI

The following is not exhaustive, but it is a working list of what a professional firm should have in place around an on-premise AI system. The specifics scale with the sensitivity of information the system will handle.

Physical security

The system is a physical machine. It should live somewhere physically secure — a locked comms cupboard or server room, not a shelf in the reception area.

Access to the physical device should be limited to specific staff and should be logged where practicable. For firms handling particularly sensitive information (defence, health, financial), physical security requirements may be more specific — for example, ensuring the device cannot be easily removed from the building.

Network segmentation

The AI system should be network-connected in a way that isolates it appropriately from the rest of the firm’s environment and from the internet.

For a firm using the system entirely offline (a genuine strength of this architecture), it may not need internet access at all after initial installation. Removing internet access is a substantial simplification of the security surface — the device cannot be attacked over the internet if the internet does not reach it.

Where internet access is needed for updates or monitoring, it should be limited and monitored.

Authentication and access control

Access to the AI system should require authentication. This sounds obvious; a surprising number of on-premise deployments end up with the AI interface on an internal URL that anyone on the office network can reach without login.

For firms of any size, the AI interface should sit behind SSO or an equivalent authentication mechanism, with access limited to staff who need it. Different access levels may be appropriate — some staff need summarisation only, some need query access to particular document corpora, some (very few) need administrative access.

Encryption

Data at rest on the AI system should be encrypted, using standard mechanisms (LUKS on Linux, BitLocker on Windows). This does not prevent a compromised system from being read, but it prevents easy data recovery from a stolen or discarded drive.

Where the system communicates with other systems on your network, TLS should be used, and certificates managed appropriately.

Logging and audit

Interactions with the AI system should be logged in a way that lets the firm reconstruct usage if needed. For notifiable-breach purposes, the ability to say “here is what was queried on this date, by whom, involving what documents” is exactly what a firm cannot do when staff use cloud AI on personal devices — and it is what an on-premise system can uniquely provide, if configured to.

The logs themselves need to be protected. A log of AI queries is a description of what documents your firm has been working on and how; it is sensitive information in its own right.

Update management

The AI system runs software that will have security updates. Someone in the firm must own applying them. This is often the point where firms handle initial setup well and then let it drift — the model runs, no one notices, and 18 months later it is running on unpatched software.

A proper support arrangement with whoever installed the system should include update management. If you self-install, you own this.

Backup and recovery

The AI system holds configuration, models, and possibly a document corpus. All of it should be backed up. Backups themselves are personal information holdings and must be secured to at least the same standard.

Recovery from backup should be tested. A backup that has never been tested is a hope, not a control.

Staff training

Everyone who uses the system should know what it does, what it does not do, and what to do if they see something unexpected. This is not a large amount of training; it is enough to prevent the classic failure modes (staff pasting things they shouldn’t, sharing outputs to inappropriate channels, treating AI output as verified without checking).

Where firms most often fail

From what we see, the two most common failures are:

1. Treating installation as the whole job. A well-installed system three years ago, never updated, with the interface still accessible on the internal network without authentication, is a worse security position than the cloud tool it replaced. Ownership needs to continue past install day.

2. No integration with the firm’s existing security processes. The AI system should be in the same incident response plan, the same backup regime, the same access review, and the same audit cycle as everything else. If it lives in a separate world, it will drift.

Neither of these is an argument against on-premise AI. Both are arguments for treating it as a proper piece of firm infrastructure rather than a curio.

What we do about this in scoping

When we scope an on-premise system, security is part of the conversation from the start. Specifically:

• Physical location and access
• Network placement and segmentation
• Authentication and integration with existing SSO where present
• Encryption and data-at-rest handling
• Logging and audit
• Update responsibility and cadence
• Backup and recovery
• Integration with the firm’s incident response

For most firms, this is a straightforward conversation because the firm already has answers to these questions for other systems and the AI system fits into the same framework. For firms that don’t — small practices with limited IT capability — the conversation is longer and the recommendation may include supporting infrastructure changes.

We will not deliver an on-premise system into an environment where it will be a security liability. That is not caution; it is that the value proposition of on-premise AI depends on the security being real. A poorly secured on-premise system that gets compromised is a materially worse outcome than the cloud AI risk it was meant to solve.