You wouldn’t give a new hire the keys on day one

In this article: Why AI governance matters in the skills sector, and how providers can innovate responsibly with clear oversight, auditability and human accountability. | 6 minute read.

In the rapidly evolving world of vocational training and apprenticeships, the conversation surrounding Artificial Intelligence often stalls at a single, existential question: Can we trust it?

For providers operating under the watchful eyes of Ofsted, the Department for Education (DfE), and rigorous funding rules, this anxiety is entirely rational. In a highly regulated environment, the stakes of a "bad" AI decision are not merely a poor user experience. An unmanaged AI system can not only jeopardize a learner’s progress, but trigger funding clawbacks, result in a failing inspection grade, or compromise vital safeguarding protocols.

But the framing of AI adoption as a binary choice – to trust or not to trust – is a misguided approach. Trust, in any context, is not a one-time gift nor one single, irreversible decision. It is a relationship built through oversight, boundaries, and proven performance.

Speaking at the AELP National Conference earlier this month, Bud Systems CEO John Ingram proposed a more practical lens: 

John's argument was straightforward: that is exactly how agentic AI should be deployed. Not as a system you simply ‘switch on’ and hope for the best, but as something you vet, onboard, manage, measure, and hold accountable, with the same discipline you would apply to any member of your team.

This is the blueprint for agentic AI. Not to treat it as a "black box" or a magic switch, but as a digital team member that must be managed and held to account – train it like you hired it!

The problem with treating autonomy as a switch

A significant portion of AI-related anxiety in the funded skills space stems from the misconception that autonomy is binary: either the AI is "on" and completely in control, or it is "off." This black-and-white view makes governance feel like a set of handcuffs, rather than a steering wheel.

The more effective model for the skills sector is to view AI autonomy as a dial. The position of that dial is not fixed; it should be adjusted based on the maturity of the AI system, the specific task at hand, and the potential risk to the learner or the provider.

How the "Autonomy Dial" operates in practice

• High autonomy (low-risk): Answering a learner’s curriculum-related query at 9:00 PM. If the AI is bounded by validated course content, the risk of a critical error is low, and the immediate value to the learner is high. A human tutor does not need to intervene in real-time.
• Human-in-the-Loop (moderate-risk): Gateway readiness, milestone approvals, and progress reviews. In these instances, AI acts as a sophisticated analyst, surfacing data and recommending a path forward. However, the final decision (the "sign-off") must remain the prerogative of a qualified professional.
• Evidence-driven expansion: As a system proves its reliability over thousands of interactions, its remit can gradually expand. This isn't based on "vibes," but on hard performance data that justifies a slight turn of the dial toward more independence.
• Non-negotiable boundaries (high-risk): Let’s take safeguarding. AI is exceptionally good at spotting signals – a change in tone, a concerning phrase, or a sudden drop in engagement – long before a busy tutor might notice. But the moment a signal is surfaced, the AI must hand the reins to a trained human. The final decision in safeguarding is a values-based choice, not a technical one, and that line must never move.

Governance: the foundation, not the blocker

There is a persistent myth that strong governance is the enemy of innovation. In reality, for regulated sectors, governance is the very thing that makes innovation possible.

Without a robust governance framework, AI is indefensible. If a provider cannot explain what their AI said, why it said it, and what data it used to reach a conclusion, they cannot responsibly deploy it. This is not just a matter of internal policy; it is a matter of survival during an audit or inspection.

The necessity of built-in auditability

There are currently AI products being marketed to the education and funded skills sector that lack a transparent decision-making history. In a less regulated market, that might be an acceptable limitation. In the skills sector, it is a serious liability.

When an Ofsted inspector asks, "What did your AI say to this learner, and on what basis?" the answer cannot be "We're not sure, but the algorithm is very advanced." The answer must be precise. An auditable AI architecture allows a provider to demonstrate:

• The exact content delivered to the learner.
• The specific boundary or data set the AI was drawing from.
• Whether the interaction was within approved parameters.
• If an escalation occurred, and who reviewed it.

Deciding which AI systems to adopt today is, in effect, deciding what governance posture you will be defending three years from now. If the audit trail isn't built into the architecture from day one, it cannot be effectively retrofitted later.

Addressing the hidden risk of algorithmic bias

One of the most critical, yet frequently overlooked, aspects of AI governance is the management of bias. AI systems learn by identifying patterns in historical data. In the skills sector, historical data often reflects historical inequalities: disparities in achievement, progression, and support based on background or demographics.

Left unmonitored, an AI system will not just mirror these inequalities; it will amplify them. For example, an AI used to "prioritize" tutor interventions might inadvertently deprioritize learners from specific backgrounds because the historical data suggests they are less likely to complete the course.

In a sector with a profound duty to promote equality, diversity, and inclusion, active monitoring for bias is not optional. This is a task that cannot be delegated to the AI itself, which brings us back to the management model:

• Are certain groups of learners being flagged for intervention more than others?
• Are the AI’s recommendations consistent across different demographics?
• Is the technology helping us to interrupt inequality, or is it quietly automating it?

'Controlled usefulness' and adjusting the autonomy dial

Responsible AI management requires moving beyond technical metrics like "uptime" or "response speed." To truly govern AI, providers must measure its impact on operational outcomes and human capacity.

The goal is controlled usefulness. We should be asking:

• Capacity recovery: How much meaningful time has been returned to tutors to focus on high-value coaching?
• Resolution accuracy: What percentage of queries are resolved safely within the validated boundary?
• Escalation calibration: Are the AI’s thresholds for human intervention set correctly, or is the system over-escalating (causing fatigue) or under-escalating (missing risks)?
• Learner velocity: Are learners receiving support at the moment of need, rather than waiting days for a human response to a simple administrative question?

These metrics provide the evidence needed to adjust the 'autonomy dial'. If the data shows that the AI is consistently accurate in a specific domain, the provider can confidently allow it more independence.

AI governance as a competitive advantage

The case for robust AI governance is sometimes framed as a constraint on what technology can do. John's argument at AELP National Conference was the opposite:

And in a sector where trust is the hardest thing to build and the hardest thing to replicate, providers do not need AI that sounds impressive in a sales demo but becomes indefensible in practice. They need AI that they can stand behind with absolute confidence.

Those who can demonstrate that their AI is explainable, auditable, and actively managed are the ones who will be able to move fastest. They will have the confidence of their learners, their employers, and their regulators behind them.

This will require a shift in perspective. Instead of asking if we can trust AI in the abstract, we must be confident we can manage it in the specific. Can we define its role? Can we bound its authority? Can we audit its decisions? And can we maintain human control where it matters most?

In a regulated market, this governance is not a footnote to innovation; it is the foundation, and the real power of AI in the skills sector isn't in its ability to replace human judgment, but in its ability to augment it – provided we are the ones holding the keys.