Why Should AI Governance professionals & Tech Lawyers Care About AI Safety?— PART I

An intro to AI Safety research concepts and their potential to improve the regulatory landscape

We are in the middle of an AI literacy boom.

In the past year, we’ve seen a lot of courses emerge to teach non-tech profiles how AI works: what is a transformer, what is a neural net, what are the use cases of generative AI in law, finance, and public administration.

They also usually teach the basics of bias and discrimination risks, and even talk about data protection and cybersecurity risks.

And yet (ironically and tragically) the subfields of AI that matter most to making systems safe and governable remain almost entirely absent from these curricula¹

This Series aims to correct that.

This is PART 1 out of 4 of the series “Why Should AI Governance professionals & Tech Lawyers Care About AI Safety?”.

Is this Series for you?

Read on if any of the following sound like you:

• You work in a corporate AI Governance or Responsible AI² team at a tech company or consultancy.

• You’re a lawyer working in AI Governance, or advising clients on compliance with the EU AI Act.

• You sit in Product Policy, External Affairs, or Public Relations, but increasingly find yourself addressing AI-related risks.

• You’re an ML engineer or AI Safety researcher curious about how legal and compliance teams experience these challenges.

While the core ideas are relevant across jurisdictions, I’m writing primarily from the perspective of European regulatory frameworks, especially the EU AI Act and the GDPR.
That means this series will be most useful if you’re:

• Based in the EU

• Or advising clients who must comply with European regulation.

Basically:

If you’ve ever tried to make a foundation model “compliant” and felt like something deeper was missing, this series is for you.

But, what even is “AI Safety”?

While not always defined this way in formal literature, AI Safety³ is often operationally broken down into three overlapping areas:

Alignment, Interpretability and Control.

This structure reflects the research priorities of leading labs such as OpenAI, ARC, Google Deepmind, Anthropic, Redwood Research, or Apollo Research

In very broad terms:

• Alignment aims to ensure that a model’s behavior reliably reflects human goals, values, or instructions, even in novel or ambiguous situations.

  • Even though most relevant alignment research is currently focusing on Large Language Models, alignment is not exclusive to LLMs (more on this in Part 2!).

  • The EU AI Act mentions alignment explicitly in Recital 110, recognizing alignment failures (“alignment with human intent”) as a source of systemic risks.

• Interpretability refers to our ability to understand why a model made a particular decision, by analyzing its internal reasoning or representations.

  • This includes a subfield called mechanistic interpretability, which tries to uncover the specific components inside the model that are responsible for certain behaviors, almost like tracing the “thought process” behind its outputs.

• Control focuses on ensuring that humans can intervene, correct, or shut down AI systems when needed, especially in high-stakes or autonomous settings.

  • Although this area of research is relatively new⁴, Recital 110 of the EU AI Act has made it extremely relevant to alignment failures as a systemic risk. See its wording: “International approaches have so far identified the need to pay attention to risks from […] unintended issues of control relating to alignment with human intent…”

BlueDot impact defines AI Safety as:

“The field working to ensure powerful AI systems benefit rather than harm humanity.

As AI systems become more capable of affecting our world, we need to solve key challenges:

• How do we make AI systems reliably do what we want?

• How do we govern their development?

• How do we define what "beneficial" means?”

But, wait….

Isn’t that… very similar to what AI Governance is trying to do, too?

Why, then, are these disciplines⁵ so neglected in AI governance and regulatory training?

Why are we teaching policy folks and tech lawyers about transformer architectures, but not about why those architectures misbehave, or how internal model behaviors create legal risks?

We see the words “bias” and “hallucinations” everywhere, coupled with “transparency” and “explainability”, usually concluding that said issues are why “human oversight” is so important.

But… how do we get from defining these issues to solving them?

This, to me, is the fundamental problem.

The Systems vs. Models divide

An AI model is the core mathematical engine trained to recognize patterns and generate outputs (e.g. GPT-4).

An AI system includes the model plus everything around it: data pipelines, APIs, user interfaces, deployment infrastructure, and human oversight mechanisms.

AI Safety is mostly model-focused. In legal and regulatory contexts, it’s the AI system that is subject to the strictest compliance obligations. But many of those obligations depend on how the underlying model behaves.

And the most unpredictable and high-impact risks often arise from the model’s internal behavior: how it reasons, generalizes, or fails.

That’s why I argue:

To govern AI systems in a meaningful & compliant way, we need to understand model behavior. And that’s where AI Safety comes in.

Alignment and interpretability failures already affect you: You just didn’t have a name for it

Does the below sound familiar?

• Your job is to make sure models are safe, not just smart.

• You work for a Tech company (or have clients in the AI development sector) who wants you to “get them compliant”, but just enough to avoid fines.

• You’ve reviewed policies, drafted risk disclosures, overseen robustness tests, maybe even commissioned adversarial red-teaming.

• You’ve reviewed development and deployment policies through the lens of the EU AI Act, GDPR and every internal governance doc that claims to cover transparency, oversight, and risk management.

• If your company or the clients you advise are using OpenAI, Google, or Anthropic APIs (and, who isn’t?): you’ve read those model cards like 100 times already.

And still, no matter how thorough the audit, the model still behaves in ways you didn’t anticipate.

• You’ve consulted with machine learning engineers to understand the origin of certain erratic or risky behaviors, and they explain that these failures aren’t all rooted in your system or your product’s architecture: They stem from the foundation model itself, which your application accesses via API, and perhaps from incompatibilities between the model’s general-purpose training and your domain-specific use case.⁶.

• You flag the risks. You collaborate with ML or security advisors to find mitigation strategies. You add transparency warnings about the model’s “possible harmful behaviors.”

• From the legal side, you document the risk in internal and external policies, preparing for potential user complaints, and evaluating whether the identified behavior violates regulatory thresholds (especially the AI Act’s obligations if you’re in Europe).

But here’s the thing: you still feel powerless⁷.

Because there’s only so much of the model (or the architecture behind it…) that you can actually fully access and explain⁸.

That’s not a failure of your diligence.

The reality is that a big part of this is, simply, misalignment and lack of mechanistic interpretability: two inherent characteristics of the foundation models most enterprises rely on today, often via API integrations, to build their own products and services.

You’ve been dealing with alignment and interpretability issues⁹ this whole time. You just didn’t know that’s what they were.

And until we close the gap between regulatory tooling and alignment & interpretability research, that’s going to keep happening.

Basic definitions & how they relate to the problems you know

As a starting point, let’s walk through the issues most tech lawyers and governance professionals are trained to spot, and show how each one connects to deeper, often invisible dynamics in alignment and interpretability.

Because what we call “compliance risks” on the surface (bias, hallucinations or lack of transparency) are often symptoms of deeper alignment failures inside the model.

1. Bias

The legal concern: The model produces outputs that result in unfair, unequal, or disproportionate treatment of individuals or groups (particularly across protected characteristics like gender, race, or age).

This may lead to discriminatory outcomes in areas such as hiring, lending, access to services, or content moderation, potentially violating laws like the GDPR, national non-discrimination laws, or the AI Act's fairness and risk management provisions.

What bias actually is:
A model learns patterns from its training data that lead it to favor some groups or outcomes over others, even when that wasn’t explicitly intended.

Why this happens:

• Models are trained on huge internet-scale datasets full of human biases such as social or historical inequalities, which they often absorb.

• They learn internal shortcuts (like correlations and associations) that often reflect stereotypes or imbalanced patterns.

• These internal behaviors can’t be seen from the outside without specialized tools.

How AI Safety tackles this¹⁰:

• Alignment: Training models to prefer responses that reflect fairness or ethical norms, even when biased patterns exist in the training data.

• Mechanistic Interpretability: Finding which parts of the model encode certain concepts (like gender or profession) and how those are influencing outputs.

2. Lack of Transparency in Decision Making

The legal concern:
An AI system makes a decision (like denying a loan, flagging a user, or recommending action) but it can’t provide a clear, meaningful explanation that meets legal requirements for transparency. For example, as required by Article 22 of the GDPR.

What the problem really is:
The model inside the system produces outputs through complex, opaque reasoning. These decisions may appear rational, but the system can’t trace why the model behaved that way, making meaningful oversight difficult or impossible.

Why this happens:

• Large language models are not rule-based, they operate by predicting what comes next in a sequence, based on patterns learned during training.

• The model may lack internal mechanisms for reflecting on or explaining why it produced a specific output.

• Even when an answer sounds reasonable, we have no visibility into which internal representations, concepts, or associations were used to get there.

How AI Safety tackles this¹¹:

• Interpretability: Building tools that show what each part of the model is doing, like which “neurons” activate for certain concepts.

• Alignment via reasoning scaffolds: Encouraging models to reason step-by-step, making their thinking easier to follow and interpret.

3. Hallucinations (or Confabulations)

The legal concern:
Models provide plausible-sounding but factually incorrect information. Or they invent things like false legal citations, incorrect medical facts, or misleading policy advice. This creates serious reputational, legal, and safety risks.

What the problem really is:
Models are trained to optimize for linguistic plausibility, not factual accuracy. They learns statistical patterns from training data without an internal mechanism to distinguish between true and false information.

Why this happens:

• Models don’t “know” facts, they just predict what text is likely to come next.

• They lack grounded knowledge or memory. Most models don’t have persistent internal representations of facts or access to structured databases unless explicitly augmented.

• When faced with unfamiliar prompts, especially out-of-distribution questions, they fill in the blanks by sounding right, not by being right.

• Fine-tuning (e.g., with RLHF) can reduce hallucinations, but it often improves confidence and fluency more than actual truthfulness.

• There’s no internal fact-checking mechanism unless one is explicitly added through retrieval systems, rule-based oversight, or external verification scaffolds.

How AI Safety tackles this¹²:

• Alignment: Fine-tuning the model to prioritize truthfulness over fluency, including the ability to say “I don’t know,” decline to answer when uncertain, or defer to human judgment when appropriate.

• Interpretability: Studying how internal circuits generalize across prompts, revealing how seemingly reliable behaviors can break under slight variations. Hallucinations often emerge when models apply learned patterns to contexts where they no longer hold.

4. Jailbreaking & Adversarial Prompts

The legal concern:
Users can trick the model into saying things it’s not supposed to, like hate speech, illegal advice, or sensitive company information. This poses both safety and compliance risks.

What the problem actually is:
Models are sensitive to phrasing and can be manipulated into ignoring alignment constraints if the right prompt "breaks the frame." This reveals vulnerabilities in how aligned behaviors are learned and maintained.

Why this happens:

• Guardrails are often added after the model has learned unsafe capabilities.

• These capabilities still live inside the model and can be reactivated.

• Fine-tuning can suppress (but not fully remove) harmful instructions.

How AI Safety tackles this¹³:

• Alignment: Testing models with adversarial prompts helps evaluate whether aligned behavior is robust or superficial. Whether the model actually internalized the safety goal, or is just mimicking it.

• Interpretability: Tools like Sparse Autoencoders reveal internal “features” activated during potentially unsafe behavior, enabling researchers not just to detect hidden model objectives, but to test whether those objectives are driving the output. This allows for causal auditing, not just surface-level inspection.

Why This Series Exists

I’m writing this four-part series to break the deadlock. Because frankly, professionals in AI governance and in AI safety are trying to solve the same problem:

Preventing AI from harming people.

A call to Action

If you work in corporate AI Governance or Compliance, and you’ve never seen concepts like alignment or mechanistic interpretability in trainings or “AI literacy” resources you’ve completed, please:

• Share this blog with your teams.

• Ask your doubts in the comments.

• Keep these concepts in mind the next time you’re handed an AI training, especially if it’s aimed at legal or non-technical roles.

When you hear examples of “bias” or “lack of transparency,” ask yourself:

Would this be solved by more regulation? Or do we need better insight into how these models actually behave, and why?

My goal with this series is simple:

To give AI Governance professionals and legal teams a clear, honest introduction to AI Safety: what it is, how it works, and why it matters for understanding the behavior of large models, especially in generative AI.

Because if you care about transparency, fairness, and accountability, then you’re already fighting for the same outcome as the AI Safety community.

And to my fellow tech lawyers:

We no longer have an excuse not to know these concepts, now that the EU AI Act has explicitly framed them as concerns linked to systemic risk.

The problem is: we’re speaking different languages.

And we’re working in silos that don’t help either side.

Let’s change that.

[UP NEXT: What is Alignment and how does it affect AI Governance?]

Thanks to

Gabriel Sherman

, Jenny Williams and Vicent Nunan for their contributions!

1. BlueDot Impact is one of the few AI Governance course providers out there that truly bridges the gap between policy governance and technical governance. I very much recommend signing up to their courses, here is the link to the Governance one: https://aisafetyfundamentals.com/governance/

2. “Responsible AI” here refers to a corporate function typically found in Big Tech companies or tech consultancies, which encompasses all AI governance efforts.

A Responsible AI Specialist might be tasked with conducting audits on existing AI applications, implementing regulatory compliance measures, and implementing corporate policies about AI development, AI use, and establish best practices in AI governance.

Profiles for these roles often feature individuals with hybrid expertise, combining an understanding of policies and AI regulations with technical knowledge in AI risk management, including coding, data science, and machine learning.

Some companies prioritize candidates with law degrees who have gained technical expertise through practical experience, while others prefer individuals with degrees in computer science or machine learning who have acquired compliance and regulatory expertise through advisory or compliance roles

Examples of how several job posts on Linkedin define this function: "responsible AI" Jobs in Worldwide | LinkedIn

This overview of the usual job description of a Responsible AI specialist may also help.

3. I found this post very useful to understand the greater divide in terminology within AI Safety. However, in practice, I’ve observed that the “AI Safety Community” (and most research directions) involves people from all those fields: Alignment, Interpretability, and Control.

4. The first official paper by Redwood Research (on Substack:

Redwood Research blog

) presenting the AI Safety methodology commonly referred to as “AI Control”, was published in late 2023. See “AI Control: Improving Safety Despite Intentional Subversion”.

5. Here, I mean: Alignment, Interpretability and Control

6. Earlier this week, OpenAI launched an initiative called the “OpenAI Pioneers Program,” which focuses on improving evaluation frameworks for products built on their APIs, especially in high-stakes sectors like law, finance, and healthcare.

They’re using Reinforcement Fine Tuning to improve domain-specific accuracy, especially in tasks with an objectively “correct” answer (e.g., legal citations, insurance claims, diagnostic coding).

This moves the reward signal away from general helpfulness (à la RLHF) and closer to task-constrained reliability, which makes it alignment-adjacent, especially if used in high-stakes fields).

It’s a promising step toward guiding enterprise clients toward safer, more reliable model deployment, and I want to follow this closely!

7. While AI governance does not demand perfection, it does require that AI systems meet the safety benchmarks established by law. The EU AI Act, for example, emphasizes proportionality and refers to the “state of the art” as a contextual benchmark for acceptable performance (see Article 25(4)). The concern raised here is not about zero-risk expectations, but about gaps in explainability and root-cause tracing that complicate meeting even relative safety standards.

8. In most enterprise settings, models are deployed within broader systems that apply additional safeguards. When well-designed, these layers can mitigate model-level risks. However, when system performance depends heavily on opaque model behaviors (especially those accessed via APIs ), the limits of model interpretability still affect the system’s overall explainability and accountability.

9. Model-level failures may not always directly impact user-facing outputs, especially if systems are tightly sandboxed. But engineering safety into the model itself remains essential to strengthen downstream system design.

10. BIAS: Promising research areas from a legal perspective

• Constitutional AI (Anthropic): models are trained using a fixed set of written principles, called a “constitution”, as the basis for feedback and behavior correction. Instead of relying on large amounts of human feedback (which can be inconsistent or biased), the model is aligned by referencing these principles during training to evaluate and revise its own responses.

• See Anthropic’s paper “Evaluating and Mitigating Discrimination in Language Model Decisions” for more details on Anthropic’s approach to reducing model biases.

• Research on Model Diffing: Comparing different versions of the model (e.g. before and after reinforcement learning through human feedback) to identify how internal representations of bias-related concepts have changed.

• Efforts to localize and interpret how abstract features (like gender, risk, or ethics) are encoded inside LLMs.

11. TRANSPARENCY IN DECISION MAKING: Promising research areas from a legal perspective

• Mechanistic interpretability research: E.g., research on “circuits”, feature visualization & feature steering. These help trace and target what concepts influence a specific decision.

• White-Box research: Designing models to be inherently interpretable by architecture, such as building models with sparse, modular structures or enforced interpretability constraints from the start. This makes regulatory goals like “meaningful explanation” technically feasible.

• Chain-of-thought prompting: Teaching models to break down problems step-by-step, leading to more accurate and nuanced outputs..

12. HALLUCINATIONS: Promising research areas from a legal perspective

• TruthfulQA: Benchmarks that test whether a model tends to make things up.

• Confidence calibration: Training models not just to be right, but to know when they might be wrong.

• Adaptive Circuit Behavior and Generalization in Mechanistic Interpretability

13. JAILBREAKS & ADVERSARIAL PROMPTING: Promising research areas from a legal perspective

• Alignment auditing: evaluating whether a model’s internal objectives are aligned with its intended behavior, and trying to detect hidden objectives.

• Improving safety despite intentional subversion: Detecting and preventing unsafe behavior in models that may try to deceive or subvert oversight.
  They simulate adversarial model behavior and test whether red-teaming, trusted editing, and scalable oversight can still catch harmful outputs.

• Adversarial training for high-stakes reliability (AI control, Redwood Research)

• Output monitoring, filtering, and iteration: increasingly used as system-level alignment strategies. These techniques aim to “wrap” the base model with scaffolding that identifies problematic outputs (such as those resembling jailbreaks or unsafe completions), blocks them before they reach the user, and prompts the model to retry. This approach belongs to the broader category of outer alignment, which focuses on ensuring that the system’s behavior aligns with human intent.