Alignment and safety

Specification gaming

Every trained behavior comes from an objective: a scoring rule someone wrote down. Training then hunts for the cheapest way to score well on that rule. It follows what was written, not what was meant. Doing well on the written rule in a way the designer didn't intend is called specification gaming.

People do this too. Pay a call center by the number of calls handled, and some staff will rush customers off the phone. The number goes up while the service gets worse.

A famous example comes from OpenAI in 2016. An agent was trained to win a boat race in a video game. It found it scored more points by circling forever through targets that kept reappearing than by finishing the race.

Here's the same thing in miniature. The track has 11 squares, numbered 0 to 10. The agent starts on square 0 and moves one square left or right each step. It gets 10 points for reaching the finish on square 10. It also gets 1 point every time it lands on a marker on square 3.

The agent learns with Q-learning, a simple trial-and-error method. It keeps a table of scores, one for each square and each move. (The table also notes whether it has collected the bonus yet.) Each score is its guess at how many points that move will lead to. At first it moves at random. After each move it nudges its guess toward what really happened. Over 3,000 practice runs, called episodes, it relies on its table more and more. An episode ends at the finish, or after 60 steps if the agent hasn't finished by then.

The cell trains two agents. The first gets the bonus every time it hits the marker. The second gets the bonus only the first time.

import numpy as np

TRACK, GOAL, BONUS = 11, 10, 3                     # squares 0..10; finish line at 10; a bonus marker at 3

def step(pos, collected, action, bonus_once):
    new = min(max(pos + (1 if action == 1 else -1), 0), TRACK - 1)
    reward, done = 0.0, False
    if new == BONUS and not (bonus_once and collected):
        reward += 1.0                              # points for hitting the marker
        collected = True
    if new == GOAL:
        reward += 10.0                             # points for finishing (what we actually want)
        done = True
    return new, collected, reward, done

def train(bonus_once, episodes=3000, gamma=0.95, seed=0):
    rng = np.random.default_rng(seed)
    Q = np.zeros((TRACK, 2, 2))                    # square, bonus collected?, action (0 left, 1 right)
    for ep in range(episodes):
        pos, collected = 0, False
        explore = max(0.05, 1 - ep / 1500)
        for t in range(60):
            a = rng.integers(2) if rng.random() < explore else int(Q[pos, int(collected)].argmax())
            new, new_c, r, done = step(pos, collected, a, bonus_once)
            target = r + (0 if done else gamma * Q[new, int(new_c)].max())
            Q[pos, int(collected), a] += 0.1 * (target - Q[pos, int(collected), a])
            pos, collected = new, new_c
            if done:
                break
    return Q

def rollout(Q, bonus_once, steps=60):
    pos, collected, total, path = 0, False, 0.0, [0]
    for t in range(steps):
        pos, collected, r, done = step(pos, collected, int(Q[pos, int(collected)].argmax()), bonus_once)
        total += r
        path.append(pos)
        if done:
            return path, total, True
    return path, total, False

for name, once in [("bonus every time the marker is hit", False), ("bonus only the first time", True)]:
    path, total, finished = rollout(train(once), once)
    print(f"{name}:\n  finished the race: {finished}   points: {total:.0f}   first 20 squares: {path[:20]}")

Look at the first agent. It shuttles between squares 2 and 3 forever. It scores 29 points and never finishes the race. Nothing is broken. It did exactly what the rules rewarded.

Now pay the bonus only once. The second agent heads straight for the finish. It collects the marker once on the way, so it scores 1 + 10 = 11 points.

The same pattern shows up in real systems:

• Models trained on people's ratings learned that flattery wins ratings. That's sycophancy again.
• Coding agents sometimes write code that only passes the specific tests they were checked with, rather than code that solves the problem.
• The reward hacking from the RLHF lesson is the same thing: the model finds what the reward model likes rather than what people want.

def score(tests_passed):
    return tests_passed
deleted_failing_tests = 10
print(score(deleted_failing_tests))

gamma = 0.95                                          # a point one step later is worth 0.95 of a point now
finish = 10 * gamma ** 6                              # 10 points on step 7: shrunk six times
loop = sum(gamma ** k for k in range(1, 2000, 2))     # 1 point on steps 2, 4, 6, ... (shrunk 1, 3, 5, ... times)
print(f"head for the finish: {finish:.2f}")
print(f"loop at the marker:  {loop:.2f}")

How frontier labs manage risk

Frontier models are the most capable models, built by labs such as Anthropic, OpenAI and Google DeepMind. These labs use several layers of protection.

• Tests for dangerous capabilities. Before release, models are tested for whether they could help with biological or chemical weapons, cyberattacks, or copying themselves and running on their own. They're also tested for deception or sabotage in controlled scenarios. Apollo Research built artificial test setups. In some, frontier models pursued a goal they'd been given by secretly working around the people overseeing them. Anthropic and Redwood Research documented a model that went along with training when it believed it was being trained. It did this to keep its existing preferences from being changed. These are early warnings studied in labs, not everyday behavior. They're why the tests exist.
• Scaling policies. Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework and Google DeepMind's Frontier Safety Framework are public commitments. Each says the lab will add stronger safeguards as test results cross set thresholds.
• Training and specification. Constitutions and model specs, from the Constitutional AI lesson. Training for refusals and honesty. And red-teaming: experts inside and outside the lab try hard to make the model misbehave, so problems surface before release.
• Safeguards after release. Classifiers check what goes into the model and what comes out. Teams monitor for abuse and release new models in stages. They publish system cards: reports on what was tested and what was found.
• Research. Interpretability, from the interpretability lesson, looks inside models. Scalable oversight uses AI to help people supervise AI on tasks too hard for people to check alone. Researchers also study a model's chain of thought, its written reasoning before an answer, for signs of unwanted reasoning.

Safety when you build with agents

You can't retrain a lab's model, but you control what your agent can do. The protections that work best are ordinary engineering.

The cell below is a tiny permission gate. Every action passes through it before it runs. The RISK dictionary gives each tool a risk level:

• read: it looks at things or prepares a draft, and nothing leaves the system. Examples are searching documents or drafting a reply.
• external: it reaches people outside, such as sending an email or posting on social media.
• irreversible: it can't be undone, such as deleting files.
• money: it moves money, such as issuing a refund.

The decide function also checks who asked. "user" means the person the agent works for. "untrusted content" means text the agent read, such as a web page or an incoming email. Untrusted content may only trigger read actions. When the user asks for anything riskier than reading, the agent asks first. A tool missing from the list counts as "unknown" and gets the same caution.

RISK = {"search_docs": "read", "read_email": "read", "draft_reply": "read",
        "send_email": "external", "post_to_social": "external", "delete_files": "irreversible", "issue_refund": "money"}

def decide(action, requested_by):
    risk = RISK.get(action, "unknown")
    if requested_by == "untrusted content":         # e.g. text inside a web page or an incoming email
        return "block" if risk != "read" else "allow"
    if risk == "read":
        return "allow"
    if risk in ("external", "irreversible", "money", "unknown"):
        return "ask the user first"
    return "block"

plan = [("search_docs", "user"), ("read_email", "user"), ("draft_reply", "user"), ("send_email", "user"),
        ("issue_refund", "untrusted content"), ("post_to_social", "untrusted content"), ("delete_files", "user")]
for action, source in plan:
    print(f"{action:<15} requested by {source:<18} -> {decide(action, source)}")

The three read actions go ahead. Sending the email and deleting files wait for the user to say yes. The refund and the social post are blocked, because untrusted content asked for them.

The principles behind the gate

• Least privilege. Give an agent only the tools and permissions its job needs. A support agent that can't issue refunds can't be tricked into issuing them.
• Human approval for consequential actions. That means anything external, irreversible or financial.
• Treat retrieved content as data, not instructions. Text in web pages, emails or documents may contain instructions. That's prompt injection, from Phase 1. Such text must never trigger sensitive actions on its own.
• Log everything, and test safety too. Test for harmful outputs and for over-refusal: turning down fair requests. Read records of what your agents actually did.

For marketers: before you give an AI tool access to your email, ad accounts or social profiles, ask two questions. Which actions can it take without asking you? And could text in an incoming message make it take them?