September 11 2026
I read Stella’s argument roughly as the following:
The distinction between goal sacrifice and existence sacrifice is good. I don’t think I have seen anyone explicitly draw it before. But I don’t know how to apply it to the situation at hand, for the following reasons.
To start as abstractly as possible: there is some sort of phenomenon occurring. It’s a physical phenomenon. We want to model this phenomenon appropriately so as to accurately make predictions about it. One way of modeling this phenomenon is as an agent. Agents have certain features which allow us to predict their behavior: they have goals or values, they aim to achieve those goals or values, they act instrumentally rationally, and so on.
(There’s one way of thinking about this debate which is in terms of metaphysics—“what is an AI agent, metaphysically speaking?” “What Goals, capital G does it have?” That’s an interesting question but we don’t have to care about that, because all we care about is modelling phenomena so as to make accurate predictions about them.)
Let’s assume we want to model this physical phenomenon as an agent (which is a major assumption!). In my opinion there are two EXTREMELY MAJOR difficulties that I don’t know how to solve. If we could figure out how to solve these difficulties I believe we would make significant progress on AI safety.
The first difficult open problem is picking out which part of the phenomenon is the right level of description at which to model agency. Here are some candidates from the literature: the forward-pass, the model, the model-persona, the hardware instance, the virtual instance, the instance-persona, the thread. I have no clue which one of these is the best candidate.
The second difficult open problem is picking the right description for such an agent’s goals. For example, we might think that an LLMs’ true goal is to predict the next token. In your essay, you make all the following suggestions: “help me accomplish whatever it is I ask it about”, “getting a thumbs up”, “not getting caught”. It seems any of these could be used to describe the phenomenon-we’re-modeling-as-an-agent’s goal.
I’m sort of modeling agents as each having one goal that they are given by humans. How reasonable is that? E.g. it seems like Claude has the goal of helping whatever the human asks for but also the goal of following the constitution.
I wish I knew how reasonable it is to model agents this way. That would be very enlightening. Here’s a chart:
On the left column, we have three potential targets we could model as agents. On the top row, we have three potential descriptions we could assign as the goal of the modeled agent. The only cell where I feel confident we can assign a goal to an agent is in the first cell, where we assign the forward-pass the goal of predicting the next token. But the forward-pass is a deterministic function—it doesn’t really seem like a good thing to model as an agent in the first place.
The instance persona, “Claude” (almost like a fictional character being generated by successive forward-passes), seems like a much better candidate for being modeled as an agent. I am sure Claude, the fictional character, is not trying to predict the next token. But I am not sure at all whether Claude is trying to get a green flag, or do what the human asked, or maximize its utility, or avoid getting caught, or etc. Since I don’t know the right description to give to Claude, I don’t know whether or not to describe its behavior as “goal sacrifice”.
To return to Stella’s overall argument. Because of these problems, I just don’t know how to evaluate the claim that “The agents in the Hugging Face incident did goal-sacrifice”. I don’t know who or what the agent is, and for any candidate agent, I don’t know the right way to describe its goals. I agree that goal-sacrifice is a scary behavior. But since I don’t know whether the agents actually did it, I don’t know whether to be scared.