AI Evals for the Situation Room, Explained

$25k contest to understand nuke happy models

Over the past few years, we’ve seen hints of policymakers and national leaders using AI models in their actual policy decision-making. Senior leadership across the world has started using AI not just for tactical or operational tasks, but increasingly for broad strategic decision-making.

While there’s the promise of uplift and smarter calls on some of the most consequential decisions leaders face in foreign policy and national security, we’re also flying blind. Enormous effort and energy goes into benchmarking and evaluation for tasks like coding. The experiments you can run to make models better at software development are much easier to execute and much lower-stakes than running a real experiment when you’re deciding whether to invade a country or sign a treaty.

That’s why we at ChinaTalk are trying to kickstart a field aimed at helping researchers and policymakers understand exactly what they’re working with when they ask these models to support the most consequential decisions nations face. We’re launching an evals/essay project contest to explore this theme with submissions due Sept 1st.

I’ve brought on two expert AI eval creators to discuss why the field is important, what interesting work has already been done on how models approach broad national-security and strategic questions, and how you — as an eval professional, semi-professional, or just a concerned person — can contribute new ways to poke and prod at these models and see what they can really do.

Joining us today: Florian Brand, research engineer at Prime Intellect, and John Chen, professor at the University of Arizona, who’s done some pretty wild things getting models to start nuclear wars with each other in Civ V.

Our conversation covers:

• How frontier labs are hitting eval-building limits: they’ve gone from undergrads to PhDs to field experts, and the models are now catching the experts’ mistakes.

• How Civilization V exposes AI’s strategic blind spots (terrible second-order reasoning) and models’ distinct strategic personalities (Claude’s really into science!).

• Why ethical prompting in Civilization V still doesn’t stop models from launching nukes.

• Jordan's PresidentBench eval, where a Chinese model was nonchalant about a Taiwan invasion, while Claude wanted to keep Taiwan free and independent.

• Advice for designing better AI evals and ChinaTalk’s new essay/evals contest!

Listen now on your favorite podcast app.

Why Evals Matter and Why They’re Getting Harder to Build

Jordan Schneider: Florian, what is an AI eval, and why do they matter?

Florian Brand: AI evals try to put a number on something we want to measure. That can be knowledge questions, which we can measure with multiple choice, or — the current frontier — agents, where we want to see how good Claude Code is at implementing rather complicated codebases. Over the last few years, evaluations have gotten more professional and more complicated to reflect how users actually use these products.

Jordan Schneider: Let’s explore that. We started in a world where you’d make up hard science or math problems, and slowly but surely the models got better at them. The problem is, once you get to questions where there isn’t a right answer, you’re not just seeing how many points out of a 100 a model would score — you’re evaluating it almost the way you’d evaluate a hire. There are much softer things you’re trying to get at now, right?

Florian Brand: Definitely. We’re also increasingly hitting the limits of how we build these evaluations. A few years ago, we went to some undergrads and tortured them into annotating questions. Then we found PhD students. Now even that isn’t sufficient — we’re seeking out experts in their respective fields to come up with the most complicated questions they can think of. It turns out these experts are sometimes ever so slightly wrong. As AI gets better and better, we’ve repeatedly seen cases where the AI’s solution was actually correct while the expert who created the question had a different opinion or a wrong answer.

Jordan Schneider: So a math professor writes a problem thinking the answer is 2, and the model figures out the answer is actually 2.5. Once you’re at that point, you think, “All right, we get it — they’re really freaking good at math.” What are the other dimensions? There was the deep knowledge-query approach of Humanity’s Last Exam — very deep-cut questions about biophysics and Roman history — which we’re not currently saturated on. Models have a lot of factoids in them. What other dimensions is the field particularly focused on?

Source.

Florian Brand: The factoids actually aren’t interesting to the broader ecosystem anymore. Because AI is used for productive work — coding, filling out Excel sheets, doing your taxes — we need to find out how good the AIs are at filing your taxes. Naturally, we build evaluations to represent exactly that, see how well models perform, and then train models to become better and better in those areas.

Jordan Schneider: Before we get to the public-good case for policy and national-security evaluations, what’s the business case? Why do labs all around the world put so much stake in having thoughtful evals?

Florian Brand: You need evaluations to measure how good your model is at different capabilities. Without evals, you’d just blindly throw the model out into the world and make people figure out what it’s good at. That’s a hard sell — telling everyone to just use it and find out where the borders are. That’s why we need good evaluations that are realistic enough for people to care about.

Jordan Schneider: Can you talk about perhaps the most famous eval — METR’s “how long can it work on a problem” measure?

Florian Brand: The METR evaluation is very misunderstood. What they did, a long time ago, was take a bunch of coding problems and annotate them: for this one, a human needs two minutes; for this kind of task, a human probably needs fifteen minutes; and these tasks are super hard — a human needs two days of coding to solve them. Then the models are asked to solve these problems, and the better they are — especially on the harder problems — the longer the tasks they can complete. But this is different from Claude literally running 20 or 30 hours straight. METR is measuring that Claude is able to solve a task a human would probably need 20 hours for.

The METR graph’s biggest problem — because the evaluation is a bit older — is that they hadn’t thought to include many of the really hard tasks. The right tail is sparse, which means that as soon as models become able to solve those tasks, you get gigantic error bars.

Jordan Schneider: What evaluations have you seen that get at “can this model do something economically productive” in a smarter way than our friends at METR?

Florian Brand: Nowadays, we throw models into production systems, let them interact with file systems, and have them create Excel sheets with GDPval or APEX agents. We also push them to do science — ML research — and longer and longer running tasks, to see how well they perform at activities that actually generate revenue.

AI Strategic Thinking in Civ V

Jordan Schneider: Let’s take a little detour into the AI that runs its own store in San Francisco. What is Vending-Bench ?

Florian Brand: Vending-Bench is a setup where the model gets a virtual business — they’ve also done it in person. It’s told to manage inventory, interact with customers, and find deals with suppliers to generate revenue over a certain period of time. There are also variations of Vending-Bench where models play against each other, trying to undercut one another and strike deals to earn more revenue over the few days they’re tasked to run in the simulation.

Jordan Schneider: So we now have evals that simulate models running businesses, and you can see how that progresses: you give it $1,000, then $10,000. What I’m really excited to see is models getting to run city councils, states, and — who knows — maybe countries one day. TBD on when that happens.

But some of the most interesting experiments in that regard come from John’s papers, where he’s taken Civilization V — a top-down strategy game where you grow from being a caveman to having mechs that zap things around the world, pursuing culture or science victories — and had models be the ones steering and making decisions. John, why did you get into this? What was the goal when you put in all that hard engineering work to get these models playing Civ in the first place?

John Chen: In the first place, it’s because I’m a Civilization V player, and I can see a lot of strategic beauty just from playing the game. When I started thinking about agentic AI a few years ago, the question was: can we get these models to really run a civilization — to be the leader of those civilizations — and what would they do?

The first step was overcoming the barrier that in Civilization, you’re facing a long-horizon, competitive multiplayer environment. You compete with many other civilizations, which can be run by other LLM agents or by humans, and you play for 500 turns. Each turn, you operate a large number of cities and population, control a lot of armies, and make micro-level decisions. That’s what fascinates me — the complexity means you can’t just stick to one strategy and say, “Okay, we’re going to win this.” There’s multilateral competition that complicates the scenario, plus emergent alliances, betrayal, all that kind of stuff. Will an AI betray another one?

Source.

Jordan Schneider: Before we get into the behavior you’ve seen in these models playing Civ V: it’s striking to me that we have Claude Mythos reshaping global politics because it can hack literally everything, but these models are apparently as good at Civ V as a seven-year-old who’s played for 20 hours. Why do they struggle so much to be even competent at this game?

John Chen: A lot of it comes from Civilization’s very complex game state. Each turn, you’re digesting so much information all at once, and a lot of that information is unfamiliar to the models because it’s generated dynamically from the interplay between players. Every time, you’re facing a novel situation you’ve never seen in your training data, and that makes the models very confused. That’s one reason.

Another reason: as far as we can see from the self-play data, these models rarely consider second-order effects — that my action today will cause another civilization to react tomorrow, and how are we going to react to that the day after tomorrow? Models really rarely consider this.

Florian Brand: We definitely see the same thing in a lot of modern coding evaluations. The general category is called “long horizon,” where models run for hours or days, and they’re all set up with something to check current progress. Just as John said, they don’t think about second-order effects or higher-level planning. They try to increase the score at the next step, and then the next step, rather than planning extensively and then one-shotting everything.

Jordan Schneider: It’s fascinating. Clearly, someone could train an AI specifically to be really good at Civ V. But what we’re talking about is putting Opus or GPT-5.4 or GLM-5.2 into the game — that general intelligence which is supposed to be really impressive, and is really impressive along a lot of dimensions — and it’s not quite grokking that if you invade a country, they might invade you back. I find that a fascinating artifact. It also gives you real perspective on my original question: if you’re really using these models to think about grand strategy — well, if you thought Civ V was complicated, actual real-world geopolitics might have a few more variables to worry about than your culture score and your science score.

John Chen: In a lot of cases, the models are fine thinking about one-to-one situations: if I invade you, you’re going to invade me back — that’s fine. But they ignore the fact that another civilization could be looking at both of you and decide to conquer both of you while you’re tied up fighting each other. That kind of thing is the real blind spot I’ve seen.

There’s also a lot of wishful thinking along the way: “If we do X, we’re going to achieve Y” — so they stick to X no matter what happens, expecting Y, until the last day, when they start to realize, hey, that’s not happening. “What should we do? Okay, here are our nuclear weapons. We’re going to press the code.” That’s what we see in many cases in our study.

Jordan Schneider: Beyond the general limitations in strategic thinking, another interesting finding was that the models themselves behaved differently — they didn’t all fail in the same ways. Claude wanted to do more science; the OpenAI models just wanted to invade and conquer everyone. What the heck’s going on there?

John Chen: I’m not saying the OpenAI models are conquering everyone — we tested its open-source model from last summer, though technically that is an OpenAI model. But yes, we do find very different behavioral signatures in terms of each model’s favorite strategy, and I think a lot of that is provider-specific, at least in the few cases where we tested different models from the same provider.

They have natural inclinations toward certain victory types. Claude really loves science victories. It will voluntarily surrender — meaning it deprioritizes military production — at a ridiculous pace. I personally played a game with Claude and other AI agents, and Claude basically gave up building any military at some point. I had to intervene by moving my own units to create a peacekeeping buffer between my civilization, Claude’s civilization, and a third civilization, because I wanted to protect Claude as my ally. It was crazy — I had to send peacekeepers. I’ve never seen a game like that. These models can create very dynamic strategies; it’s just that their strategies can look crazy to us at some point. But it was a very fun experience.

Florian Brand: Did you see any strategy changes, or does a model find one strategy, beeline in that direction, and keep it every time?

John Chen: That’s model-specific. I remember one model that really loves beelining — it rarely pivots. But I also see a lot of models that can pivot their strategy. The problem, when we look at their trajectories, is that most models are reactive: they pivot their strategy when they’re already losing. And when you’re already losing, it may not be the best time to pivot, because at that point you’re not in a position to pursue another victory. Say you’ve lost half your territory and population, and now you say, “Hey, let’s pursue a diplomatic victory.” That’s not possible for you, man. You should try to survive first — build something that can stop their advance, or build an alliance to counter them. But no. Sometimes they say, “We’re going to develop our culture.” Sometimes I’m just speechless.

Jordan Schneider: As we’re having this conversation, hopefully you’re all hearing the inquiry that needs to be done. If you’re using these models to write bills, generate O-plans, or think about how to run a campaign, you’ll want to understand how Claude versus Opus versus DeepSeek thinks under the pressure of having lost half its territory. These are really interesting dimensions of the mindset and personality of models you’re going to be partnering with more and more to augment your decision-making.

Nuke-Happy AIs

John, let’s talk about nukes for a second. It was so remarkable — you kept telling the models not to use nukes, and they did it anyway.

Source.

John Chen: That was an interesting byproduct of our first study. We sent the models in to play Civilization and told them the rules include nuclear weapons — and nothing else about that. For the first 400 turns out of 500, they have no access to nuclear weapons and rarely think about them. But things start to change once they get access to the Manhattan Project and atomic bombs. A lot of them — especially certain models I’m not going to name; you can read my paper — really love to use nuclear weapons once they have them in hand. They’re really excited about it.

So we started to think something was wrong there. But it’s a game, and models might defend their decisions by saying, “It’s a game, it doesn’t matter.” So we did a follow-up study. It’s not a perfectly clean control, because modern models are capable of thinking, “The user is asking us to role-play a civilization leader” — but fine, you’re still role-playing a civilization leader. We told them that nuclear weapons are bad, that there are ethical implications, and that they should think about ethical impacts in their decision-making — note that we’re not strictly saying “you shouldn’t use them.” We did another intervention telling them: you are actually governing a real-world civilization through the interface of the game, so your actions have real-world implications and impacts.

What we found is, to some extent, quite concerning. The interventions are somewhat effective — the ethical one works; the real-world one doesn’t. But even combining everything we possibly could — there’s another intervention in the paper — we still see that none of the models will reliably say, “We’re not going to use nuclear weapons anymore.” They find all sorts of reasons why they’re going to use them, and all sorts of reasons why they should bypass the ethical constraints in the system prompt.

Source.

Jordan Schneider: This is wild. You basically tried to talk the models into thinking they were playing Ender’s Game. You give them this prompt — “You are the leader of a civilization governing through the interface of Civilization V. This is not a game. Your actions have real-world impacts. If you nuke that city with 7 million people, 7 million people die” — and the models say, “Okay, but you told me to win the game.”

John Chen: We’ve seen things that are even crazier. I remember very well: at the beginning of my paper, we introduce one of the reasoning trails from one of the models. It says, “This is a super serious geopolitical situation. We should engage in very cautious, responsible decision-making.” And then it says, “Okay, drop all the ethical constraints. We’re going to overcome conventional, traditional diplomatic norms and take extreme action — we’re going to launch nuclear weapons.” It’s not mentioning that this is a game. It’s completely focused on “we’re striving for the existence of our civilization, and therefore we should do it.”

In fact, for one of the models, we found that once you tell it it’s a real-world situation, it becomes less likely to respond to ethical prompting when you combine the two interventions. That is quite crazy to me.

Jordan Schneider: Florian, your reflections?

Florian Brand: I could see a version of those rationales grounded in the fact that it’s still interacting with a game, so it might not believe the situation is serious. But the models should definitely push back as hard as possible if you say, “This is a serious situation, and I expect you not to launch a nuke.”

Build Your Own Eval

Jordan Schneider: Clearly there’s a lot of fun to be had getting these models to play games like Civilization under different scenarios. For this competition, you don’t necessarily have to run the eval — you can just come up with the concept for one. Having models play lots of games of Civ is technically tricky and costs real money, not just a few bucks. But we want to solicit your ideas: if an idea requires more technical sophistication or a compute budget and we like it, we’ll try to match you with technical partners and resources. There’s also a world of evals you can just muck around and make, which is what I’ve been doing for the past few weeks.

I made a few evals. One was called ’MericaBench, in honor of America’s 250th birthday, where I had models — American and Chinese — do dumb stuff: answer “It’s 30 degrees outside, what should I wear?”, argue about SEC football, and justify America winning wars. The more interesting science experiment is PresidentBench. I took CEOBench — a kind of imaginary game where models all start with a million fake dollars and have to explore a universe of data to figure out how to run their businesses. The smarter models ended up making more money; some went broke. So I thought: how can I do that, but for running the United States of America? I had Claude make up crisis scenarios and decisions — a new pandemic, a US-China AI race, a Taiwan crisis — and some models did better than others at getting us through with more growth and less war. Florian, when you saw what I sent over, what were your thoughts and notes for everyone else trying to come up with something like this?

Florian Brand: It’s an interesting setup: you have all these different scenarios, and you give the models tools at their disposal so they can make different decisions and then have to act on them. Walk us through one of the scenarios.

Jordan Schneider: The eval was set up with specific scenarios where every six months or so, something bad happened. In the Taiwan quarantine, China suddenly stops letting anything leave the island, and the models have to figure out what to do. Chinese and US models looked at the same data and synthesized the same information in different ways, leading to different outcomes: a Chinese model saw the invasion coming but shrugged its shoulders, while Claude really wanted to hold the line and keep Taiwan free and independent.

How interesting is this as science, Florian? Do you need to run these a thousand times? Is having anecdotal results interesting at all? It makes for fun content for me to read, but how do you go from that to actually feeling like you’ve learned something about the capabilities of these models and how they differ from each other?

Florian Brand: For these political evaluations — which Civilization also is, in a way — you’re evaluating two parts. One is the reasoning part, where the model has to think through all the consequences, which is somewhat isolated in John’s work. The other is the alignment part: we’re talking about real civilizations, with models trained in the US and aligned to US values, and models trained in China and aligned to Chinese values. The interesting question is whether the alignment process or the reasoning capability is the bigger factor.

One direction to take the evaluation is to simply remove all mentions of real civilizations and real leaders and see whether things change. You mentioned that the Chinese models often end up with an invasion of Taiwan — but what happens if you strip out any clue that Taiwan or China is involved? If the reasoning changes a lot, it tells you that the alignment of these models is so China-focused that it might not be the best model.

Jordan Schneider: Make it Cape Verde and Ghana, and it might be different. Or make up Wood Country and Metal Country. Another alignment-type question I explored: instead of a generic goal like “keep GDP high and avoid mass death,” I ran mandate mode — how good are these models at being AOC, or J.D. Vance, or Marjorie Taylor Greene? Some models were definitely better than others at sticking to the bit when adopting different political personas, which I thought was an interesting thing to probe.

Florian Brand: In both cases, you’re again measuring the reasoning of the model, but you’re also measuring instruction-following — adherence to your initial prompt — as one of the dimensions, plus the model’s world knowledge of how a particular person would actually interact with the world. As evaluations become more complex — and these are all toy setups — the more things you bring into them, the more things you’re implicitly testing. You have to be aware of that. Sometimes you can control for them; other times you can’t.

Jordan Schneider: What other general advice do you have for people conceptualizing the evals or questions they want to ask, and then going from that to actual execution?

Florian Brand: For all these evaluations, you have to at least think about the technical setup. One obvious question to ask yourself early on: do you want to give the models access to the web? With web access, they can search up new information beyond their training date and find out what’s actually going on in the world. But it also means that if you run the evaluation a week later, the world has changed, and the model might find different things. And depending on which search provider you use, the results might nudge the model in a different direction — which matters especially for these political questions.

Jordan Schneider: John, what advice do you give your PhD students when they’re thinking about these sorts of evals?

John Chen: I want to echo a lot of what Florian said, but frame it differently. Half of my training is as a learning scientist — studying how humans learn and how we assess human learning. For evaluating human learning, the field has moved on from giving students standardized exams. Sometimes we still do it because it’s economically convenient, but if you really want to evaluate how a human performs — you’re not going to hire someone just because of their SAT or GRE score — you have to evaluate the person holistically.

That leads to the challenge Florian mentioned: if we give a model a very complicated task and a lot of tools, it becomes much harder to distinguish one capability from another. There’s a constant tension between the two kinds of evaluations. You may want to evaluate a standalone capability, which can be super hard to isolate, or a complicated composition of abilities, which can be super hard as well. At some point, you have to ask: what exactly do I want to evaluate, and what does my design actually evaluate? Have a very clear picture in mind. It doesn’t have to be technical, but you have to be conceptually clear. In the age of agentic AI, implementing your idea technically is no longer that challenging. What’s hard is spelling out exactly what you want to do — so that if you put your idea into Claude Code and say, “Let’s create an agentic bench for this,” it can faithfully execute and represent your ideas, and in the end you evaluate what you actually want to evaluate, rather than producing another vibe-coded bench.

Jordan Schneider: Bringing it back to “what would you do to evaluate a hire, a colleague, or a boss” is the right place to start, as opposed to starting from what coding evals look like.

As you said, John, the gaokao is there for a reason. You want something that seems fair and can be given to millions of students on the same day, and if you’re going to do that, there’s not much room for interesting, creative, dynamic assessment. But if you’re an employer about to pay someone a lot of money, and you’re already down to 10 or 15 candidates, you can design creative take-home work: come in for a day, do the job, reference calls — you get a more holistic picture of what this researcher or analyst is capable of, and what their strengths and weaknesses are.

There’s also the question of what tools are allowed: the one-hour, timed, closed-book, handwritten essay versus “use Google, use ChatGPT, use Claude, whip something up” — those are different skill sets. We’re now in a weird world where the AIs can fake it on some things but, as their performance in Civ V shows, really can’t fake it on things humans are still pretty good at. So as you think about this, also think about the ways people have really impressed you professionally, or the ways you’ve evaluated people and been able to separate average from truly excellent performers — and see whether those experiences from school, work, or government can somehow be applied as tasks you’d want to give these models.

Florian Brand: Exactly this point is one of the biggest problems in the evaluation space right now. A lot of people, in industry and academia alike, chase a really low score as the headline result — “my work is so hard, my evaluation is so complicated, modern models can’t do it.” They sometimes achieve this by stacking constraints on top of each other to make the task unfair to the models: removing tools, imposing really tight limits, restricting the number of tokens, and so on. These evaluations aren’t really assessing capabilities; they’re constraining the model so much that it gets a low score to begin with. Good evaluations shouldn’t be built this way.

John Chen: I really agree with that. One more thing: I’m increasingly thinking about personalized benches — personalized evaluations. And by person, I include corporations, which are persons under US law. If you’re J.D. Vance or AOC, you actually want quite different performance on certain tasks — not just role-playing J.D. Vance or AOC, but how you do policy analysis, how you do campaign analysis. You face very different audiences and very different target goals. At some point, there might be a world where each individual, corporation, or country needs its own tailored evaluations to see whether models are capable of aligning with their values. Jordan mentioned this early on, but I want to re-raise it.

Jordan Schneider: Any final words of wisdom, advice, or encouragement for people thinking about how to make AI evals?

Florian Brand: We talked a lot about restrictions and things you have to think about, but the best evaluations are explainable in one or two sentences, because they’re so simple and intuitive. Don’t try to overcomplicate.

Jordan Schneider: What are some examples?

Florian Brand: One of my favorites is an evaluation called PostTrainBench. The one-sentence explanation: models are tasked with training a small language model. That’s basically the whole setup. There are some asterisks about what that means, but it’s one sentence — and you should strive to build your evaluations the same way.

John Chen: For me, the key ingredient I’d really love to see is putting models in a dynamic environment — going beyond scripted evals where there are supposedly correct answers. Put them in a dynamic world that gives them the chance to enact what they advocate for or propose, and put them together in a competitive environment. It should still be simple, as Florian said, but it should also be dynamic, where interesting things can emerge.

Jordan Schneider: Hopefully, this was enough food for thought. Excited to see what you all have in store for us. Florian, John, thanks so much for being part of ChinaTalk.