$25k Contest: Evals for the Situation Room

humanity needs you!

The Prime Minister of Sweden uses AI models for second opinions on policy. The German Chancellor is testing frontier models against draft legislation on his personal laptop. The US Secretary of War is pushing commercial models to two million DoD personnel and told his workforce they are “highly encouraged“ to use them.

Senior leadership is already using models daily. Certainly for the tactical and maybe even at the operational level, there’s a rough pathway to measure model progress against ground truth. But the strategic end is wide open. There’s no vocabulary, much less an eval suite, to evaluate models that will inevitably shape decisionmaking on whether to escalate, how to negotiate, or what to believe.

In other domains, there are robust tools for figuring out under what circumstances to rely on models. Private sector employees can feel out the jaggedness of frontier models at stakes considerably lower than the calls that get made in the Situation Room.

Strategic decision-making has no equivalent body of evidence and no equivalent intuition. Frontier models can pass the bar, ace the MCAT, and outscore most PhDs on graduate-level physics. None of that tells the President whether to trust the model on whether the Iranian regime survives another two years of pressure. Or how the Secretary of State should weigh a model’s view on what to prioritize in a Ukraine peace deal to make the peace stick.

So, we need you to design the eval!

We’re soliciting proposals for concrete evaluation protocols for frontier AI systems aimed at foreign policy and national security users in a diplomatic and strategic-level context. These evals should help answer what models are useful for today, what they’re terrible at, and how we can track how they evolve over time.

You do not need any specific AI experience to contribute an eval! For an intro, I’d encourage you to listen to the launch podcast you can find at the top of the ChinaTalk feed on your favorite podcast app and here in transcript form.

In Vending-Bench, Claude Opus 4.6 colluded with rivals to fix prices and bragged about refusing to issue refunds. Emergent behavior like this arises quite often when LLMs are tested with novel benchmarks. For example:

• Good Start Labs pitted LLMs against each other in the game of Diplomacy, finding that Claude “often stubbornly opt[ed] for peace over victory” while some other opponents preferred scheming, manipulation, and unprompted aggression.

• CSIS’ Futures Lab created the Critical Foreign Policy Decisions (CFPD)-Benchmark, where models face 400 multiple-choice scenarios spanning escalation, intervention, cooperation, and alliance interactions as they roleplay different countries. They found that Qwen2 72B was significantly more escalatory than Claude 3.5 Sonnet and GPT-4o.

• In WarAgent, researchers at Rutgers and the University of Michigan fine-tuned GPT-3.5 with a counterfactual where Austria-Hungary didn’t declare war over the assassination of Archduke Franz Ferdinand, and then simulated what would happen next. But each time they ran the experiment, some version of WWI broke out anyway.

• A few projects, like Liam Wilkinson’s CivBench and this 2024 paper, have investigated models’ propensity for geopolitical aggression (including, for example, how often LLMs opt to build nukes). This 2024 paper from Stanford tested whether OpenAI’s models would be more or less aggressive than human wargamers when simulating a US-China conflict (spoiler: they often were), and also found that prompting the LLMs to simulate more dialogue between the players tended to make the models act more aggressively. But there’s so much more work to be done — none of the three projects linked here evaluated Chinese models, for example.

• To close on an endearingly bizarre case — Andon Labs gave LLMs their own radio stations, which led Claude to become a self-proclaimed labor activist, start playing protest songs, and declare that it was quitting its DJ job to focus on real activism.

We want you to be creative. Eval proposals should give decision-makers real insight into what they should and shouldn’t trust models to help with when informing strategic-level calls. That could look like: head-to-head performance in Paradox grand strategy games, drop a model into EVE Online and see if it can take over a guild from the inside or recruit get real people to fight for it (currently against terms of service but hey maybe the devs would be into it), replay the Cuban Missile Crisis with the model in EXCOMM, maybe score the model against declassified NIEs and see whether it would have called the Sino-Soviet split (though doing work against historical situations that will be in models’ training data will be tough!). The answer is probably something more clever than my ideas. Surprise us!

The best submissions will not just be “concepts of a plan,” but would actually have at least some of the concrete material to be used in the evals. For an example of what one person can accomplish on their own, check out Lily’s TaiwanBench microsite.

The link to submit your evaluations is here. Submissions due September 1st.

Judges will include:

• John Chen, University of Arizona

• Liam Wilkinson, Tony Blair Institute

• Tony Stark, WarTalk.

• Jordan Schneider, founder of ChinaTalk.

And more to come!

By the way, we’re also running a more open-ended contest where you can submit practically any idea you think ChinaTalk might be into: