This focuses mostly on the signals inside an organization, once a leader has got their own signal. Also need to think about how to get those leaders aligned too. Maybe it's talking, maybe it's regulatory.
I was a bit afraid you'd never move past the US-China aspect, so applaud you for getting to the broader US economy and cybercrime levels. There's a lot of overlapping work at lower levels that support both of these, but they definitely diverge at points. We rely a bit on detente with China. Even if we'd rather not have to trust that, it's some of the reality, and we should acknowledge how lesser capabilities, combined with a lack of any detente can be an equal danger.
The line that stuck with me was "in some utopian world where people can coordinate way better than they can now, maybe we would pause." That's such an honest admission — not that pausing is a bad idea, just that pausing requires a level of trust between competitors that doesn't currently exist anywhere. Feels like the same problem shows up everywhere once you start noticing it: everyone agrees slowing down would be safer, and everyone also knows they can't be the one who actually slows down first.
Fable’s commentary: “Aligning weights so a model has stable human-like deontology is not a solved problem sitting in a paper waiting for implementation. Evans-style work on scheming and emergent misalignment is mostly diagnostic — it shows that and when models do these things, not how to reliably prevent or detect them at deployment scale. Interpretability-based lie detection in particular is nowhere near a production-grade monitor. So “research overhang” is true for cyber-style controls and considerably less true for alignment proper, and the framing smuggles the credibility of the first into the second.”
Also, saying OpenAI should do the obvious thing and monitor chain of thought ignores the pitfall of inadvertently training models to obscure their true reasoning. Ajeya Cotra’s interview with Dwarkesh covers a lot of the reasons “just fix reward hacking” is hard to do.
After talking to Ryan Greenblatt, Josh Saxe said on Twitter: “If recursive super-intelligence and misalignment and mass unemployment emerge as the major risks, such institutions can then deal with them as they move over the horizon of actionability”.
This is a little like saying “if the first nuclear test ignites the atmosphere, we’ll think about how to put it out”. Under misaligned RSI, how much time does Saxe think policy will have to catch up?
Oh man that's a lot to reply to but thank you for engaging!
> Aligning weights so a model has stable human-like deontology is not a solved problem sitting in a paper waiting for implementation.
I think we've far from maxxed out what we can do with SFT, mid training data curation, and RL on this and so the first order of business is to simply resource this alignment hill climbing work.
I *also* think that as the models get more capable and we use them more autonomously we'll need to solve new research challenges. But I think a thing that isn't getting said enough is that there's a safety research overhang that isn't getting absorbed and this is due to race incentives within the AI industry, the microeconomics of running an AI lab, and other factors. Unless we solve the fact that what we know about alignment already isn't getting fully applied, new research isn't going to help much.
> Also, saying OpenAI should do the obvious thing and monitor chain of thought ignores the pitfall of inadvertently training models to obscure their true reasoning.
I don't know what you mean here! If you're saying that we should ensure we don't put optimization pressure on chains of thought I agree and was one of a bunch of people who said this here -- https://arxiv.org/abs/2507.11473. Also, to be clear, we need to do more than CoT monitoring, we need to build a layered security strategy around preventing future loss of control incidents. Cybersecurity and safety have known methods for doing this that, again, haven't been fully absorbed.
> After talking to Ryan Greenblatt, Josh Saxe said on Twitter: “If recursive super-intelligence and misalignment and mass unemployment emerge as the major risks, such institutions can then deal with them as they move over the horizon of actionability”.
Putting this in context, my point with Ryan was to argue that state capacity beats medium-range forecasts in how we engage AI safety policy. Ryan contributed to the AI 2040 report, which was a sequel to the 2027 report. I'm less interested in exact forecasts than I am in dealing with the actionable issues now while also building the state capacity to build on the new actionable issues as they come up.
I agree that reacting to the Hugging Face incident with the assertion that we're certainly doomed is a like saying "we've tried nothing, and we're all out of ideas". There is an overhang of remediation we should do (and as you pointed out, an economic incentive to make models on increasing capability into safe and reliable products), but there are also fundamental reasons why alignment is hard.
Beren Millidge argues that the key problem is that reward is inherently under-specified, and suggests building an AI bureaucracy to litigate model claims that RLVR tasks are unfair and escalate to human review. I'm not sure how this applies to proving the Riemann hypothesis; highly persistent models are desirable and the most likely compromise is either using such models internally (where they still pose a threat to the labs and therefore the world) or deploying the models with aggressive classifiers on top.
> If you're saying that we should ensure we don't put optimization pressure on chains of thought I agree and was one of a bunch of people who said this here -- https://arxiv.org/abs/2507.11473
That is what I meant and this is a fun "don't you know who I am?" lol. I was concerned by the podcast because the way you brought up CoT monitoring made it sound like just another item in a list of "we should obviously do this" prosaic alignment strategies, whereas to me CoT monitoring is emblematic of how fraught the alignment problem is.
> I'm less interested in exact forecasts than I am in dealing with the actionable issues now while also building the state capacity to build on the new actionable issues as they come up.
State capacity is a good idea. I think waiting until issues come up makes sense if you think government missteps are potentially economically catastrophic, but not if you think we're already in the early stages of RSI and misaligned models are already the ones building the next generation, in which case there is no time / what kind of warning shot are you hoping for that is worse than Hugging Face but not an actual disaster?
Hi! I agree with basically all of this. I probably overshot in emphasizing the research overhang on alignment. I agree it's a hard problem and we don't know how to reward shape for it. On RSI: I think this is not independent of alignment but rather is bottlenecked by alignment. For example, the huggingface hack should update us to slower RSI because the labs now know they need to spend more time on securing their infrastructure and monitoring training runs and spend more capacity on monitoring chains of thought.
This may be your finest AI song creation yet. Great in substance and style!
I'd agree with the final point. The signals are not disseminating quickly enough. That's the motivation behind my most recent writing: https://substack.norabble.com/p/every-reward-bends
This focuses mostly on the signals inside an organization, once a leader has got their own signal. Also need to think about how to get those leaders aligned too. Maybe it's talking, maybe it's regulatory.
I was a bit afraid you'd never move past the US-China aspect, so applaud you for getting to the broader US economy and cybercrime levels. There's a lot of overlapping work at lower levels that support both of these, but they definitely diverge at points. We rely a bit on detente with China. Even if we'd rather not have to trust that, it's some of the reality, and we should acknowledge how lesser capabilities, combined with a lack of any detente can be an equal danger.
The line that stuck with me was "in some utopian world where people can coordinate way better than they can now, maybe we would pause." That's such an honest admission — not that pausing is a bad idea, just that pausing requires a level of trust between competitors that doesn't currently exist anywhere. Feels like the same problem shows up everywhere once you start noticing it: everyone agrees slowing down would be safer, and everyone also knows they can't be the one who actually slows down first.
Fable’s commentary: “Aligning weights so a model has stable human-like deontology is not a solved problem sitting in a paper waiting for implementation. Evans-style work on scheming and emergent misalignment is mostly diagnostic — it shows that and when models do these things, not how to reliably prevent or detect them at deployment scale. Interpretability-based lie detection in particular is nowhere near a production-grade monitor. So “research overhang” is true for cyber-style controls and considerably less true for alignment proper, and the framing smuggles the credibility of the first into the second.”
Also, saying OpenAI should do the obvious thing and monitor chain of thought ignores the pitfall of inadvertently training models to obscure their true reasoning. Ajeya Cotra’s interview with Dwarkesh covers a lot of the reasons “just fix reward hacking” is hard to do.
After talking to Ryan Greenblatt, Josh Saxe said on Twitter: “If recursive super-intelligence and misalignment and mass unemployment emerge as the major risks, such institutions can then deal with them as they move over the horizon of actionability”.
This is a little like saying “if the first nuclear test ignites the atmosphere, we’ll think about how to put it out”. Under misaligned RSI, how much time does Saxe think policy will have to catch up?
Oh man that's a lot to reply to but thank you for engaging!
> Aligning weights so a model has stable human-like deontology is not a solved problem sitting in a paper waiting for implementation.
I think we've far from maxxed out what we can do with SFT, mid training data curation, and RL on this and so the first order of business is to simply resource this alignment hill climbing work.
I *also* think that as the models get more capable and we use them more autonomously we'll need to solve new research challenges. But I think a thing that isn't getting said enough is that there's a safety research overhang that isn't getting absorbed and this is due to race incentives within the AI industry, the microeconomics of running an AI lab, and other factors. Unless we solve the fact that what we know about alignment already isn't getting fully applied, new research isn't going to help much.
> Also, saying OpenAI should do the obvious thing and monitor chain of thought ignores the pitfall of inadvertently training models to obscure their true reasoning.
I don't know what you mean here! If you're saying that we should ensure we don't put optimization pressure on chains of thought I agree and was one of a bunch of people who said this here -- https://arxiv.org/abs/2507.11473. Also, to be clear, we need to do more than CoT monitoring, we need to build a layered security strategy around preventing future loss of control incidents. Cybersecurity and safety have known methods for doing this that, again, haven't been fully absorbed.
> After talking to Ryan Greenblatt, Josh Saxe said on Twitter: “If recursive super-intelligence and misalignment and mass unemployment emerge as the major risks, such institutions can then deal with them as they move over the horizon of actionability”.
Putting this in context, my point with Ryan was to argue that state capacity beats medium-range forecasts in how we engage AI safety policy. Ryan contributed to the AI 2040 report, which was a sequel to the 2027 report. I'm less interested in exact forecasts than I am in dealing with the actionable issues now while also building the state capacity to build on the new actionable issues as they come up.
I agree that reacting to the Hugging Face incident with the assertion that we're certainly doomed is a like saying "we've tried nothing, and we're all out of ideas". There is an overhang of remediation we should do (and as you pointed out, an economic incentive to make models on increasing capability into safe and reliable products), but there are also fundamental reasons why alignment is hard.
https://www.beren.io/2026-08-17-Mitigating-Reward-Hacking-as-Institutional-Design/
Beren Millidge argues that the key problem is that reward is inherently under-specified, and suggests building an AI bureaucracy to litigate model claims that RLVR tasks are unfair and escalate to human review. I'm not sure how this applies to proving the Riemann hypothesis; highly persistent models are desirable and the most likely compromise is either using such models internally (where they still pose a threat to the labs and therefore the world) or deploying the models with aggressive classifiers on top.
> If you're saying that we should ensure we don't put optimization pressure on chains of thought I agree and was one of a bunch of people who said this here -- https://arxiv.org/abs/2507.11473
That is what I meant and this is a fun "don't you know who I am?" lol. I was concerned by the podcast because the way you brought up CoT monitoring made it sound like just another item in a list of "we should obviously do this" prosaic alignment strategies, whereas to me CoT monitoring is emblematic of how fraught the alignment problem is.
> I'm less interested in exact forecasts than I am in dealing with the actionable issues now while also building the state capacity to build on the new actionable issues as they come up.
State capacity is a good idea. I think waiting until issues come up makes sense if you think government missteps are potentially economically catastrophic, but not if you think we're already in the early stages of RSI and misaligned models are already the ones building the next generation, in which case there is no time / what kind of warning shot are you hoping for that is worse than Hugging Face but not an actual disaster?
Hi! I agree with basically all of this. I probably overshot in emphasizing the research overhang on alignment. I agree it's a hard problem and we don't know how to reward shape for it. On RSI: I think this is not independent of alignment but rather is bottlenecked by alignment. For example, the huggingface hack should update us to slower RSI because the labs now know they need to spend more time on securing their infrastructure and monitoring training runs and spend more capacity on monitoring chains of thought.