helpful framing and agree but instead of a binary categorization, stumbler : optimizer, i find it helpful to ask: “what is the manifold ___ is optimizing over?”
maybe naive but i think there is more that separates e.g. obama vs trump vs tim cook than different initial conditions
also, i am an anthropic shill, but i think it is worth pointing out they intentionally tried to set up the corporate governance with structural constraints so the hinge is not just Dario’s personality.
this did not work with OAI but I sincerely believe Anthropic (the company) has a better shot at staying aligned
Seems like it should be possible to be an optimizer who is optimizing for something complex and/or morally good, though? In which case “sacrificing [resources, power] for principles” doesn’t always conflict with optimization… Like maybe some kind of instrumental convergence is happening for human optimizers where they all need to max out resource/power gathering in the early/mid term but they should be cashing out more and more in favor of whatever they’re actually optimizing towards as time goes on. Unless you’re saying that the optimizers who are optimizing towards simple power acquisition for its own sake are the only ones that end up with significant power bc the others cash out “too early”…?
I think this is directionally interesting, but it fails in one critical way: sample size. For example, if there were 500,000 people working in AI before ChatGPT, and maybe 1% of them were good enough leaders to become a lab CEO, that would leave a pool of 5,000 potential lab CEOs competing. Now imagine that 5% of those people are optimizers. That gives us 250 optimizers versus 4,750 non-optimizers. With a sample size like that, you could still easily end up with a non-optimizer lab CEO in many cases, even if being an optimizer provides a 2–3x force multiplier.
it's a good point! but keep in mind that the type of optimizers i'm talking about here tend to be optimizing *for power* and have potentially many paths towards it. eg sam altman was doing many other things before becoming involved with openai and doesn't particularly have AI expertise himself, perhaps demis isn't an optimizer but sundar in fact controls deepmind, etc. i agree that more strict and technical / optimizer-neutral qualifications necessary for position do reduce the odds, eg chief scientists tend to be much more truthseeking than CEOs.. but simultaneously not all hypercompetent AI researchers actually particularly want to *chase power* in this way, and if the path to power is clearly through ai expertise then the optimizers will try to gain it. it's not such a clean selection dynamic as you present
Really interesting read. I feel like an optimizer until I see people that really are. Reminds me a little bit about the book "playing to win" about esports competitions. Most people say they want to win, but don't actually do the things they need to to do it.
Suppose you're very unalligned with humans, rushing to advance AI capabilities makes more sense? In the worst case imagine you're a sadist, if others develop AI there may be almost zero sadism in the world the way you like it. Meanwhile someone who wants happy rainbows could still get that wish by having the competitor develop it.
>the collective can constrain optimizers though. the important thing, the only thing that works, is to change the rules of the game. they optimize under the constraints and resources of their environment. so we, as the environment, have to ensure the outcomes that are reachable by them are acceptable to us.
IMO capitalism has made lots of progress here in shifting would be napoleons to industry
I wonder if optimizers mostly have more clear and legible goals than other people?
Reminds me of playing historical strategy games - do you play the strategy which maximizes the probability you'll win, or the strategy which is more historically accurate, even if the game doesn't model that which made the historically accurate strategy more optimal than the meta in game strategy.
helpful framing and agree but instead of a binary categorization, stumbler : optimizer, i find it helpful to ask: “what is the manifold ___ is optimizing over?”
maybe naive but i think there is more that separates e.g. obama vs trump vs tim cook than different initial conditions
also, i am an anthropic shill, but i think it is worth pointing out they intentionally tried to set up the corporate governance with structural constraints so the hinge is not just Dario’s personality.
this did not work with OAI but I sincerely believe Anthropic (the company) has a better shot at staying aligned
Seems like it should be possible to be an optimizer who is optimizing for something complex and/or morally good, though? In which case “sacrificing [resources, power] for principles” doesn’t always conflict with optimization… Like maybe some kind of instrumental convergence is happening for human optimizers where they all need to max out resource/power gathering in the early/mid term but they should be cashing out more and more in favor of whatever they’re actually optimizing towards as time goes on. Unless you’re saying that the optimizers who are optimizing towards simple power acquisition for its own sake are the only ones that end up with significant power bc the others cash out “too early”…?
oh hey, that title photo looks familiar...
I think this is directionally interesting, but it fails in one critical way: sample size. For example, if there were 500,000 people working in AI before ChatGPT, and maybe 1% of them were good enough leaders to become a lab CEO, that would leave a pool of 5,000 potential lab CEOs competing. Now imagine that 5% of those people are optimizers. That gives us 250 optimizers versus 4,750 non-optimizers. With a sample size like that, you could still easily end up with a non-optimizer lab CEO in many cases, even if being an optimizer provides a 2–3x force multiplier.
it's a good point! but keep in mind that the type of optimizers i'm talking about here tend to be optimizing *for power* and have potentially many paths towards it. eg sam altman was doing many other things before becoming involved with openai and doesn't particularly have AI expertise himself, perhaps demis isn't an optimizer but sundar in fact controls deepmind, etc. i agree that more strict and technical / optimizer-neutral qualifications necessary for position do reduce the odds, eg chief scientists tend to be much more truthseeking than CEOs.. but simultaneously not all hypercompetent AI researchers actually particularly want to *chase power* in this way, and if the path to power is clearly through ai expertise then the optimizers will try to gain it. it's not such a clean selection dynamic as you present
Really interesting read. I feel like an optimizer until I see people that really are. Reminds me a little bit about the book "playing to win" about esports competitions. Most people say they want to win, but don't actually do the things they need to to do it.
Suppose you're very unalligned with humans, rushing to advance AI capabilities makes more sense? In the worst case imagine you're a sadist, if others develop AI there may be almost zero sadism in the world the way you like it. Meanwhile someone who wants happy rainbows could still get that wish by having the competitor develop it.
>the collective can constrain optimizers though. the important thing, the only thing that works, is to change the rules of the game. they optimize under the constraints and resources of their environment. so we, as the environment, have to ensure the outcomes that are reachable by them are acceptable to us.
IMO capitalism has made lots of progress here in shifting would be napoleons to industry
I wonder if optimizers mostly have more clear and legible goals than other people?
Reminds me of playing historical strategy games - do you play the strategy which maximizes the probability you'll win, or the strategy which is more historically accurate, even if the game doesn't model that which made the historically accurate strategy more optimal than the meta in game strategy.
there are many fields besides ai in which we will need more agency