All of Zvi Mowshowitz's Comments + Replies
If you wish to make an apple pie, you must first become dictator of the universe [draft for comment]
Focus will be on the actual arguments in section on optimization pressure, since that seems to be the true objection here - previous sections seem to be rhetoric and background, mostly accepting the theoretical basis for the discussion.
I take it this essay presumes that the pure version of the argument is true - if you were so foolish as to tell a sufficiently capable AGI 'calculate as many digits of Pi as possible' with no mitigations in place, and it has the option to take over the world to do the calculation faster, it's going to do that.
However I inter... (read more)
11yThanks a lot, Zvi.
Meta-level: I think to have a coherent discussion, it is important to be clear
about which levels of safety
[https://twitter.com/jasoncrawford/status/1671553789017595907] we are talking
about.
* Right now I am mostly focused on the question of: is it even possible for a
trained professional to use AI safely, if they are prudent and reasonably
careful and follow best practices?
* I am less focused, for now, on questions like: How dangerous would it be if
we open-sourced all models and weights and just let anyone in the world do
anything they wanted with the raw engine? Or: what could a terrorist group do
with access to this? And I am not right now taking a strong stance on these
questions.
And the reason for this focus is:
* The most profound arguments for doom claim that literally no one on Earth can
use AI safely, with our current understanding of it.
* Right now there is a vocal “decelerationist” group saying that we should
slow, pause, or halt AI development. I think this argument mostly rests on
the most extreme and IMO least tenable versions of the doom argument.
With that context:
We might agree, at the extreme ends of the spectrum, that:
* If a trained professional is very cautious and sets up all of the right
goals, incentives and counter-incentives in a carefully balanced way, the AI
probably won't take over the world
* If a reckless fool puts extreme optimization pressure on a superintelligent
situationally-aware agent with no moral or practical constraints, then very
bad things might happen
I feel like we are still at different points in the middle of that spectrum,
though. You seem to think that the balancing of incentives has to be pretty
careful, because some pretty serious power-seeking is the default outcome. My
intuition is something like: problematic power-seeking is possible but not
expected under most normal/reasonable scenarios.
I have a hunch that the crux has something to
I think it's an important crux of its own which level of such safety is necessary or sufficient to expect good outcomes. What is the default style of situation and use case? What can we reasonably hope to prevent happening at all? Do our 'trained professionals' actually know what they have to do, especially without being able to cheaply make mistakes and iterate, if they do have solutions available? Reality is often so much stupider than we expect.
Saying 'it is possible to use a superintelligent system safely' would, if true, be highly insufficient, unless... (read more)