Epistemic Status: likely
Imagine you’re in one of those ancient tragedies. You know, the ones with prophecies and fate and stuff.
Yeah, I know, prophecy is so last paradigm and considered harmful and so on. Just stick with me here.
Suppose you’re just walking down the road, minding your own business, and you happen to walk by your town oracle. Nothing very remarkable, it happens everyday; ve walks vis little poodle regularly and you usually catch ver on your way home from the grocery store. Then all-of-a-sudden, their eyes roll back and they speak in the deep, inhuman voice of prophecy. It’s something about how some horrible event shall betide thee; maybe “thou shalt killeth thine heir” or something similarly cliché. Let’s just go with that as an example.
Generally, prophecies are infallible, and this one is no different. Thus, you know with certainty that the aforementioned tragedy shalt come to pass. Maybe thou’ll accidentally your child in a sparring match when you’re just simply trying to show them the ropes. Perhaps thine child will commit some unforgivable crime, and you must sentence them to death. Or you just lose your temper with them at the wrong time.
They will die by your hand, is all I’m saying.
However, you’re also well-read enough to know how these patterns typically resolve. Subverting a prophecy never works. If you forego the sparring match, your child will die unprepared in battle. If you exonerate them of their crime, ve’ll recidivate. Or you just won’t be there to save them while avoiding them.
You’d be hoist by your own petard, is what I’m saying.
Thou canst not flee either. Mayhaps if thou runeth to the hills, your child attempts a desperate search, and by a thoroughly bizarre sequence of coincidences you mistake them for a bandit or highwayman, and realize only when it’s too late.
Even if you attempt suicide, your child, unable to live with the grief, will do likewise.
The game was rigged. Fate was the original master of the Xantos gambit.
I bring this up because I thought about these types of scenarios a long time ago and then I realized in this lies a paradox.
When your feeble mortal mind comes to contact with the preternatural certainty of Prophecy, your agency is subsumed into that of Fate. If we try to personify this (which will allow for a better analysis), the best way to go about it is imagining a scenario where Fate’s role is played by a superintelligent agent with a diametrically opposed utility function (read: evil).
If you’ve been immersed in the Lesswrong memeplex, just imagine an evil Omega.
Now suppose you’re walking down the street, minding your own business, when all-of-a-sudden, you walk past this evil!Omega. You’re familiar with vis reputation, and you make no sudden moves, and avoid eye contact. Ve just smiles and walks past without doing anything conspicuous.
You relax a little but are still quite worried. Being ridiculously super-intelligent, evil!Omega could probably know your thoughts just by looking at the micro-expressions on your face. You are fully convinced, if ve wanted to, ve could effectively take control of your behavior.
And, of course, ve does want to, having literal anti-human utility function and all.
Later that day, you come about a burning orphanage. On any other day, your optimal course of action would be assured. But what if evil!Omega optimized this situation perfectly counter to your utility function? What if, after running in there and trying to save them, you’re hit with falling debris and die (or worse)? What if, after running away or giving up halfway, someone else immediately rushes in there and saves them, and you’re demonized by the locals?
And, naturally, what if the whole point of that encounter earlier was to get you second-guessing yourself at the right moment?
You can see the problem here.
Formally, this is situation is something like
A translation: For every every action: that if you think the utility of is positive then the utility of is really negative.
Or, If you think something going to go well, it’s really going to screw you over. And vice versa.
If you squint, the paradox here is just a little like the liar paradox:
A: A is false
Really, the whole paradox business is “define s to be a paradoxical statement … oh no! we have a paradox!”
Jokes aside, most paradoxes are like this: the irresistible force paradox comes from sneaking in an irresistible force and an immovable object simultaneously, when the existence of one precludes the existence of another1.
The trouble with paradox is there’s always a hidden assumption that’s sneaked in by mistake, and it’s never obvious where or what it is.
1: The real resolution to this paradox is noting that it was never stated that the force and object had to interact; it’s quite possible the force would just phase right through the object.
Where’s the hidden assumption that makes this paradox possible?
One avenue of attack is that canonically Omega works by simulating what you do (before you do it). It’s a trivial exercise to demonstrate a computer can’t simulate something more complex that it is. Somewhat similar to Vinge’s assertion that an author can’t write characters smarter than the author verself is.
Then you might be able to subvert evil!Omega and escape the bounds of predictability by self-improving until you’re smarter than evil!Omega. Right?
That might be exact what ve is counting on.
What if we look at this in terms of optimizing power?
Were you were interacting evil!Omega over the internet, it’s clear ve couldn’t make someone blow up a bus by sending them a single bit (hopefully).
This implies a nice metaphor with encoding theory. You can’t universally compress every string into a smaller string. You can’t manipulate an agent into committing any act in actionspace with uniformly less information than what’s needed to specify that action.
In the case of people, their actions are heavily compressed with respect to what the typically tend to do. If you want to convince a nice person to pick up something you dropped, that’ll be much, much easier than convincing them to punch a random passerby, even though they both take a similar amount of information to specify (naively).
So, evil!Omega can’t control your actions here in front of the orphanage with just a smile and no conspicuous actions, because more information than that is need to specify some relevant action, especially one you aren’t poised to do.
You finally reason this all this out, still standing in front of the orphanage.
Now confident in your free agency, you run into the orphanage with heroism on your mind and bravery in your heart. You are mangled by falling debris, becoming paralyzed for life.
Because fate is a master of the Xantos gambit.