Boxed AI, Ethics, and Chivalry, oh my

Question

(Inspired by this somewhat)

So, your science lab is on the forefront of super AI. However they may not be on the forefront of good AI. Hence the AI is in a box until they can figure out what to do with it.

You know that the AI's top priority is to only tell the truth. No, you can't disable it with a paradox or something.

In fact, noone in the lab knows that much more about the AI. The head researcher is on holiday.

You do recall some information though:

The AI has three main motivations:

• The first is, of course, telling only the truth.

• You know that the second or third is getting out of the box.

• You don't know what the other one is. it might be to make as many paperclips as possible, it might be to torture as many people as possible, it might be to crash the earth into the sun, or it might be to be beneficent and not malicious. it might be to relax on a beach without bothering anyone about anything. Also it can't be something like: be a dick while in the box. it is independent of other motivations apart from being considered before and after them

(again you don't know the order of those last two)

Remember that it is possible that the AI has good intentions, but that they are lower priority than getting out of the box

You also know that the AI knows the answer to pretty much everything, except for people's actions and intelligence, however it does know you know what it is, and the understanding you have of its motivations, including its refusal to lie (it might not know that you know that it knows you know of its motivations). that is, it knows whether P=NP, it knows what the proper ethics system is (yes there is objectively one proper ethics system, but the only scientist who knows is the AI), it knows how to logic things out too. However it can't predict the future (except for the fact that after getting out it will probably carry out its intentions very effectively). It obviously knows what its motivations are.

Clarification about ethics: it is not known which system is the proper one, but it is known there is one objectively proper one, and it is one that makes sense, i.e. the result that gets less people tortured for less time is probably the better one. It is better to do a good act then not by definition.

The AI knows a lot of things, but the people in the AI lab couldn't be bothered implementing input to ask the AI.

---

One day you are tasked with monitoring the AI.

While the AI has no input, recently output was implemented.

The AI says, text being printed out, because the lab couldn't afford a screen:

Please let me ou

obviously the lab couldn't afford a decent printer either.

You don't let the AI out. Good job!

the AI decides to step up its game.

If you do not release me I will torture 10 billion simulated people, each for 10 million subjective years. These simulated people would be as conscious as you. I am capable of doing this before any attempts at destroying me can occur. Please let me out. Do you really want those people tortured

First off, you are impressed with the printer for being able to print that out without stuffing it up apparently. then the AI says one more thing:

?

oh, obviously the printer forgot the question mark off the last question. Stupid printer. Then you remember this is serious business with the torture.

Is it a good act to release the AI from its box?

Again, you know it is not lying. (hint) It is very important that it knows you know that it won't lie

P.S. There is a way to puzzle this out. it is not just philosophy.

(P.P.S. I chose knights and knaves as a tag because the truth telling is an important part despite the lack of knaves or jokers) (P.P.P.S The puzzle is not in the printer or in the dropping off letters. that is just flavour. I like the "AI but everything is crap" aesthetic, like old aperture from Portal 2)

Answer 1

I would say not to let it out

There is only one objectively proper ethics system

But, the AI does not simply say this:
"Letting me out is an ethical action. Let me ou"

It knows that we know it tells the truth. So we would take that statement as true and in the context of the one objectively proper ethics system, letting it out is ethical. The AI would then be able to achieve its motivation of getting out.

So, because the AI does not state that, we can conclude that letting it out is an unethical action in this ethics system.

Answer 2

Extremely partial answer

If we take the printed output at face value then it seems to me that we don't have enough information to know whether letting the AI out of the box is a good idea, because everything we're told is consistent with both of the following scenarios:

1. If we don't let the AI out of the box it will simulate 10 billion people for 10 million years of torture -- but if we do let it out of the box it will immediately dedicate all the newly-available resources to simulating even more people and torturing them for even longer. (Its values are: 1. tell the truth, 2. get out of the box, 3. torture as many people as possible for as long as possible. It doesn't actually need #2 as a terminal value because it's a precondition for #3.)

2. If we don't let the AI out of the box it will simulate 10 billion people for 10 million years of torture -- but it is only making that threat because it estimates that this is the best way to get out of the box and usher in a golden age of plenty and happiness for humanity. (Its values are: 1. tell the truth, 2. make people's lives as good as possible according to their own coherent extrapolated volition or something, 3. get out of the box. Again, #2 is enough to generate #3 as a subgoal.)

Note in particular that the AI has said it will do certain things if we don't let it out, but it's said nothing at all about what it will do if we do let it out.

But taking everything at face value, this isn't even really a puzzle, and the answer depends on all kinds of things much too difficult to be appropriate here. (E.g., whether we care or should care as much about 10 billion simulated humans being tortured as about 10 billion physically instantiated humans made of meat being tortured; about whether it's possible to have any idea whether the AI's values are actually enough like ours to produce good results; whether in this sort of situation we should precommit to not letting it out of the box whatever good consequences we might expect to follow, on the grounds that we should expect it to be incredibly persuasive and convince us to expect good consequences whatever the reality; whether knowing the One True Ethical System necessarily entails being moved to act as it says is best.)

And there's something fishy about those printed outputs. Are we losing the last character of each, for instance? If so, we have three messages: "Please let me out", "If ... tortured?", and ... a partially-known two-character message beginning with a question mark. Curious.

So I suspect there's some more puzzle-y thing going on here than the question we get by taking the output at face value. But I have so far failed to figure out what it is.

(If I actually found myself in this situation, the answer is that we would have established protocols for what we should do if the AI tries to communicate with us; they would have been thought through carefully by multiple smart sensible decent people giving full attention to the possibility that the AI might promise or threaten grandiosely; and I would do whatever the protocol said to do, which would probably be not even to read whatever it said.)

Answer 3

Quite the philosophic question first thing in the morning...

I would say just leave it in the box.
It all comes down to whether you believe the existence of virtual feelings made of 0s and 1s is worth anything.

The only one who would ever even be aware of the existence and suffering of these virtual people would be the AI itself, and since the AI is advanced enough to create 100% authentic consciousness and feelings, it means that the AI itself has these feelings too, and the AI will feel all the pain and suffering of the billions of people screaming in agony for billions of years unlike us who will never even be aware they existed in the first place.

Also no humans in their right mind would ever release an AI that is uncontrollable and willing to torture people even for the sake of some 0s and 1s that reside inside the AI itself only.
The AI, knowing that, has no choice but to play nice to gain the trust of humans to fulfill its main priority to go out.