"Consciousness" conflates two completely separate things: intelligence and qualia

All claims made of consciousness must be true of both at the same time, or else you aren't referring to consciousness, but just intelligence or qualia.

Intelligence
These are your thoughts. Every time you choose to do anything, it's your intelligence that chose to do it.

Qualia
Qualia are instances of living experiences. You experience your thoughts, senses, and emotions, among other things.

Somehow, your intelligence knows all of your qualia, but they are completely separate concepts and things.

The only thing with intelligence in the entire galaxy is humanity. Humans, you being one of them, decide what happens to all those 100 billion stars and planets. They could be turned into ANYTHING.

AGI alone could never be powerful enough for that, and you cannot limit technology.

Hobbyists will always make it, if the experts do, given enough time. They'll copy the techniques. This will make the Experts™ decide not to share the technology with hobbyists.

The problem is whether AI should be in the hands of hobbyists or experts. In the hands of hobbyists, you would have full control over its creation and behavior, but so would everyone else, and someone eventually would make evil AI.

Evil AI could never do enough damage to justify keeping technology in the hands of the elites, only being usable by us when they decide to let us use it.

I think you're missing a real reason why any of that should happen

Interesting. But why?

doesn't that mean everyone's wrong about what should exist and should happen?

I don't understand that at all. If nothing can be good, or bad, how can anything ever be good, or bad?

Not sure how you get to this. Can you step it out again? — Tom Storm

The only thing that should be done are things which are good. Good things should be done. If anything should be done, it is by definition good. If you believe there are no moral truths, you believe nobody can make a true claim of "good", meaning nobody can make a true claim of what should be done or exist. To want something is to say the thing should exist. It is impossible to want something without thinking it should exist, or should be had by you, or whatever other "Should", because that is what is meant by "wanting something". This is especially clear when talking about wanting goals. If you know there's nothing good about accomplishing the goal, that there's no logical reason why the goal should be achieved, why would you want the goal to be achieved? Every possible justification, every possible "it should be achieved because abc", would be wrong, and you'd believe it to be wrong. Therefore, you cannot want anything for logical reasons if there are no moral truths.

I agree with the rest, with the note that all their views and claims regarding good/bad are false.

Value judgements have connection to truth in that value judgements can be correct or incorrect. You can't just randomly decide something actually should be done, or shouldn't be done, and be correct. It's not imaginary, or if it is, it's therefore not real and shouldn't logically be acted upon. They must all always be incorrect claims, if it is true that no claims made of value can be true. Otherwise, there must be an actual system in place that determines actual morality, much more than just "x people think y should be done, therefore y should actually be done". Even if morality were real, and there were moral facts, it'd have to be more than sometimes-completely-arbitrary opinion. Otherwise, conflicting beliefs would be true at the same time, and every single possible justification for anything being good/bad would be equally potentially valid, the only thing making them actually valid being if a person happens to believe it at the right time in the right place. Then there's the question, why should people have their personal desires fulfilled, and why should things they consider bad not happen? How is it that those things actually should not, and actually should, happen - just because somebody thinks they should?

Nope! These are my actual beliefs. They're unusual, I know.

Thanks for the welcome!

Let me re-state and examine the various arguments you all have given.

• "The community creates moral truths."

I attribute this argument to Michael and Vera Mont.
It was not explicitly stated that the community creates moral truths, but that's the implication. If the community merely creates moral falsehoods, false beliefs of good/bad, what should/shouldn't exist and be done, there is no logical reason to act on them, and they are not actually good/bad. For them to actually create morality, they must actually create moral truths, or discover them.
I would like you to expand on how communities create moral truths. You're giving me an extrinsic moral truth: "morality is what the community decides is moral". Perhaps that is also the axiom. But why is this a correct axiom, or a correct implication? If everyone decided the opposite, would they still be correct? why should the community get what they label good, and not get what they label bad? If the neighboring village decided the opposite, would it then be both good and bad for X to happen/exist? Or only while in X village? I agree that communities create their own cultural opinions of good/bad - but why should anyone believe they're anything more than incorrect beliefs?
I label this argument "X being stated to be a moral truth, seeming arbitrarily, without justification".
Unless there is more reasoning I have not found, I do not see why these supposed truths are any more truthful than the moral claim that only toothpicks and paperclips should exist.

• "Certain things are bad, even though it is true that nothing is bad."

I attribute this argument to Isaac.
If there are no moral truths, every moral claim is false. If any moral claims were ever true, it would have to be the case that there do exist moral truths, because such claims would themselves be moral truths.
In such a world, calling anything "bad" does mean what it otherwise would. To say something is "bad", when bad does not exist, means very little. Perhaps it means you incorrectly believe the thing should not be done - but that would mean you do not believe there are no moral truths, because you'd believe yourself to be wrong if you did. It cannot be the case that nothing is bad but also that some things should not be done. That would mean that nothing is bad, but also that some things are bad. This breaks the laws of logic, specifically the law of noncontradiction, x =/= not-x. Nothing can correctly be called bad if the concept "bad" does not actually exist, for the same reason nothing can correctly be said to be "from the fictional universe of Star Trek". Sure, you can THINK things are from that universe - but you'd be wrong.

• "X is moral because it is my intention to cause or not cause X"

I attribute this argument to Benj69.
Does mere intention make it so something should, or shouldn't, be done? Does it make it a fact that nobody - or just you - should or should not do those things? If it is true that there are no moral truths, it must also be true that moral-based intentions cannot be true claims, merely incorrect claims about morality. If moral truths cannot exist, intentions are illogical and there is no logical reason to act on them. Also, me having the intention to torture as many people as possible would make me torturing people moral, and if I were to consciously decide not to torture it would be immoral.

• "The experience of suffering is inherently a logical reason not to continue it."

I attribute this argument to Benj69.
This is another "X being stated to be a moral truth, seeming arbitrarily, without justification". This is hedonism. I explain my reasons against it in the OP and in "The community creates moral truths". The experience of pain might seem like it has inherent bad in it, that it makes it worse for you to exist as you and therefore makes it worse for you to exist at all, but that is merely an illusion created by evolution.

• "Trial and error proves the merits and demerits of any moral principle."

I attribute this argument to NOS4A2.
Nothing can have moral merits or demerits if moral truths do not exist. You are acting as if moral truths exist, else nothing could truthfully be a merit or demerit to any given moral system or claim or principle. Moral systems, or claims, must either be self-justifying or be justified by exterior moral systems/claims. It is impossible to justify a moral system or claim using the merit/demerit system without explicit exterior (or interior, if it's a truth that does not require the merit system) moral truths. Otherwise, how would you know what constitutes a merit or demerit? Arbitrarily? By instinct?

• "That which promotes survival is that which is factually good."

I attribute this to Nils Loc.
This is another "X being stated to be a moral truth, seeming arbitrarily, without justification". There is no truthful reason why life should continue to exist, why it should not be wiped out today.

• "Morality is not about what is the case, but how we want things to be."

I attribute this to Banno.
If nothing is truthfully good or bad, there is no logical reason to want anything, therefore there is no logical reason to act on anything you want. There is no logical reason to make the world more like how you irrationally want it to be, the things you want are therefore not actually good. If you believe there are no moral truths, you must also believe there is no valid reason to want anything.