Nick Bostrom on singletons
Can we trust evolutionary development to take our species in broadly desirable directions?
[…]
What we shall call the Panglossian view maintains that this past record of success gives us good grounds for thinking that evolution (whether biological, memetic, or technological) will continue to lead in desirable directions. This Panglossian view, however, can be criticized on at least two grounds. First, because we have no reason to think that all this past progress was in any sense inevitable‒‐much of it may, for aught we know, have been due to luck. And second, because even if the past progress were to some extent inevitable, there is no guarantee that the melioristic trend will continue into the indefinite future.
[…]
Evolution made us what we are, but no fundamental principle stands in the way of our developing the capability to intervene in the default course of events in order to steer future evolution towards a destiny more congenial to human values.
Directing our own evolution, however, requires coordination. If the default evolutionary course is dystopian, it would take coordinated paddling to turn the ship of humanity in a more favorable direction. If only some individuals chose the eudaemonic alternative while others pursued fitness‐maximization, then, by assumption, it would be the latter that would prevail. Fitness‐maximizing variants, even if they started out as a minority, would be preferentially selected for at the expense of eudaemonic agents, and a process would be set in motion that would inexorably lead to the minimization or disappearance of eudaemonic qualities, and the non‐eudaemonic agents would be left to run the show.
To this problem there are only two possible solutions: preventing non‐eudaemonic variants from arising in the first place, or modifying the fitness function so that eudaemonic traits become fitness‐maximizing.
[…]
A singleton could be a rather minimalist structure that could operate without significantly disrupting the lives of its inhabitants. And it need not prohibit novelty and experimentation, since it would retain the capacity to intervene at a later stage to protect its constitution if some developments turned malignant. Increased social transparency, such as may result from advances in surveillance technology or lie detection, could facilitate the development of a singleton. Deliberate international political initiatives could also lead to the gradual emergence of a singleton, and such initiatives might be dramatically catalyzed by ‘wild card’ events such as a series of cataclysms that highlighted the disadvantages of a fractured world order. It would be a mistake to judge the plausibility of the ultimate development of a singleton on the basis of ephemeral trends in current international affairs. The basic conditions shaping political realities may change as new technologies come online, and it is worth noting that the longterm historical trend is towards increasing scope of human coordination and political integration. If this trend continues, the logical culmination is a singleton.
[…]
A singleton could take a variety of forms and need not resemble a monolithic culture or a hive mind. Within the singleton there could be room for a wide range of different life forms, including ones that focus on non‐eudaemonic goals. The singleton could ensure the survival and flourishing of the eudaemonic types by restricting the ownership rights of non‐eudaemonic entities, by subsidizing eudaemonic activities, by guaranteeing the enforcement of property rights, by prohibiting the creation of agents with human‐unfriendly values or psychopathic tendencies, or in a number of other ways. Such a singleton could guide evolutionary developments and prevent our cosmic commons from going to waste in a first‐come‐first‐served colonization race.
[…]
We do not know that a dystopian scenario is the default evolutionary outcome. Even if it is, and even if the creation of a singleton is the only way to forestall ultimate catastrophe, it is a separate question what policies it makes sense to promote in the here and now. While creating a singleton would help to reduce certain risks, it may at the same time increase others, such as the risk that an oppressive regime could become global and permanent. If our preliminary study serves to draw attention to some possibly non‐obvious considerations and to stimulate more rigorous analytic work, its purpose will have been achieved.
https://www.nickbostrom.com/fut/evolution.html
It could act merely as a subtle enforcer of certain background conditions that could serve, e.g. to guarantee security or to administer some other minimal governmental tasks. […] When considering the characteristics of a singleton it would be a mistake to assume that it would necessarily possess the attributes commonly associated with large human bureaucracies — rigidity, lack of imagination, inefficiency, a tendency to micro-manage and to expand its own powers, etc. This would be true of some possible singletons but it might not be true of others.
The concept of a singleton is thus much broader and more abstract than concept of a world government. A world government (in the ordinary sense of the word) is only one type of singleton among many.
Nevertheless, all the possible singletons share one essential property: the ability to solve global coordination problems. Intelligent species that develop the capability to solve global coordination problems, such as those listed in the next section, may in the long run develop along very different trajectories than species that lack this capacity.
[…]
A major risk with creating a singleton is that it would turn out to be a bad singleton. Smaller units of decision-making, such as states, can also turn bad. But if a singleton goes bad, a whole civilization goes bad. All the eggs are in one basket.
Anders Sandberg on transhumanism
Anders Sandberg characterises the essence of transhumanism as:
Questioning the necessity and optimality of the current human condition, as well as suggesting that methods to improve it might be both feasible and desirable. If we were living in a fantasy world transhumanists would no doubt argue in favor of using cutting edge magic to improve life.
Sandberg notes that, in contrast, Nick Bostrom sometimes emphasises “the desire to explore the posthuman realm, the states of being that are currently unavailable to us”.
But:
In practice there [are] plenty of transhumanists who are not terribly interested in becoming radically posthuman - a few extra centuries in comfort with enhanced minds and bodies is all right with them. They want to personally explore the nicer reaches of the human modes of being and maybe some near posthuman modes. But the common theme is that they do not see the current limitations as desirable, and think (with varying levels of confidence and evidence) that there are or will be ways of overcoming them.
Elsewhere, Sandberg writes:
All things considered, the human condition has many things going for it. Unfortunately, there are some problems. The need to sleep. Hangovers. Pain. Forgetfulness. Bad judgement. Cruelty. Depression. Ageing. Mortality. To name a few.
One approach is to try to accept these limitations and pains. Learning to live with adversity can sometimes be good for a person—it might teach them perseverance or humility . . . Unfortunately, adversity can also numb, harden, or crush us—and surely we should not just accept cruelty or ignorance as a fact of life.
Another approach is to try to fix our limitations and problems. This is the goal of human enhancement: if we are forgetful, maybe we can improve our brains to forget less, for example by taking drugs that increase neural plasticity. To counteract ageing we might use gene therapy to increase production of enzymes that decline with age, remove aged and ill cells, or add fresh stem cells.
Human enhancement is all around us. The morning coffee or tea contains stimulating caffeine that counteracts sleepiness. Vaccines are a form of collective, global immunity against illnesses we may have never encountered. Less invasively, most of us live our lives with smartphones connecting us to a sizeable fraction of humanity and its knowledge. We are never alone, never lost, never bored, able to record anything. Our medieval ancestors would find our (long, healthy, rich) lives superhuman.
Robin Hanson on world government and collective suicide
If your mood changes every month, and if you die in any month where your mood turns to suicide, then to live 83 years you need to have one thousand months in a row where your mood doesn’t turn to suicide. Your ability to do this is aided by the fact that your mind is internally divided; while in many months part of you wants to commit suicide, it is quite rare for a majority coalition of your mind to support such an action.
[…]
When there are powers large enough that their suicide could take down civilization, then the risk of power suicide becomes a risk of civilization suicide. Even if the risk is low in any one year, over the long run this becomes a serious risk.
[…]
Alas, central power risks central suicide, either done directly on purpose or as an indirect consequence of other broken thinking. In contrast, in a sufficiently decentralized world when one power commits suicide, its place and resources tend to be taken by other powers who have not committed suicide. Competition and selection is a robust long-term solution to suicide, in a way that centralized governance is not.
This is my tentative best guess for the largest future filter that we face, and that other alien civilizations have faced. The temptation to form central governments and other governance mechanisms is strong, to solve immediate coordination problems, to help powerful interests gain advantages via the capture of such central powers, and to sake the ambition thirst of those who would lead such powers. Over long periods this will seem to have been a wise choice, until suicide ends it all and no one is left to say “I told you so.”
https://www.overcomingbias.com/2018/11/world-government-risks-collective-suicide.html
Future filters could result from either too little or too much coordination. To prevent future filters, I don’t know if it is better to have more or less world government.
https://www.overcomingbias.com/2018/05/two-types-of-future-filters.html
Joe Carlsmith on non-naturalist realism vs anti-realism
Utopia, for [the non-naturalist realist], was always the promise of something more than e.g. joy, love, creativity, understanding — it was the promise of those things, with an extra non-natural sauce on top. A Utopia with no sauce is an empty shell, the ethical analog of a phenomenal zombie. It looks right, but the crucial, invisible ingredient is missing.
How one reacts here is ultimately a question of psychology, not metaphysics. Some robots are envelope-ranking-maximizers, and they won’t change their goals just because the envelope probably doesn’t exist. But I think we should be wary of assuming too quickly that we’re like this.
[…]
[On the anti-realist picture] it’s clearer to me what I’m doing, in deciding whether to help the deer, create Utopia, etc, given the posited metaphysical clarity. I feel somehow more grounded, more like a creature in the real, raw, beautiful world, with my eyes open, taking full responsibility for my actions, and less like I’m somehow playing pretend, seeking guidance from some further realm or set of facts that I secretly suspect does not exist.
[…]
To me, what this currently looks like is a place where the choice about what sort of world to create is in a deep way on us. Just as there is no theistic God to tell us what to do, so there is no further realm of normative facts to tell us, either. We have to choose for ourselves. We have to be, as it were, adults. To stand together, and sometimes alone, amidst the beauty and horror and confusion of the world; to look each other and ourselves in the eye; to try to see as clearly as possible what’s really going on, what our actions really involve and imply and create, what it is we really do, when we do something; and then, to choose.
This isn’t to say we can’t mess up: we can — deeply, terribly, irreparably. But messing up will come down to some failure to understand our actions, ourselves, and each other, and to act on that understanding; some veil between us and the real world we inhabit; not some failure to track, in our decisions, the True Values hidden in some further realm. And when we don’t mess up — when we actually find and build and experience what it is we would seek if we saw with full clarity — what we will get isn’t the world, and therefore something else, the “goodness” or “value” of that world, according to the True Values beyond the world. It will be just the world, just what we and others chose to do, or to try to do, with this extraordinary and strange and fleeting chance, this glimpse of whatever it is that’s going on.
https://handsandcities.com/2021/01/03/the-despair-of-normative-realism-bot/
Bernard Williams on Sidgwick and the ambitions of ethics
My own view is that no ethical theory can render a coherent account of its own relation to practice: it will always run into some version of the fundamental difficulty that the practice of life, and hence also an adequate theory of that practice, will require the recognition of what I have called deep dispositions; but at the same time the abstract and impersonal view that is required if the theory is to be genuinely a theory cannot be satisfactorily understood in relation to the depth and necessity of those dispositions. Thus the theory will remain, in one way or another, in an incoherent relation to practice.
Making Sense of Humanity (1995), “The point of view of the universe: Sidgwick and the Ambitions of Ethics”
John Maynard Keynes on Sidgwick
He never did anything but wonder whether Christianity was true and prove that it wasn’t and hope that it was.
[…]
There is no doubt about his moral goodness. And yet it is all so dreadfully depressing- no intimacy, no clear-cut crisp boldness. Oh, I suppose he was intimate but he didn’t seem to have anything to be intimate about except his religious doubts. And he really ought to have got over that a little sooner; because he knew that the thing wasn’t true perfectly well from the beginning.