Nick Bostrom on singletons
Can we trust evolutionary development to take our species in broadly desirable directions?
[…]
What we shall call the Panglossian view maintains that this past record of success gives us good grounds for thinking that evolution (whether biological, memetic, or technological) will continue to lead in desirable directions. This Panglossian view, however, can be criticized on at least two grounds. First, because we have no reason to think that all this past progress was in any sense inevitable‒‐much of it may, for aught we know, have been due to luck. And second, because even if the past progress were to some extent inevitable, there is no guarantee that the melioristic trend will continue into the indefinite future.
[…]
Evolution made us what we are, but no fundamental principle stands in the way of our developing the capability to intervene in the default course of events in order to steer future evolution towards a destiny more congenial to human values.
Directing our own evolution, however, requires coordination. If the default evolutionary course is dystopian, it would take coordinated paddling to turn the ship of humanity in a more favorable direction. If only some individuals chose the eudaemonic alternative while others pursued fitness‐maximization, then, by assumption, it would be the latter that would prevail. Fitness‐maximizing variants, even if they started out as a minority, would be preferentially selected for at the expense of eudaemonic agents, and a process would be set in motion that would inexorably lead to the minimization or disappearance of eudaemonic qualities, and the non‐eudaemonic agents would be left to run the show.
To this problem there are only two possible solutions: preventing non‐eudaemonic variants from arising in the first place, or modifying the fitness function so that eudaemonic traits become fitness‐maximizing.
[…]
A singleton could be a rather minimalist structure that could operate without significantly disrupting the lives of its inhabitants. And it need not prohibit novelty and experimentation, since it would retain the capacity to intervene at a later stage to protect its constitution if some developments turned malignant. Increased social transparency, such as may result from advances in surveillance technology or lie detection, could facilitate the development of a singleton. Deliberate international political initiatives could also lead to the gradual emergence of a singleton, and such initiatives might be dramatically catalyzed by ‘wild card’ events such as a series of cataclysms that highlighted the disadvantages of a fractured world order. It would be a mistake to judge the plausibility of the ultimate development of a singleton on the basis of ephemeral trends in current international affairs. The basic conditions shaping political realities may change as new technologies come online, and it is worth noting that the longterm historical trend is towards increasing scope of human coordination and political integration. If this trend continues, the logical culmination is a singleton.
[…]
A singleton could take a variety of forms and need not resemble a monolithic culture or a hive mind. Within the singleton there could be room for a wide range of different life forms, including ones that focus on non‐eudaemonic goals. The singleton could ensure the survival and flourishing of the eudaemonic types by restricting the ownership rights of non‐eudaemonic entities, by subsidizing eudaemonic activities, by guaranteeing the enforcement of property rights, by prohibiting the creation of agents with human‐unfriendly values or psychopathic tendencies, or in a number of other ways. Such a singleton could guide evolutionary developments and prevent our cosmic commons from going to waste in a first‐come‐first‐served colonization race.
[…]
We do not know that a dystopian scenario is the default evolutionary outcome. Even if it is, and even if the creation of a singleton is the only way to forestall ultimate catastrophe, it is a separate question what policies it makes sense to promote in the here and now. While creating a singleton would help to reduce certain risks, it may at the same time increase others, such as the risk that an oppressive regime could become global and permanent. If our preliminary study serves to draw attention to some possibly non‐obvious considerations and to stimulate more rigorous analytic work, its purpose will have been achieved.
https://www.nickbostrom.com/fut/evolution.html
It could act merely as a subtle enforcer of certain background conditions that could serve, e.g. to guarantee security or to administer some other minimal governmental tasks. […] When considering the characteristics of a singleton it would be a mistake to assume that it would necessarily possess the attributes commonly associated with large human bureaucracies — rigidity, lack of imagination, inefficiency, a tendency to micro-manage and to expand its own powers, etc. This would be true of some possible singletons but it might not be true of others.
The concept of a singleton is thus much broader and more abstract than concept of a world government. A world government (in the ordinary sense of the word) is only one type of singleton among many.
Nevertheless, all the possible singletons share one essential property: the ability to solve global coordination problems. Intelligent species that develop the capability to solve global coordination problems, such as those listed in the next section, may in the long run develop along very different trajectories than species that lack this capacity.
[…]
A major risk with creating a singleton is that it would turn out to be a bad singleton. Smaller units of decision-making, such as states, can also turn bad. But if a singleton goes bad, a whole civilization goes bad. All the eggs are in one basket.
https://www.nickbostrom.com/fut/singleton.html