Sam Altman on The Merge
I think a merge is probably our best-case scenario. If two different species both want the same thing and only one can have it—in this case, to be the dominant species on the planet and beyond—they are going to have conflict. We should all want one team where all members care about the well-being of everyone else.
Although the merge has already begun, it’s going to get a lot weirder. We will be the first species ever to design our own descendants. My guess is that we can either be the biological bootloader for digital intelligence and then fade into an evolutionary tree branch, or we can figure out what a successful merge looks like.
Vitalik Buterin on superintelligence: merge or die
Across the board, I see far too many plans to save the world that involve giving a small group of people extreme and opaque power and hoping that they use it wisely. And so I find myself drawn to a different philosophy, one that has detailed ideas for how to deal with risks, but which seeks to create and maintain a more democratic world and tries to avoid centralization as the go-to solution to our problems.
[…]
Unless we create a world government powerful enough to detect and stop every small group of people hacking on individual GPUs with laptops, someone is going to create a superintelligent AI eventually - one that can think a thousand times faster than we can - and no combination of humans using tools with their hands is going to be able to hold its own against that. And so we need to take this idea of human-computer cooperation much deeper and further.
A first natural step is brain-computer interfaces. Brain-computer interfaces can give humans much more direct access to more-and-more powerful forms of computation and cognition, reducing the two-way communication loop between man and machine from seconds to milliseconds. This would also greatly reduce the “mental effort” cost to getting a computer to help you gather facts, give suggestions or execute on a plan.
Later stages of such a roadmap admittedly get weird. In addition to brain-computer interfaces, there are various paths to improving our brains directly through innovations in biology. An eventual further step, which merges both paths, may involve uploading our minds to run on computers directly.
[…]
If we want a future that is both superintelligent and “human”, one where human beings are not just pets, but actually retain meaningful agency over the world, then it feels like something like this is the most natural option. There are also good arguments why this could be a safer AI alignment path: by involving human feedback at each step of decision-making, we reduce the incentive to offload high-level planning responsibility to the AI itself, and thereby reduce the chance that the AI does something totally unaligned with humanity’s values on its own.
https://vitalik.eth.limo/general/2023/11/27/techno_optimism.html
Scott Alexander on conservatism self-preservation-against-optimality urge
I think that it might be reasonable to have continuation of your own culture as a terminal goal, even if you know your culture is “worse” in some way than what would replace it. There’s a transhumanist joke — “Instead of protecting human values, why not reprogram humans to like hydrogen? After all, there’s a lot of hydrogen.” There’s way more hydrogen than beautiful art, or star-crossed romances, or exciting adventures. A human who likes beautiful art, star-crossed romances, and exciting adventures is in some sense “worse” than a human who likes hydrogen, since it would be much harder for her to achieve her goals and she would probably be much less happy. But knowing this does not make me any happier about the idea of being reprogrammed in favor of hydrogen-related goals. My own value system might not be objectively the best, or even very good, but it’s my value system and I want to keep it and you can’t take it away from me. I am an individualist and I think of this on an individual level, but I could also see having this self-preservation-against-optimality urge for my community and its values.
(I’ve sometimes heard this called Lovecraftian parochialism, based on H.P. Lovecraft’s philosophy that the universe is vast and incomprehensible and anti-human, and you’ve got to draw the line between Self and Other somewhere, so you might as well draw the line at 1920s Providence, Rhode Island, and call everywhere else from Boston all the way to the unspeakable abyss-city of Y’ha-nthlei just different degrees of horribleness.)
https://slatestarcodex.com/2016/07/25/how-the-west-was-won/
Quote conservatism transhumanism moral philosophy scott alexander
Scott Alexander on liberalism and universal culture
On liberalism
Liberalism is a technology for preventing civil war. It was forged in the fires of Hell — the horrors of the endless seventeenth century religious wars. For a hundred years, Europe tore itself apart in some of the most brutal ways imaginable — until finally, from the burning wreckage, we drew forth this amazing piece of alien machinery. A machine that, when tuned just right, let people live together peacefully without doing the “kill people for being Protestant” thing. Popular historical strategies for dealing with differences have included: brutally enforced conformity, brutally efficient genocide, and making sure to keep the alien machine tuned really really carefully.
https://slatestarcodex.com/2017/06/21/against-murderism/
On “Western” culture vs universal culture
I am pretty sure there was, at one point, such a thing as western civilization. I think it included things like dancing around maypoles and copying Latin manuscripts. At some point Thor might have been involved. That civilization is dead. It summoned an alien entity from beyond the void which devoured its summoner and is proceeding to eat the rest of the world.
[…]
“Western medicine” is just medicine that works. It happens to be western because the West had a technological head start, and so discovered most of the medicine that works first. But there’s nothing culturally western about it; there’s nothing Christian or Greco-Roman about using penicillin to deal with a bacterial infection. Indeed, “western medicine” replaced the traditional medicine of Europe — Hippocrates’ four humors — before it started threatening the traditional medicines of China or India. So-called “western medicine” is an inhuman perfect construct from beyond the void, summoned by Westerners, which ate traditional Western medicine first and is now proceeding to eat the rest of the world.
[…]
If western medicine is just medicine that works, soda pop is just refreshment that works.
[…]
Sushi has spread almost as rapidly as Coke. But in what sense has sushi been “westernized”? Yes, Europe has adopted sushi. But so have China, India, and Africa. Sushi is [just] another refreshment that works.
Here’s what I think is going on. Maybe every culture is the gradual accumulation of useful environmental adaptations combined with random memetic drift. But this is usually a slow process with plenty of room for everybody to adjust and local peculiarities to seep in. The Industrial Revolution caused such rapid change that the process become qualitatively different, a frantic search for better adaptations to an environment that was itself changing almost as fast as people could understand it.
The Industrial Revolution also changed the way culture was spatially distributed. When the fastest mode of transportation is the horse, and the postal system is frequently ambushed by Huns, almost all culture is local culture. England develops a culture, France develops a culture, Spain develops a culture. Geographic, language, and political barriers keep these from intermixing too much. Add rapid communication — even at the level of a good postal service — and the equation begins to change. In the 17th century, philosophers were remarking (in Latin, the universal language!) about how Descartes from France had more in common with Leibniz from Germany than either of them did with the average Frenchman or German. Nowadays I certainly have more in common with SSC readers in Finland than I do with my next-door neighbor whom I’ve never met.
Improved trade and communication networks created a rapid flow of ideas from one big commercial center to another. Things that worked — western medicine, Coca-Cola, egalitarian gender norms, sushi — spread along the trade networks and started outcompeting things that didn’t. It happened in the west first, but not in any kind of a black-and-white way. Places were inducted into the universal culture in proportion to their participation in global trade; Shanghai was infected before West Kerry; Dubai is further gone than Alabama. The great financial capitals became a single cultural region in the same way that “England” or “France” had been a cultural region in the olden times, gradually converging on more and more ideas that worked in their new economic situation.
Let me say again that this universal culture, though it started in the West, was western only in the most cosmetic ways. If China or the Caliphate had industrialized first, they would have been the ones who developed it, and it would have been much the same. The new sodas and medicines and gender norms invented in Beijing or Baghdad would have spread throughout the world, and they would have looked very familiar. The best way to industrialize is the best way to industrialize.
Universal culture is the collection of the most competitive ideas and products. Coca-Cola spreads because it tastes better than whatever people were drinking before. Egalitarian gender norms spread because they’re more popular and likeable than their predecessors. If there was something that outcompeted Coca-Cola, then that would be the official soda of universal culture and Coca-Cola would be consigned to the scrapheap of history.
The only reason universal culture doesn’t outcompete everything else instantly and achieve fixation around the globe is barriers to communication. Some of those barriers are natural — Tibet survived universalization for a long time because nobody could get to it. Sometimes the barrier is time — universal culture can’t assimilate every little hill and valley instantly. Other times there are no natural barriers, and then your choice is to either accept assimilation into universal culture, or put up some form of censorship.
Michael Nielsen is having second thoughts
My work has nearly all been about accelerating science and technology. And over the last year I find myself very conflicted. I’ve begun to question whether accelerating science and technology is a good idea at all.
https://michaelnotebook.com/ti/index.html
Quote michael nielsen catastrophic risk vulnerable world hypothesis nick bostrom
Tyler Cowen on Malthus on vice
So one way to read Malthus is this: if a society is going to have any prosperity at all, the people in that society either will be morally quite bad, or they have to be morally very, very good, good enough to exercise that moral restraint. Alternatively, you can read Malthus as seeing two primary goals for people: food and sex. His accomplishment was to show that, taken collectively, those two goals could not easily be obtainable simultaneously in a satisfactory fashion. In late Freudian terms, you could say that eros/sex amounts to the death drive, but again painted on a collective canvas and driven by economic mechanisms.
Malthus also hinted at birth control as an important social and economic force, especially later in 1817, putting him ahead of many other thinkers of his time. Birth control was widely practiced for centuries through a variety of means, and Malthus unfortunately was not very specific. He did call it “unnatural,” and the mainstream theology of his Anglican church condemned it, as did many other churches. But what did he really think? Was this unnatural practice so much worse than the other alternatives of misery and vice that his model was putting forward? Or did Malthus simply fail to see that birth control could be so effective and widespread as it is today? It doesn’t seem we are ever going to know.
From Malthus’s tripartite grouping of vice, moral restraint, and misery, two things should be clear immediately. The first is why Keynes found Malthus so interesting, namely that homosexual passions are one (partial) way out of the Malthusian trap. The second is that there is a Straussian reading of Malthus, namely that he thought moral restraint, while wonderful, was limited in its applicability. So maybe then vice wasn’t so bad after all? Is it not better than war and starvation?
I don’t buy the Straussian reading as a description of what Malthus really meant. But he knew it was there, and he knew he was forcing you to think about just how bad you thought vice really was. Malthus for instance is quite willing to reference prostitution as one possible means to keep down population. He talks about “men,” and “a numerous class of females,” but he worries that those practices “lower in the most marked manner the dignity of human nature.” It degrades the female character and amongst “those unfortunate females with which all great towns abound, more real distress and aggravated misery are perhaps to be found, than in any other department of human life.”
How bad are those vices relative to starvation and population triage? Well, the modern world has debated that question and mostly we have opted for vice. You thus can see that the prosperity of the modern world does not refute Malthus. We faced the Malthusian dilemma and opted for one of his options, namely vice. It’s just that a lot of us don’t find those vices as morally abhorrent as Malthus did. You could say we invented another technology that (maybe) does not suffer from diminishing returns, namely improving the dignity and the living conditions of those who practice vice. Contemporary college dorms seem pretty comfortable, and they have plenty of birth control, and of course lots of vice in the Malthusian sense. While those undergraduates might experience high rates of depression and also sexual violation, that life of vice still seems far better than life near the subsistence point. I am not sure what Malthus would think of college dorm sexual norms (and living standards!), but his broader failing was that he did not foresee the sanitization and partial moral neutering of what he considered to be vice.
Quote tyler cowen thomas malthus scarcity moral philosophy evolution value drift