Tyler Cowen's core view on AI

My core view: if we humans can get more intelligence into our hands, if we can't turn that into to a positive... what is it we're hoping for? This is our big chance! Let's not blow it.

Episode 41: Is This the AI Renaissance?

Digital Minds or Butlerian Accord? (Part 1)

Two scenarios for 2100 that might be choiceworthy:

1. Digital Minds: digital minds rule the Earth. There was a peaceful transition of power; humans think the 21st century went well and are optimistic about the future.
2. Butlerian Accord: biological humans rule the Earth. They have agreed not to develop AI systems that replace them, and to regulate other technologies that greatly increase extinction risk.

Which would be preferable? Would either be acceptable?

---

Let's imagine each these scenarios have come to pass (bracketing the question of how it actually happened).

If you like the Digital Minds scenario, you may think some or all of:

1. Digital minds will live far better lives than biological humans; there will also be far more of them.
2. Biological humans are overall better off in this scenario.
3. Digital minds will have values somewhat (or very) different from our own, and that's fine.
4. We should think of digital minds as our descendants, not as an alien species.
5. It would be good if our descendants become grabby and occupy a large local chunk of the Universe, rather than letting other civilisations take our place. "Digital Minds" soon increases the chance of this.

If you dislike the Digital Minds scenario, you may think some or all of:

1. Digital Minds will not be conscious, or otherwise capable of wellbeing.
2. Digital Minds will have values somewhat (or very) different from our own, and those values will be worse, objectively speaking.
3. Digital Minds will have values somewhat (or very) different from our own, and I don't like that.
4. We should think of digital minds as an "other" that we dislike.
5. Digital Minds will experience extraordinary suffering, or other kinds of disvalue.

---

If you like Butlerian Accord, you may think some or all of:

1. The human condition is already pretty good at 21st century levels of technology. Same goes for non-human animals.
2. A human civilisation based on a Butlerian Accord can be stable, and flourish for millennia.
3. We should shift to "Digital Minds" eventually, but not before we've thought long and hard about how to maximise the chance that shift goes well.

If you dislike Butlerian Accord, you may think some or all of:

1. We currently live in a Darwinian hellscape; superintelligent digital minds are the only way out of that.
2. A Butlerian Accord would not last.
3. ...

---

What other big reasons are there to like or dislike each scenario?

Who will rule the Earth in 2100?

Scenarios for 2100, with my probability guesstimates:

(1) (30%) Biological humans rule the Earth¹.

(2) (50%) Digital minds rule the Earth^{2 3}.

(3) (20%) Something very different to the above .

The shift from (1) to (2) or (3) might happen—or at least get baked in—within 20 years. I guess 10-30% chance of that.

The short story for (2) goes: the capabilities of AI systems surpass those of humans; AI systems pursue their own values.

The main ways that (1) might continue:

(a) (20%) A global agreement prevents the development of the most dangerous AI systems. ⁴

(b) (10%) A global disaster sets back the development of AI systems (along with much else) by decades. E.g. a major nuclear war.

(c) (<10%) A technological barrier prevents superhuman AI before 2100.

How might we get (3)?

(a) Something causes human extinction before digital minds can sustain themselves without us

(b) Aliens.

(c) Simulation ends.

(d) ???

As Holden says, we should not expect business as usual.

When you're writing to learn, very short posts are fine

When I draft a post for this journal, I often write several good paragraphs and then run into a lot of questions. I start to think about how I can make my writing more understandable. An hour or two later, I'm out of steam, and it's time to move on with my day.

Why? This is a study journal. I'm mostly writing for myself, and, on occasion, a small audience of high-context peers.

The journal format is partly about "getting thoughts out of my head" so that they are easier to interrogate. It's also about providing a partial record of thoughts I'd like to reconnect with later.

I'm going to try going hard on this style for a few weeks.

I'll write the journal I want to have.

@mfweof and @RokoMijic on naturalistic metaethics

Anything moral is only moral because it won, and it won for some reason other than being moral.

—@mfweof

The past is not more ethical than the present. The past is (relative to current standards) less ethical, but so is the future!

The maintenance of present day human values are bound to present day technology in the same way thay the maintenance of medieval hierarchies was bound to the function of knights on the battlefield. Technology is not value neutral, so if you want to upgrade your technology without destroying your values you need to do some kind of alignment work. You need to constrain that technology so that it doesn't do the thing it wants to do in the most efficient way possible: you need to make a gun that only knights can use.

—@RokoMijic

See also: Robin Hanson, Most AI fear is Future Fear.

Eliezer Yudkowsky (1999) on a world with nanotechnology

Unless you've heard of nanotechnology, it's hard to appreciate the magnitude of the changes we're talking about.  Total control of the material world at the molecular level is what the conservatives in the futurism business are predicting.

Taking 10^17 ops/sec as the figure for the computing power used by a human brain, and using optimized atomic-scale hardware, we could run the entire human race on one gram of matter, running at a rate of one million subjective years every second.

Frequently Asked Questions about the Meaning of Life

Kendall Roy on Logan Roy

He was comfortable with this world. And he knew it. And he liked it.

And I say, amen to that.

Tom Holland on Paul’s Letter to the Galatians

COWEN: Which Gospel do you view as most foundational for Western liberalism and why?

HOLLAND: I think that that is a treacherous question to ask because it implies that there would be a coherent line of descent from any one text that can be traced like that. I think that the line of descent that leads from the Gospels and from the New Testament and from the Bible and, indeed, from the entire corpus of early Christian texts to modern liberalism is too confused, too much of a swirl of influences for us to trace it back to a particular text.

If I had to choose any one book from the Bible, it wouldn’t be a Gospel. It would probably be Paul’s Letter to the Galatians because Paul’s Letter to the Galatians contains the famous verse that there is no Jew or Greek, there is no slave or free, there is no man or woman in Christ. In a way, that text — even if you bracket out and remove the “in Christ” from it — that idea that, properly, there should be no discrimination between people of different cultural and ethnic backgrounds, based on gender, based on class, remains pretty foundational for liberalism to this day.

I think that liberalism, in so many ways, is a secularized rendering of that extraordinary verse. But I think it’s almost impossible to avoid metaphor when thinking about what the relationship is of these biblical texts, these biblical verses to the present day. I variously compared Paul, in particular in his letters and his writings, rather unoriginally, to an acorn from which a mighty oak grows.

But I think actually, more appropriately, of a depth charge released beneath the vast fabric of classical civilization. And the ripples, the reverberations of it are faint to begin with, and they become louder and louder and more and more disruptive. Those echoes from that depth charge continue to reverberate to this day.

Carl Schmitt on liberalism

Otto Petras on the conditions for religious awe

A religion that one understands is, for he who understands, no longer a religion. For by comprehending it, he stands above it; he surveys its conditions and possibilities, and to the extent that he does so he no longer feels like the unconditional object of religious demands. One can be possessed and awe-struck only as long as one does not understand how and why that occurs.

Chris Cullen: what do you want to naturalise?

We become our habits. So it's crucial to ask: what do you want to naturalise? What do you want to come easily?

Whatever the heart frequently dwells upon becomes the shape of the heart.

Richard Meadows & Nassim Taleb on FU money

Humphrey Bogart used to keep a $100 bill in his dresser drawer at all times—a decent chunk of change in the 1920s. He referred to it as his ‘fuck-you money’, because it meant he’d never be forced to take a crappy part. According to Bogie, the only good reason for making money was “so you can tell any son-of-a-bitch in the world to go to hell”.

— Richard Meadows

---

A sum large enough to get most, if not all, of the advantages of wealth (the most important one being independence and the ability to only occupy your mind with matters that interest you) but not its side effects, such as having to attend a black-tie charity event and being forced to listen to a polite exposition of the details of a marble-rich house renovation.

Money buys freedom: intellectual freedom, freedom to choose who you vote for, to choose what you want to do professionally. But having what I call “fuck you” money requires a huge amount of discipline. The minute you go a penny over, then you lose your freedom again.

— Nassim Taleb

The last homily of Pope Pius XII

Our mouths are filled with the word "love".

"But I, before anyone else, didn't know how to define it.

Our mouths are filled with the word "beauty".

But I, before anyone else, didn't know how to receive it.

For this, I ask you forgiveness.

Please, forgive me.

At times we confound love with madness.

Beauty with ecstasy.

History has repeated itself.

Madness and ecstasy have once again proven to be irresistible temptations, but they always end the way they did on Ventotene.

With unjust death.

In this case, of a good and innocent priest.

There is a life of happiness to be found in the sphere of gentleness, kindness, mildness, lovingness.

We must learn to be in the world.

And the Church must contemplate the idea of opening up to the love that is possible, in order to fight against the love that is aberrant.

All this, John Paul III, with great humility, calls "the middle way."

In the past few days I have understood.

It's not the middle way.

It is the way.

Ever since I came back, you've been asking yourselves all sorts of questions.

Is he the father or the son?

Is he God or the Holy Spirit?

Is he man or is he Jesus?

Did he wake up or did he rise from the dead?

Is he a saint or is he an imposter?

Is he Christ or is he the Antichrist?

Is he alive or is he dead?

It doesn't matter.

You know what is so beautiful about questions?

It's that we don't have the answers.

In the end, only God has the answers.

They are his secret.

God's secret, which only He knows.

That is the mystery in which we believe.

And that is the mystery which guides our conscience.

And now I would like to come down among you, and do what I have wanted to do since the first moment: embrace you, one by one.

Marc Andreessen on the heros we're allowed to have

The anti-hero is the portrait of the Nietzschean superhero that we are allowed to have. Tony Soprano, Walter White, Don Draper. We can have someone who does Great Things, so long as that person is fundamentally bad by the standards of modern morality.

We are not allowed to have the full version of the Nietzschean superman doing something outstanding. We're not allowed Napoleon figures, the building of the pyramids, Beethoven, or even the person who built the transcontinental railroad, the car industry, that sort of thing.

The full Nietzschean superman is the person who says 'I really am going to rule the world, and rule it much better'. Those narratives are gone. They're too scary. They're absolutely frightening, because if we rediscover that kind of morality it would upend our entire order.

Email to Tyler on Bernard Williams & effective altruism

In your MacAskill interview, and again in the St Andrews talk, I heard you channeling Bernard Williams on Philosophy as a Humanistic Discipline and especially "The Human Prejudice".

I agree that Williams on philosophy and impartiality is an important message for EA. I pushed this line in conversations with Will MacAskill and others in 2015, and with several other Oxford figures since then. I'm surely not the ideal advocate, but in the replies I mostly heard a lot of "ugh, Bernard" followed by weak arguments against superficial misreadings of his work. People seemed very much in the mode of "devalue and dismiss".

My best EA Forum post is also my least popular:

https://forum.effectivealtruism.org/posts/G6EWTrArPDf74sr3S/bernard-williams-ethics-and-the-limits-of-impartiality

Williams' low status within EA is surprising given how seriously Derek Parfit took him as a peer. I understand that Williams was often seen as using non-kosher methods and unkind remarks in his philosophical writing and conversation, and was intensely disliked by some of his peers. So I suspect that much of his neglect is driven by residual animosity in the Oxford crowd. But they ignore this kind of thing and just "take the ideas seriously"... right...?

There are some notable exceptions. For example, Thomas Moynihan is somewhat associated with the Oxford EA scene, and appropriately rates Bernard Williams. Unsurprisingly, Tom has a background in "continental" philosophy.

You've not blogged much about Williams. How about it? E.g.

(1) Was Williams a pragmatist in denial, per Rorty's review of Truth and Truthfulness? Why did he resist Rorty?

(2) What prioritisation errors are made by those who go too far with impartiality? 

On (2): if EAs stopped going "too far" with impartiality, I think we'd see the EA portfolio shift a bit towards catastrophic risk and away from existential risk. The current strong focus on x-risk can be seen as another form of the 51:49 bet.

A couple years ago one of the more influential EAs told me that rejecting the 51:49 bet is a form of egoism. We should not care about our personal chances of survival: we should just follow the rule that maximises EV across all possible worlds. I replied that ecological rationality beats axiomatic rationality in the world I care about. But if you think impartial reasons are the only reasons that count, you can't justify your "arbitrary" care for this particular world over others.

And with that—and your remarks on the useful generativity of a mistake taken seriously—we're back to Nietzsche's remarks on Plato:

It seems that in order to inscribe themselves upon the heart of humanity with everlasting claims, all great things have first to wander about the earth as enormous and awe- inspiring caricatures: dogmatic philosophy has been a caricature of this kind--for instance, the Vedanta doctrine in Asia, and Platonism in Europe. Let us not be ungrateful to it, although it must certainly be confessed that the worst, the most tiresome, and the most dangerous of errors hitherto has been a dogmatist error--namely, Plato's invention of Pure Spirit and the Good in Itself. But now when it has been surmounted, when Europe, rid of this nightmare, can again draw breath freely and at least enjoy a healthier--sleep, we, WHOSE DUTY IS WAKEFULNESS ITSELF, are the heirs of all the strength which the struggle against this error has fostered. It amounted to the very inversion of truth, and the denial of the PERSPECTIVE--the fundamental condition--of life, to speak of Spirit and the Good as Plato spoke of them; indeed one might ask, as a physician: "How did such a malady attack that finest product of antiquity, Plato? Had the wicked Socrates really corrupted him? Was Socrates after all a corrupter of youths, and deserved his hemlock?" But the struggle against Plato, or--to speak plainer, and for the "people"--the struggle against the ecclesiastical oppression of millenniums of Christianity (FOR CHRISITIANITY IS PLATONISM FOR THE "PEOPLE"), produced in Europe a magnificent tension of soul, such as had not existed anywhere previously; with such a tensely strained bow one can now aim at the furthest goals. 

Peter

P.S. Nietzsche's thoughts on effective altruism, according to ChatGPT.

Jonathan Bi on how to live with a Girardian worldview

I compared Girard to my Virgil in the sense that he was able to rescue me through Hell. He was able to show me how to purge more milder forms of perversion.

But, just as Virgil couldn't take Dante all the way to heaven, neither could Girard. Girard kind of just retreats.

What I'm about to share with you is mostly my own creative interpretations on top of Girard.

I think there's in general two solutions, once you've identified there's a metaphysical and there's a physical desire. One wing, and I think this is what Girard leans to, is to say this metaphysical--this is the Buddhist as well as the Girardian way--is to say this metaphysical desire, this desire for being, it's completely perverse. It's _always_perverse, whether from Girard's perspective, because it's essentially a desire to be God. This is why it's satanic. You're desiring persistence; you're desiring power; you're desiring reality. If you push those far enough, those are the metaphysical qualities of the Judeo-Christian God. So, Girard actually sees metaphysical desire as the original sin, as the satanic drive to rival God in his metaphysical splendor.

And the Buddhists--right--we don't have to go into that, but long story short, these metaphysical qualities are not possible in the world. Emptiness is what permeates the world. So, this is a fundamentally wrong sort of desire.

So, for the Christians and Buddhists, the way to good health is to completely get rid of metaphysical desire, to be only concerned by the object physical desire.

There's another, however, strand of thinking, and probably most popular amongst the Germans, in Hegel, is to say there is actually a healthy way--the Germans, and Plato actually, which we'll talk about--there actually is a healthy way to exist in society. And the way, long story short, to do so is for your metaphysical and your physical desires to align.

That is to say: if you really like to do philosophy, don't hang out with a bunch of people who are industrialists. Hang out with a bunch of philosophers, so that the somewhat partial spectator, as we've discussed, will naturally _align_with your normative values, with your physical desires, and thus you'll receive recognition and a form of reality.

https://www.econtalk.org/johnathan-bi-on-mimesis-and-rene-girard/

Nick Bostrom on differential technological development

The Principle of Differential Technological Development

Retard the development of dangerous and harmful technologies, especially ones that raise the level of existential risk; and accelerate the development of beneficial technologies, especially those that reduce the existential risks posed by nature or by other technologies Bostrom, 2002).

The principle of differential technological development is compatible with plausible forms of technological determinism. For example, even if it were ordained that all technologies that can be developed will be developed, it can still matter when they are developed. The order in which they arrive can make an important difference – ideally, protective technologies should come before the destructive technologies against which they protect; or, if that is not possible, then it is desirable that the gap be minimized so that other countermeasures (or luck) may tide us over until robust protection become available. The timing of an invention also influences what sociopolitical context the technology is born into. For example, if we believe that there is a secular trend toward civilization becoming more capable of handling black balls, then we may want to delay the most risky technological developments, or at least abstain from accelerating them.

Nick Bostrom on "turnkey totalitarianism"

Developing a system for turnkey totalitarianism means incurring a risk, even if one does not intend for the key to be turned.

One could try to reduce this risk by designing the system with appropriate technical and institutional safeguards. For example, one could aim for a system of ‘structured transparency’ that prevents concentrations of power by organizing the information architecture so that multiple independent stakeholders must give their permission in order for the system to operate, and so that only the specific information that is legitimately needed by some decision-maker is made available to her, with suitable redactions and anonymization applied as the purpose permits. With some creative mechanism design, some machine learning, and some fancy cryptographic footwork, there might be no fundamental barrier to achieving a surveillance system that is at once highly effective at its official function yet also somewhat resistant to being subverted to alternative uses.

How likely this is to be achieved in practice is of course another matter, which would require further exploration. Even if a significant risk of totalitarianism would inevitably accompany a well-intentioned surveillance project, it would not follow that pursuing such a project would increase the risk of totalitarianism. A relatively less risky well-intentioned project, commenced at a time of comparative calm, might reduce the risk of totalitarianism by preempting a less-wellintentioned and more risky project started during a crisis. But even if there were some net totalitarianism-risk-increasing effect, it might be worth accepting that risk in order to gain the general ability to stabilize civilization against emerging Type-1 threats (or for the sake of other benefits that extremely effective surveillance and preventive policing could bring).

Naturalism, pragmatism, impartiality

Joshua Greene's naturalism

Joshua Greene sees morality as part of the natural world ¹. It emerges from evolution, because certain kinds of cooperation are adaptive.

In his breakdown:

• Morality helps individuals cooperate ². Codes of morality operate within groups, but vary between groups.
• "Metamorality" helps groups cooperate. For groups to cooperate, they need to find a common currency, even if they have quite different codes of morality.

On this picture, both morality and metamorality stem from a practical problem—how can creatures like us flourish, get along, and have many descendants?

What Greene calls "modern ethics" is mostly concerned with the question: how can we improve our metamorality?

Greene, following [Jeremy Bentham](https://notes.pjh.is/=Jeremy Bentham), thinks that the best candidate for a common currency to support metamorality is: quality of experience. If you 5-whys ask people why they care about something, it often comes down to the quality of their experience—crudely: whether it amounts to pleasure or suffering, an experience they would gladly choose or strive to avoid.

Naturalism and impartiality

Where does the ideal of impartiality fit into this picture?

Moral philosophers and people associated with the effective altruism movement often characterise impartiality as:

Giving equal weight to everyone's interests.

This is often cached out in negative terms, as:

Trying to avoid giving undue weight to particular interests based on (putatively) non-morally-relevant factors such as race, gender, proximity in space and time, nationality, species membership, or substrate.

Usually, such lists are motivated by a positive claim about what kinds of things are morally-relevant, such as capacity for wellbeing, or sentience.

Let's bracket the question of what our positive claim, and our list of morally-irrelevant factors, should look like. Let's instead zoom in on the "undue weight" part.

At the level of within-group morality, one can tell a story about how norms of (relatively) equal treatment and fairness could emerge as a stable equilbria, grounded upon (relatively) equal distributions of power in forager societies. ³

At the level of between-group metamorality, we can tell a similar story: groups will be reluctant to cooperate if they think their interests are unduly neglected (though they may in fact accept a bunch of unfair treatement, if their other options—e.g. conflict—seem worse).

So how do we cash out "undue weight" in naturalistic terms? Well, the obvious candidate is: non-adaptive.

In moral philosophy and effective altruism, the "impartiality story" is not told as though it's based on evolutionary logic of adaptive bargains and long-run equilibria. The story is not "we should be impartial in order to solve our co-operation problems and maximise reproductive fitness". But rather, the idea is "we should impartially promote the good because... well... that's the right thing to do".

This seems like a case where your metaethics, or the story you tell about what we're up to when we do moral philosophy, rather matters.

Bernard Williams, discussing the limits to impartiality, approvingly quotes Max Stirner:

The tiger who attacks me is in the right, and so am I when I strike him down. I defend against him not my right, but myself.

—Max Stirner

Williams imagines that humanity is threatened by an alien civilisation, and suggests that in such a situation, it would not be appropriate ask:

(a) What outcome would maximise value according to our best impartial axiology?

For Williams as for Greene, our impartiality axiology is ultimately a tool that's supposed to serve us, not an external imperative that we are supposed to serve. Thoughts to the contrary are "a relic of a world not yet fully disenchanted", i.e. the artefact of a non-secular worldview ⁴.

Rather, he thinks we should ask:

(b) What outcome would be best for us (or according to us)?

The choice between (a) and (b) isn't just academic: we face situations where we must decide which of these questions to ask now, soon, and in the long-run:

1. (now) Western liberalism vs Political Islam; West vs China; etc
2. (soon?) Digital Minds
3. (long-run) Alien civilisations

Naturalism vs non-naturalism; pragmatism and normativity

For the naturalist, moral and metamoral questions are quite empirical. In the current environment, which ideals actually work?

(The normativity within "actually work" boils down to "adaptive fitness", whether we recognise this or not.)

As Bostrom & Schulman reminded us recently ⁵, we can reasonably reject the insistence of many philosophers that there's a fundamental distinction between descriptive and normative claims. After reflecting on Pragmatism last autumn, I became more confident is that this distinction is not, in fact, fundemental. Rather, things are blurry—there's no such thing as a purely descriptive claim, when we begin our enquiry we're always already bound up in the normative project of being humans trying to get by in the world, with various aims and agendas baked in ⁶.

On the naturalistic perspective, we started out trying to solve a practical problem—how can we get along with groups with which, on the surface, we don't share much in common—and ended up mistakenly thinking of ourselves as doing something else (seeking the (meta)moral truth, then following that). If that's our self-image, we won't be so concerned with empirics: we'll just try whatever our culture and our moral philosophers come up with, and if it "works" in the naturalistic sense, all good. Otherwise, we'll get wiped out, perhaps gradually, perhaps quickly.

The non-naturalist impartialist thinks there is an external, non-human standard, so they are likely to be more interested in revisionary or revolutionary maximisation—maximisation of whatever they think is valuable, independently of humans—and chafing against the constraints imposed by being the kinds of creatures we find ourselves to be.

On the naturalistic perspective, we'll be more concerned with thinking about what flavours of impartiality are going to work well for us over the long run. We'd be more inclined to, like the pragmatist, keep coming back to the question: what problem are we actually trying to solve here?

Digital minds: descendents or rivals?

Should we think of digital minds as our descendents, or our rivals?

If they are descendents, we can think of them as children—different from us in important ways, but carrying on the flame.

If they are rivals, well—they are rivals. If they inherit the future, then we have, in some important sense, lost.

If you think that digital minds will inherit the future whether we like it or not, the main way in which this matters is how it affects your attitudes toward this future today. And so perhaps we should make more effort to like it.