Joe Carlsmith on the utilitarian dream

I think there’s a certain type of person who comes to philosophy and encounters ideas — a certain set of kind of simple and kind of otherwise elegant or theoretically attractive ideas. The ideas I most think of as in this cluster — and this is all separable — is total utilitarianism, Bayesianism, expected utility reasoning.

I remember at one point I was talking with a friend of mine, who used to be a utilitarian, about one of his views. And I started to offer a counterexample to his views, and he just cut me off and he was like, “Joe, I bite all the bullets.” I was like, “You don’t even need to hear the bullets?” He’s like, “Yeah. It’s like, whatever. Does it fall out of my view? I bite it.”

So I think that a certain kind of person in this mindset can feel like, “Sure, there are bullets I need to bite for this view. I need to push fat men off of bridges; I need to create repugnant conclusions, even with a bunch of hells involved.” All this sort of stuff. But they feel like, “I am hardcore, I am rigorous, I have theorems to back me up. My thing is simple; these other people’s theories, they’re janky and incomplete and kind of made up.”

It just has this flavor of you’re kind of unwilling to look the truth in the face. Like, make the lizards… Sorry, the lizards: One conclusion that falls out of total utilitarianism is the idea that for any utopia, there’s a better world with kind of a sufficient number of barely happy lizards plus arbitrary hells.

Infinite ethics just breaks this narrative. And that’s part of why I wanted to work on this topic: I felt like I saw around me some people who were too enamored of this utilitarian dream, who thought it was on better theoretical foundations than I think it is, who felt like it was more of a default, and more of a kind of simple, natural foundation than I think it is.

You’re going to have to start giving stuff up. You’re going to be incomplete. You’re going to start playing a game that looks more similar to the game you didn’t want to play before, and more similar to the game that everyone else was playing. You’re not going to be able to say, “I’m just going to bite whatever bullets my theory says to bite.” You’re running out of track, or to the extent you’re on a crazy train, the crazy train just runs out of track. There are just horrific bullets if you want to bite them, but it’s just a very different story and you’re much more lost.

I think that’s an important source of humility for people who are drawn to this perspective. I think they should be more lost than they were if they were really jazzed by like, “I know it’s total utilitarianism. I’m so hardcore. No one else is willing to be hardcore.” I’m like, I think you should spend some time with infinite ethics and adjust your confidence in the position accordingly.

https://80000hours.org/podcast/episodes/joe-carlsmith-navigating-serious-philosophical-confusion/

Joe Carlsmith against "non-naturalistic realism or bust"

Are you so sure you even know what this debate is about – what it means for something to matter, or to be “natural,” or to be “mind-dependent”— let alone what the answer is? So sure that if it’s only raw nature – only joy, love, friendship, pain, grief, and so on, with — then there’s nothing to fight for, or against? I, for one, am not.

So “you should have higher credence on naturalism-but-things-matter” is the immediate objection here. Indeed, I think this objection cautions wariness about the un-Bayesian-ness of much philosophical discourse. Some meta-ethicist might well declare confidently “if naturalism is true, then nothing matters!” But they are rarely thinking in terms of quantitative credences on “ok but actually maybe if naturalism is true some things matter after all,” or about the odds at which they’re willing to bet.

[...]

Remember, nihilism – the view that there are no normative facts – is distinct from what we might call “indifference-ism” – that is, the view that there are normative facts, and they actively say that you should be indifferent to everything. On nihilism, indifference is no more normatively required, as a response to the possibility of innocent children being burned alive, than is intense concern (or embarrassment, or desire to do a prime number of jumping jacks). Conditional on nihilism, nothing is telling you not to care about yourself, or your family, or those children: you’re absolutely free to do so. And plausibly – at least, if your psychology is similar to many people who claim to accept something like nihilism – you still will care.

[...]

My own sense is that most familiar, gloomy connotations of nihilism aren’t centrally about meta-ethics at all. Rather, they are associated more closely with a cluster of psychological and motivational issues related to depression, hopelessness, and loss of connection with a sense of care and purpose. Sometimes, these issues are bound up with someone’s views about the metaphysics of normative properties and the semantics of normative discourse (and sometimes, we grope for this sort of abstract language in order to frame some harder-to-articulate disorientation). But often, when such issues crop up, meta-ethics isn’t actually the core explanation. After all, the most meta-ethically inflationary realists, theists, and so on can see their worlds drain of color and their motivations go flat; and conversely, the most metaphysically reductionist subjectivists, anti-realists, nihilists and so on can fight just as hard as others to save their friends and families from a fire; to build flourishing lives and communities; to love their neighbors as themselves. Indeed, often (and even setting aside basic stuff about mental health, getting enough sleep/exercise, etc), stuff like despair, depression, and so on is often prompted most directly by the world not being a way you want it to be — e.g., not finding the sort of love, joy, status, accomplishment and so forth that you’re looking for; feeling stuck and bored by your life; feeling overwhelmed by the suffering in the world and/or unable to make a difference; etc — rather than by actually not wanting anything, or by a concern that something you want deeply isn’t worth wanting at the end of the day. Meta-ethics can matter in all this, yes, but we should be careful not to mistake psychological issues for philosophical ones – even when the lines get blurry.

https://joecarlsmith.com/2022/10/09/against-the-normative-realists-wager

See also: Simon Blackburn on people who think that anti-realism entails nihilism

Despair is a sin and misery is a choice.

200 IQ and made of meat

Embryo selection probably works, and may arrive soon. So we can imagine a world where we have 10,000 adults who are roughly twice as intelligent, happy and healthy (and so on) compared to the most fortunate humans today.

What would happen in that world?

(1) Evolutionary perspective: these (trans)humans would attain most positions of power. They would form fertile factions. Within a few generations, the "old model" humans would be a minority of the population.

(2) Religious perspective: these beings would be so wise and benevolent that they would moderate their own power. They would make decisions in the interests of the social welfare function, taking care to minimise their violations of deontological side-constraints.

We should expect something closer to (1) than (2). Such is the way of things.

How do we feel about the prospect of (1)? It'll depend on the details, but broadly I think people today are happy with the idea of happier healthier and more intelligent humans inheriting the earth. We send our children to school.

Now: replace these transhumans with digital minds, which are made of sand instead of meat. Feels different, but do you endorse that feeling?

Atoms are atoms.¹

https://www.mit.edu/people/dpolicar/writing/prose/text/thinkingMeat.html).

What is AI alignment?

"AI alignment" means different things to different people.

My preferred definition, following many others, is:

An AI system is aligned if it does what its operator intends.

Why not just say that an AI system works if it does what its operator intends?

The reason is that AI systems are more agentic. To understand their behaviour, we adopt the intentional stance; we explain their behaviour partly by ascribing them values.

For an AI system to do what its operator intends, it must share values with its operator to a sufficient degree. Values do not need to perfectly shared in order for the system to do what the operator intends.

The same holds for a manager-employee relationship. In such a relationship, values are not perfectly shared, but they're shared enough that the employee does what the manager intends.

How closely do values need to be shared, for an AI system do to what its operator intends? And how easily can we build such systems? That's mostly an empirical question, and it depends on the use case.

The case of GPT-4 is somewhat encouraging: the system is remarkably good at understanding user intent; better, in fact, than many human colleagues.

Tyler Cowen's core view on AI

My core view: if we humans can get more intelligence into our hands, if we can't turn that into to a positive... what is it we're hoping for? This is our big chance! Let's not blow it.

Episode 41: Is This the AI Renaissance?

Digital Minds or Butlerian Accord? (Part 1)

Two scenarios for 2100 that might be choiceworthy:

1. Digital Minds: digital minds rule the Earth. There was a peaceful transition of power; humans think the 21st century went well and are optimistic about the future.
2. Butlerian Accord: biological humans rule the Earth. They have agreed not to develop AI systems that replace them, and to regulate other technologies that greatly increase extinction risk.

Which would be preferable? Would either be acceptable?

---

Let's imagine each these scenarios have come to pass (bracketing the question of how it actually happened).

If you like the Digital Minds scenario, you may think some or all of:

1. Digital minds will live far better lives than biological humans; there will also be far more of them.
2. Biological humans are overall better off in this scenario.
3. Digital minds will have values somewhat (or very) different from our own, and that's fine.
4. We should think of digital minds as our descendants, not as an alien species.
5. It would be good if our descendants become grabby and occupy a large local chunk of the Universe, rather than letting other civilisations take our place. "Digital Minds" soon increases the chance of this.

If you dislike the Digital Minds scenario, you may think some or all of:

1. Digital Minds will not be conscious, or otherwise capable of wellbeing.
2. Digital Minds will have values somewhat (or very) different from our own, and those values will be worse, objectively speaking.
3. Digital Minds will have values somewhat (or very) different from our own, and I don't like that.
4. We should think of digital minds as an "other" that we dislike.
5. Digital Minds will experience extraordinary suffering, or other kinds of disvalue.

---

If you like Butlerian Accord, you may think some or all of:

1. The human condition is already pretty good at 21st century levels of technology. Same goes for non-human animals.
2. A human civilisation based on a Butlerian Accord can be stable, and flourish for millennia.
3. We should shift to "Digital Minds" eventually, but not before we've thought long and hard about how to maximise the chance that shift goes well.

If you dislike Butlerian Accord, you may think some or all of:

1. We currently live in a Darwinian hellscape; superintelligent digital minds are the only way out of that.
2. A Butlerian Accord would not last.
3. ...

---

What other big reasons are there to like or dislike each scenario?

Who will rule the Earth in 2100?

Scenarios for 2100, with my probability guesstimates:

(1) (30%) Biological humans rule the Earth¹.

(2) (50%) Digital minds rule the Earth^{2 3}.

(3) (20%) Something very different to the above .

The shift from (1) to (2) or (3) might happen—or at least get baked in—within 20 years. I guess 10-30% chance of that.

The short story for (2) goes: the capabilities of AI systems surpass those of humans; AI systems pursue their own values.

The main ways that (1) might continue:

(a) (20%) A global agreement prevents the development of the most dangerous AI systems. ⁴

(b) (10%) A global disaster sets back the development of AI systems (along with much else) by decades. E.g. a major nuclear war.

(c) (<10%) A technological barrier prevents superhuman AI before 2100.

How might we get (3)?

(a) Something causes human extinction before digital minds can sustain themselves without us

(b) Aliens.

(c) Simulation ends.

(d) ???

As Holden says, we should not expect business as usual.

When you're writing to learn, very short posts are fine

When I draft a post for this journal, I often write several good paragraphs and then run into a lot of questions. I start to think about how I can make my writing more understandable. An hour or two later, I'm out of steam, and it's time to move on with my day.

Why? This is a study journal. I'm mostly writing for myself, and, on occasion, a small audience of high-context peers.

The journal format is partly about "getting thoughts out of my head" so that they are easier to interrogate. It's also about providing a partial record of thoughts I'd like to reconnect with later.

I'm going to try going hard on this style for a few weeks.

I'll write the journal I want to have.

@mfweof and @RokoMijic on naturalistic metaethics

Anything moral is only moral because it won, and it won for some reason other than being moral.

—@mfweof

The past is not more ethical than the present. The past is (relative to current standards) less ethical, but so is the future!

The maintenance of present day human values are bound to present day technology in the same way thay the maintenance of medieval hierarchies was bound to the function of knights on the battlefield. Technology is not value neutral, so if you want to upgrade your technology without destroying your values you need to do some kind of alignment work. You need to constrain that technology so that it doesn't do the thing it wants to do in the most efficient way possible: you need to make a gun that only knights can use.

—@RokoMijic

See also: Robin Hanson, Most AI fear is Future Fear.

Eliezer Yudkowsky (1999) on a world with nanotechnology

Unless you've heard of nanotechnology, it's hard to appreciate the magnitude of the changes we're talking about.  Total control of the material world at the molecular level is what the conservatives in the futurism business are predicting.

Taking 10^17 ops/sec as the figure for the computing power used by a human brain, and using optimized atomic-scale hardware, we could run the entire human race on one gram of matter, running at a rate of one million subjective years every second.

Frequently Asked Questions about the Meaning of Life

Kendall Roy on Logan Roy

He was comfortable with this world. And he knew it. And he liked it.

And I say, amen to that.

Tom Holland on Paul’s Letter to the Galatians

COWEN: Which Gospel do you view as most foundational for Western liberalism and why?

HOLLAND: I think that that is a treacherous question to ask because it implies that there would be a coherent line of descent from any one text that can be traced like that. I think that the line of descent that leads from the Gospels and from the New Testament and from the Bible and, indeed, from the entire corpus of early Christian texts to modern liberalism is too confused, too much of a swirl of influences for us to trace it back to a particular text.

If I had to choose any one book from the Bible, it wouldn’t be a Gospel. It would probably be Paul’s Letter to the Galatians because Paul’s Letter to the Galatians contains the famous verse that there is no Jew or Greek, there is no slave or free, there is no man or woman in Christ. In a way, that text — even if you bracket out and remove the “in Christ” from it — that idea that, properly, there should be no discrimination between people of different cultural and ethnic backgrounds, based on gender, based on class, remains pretty foundational for liberalism to this day.

I think that liberalism, in so many ways, is a secularized rendering of that extraordinary verse. But I think it’s almost impossible to avoid metaphor when thinking about what the relationship is of these biblical texts, these biblical verses to the present day. I variously compared Paul, in particular in his letters and his writings, rather unoriginally, to an acorn from which a mighty oak grows.

But I think actually, more appropriately, of a depth charge released beneath the vast fabric of classical civilization. And the ripples, the reverberations of it are faint to begin with, and they become louder and louder and more and more disruptive. Those echoes from that depth charge continue to reverberate to this day.

Carl Schmitt on liberalism

Otto Petras on the conditions for religious awe

A religion that one understands is, for he who understands, no longer a religion. For by comprehending it, he stands above it; he surveys its conditions and possibilities, and to the extent that he does so he no longer feels like the unconditional object of religious demands. One can be possessed and awe-struck only as long as one does not understand how and why that occurs.

Chris Cullen: what do you want to naturalise?

We become our habits. So it's crucial to ask: what do you want to naturalise? What do you want to come easily?

Whatever the heart frequently dwells upon becomes the shape of the heart.

Richard Meadows & Nassim Taleb on FU money

Humphrey Bogart used to keep a $100 bill in his dresser drawer at all times—a decent chunk of change in the 1920s. He referred to it as his ‘fuck-you money’, because it meant he’d never be forced to take a crappy part. According to Bogie, the only good reason for making money was “so you can tell any son-of-a-bitch in the world to go to hell”.

— Richard Meadows

---

A sum large enough to get most, if not all, of the advantages of wealth (the most important one being independence and the ability to only occupy your mind with matters that interest you) but not its side effects, such as having to attend a black-tie charity event and being forced to listen to a polite exposition of the details of a marble-rich house renovation.

Money buys freedom: intellectual freedom, freedom to choose who you vote for, to choose what you want to do professionally. But having what I call “fuck you” money requires a huge amount of discipline. The minute you go a penny over, then you lose your freedom again.

— Nassim Taleb

The last homily of Pope Pius XII

Our mouths are filled with the word "love".

"But I, before anyone else, didn't know how to define it.

Our mouths are filled with the word "beauty".

But I, before anyone else, didn't know how to receive it.

For this, I ask you forgiveness.

Please, forgive me.

At times we confound love with madness.

Beauty with ecstasy.

History has repeated itself.

Madness and ecstasy have once again proven to be irresistible temptations, but they always end the way they did on Ventotene.

With unjust death.

In this case, of a good and innocent priest.

There is a life of happiness to be found in the sphere of gentleness, kindness, mildness, lovingness.

We must learn to be in the world.

And the Church must contemplate the idea of opening up to the love that is possible, in order to fight against the love that is aberrant.

All this, John Paul III, with great humility, calls "the middle way."

In the past few days I have understood.

It's not the middle way.

It is the way.

Ever since I came back, you've been asking yourselves all sorts of questions.

Is he the father or the son?

Is he God or the Holy Spirit?

Is he man or is he Jesus?

Did he wake up or did he rise from the dead?

Is he a saint or is he an imposter?

Is he Christ or is he the Antichrist?

Is he alive or is he dead?

It doesn't matter.

You know what is so beautiful about questions?

It's that we don't have the answers.

In the end, only God has the answers.

They are his secret.

God's secret, which only He knows.

That is the mystery in which we believe.

And that is the mystery which guides our conscience.

And now I would like to come down among you, and do what I have wanted to do since the first moment: embrace you, one by one.

Marc Andreessen on the heros we're allowed to have

The anti-hero is the portrait of the Nietzschean superhero that we are allowed to have. Tony Soprano, Walter White, Don Draper. We can have someone who does Great Things, so long as that person is fundamentally bad by the standards of modern morality.

We are not allowed to have the full version of the Nietzschean superman doing something outstanding. We're not allowed Napoleon figures, the building of the pyramids, Beethoven, or even the person who built the transcontinental railroad, the car industry, that sort of thing.

The full Nietzschean superman is the person who says 'I really am going to rule the world, and rule it much better'. Those narratives are gone. They're too scary. They're absolutely frightening, because if we rediscover that kind of morality it would upend our entire order.

Email to Tyler on Bernard Williams & effective altruism

In your MacAskill interview, and again in the St Andrews talk, I heard you channeling Bernard Williams on Philosophy as a Humanistic Discipline and especially "The Human Prejudice".

I agree that Williams on philosophy and impartiality is an important message for EA. I pushed this line in conversations with Will MacAskill and others in 2015, and with several other Oxford figures since then. I'm surely not the ideal advocate, but in the replies I mostly heard a lot of "ugh, Bernard" followed by weak arguments against superficial misreadings of his work. People seemed very much in the mode of "devalue and dismiss".

My best EA Forum post is also my least popular:

https://forum.effectivealtruism.org/posts/G6EWTrArPDf74sr3S/bernard-williams-ethics-and-the-limits-of-impartiality

Williams' low status within EA is surprising given how seriously Derek Parfit took him as a peer. I understand that Williams was often seen as using non-kosher methods and unkind remarks in his philosophical writing and conversation, and was intensely disliked by some of his peers. So I suspect that much of his neglect is driven by residual animosity in the Oxford crowd. But they ignore this kind of thing and just "take the ideas seriously"... right...?

There are some notable exceptions. For example, Thomas Moynihan is somewhat associated with the Oxford EA scene, and appropriately rates Bernard Williams. Unsurprisingly, Tom has a background in "continental" philosophy.

You've not blogged much about Williams. How about it? E.g.

(1) Was Williams a pragmatist in denial, per Rorty's review of Truth and Truthfulness? Why did he resist Rorty?

(2) What prioritisation errors are made by those who go too far with impartiality? 

On (2): if EAs stopped going "too far" with impartiality, I think we'd see the EA portfolio shift a bit towards catastrophic risk and away from existential risk. The current strong focus on x-risk can be seen as another form of the 51:49 bet.

A couple years ago one of the more influential EAs told me that rejecting the 51:49 bet is a form of egoism. We should not care about our personal chances of survival: we should just follow the rule that maximises EV across all possible worlds. I replied that ecological rationality beats axiomatic rationality in the world I care about. But if you think impartial reasons are the only reasons that count, you can't justify your "arbitrary" care for this particular world over others.

And with that—and your remarks on the useful generativity of a mistake taken seriously—we're back to Nietzsche's remarks on Plato:

It seems that in order to inscribe themselves upon the heart of humanity with everlasting claims, all great things have first to wander about the earth as enormous and awe- inspiring caricatures: dogmatic philosophy has been a caricature of this kind--for instance, the Vedanta doctrine in Asia, and Platonism in Europe. Let us not be ungrateful to it, although it must certainly be confessed that the worst, the most tiresome, and the most dangerous of errors hitherto has been a dogmatist error--namely, Plato's invention of Pure Spirit and the Good in Itself. But now when it has been surmounted, when Europe, rid of this nightmare, can again draw breath freely and at least enjoy a healthier--sleep, we, WHOSE DUTY IS WAKEFULNESS ITSELF, are the heirs of all the strength which the struggle against this error has fostered. It amounted to the very inversion of truth, and the denial of the PERSPECTIVE--the fundamental condition--of life, to speak of Spirit and the Good as Plato spoke of them; indeed one might ask, as a physician: "How did such a malady attack that finest product of antiquity, Plato? Had the wicked Socrates really corrupted him? Was Socrates after all a corrupter of youths, and deserved his hemlock?" But the struggle against Plato, or--to speak plainer, and for the "people"--the struggle against the ecclesiastical oppression of millenniums of Christianity (FOR CHRISITIANITY IS PLATONISM FOR THE "PEOPLE"), produced in Europe a magnificent tension of soul, such as had not existed anywhere previously; with such a tensely strained bow one can now aim at the furthest goals. 

Peter

P.S. Nietzsche's thoughts on effective altruism, according to ChatGPT.