Tim Urban on cosmology
Go hug someone pic.twitter.com/I63AaP1xQu
— Tim Urban (@waitbutwhy) September 16, 2021
Robin Hanson on value drift
Value drift is just a generic problem for humans, ems or AI. It’s just what’s always happened so far. It’s the default for what will happen in the future. If you hate it you’re in trouble because it’s really really likely. Some people for some reason think that value drift in humans is bounded well, while value drift in machines is not only unbounded but happens quickly and I don’t really see the grounds for that. […] The main thing is that in the past when value drift happened change was so slow that you didn’t see it in your lifetime so you didn’t worry very much about it. As change gets faster and your lifetimes get longer your life will encompass more value drift. And then whether it’s humans or machines or whatever you will see it, and if you don’t like it then you will see something you don’t like.
John Richardson on Nietzsche’s metaethics
[Nietzsche discusses] three main ways of valuing: the body’s, the moral agent’s, and his own. We could also call these animal, human, and superhuman valuing. Each has its own “semantics” (or “intentionality”); that is, its own way of positing its values as good. So Nietzsche has what might look like three separate and inconsistent metaethical positions but that are really three elements in a unified account of valuing.
[…]
Our drives value simply by using signs to steer by (toward). They “see” or “interpret” their values as good just by using them this way. They don’t posit them as “true” to anything outside them. Instead they judge and adjust these signs as they learn how well they “pay off” in expanding power. As we saw, the drives don’t recognize what they’re doing as they value. They don’t see the “frame” of their valuing around their values; they lack the perspectivist truth. But they refrain from the externalist mistake of thinking their values tasked to match real goods outside.
By contrast Nietzsche thinks that our agential valuing does make that externalist posit. This is one of its main impositions on our drive-valuing. In order to “tame” the latter for social life, the habit of obeying external norms needs to be inculcated. It’s to license this habit of obedience that the conviction is gradually ingrained that there are real values outside one’s valuing that one needs to align it toward.
[…]
The historical character of this posit and the way it is overlaid on a deeper valuing that doesn’t make it suggest the contingency of such externalism. They support Nietzsche’s optimism that human can find a way to grow out of what is only a (deeply settled) bad habit.
[…]
Nietzsche’s frequent expressions of [error theory] are unsurprising given what we’ve just seen: they apply to our agential, moral valuing which does indeed claim its values to be real—which they’re not.
[…]
But this error-theory does not apply to the two other ways of valuing in Nietzsche’s scenario. Bodily valuing makes no truth-claim, and his own valuing does, but a different one that (we’ll see) has a chance to be true. Nietzsche denies that all valuing makes the mistake of positing its values as real. And why indeed would he allow our agential-moral valuing to represent valuing in general? Human is “the sick animal” due precisely to the defective way it values. Nietzsche’s return to “natural” values is his effort to bring our conscious and worded values into healthy alignment with our drive-valuing; this will include undoing that false posit.
[…]
Nietzsche justifies his values by direct appeal to the values we already have. He tries to point out values we have without noticing them. The “ought” is supplied not from outside but by what the person values already. He claims only to offer the means by which that valuing will want to improve itself.
By his perspectivism, Nietzsche gives credit to our existing values as the only determiners of what’s good for us. So his appeal is ultimately to these. But our valuing of these values includes a will and ability to improve them, in the two fundamental respects we noticed in §1.4. We will to improve them as signs for power—a will embedded deeply in us just as living things. We also will to improve our values in how well they face the truth—a will bred into us humans and indeed distinctive of our kind. These deep aims function as second-order or meta-values, criteria by which we will to improve our first-order values.
John Richardson, Nietzsche’s Values, Chapter 1
Elijah Millgram’s pragmatic critique of internalism
That motivations fail to agglomerate is exhibited in the most striking logical feature of internalism (and of its cruder relative, instrumentalism), namely, that one’s bottom-line desires and projects are incorrigible.
[…]
You want what you want, and someone who insists that you are wrong to do so, when mistakes about such things as how to get what you want are not at issue, is just bluffing.
Subjective motivations can change, in all manner of ways, but they cannot be corrected, and this means that nothing could count as the rational investigation, on the part of such a creature, as to whether its bottom-line guidelines and priorities were correct. Since the creatures do not correct their own motivations, the design strategy is reasonable only if they do not need to; in other words, only if, for the most part, the designer can equip them with motivations (or ensure that they pick up motivations from their surroundings) that will not need correction. That in turn is feasible only if the designer can anticipate the practical problems his creatures will face, and only if the guidelines his creature would need to negotiate them are sufficiently compact to be stored and accessed. Given plausible cognitive constraints on processing, memory, and so on, that in turn requires that the environment the creature is anticipated to face be both stable and simple.
Coloring in the line drawing, we see that [Bernard] Williams’s alethic state of nature is something on the order of a tourist-brochure version of a village in the hills of Provence, where life goes on as it has since time immemorial. The villagers work their plots of land, growing the same grains and vegetables they always have; they herd their sheep and goats; they bake rustic bread and knit rustic clothes; they hunt rabbits and deer; they build houses out of the local stone; they marry and raise children; when they get old, they sit outside the village pub and drink pastis; they play boules in the park; eventually, they die, and are buried in the cemetery behind the church. The internalist design solution is satisfactory for this form of life. The designer knows that his peasants will have to work the fields, so when it comes time to own a field and work it, they come to have a desire to do so. They need to be made to reproduce, and thus are built so that, when they get old enough, they will want to have children, or anyway want to do things that as a predictable side effect produce children. Not all of a subjective motivational set need be hardwired, of course; a disposition to mimic others, and to learn and adopt one’s elders’ thick ethical concepts, will keep the games of boules going and the pastis flowing. Because life in the mythical village never changes, there is no need to delegate to the peasants themselves the task of investigating what their motivations ought to be, and no need to equip them to correct their motivations; thus, there is no need to complicate their cognitive or normative systems with the gadgetry that would take.
[…]
Analytic philosophy has done something that is quite peculiar: instead of making sense of humanity, we have been philosophizing for the inhabitants of a romantic fantasy of traditional peasant life.
[…]
Instrumentalist (or “Humean”) theories of practical reasoning are how philosophers talk through the strategy of hardwiring designated objectives into an organism, so that it can execute a life plan suitable to a stable environment. Your environment is no longer stable enough for relying on desires to be a decent strategy. Instrumentalists (“Humeans”) have a view of practical rationality suitable for a cruder, simpler species.
The Great Endarkenment, ‘D’où venons-nous . . . Que sommes nous . . . Où allons-nous?’
quote elijah millgram internalism pragmatism bernard williams
John Richardson on Nietzsche’s naturalism
Nietzsche thinks he cares more about truth than other philosophers do. This is partly because he is not in thrall to a moral bias, but also because he understands better the kind of truth there really can be—the kind humans can and do have. So he rewrites philosophers’ previous idea of truth while still giving it preeminent value.
[…]
In announcing these truths he contributes to what he thinks is a prolonged, ineluctable process by which our modern scientific will to truth finally faces the truth about values—the last and hardest topic for it to face. As these truths are exposed, our culture, and the rest of the world through it, is confronted with a great spiritual crisis and challenge: How can and will we go on to value once we have uncovered these truths about our valuing? How can we value, now for the first time, honestly (i.e., while facing the truth about what we’re doing)?
[…]
It is extremely difficult to do so because this truth tends to undermine our [values] […] insofar as they involve a framing claim that these things (that are valued) are really, independently good. For the truth, Nietzsche holds, is that all values are dependent on valuings—are “perspectival.”
[…]
Recognizing Nietzsche’s idea of values as signs is the key to much of his thought about them. Seeing a value as a sign, we see why he insists that it’s not only humans that value. Animals are clearly responsive to signs in their perceptual discernments. So a predator may employ a certain smell as a sign of prey. And we can see ways that plants are responsive to signs as well. Nietzsche holds that willing (or aiming) is something that all organisms do. It depends not at all on consciousness.
[…]
Our human values, as worded, are distinctive in being held in common, as norms. They are accepted because this is “how one values” in the community to which one belongs. They thus serve a “herding” function, which strengthens the group but at the expense of members’ individuality.
John Richardson, Nietzsche’s Values, Preface
Thoughts on Robin Hanson and David Deutsch on predicting the future
David Deutsch has an influential book that contains statements like the following:
The future of civilization is unknowable, because the knowledge that is going to affect it has yet to be created.
Unfortunately, the closest he comes to explaining this claim is:
The ability of scientific theories to predict the future depends on the reach of their explanations, but no explanation has enough reach to predict the content of its own successors — or their effects, or those of other ideas that have not yet been thought of. Just as no one in 1900 could have foreseen the consequences of innovations made during the twentieth century — including whole new fields such as nuclear physics, computer science and biotechnology — so our own future will be shaped by knowledge that we do not yet have.
In the same book Deutsch writes:
The philosopher Roger Bacon (1214–94) […] foresaw the invention of microscopes, telescopes, self-powered vehicles and flying machines — and that mathematics would be a key to future scientific discoveries.
Later, he predicts:
Illness and old age are going to be cured soon — certainly within the next few lifetimes — and technology will also be able to prevent deaths through homicide or accidents by creating backups of the states of brains, which could be uploaded into new, blank brains in identical bodies if a person should die. Once that technology exists, people will consider it considerably more foolish not to make frequent backups of themselves than they do today in regard to their computers. If nothing else, evolution alone will ensure that, because those who do not back themselves up will gradually die out. So there can be only one outcome: effective immortality for the whole human population, with the present generation being one of the last that will have short lives.
So—it seems like we should take this unknowability claim with a big pinch of salt. A more plausible slogan would be:
The future of civilization is very hard to know, partly because the knowledge that is going to affect it has yet to be created.
Anyway, with this context I expected that Deutsch would be quite critical of Robin Hanson’s approach to prediction. In fact, during their recent hour-long discussion, Deutsch did not identify any particular cases where he thought Hanson’s approach was widlly off. After discussing several cases (including the cost of solar power; demographic projections; the grabby aliens model), he said:
DD: You haven’t yet given an example of something where I would disagree with you that it’s worth investigating.
And then, towards the end of the discussion:
DD: Everything you are doing is legitimate and indeed morally required, and it’s a bit of a scandal that more people aren’t doing it. But I think the same is true of all fundamental theories, branches of knowledge.
In closing, Deutsch summarised his position:
DD: In short, the place where [probability] is dangerous is where the thing you are predicting depends on the future growth of knowledge. You’ve given examples where it still works even then, e.g. you’ve mentioned the idea where stock prices will be a random walk [even when knowledge accumulates].
He continues:
DD: But there are cases where it’s very misleading. So for example, [if one says] that all long-lived civilisations in the past have failed, and therefore ours will—that’s illegitimate. Because it’s making an assumption that the frequency is the probability. And here I have a substantive theory that says why it isn’t. Namely that our civilisation is different from all the others.
But then:
But it doesn’t matter—even if I didn’t know that theory, I would still say it was illegitimate to extrapolate the future of our civilisation based on past civilisations. Because all of them depended on the future growth of knowledge. And if you look in detail about how they failed, they all failed in different ways, but one thing you can say about it is that in all cases, more knowledge would have saved them.
He says he would say its illegitimate to extrapolate even if he didn’t have a theory as to why—but does not explain why that’s the case, instead just restates the theory he actually has.
Notably, his central “principle of optimism” (“all evils are caused by lack of knowledge) is a claim about the future of our civilisation based on… extrapolation from past civilisations.
My suspicion is that Deutsch doesn’t have a crisp way to distinguish (legitimate) prediction from (illegitimate) prophecy, and that often he just labels as “prophecy” the predictions that he does not want to seriously engage. At least, this is what I think I’ve repeatedly seen going on during discussions of existential risk that threaten his principle of optimism.
At the level of theory, Deutsch concedes:
DD: I admit that I think the connection between risk and what you might call probability, the reason why risks can be approximated by probabilities, and also the reason why frequencies, in certain situations, can be approximated by probabilities, is an unsolved problem. And I think it’s a very important problem and if I wasn’t working on other things I would be working on that.
Deutsch is right to emphasise is that if you’re going to extrapolate trends or make reference class comparisons, it’s worth trying to articulate the grounds on which you’ve selected your reference classes, and reasons the trend might not continue. In his language: predictions are always underwritten by explanations, implicit or otherwise, and if you make them explicit you may be able to improve on them. That said, it’s surprising how often a prior in favour of simple extrapolations can peform better than more complicated models (see e.g. COVID-19, spring 2020).
Deutsch is also right to emphasise that probability estimates can easily be thrown wildly off due to unknown unknowns or other kinds of model error. This is not controversial, but it is often forgotten, and this mistake can be expensive.
DD: I think it’s reasonable if your best explanations imply that.
DD: You mean not only that the graphs are the same, you mean that you expect solar power technology to be dependent on making factories which use materials of a certain kind which aren’t going to be throttled by a hostile foreign power and so on.
DD: It is simply wrong to use probability in this way and it would be better to make the assumptions explicit.
I note that Deutsch does not offer an argument for the “it is simply wrong to use probability in this way” claim, in much the way that Mervyn King and John Kay fail to in their book. In particular, he does not discuss or argue against the Bayesian notion of assigning subjective probabilities to beliefs.
In a talk titled Knowledge Creation and It’s Risks, Deutsch says:
Outcomes can’t be analysed in terms of probability unless we have specific explanatory models that predict that something is or can be approximated as a random process, and predicts the probabilities. Otherwise one is fooling oneself, picking arbitrary numbers as probabilities and arbitrary numbers as utilities and then claiming authority for the result by misdirection, away from the baseless assumptions.
For example, when we were building the Hadron collider, should we not switch it on just in case it destroys the universe? Well either the theory that it will destroy the universe is true, or the theory that it’s safe is true. The theories don’t have probabilities. The real probability is zero or one, it’s just unknown. And the issue must be decided by explanation, not game theory. And the explanation that it was more dangerous to use the collider than to scrap it, and forgo the resulting knowledge, was a bad explanation, because it could be applied to any fundamental research.
He’s right that—objectively—either the collidor will destroy the universe or it won’t. But it seems fine for the Bayesian to say: in cases like this, with evidence like this, I expect my beliefs about the objective world to be correct X% of the time.
Towards the end of the dialog, Deutsch accidentally states a prediction in rather Bayesian-sounding terms:
DD: Conditional on our species not surviving this century, I think it is overwhelmingly likely that the reason is one we have not thought of yet.
He notices the mistake, and corrects himself:
DD: As a side remark, you’ve caught me in an illegitimate use of probability. When I said that conditional on our species being destroyed it’s overwhelmingly likely to be [due to a reason we have not thought of]… I shouldn’t have said that. This just shows how deeply this mistaken notion of ideas having probability has permeated our culture. Even though I hate it, I can’t help using it.
At the end of the dialog, I was still left waiting for Deutsch’s explanation for why assigning subjective probabilities is not a sensible thing to do1. Perhaps that will not be forthcoming. I would gladly settle, instead, for his object-level discussion of the Vulnerable World Hypothesis 2.
Edit (2021-12-29): Tyler Cowen comments on his interview with David Deutsch:
Deutsch convinced me there’s often a lot of hot air behind Popper. He didn’t argue very well on Popper’s behalf. Deutsch is way smarter than I am but he seemed to me in some fundamental ways a dogmatist, and not really able to defend Popper very well. He’s made up his mind and you get a particular kind of emphatic statement, but I thought that at the philosophical level his defences were weak. […] There’s this odd feature of Popperianism. It somehow attracts a lot of dogmatists. I don’t know why.
Edit (2021-12-27): since writing this, Joesph Walker pointed me to a 2016 paper by Deutsch, which contains an extended discussion of the Popperian conception of scientific explanation which Deutsch favours, and contrasts it to Bayesian conception. I’ve not yet had a chance to give it a proper read. I will update this footnote/post when I do.↩︎
For what it’s worth: my current take is that Deutsch is right to worry about widespread pessimism about technology, but his reaction is too sweeping. I worry that the slogan he presents as the principle of optimism is used by technologists to wave away legitimate concerns about catastrophic and existential risk. In many areas (e.g. nuclear power)—and probably on average—I guess Deutsch is right that we need more techno-optimism. But in some areas, I suspect we need discouragement and regulation, with a view to pulling off some strategy of differential technological development.↩︎
writing robin hanson david deutsch futurism bayesianism radical uncertainty