Very Big If True, therefore probably wrong and/or crazy (part 1)
So you think you’ve had an extremely important insight which is widely underrated. In that case, a sensible early reaction is to pause, and to ask:
- Am I missing something?
- Am I crazy?
Often, (1) will turn out to be true. You may be able to work this out for yourself, or you may need to do it through research and conversation.
Less often, (2) will be your problem. That’s beyond the scope of this blog post.
Sometimes, though, you will actually be on to something—probably not exactly right, but at least on the right track.
One promising sign is that you have a good story to tell about:
- What “edge”—or stroke of good fortune—might explain how I’ve had this insight, yet others haven’t? `` For example, if you are Leo Szilard, you’re unusually likely to have “big if true” type insights about nuclear physics. You’re a trained physicist, and one of your peers just annoyed you.
If you think your insight is worth pursuing, your early steps should be taken in the spirit of “testing”. You should make great effort to expose your insight to scrutiny, and only gradually scale up the bets you place on it. If you are misguided, you want someone to show you why, as soon as possible. It is critical to hold onto the “why this is wrong?” attitude, rather than absorbing the idea into your identity [1].
Holden Karnofsky is an interesting example of someone doing this well, somewhat in public, right now.
Who else?
I haven’t finished this blog post. But I’m committed to publishing something every day, so here goes. Let’s call it “part 1”.
[1] It’s also critical to ask: is this insight an information hazard? Most insights have mixed consequences when they become widely known—some may have very strongly negative consequences, on net. The example of Szilard comes to mind again.
Robin Hanson on the future as reality
The future is not the realization of our hopes and dreams, a warning to mend our ways, an adventure to inspire us, nor a romance to touch our hearts. The future is just another place in space-time. Its residents, like us, find their world mundane and morally ambiguous.
[…]
New habits and attitudes result less than you think from moral progress, and more from people adapting to new situations. So many of your descendants’ strange habits and attitudes are likely to violate your concepts of moral progress; what they do may often seem wrong. Also, you likely won’t be able to easily categorize many future ways as either good or evil; they will instead just seem weird. After all, your world hardly fits the morality tales your distant ancestors told; to them you’d just seem weird. Complex realities frustrate simple summaries, and don’t fit simple morality tales.
The Age of Em: Introduction
Nietzsche’s mountain
Those who can breathe the air of my writings know that it is an air of the heights, a strong air. One must be made for it. Otherwise there is no small danger one may catch cold in it. The ice is near, the solitude tremendous . . . Philosophy, as I have so far understood and lived it, means living voluntarily among ice and high mountains—seeking out everything strange and questionable in existence, everything so far placed under a ban by morality.
—Nietzsche, Preface to Ecce Homo
Nietzsche wasn’t climbing Parfit’s mountain
Andrew Huddlestone offers a good treatment of Parfit on Nietzsche.
My summary:
- Parfit really wants convergence because he is committed to moral intuitionism in the tradition of Sidgwick. On this picture, epistemic peers in ideal conditions should converge on a cluster of “self-evident” normative axioms—on pain of intractable disagreement about whose “self-evident” intuitions to trust.
- Compare mathematics, geometry and logic, where there does seem to be a common core of widely shared “self-evident” intuitions, that can be taken as axioms (even though non-Western thought has some variations).
- Parfit takes Nietzsche seriously, thinks of him as an epistemic peer on par with (e.g. Kant). So he needs to either dissolve merely apparent disagreement, or to explain Nietzsche’s disagreement in terms of non-ideal epistemic conditions or mistakes that Nietzsche would have recognised as such, if they were pointed out.
- Parfit tries to paint Nietzsche as a mixture of (a) more in agreement than he seems and (b) making a couple of basic mistakes. To do this, he strawmans Nietzsche rather badly, making heavy use of unpublished journal fragments (an approach which strikes me as odd… to the point of underhand?).
- Parfit doesn’t give a plausible account of Nietzsche’s normative views (anti-egalitarianism; suffering sometimes non-instrumentally good) or his meta-axiological views (Huddlestone thinks they’re underdetermined by Nietzsche’s writings, but Nietzsche definitely didn’t hold the “normativity requires God” thesis which Parfit attributes to him).
Nick Bostrom on the influence of moral prophets
It is not too soon to call for practitioners to express a commitment to safety, including endorsing the common good principle and promising to ramp up safety if and when the prospect of machine superintelligence begins to look more imminent. Pious words are not sufficient and will not by themselves make a dangerous technology safe: but where the mouth goeth, the mind might gradually follow.
—Superintelligence, Chapter 15: Crunch Time
And out of his mouth goeth a sharp sword, that with it he should smite the nations: and he shall rule them with a rod of iron: and he treadeth the winepress of the fierceness and wrath of Almighty God. And he hath on his vesture and on his thigh a name written, KING OF KINGS, AND LORD OF LORDS.
—Revelation 19:15
Nick Bostrom on crucial considerations
Strategic analysis is especially needful when we are radically uncertain not just about some detail of some peripheral matter but about the cardinal qualities of the central things. For many key parameters, we are radically uncertain even about their sign—that is, we know not which direction of change would be desirable and which undesirable. Our ignorance might not be irremediable. The field has been little prospected, and glimmering strategic insights could still be awaiting their unearthing just a few feet beneath the surface.
What we mean by “strategic analysis” here is a search for crucial considerations: ideas or arguments with the potential to change our views not merely about the fine-structure of implementation but about the general topology of desirability. Even a single missed crucial consideration could vitiate our most valiant efforts or render them as actively harmful as those of a soldier who is fighting on the wrong side. The search for crucial considerations (which must explore normative as well as descriptive issues) will often require crisscrossing the boundaries between different academic disciplines and other fields of knowledge. As there is no established methodology for how to go about this kind of research, difficult original thinking is necessary.
—Superintelligence, Chapter 15: Crunch Time