Five Reasons Effective Altruists Should Use Less Bayesian Reasoning
Our knowledge is a lot more limited than we think.
Many effective altruists (EAs) use extensive “bayesian reasoning.” The basic idea behind it is that you should do the following:
Be willing to assign probabilities to the likelihood of anything occurring or being true.
Update these probabilities when you learn new information.
Act on these probabilities if they suggest that certain actions are higher in value than all other actions.
Bayesian reasoning is helpful for a lot of everyday decision-making. If you’re trying to figure out whether to take a job in Los Angeles or in New York, it makes sense to try to guess how happy you’d be in each respective city. And, if you’re trying to compare career paths, you should assign probabilities to how likely you are to succeed in them.
But, to me, EAs take this kind of reasoning too far. EAs have variously tried to predict how many future humans there will be, asked non-domain experts how likely they think a catastrophic pandemic will be, and even tried to determine whether we should work on improving the lives of people in the far future.
Probably the most common (and representative) example of this, though, is the idea of AI timelines, which means trying to predict when “AGI” will be developed. Given that we have no clue what it actually takes to develop AGI or how many breakthroughs are required, it seems pretty unreasonable to me to assign an exact year to when such an outcome would occur.
But, setting that aside, these are my five reasons EAs should use less Bayesian reasoning.
If you know very little about something, your guesses are completely arbitrary.
Have you ever asked a child how far away they think New York is and then heard them guess “a hundred miles?” This child’s guess is inaccurate because they don’t know very much about the relative distances of places around the world.
I think this same line of reasoning can be applied to a lot of EA’s predictions about the future. For instance, a lot of EAs make guesses about the likelihood of AI takeover, but I don't think we have enough knowledge to know when we'll develop AGI, whether we’ll develop AGI, or what the development of AGI will look like. So, given this, it seems pretty unlikely that we could come up with any kind of accurate prediction about how likely AI takeover is.
We’re bad at making guesses in general.
Some people have undergone extensive training to be good at making forecasts over short durations of time in situations where trends can be roughly extrapolated forwards.
Most people have not undergone this training, and are, in fact, pretty bad at making guesses in general.
Most people are bad at predicting how successful they will be, who will win elections, and how long it will take for them to finish projects.
Given this, it seems like we should expect our predictions on most things to be off by a reasonable extent.
If your guesses are completely arbitrary, updating won’t bring you to the correct probability.
Bayesians like to say that, if you have no clue how likely something is, you should just pick a random number and then update your beliefs from there. The problem, though, is that, if you’re updating relative to an arbitrary number, your arbitrary number might hold too much weight
For instance, if you think there’s a 10% chance that a pandemic this century will kill more than a billion people, you might be only willing to update by a single order of magnitude each time you learn new information.
Given this, if you learned that the Chinese government has decided to stop stockpiling masks, you might reduce your probability to 1%, but, if the real probability were 10^-7, you would need an overwhelming amount of information to update your beliefs to the correct probability.
We should expect most guesses about the future to be wrong.
People are generally familiar with the idea that we’re bad at predicting the future, but I think they fail to take seriously how significant of an issue this is.
The fact is that history has been determined by an extraordinarily complex interaction of social, political, environmental, economic, and circumstantial factors. And, as a result, historically, people were very bad at predicting the future. If you had someone in 1910 try to make predictions about how the century would go, they would probably be wrong in a vast myriad of ways. They likely wouldn’t have predicted two world wars and a cold war. They wouldn’t have guessed that we’d discover the existence of galaxies. And, they wouldn’t have been able to tell you that we’d become completely digitally interconnected. Given this, I think we should also consider our own predictions about the future to likely be very wrong.
If your guesses are based on other people’s guesses, you might all be wrong.
Humans experience an anchoring bias when it comes to making predictions, so if we hear someone make a prediction, we usually make ours relative to theirs. The problem with this is that, if one person makes a very prominent prediction that is completely off, everyone will be basing their prediction on that bad prediction.
I think this is particularly concerning in domains where only a few individuals have prominence, but there's very little information to go off of. If everyone is assuming those individuals know more than they do, then everyone will have very biased guesses.
People rarely offer extraordinarily low probabilities.
Whenever I ask someone how likely they think something is, they pretty much never give a probability less than .1% unless that something is religious in nature. Given this, it seems like people systematically over estimate low probabilities because they fail to consider probabilities such as 10^-7 or 10^-53.