If you haven’t already read the work of the late Aaron Wildavsky, I would highly recommend it because he devoted much of his life’s work to the exact issue you tee up here. I’d recommend two of his books to start. The first is Risk and Culture co-authored with Mary Douglas, and the second is his absolutely remarkable Searching for Safety, which served as the inspiration for my book on Permissionless Innovation.
Here are a few choice quotes from Risk and Culture:
“Relative safety is not a static but rather a dynamic product of learning from error over time. . . . The fewer the trails and the fewer the mistakes to learn from, the more error remains uncorrected.” (p. 195)
“The ability to learn from errors and gain experience in coping with a wide variety of difficulties, has proved a greater aid to preservation of the species than efforts to create a narrow band of controlled conditions within which they would flourish for a time… “(p. 196)
“If some degree of risk is inevitable, suppressing it in one place often merely moves it to another. Shifting risks may be more dangerous than tolerating them, both because those who face new risks may be unaccustomed to them and because those who no longer face old ones may be more vulnerable when conditions change.”(p. 197)
And then in Searching for Safety, Wildavsky went on to build on that logic as he warned of the dangers of “trial without error” reasoning, and contrasted it with the trial-and-error method of evaluating risk and seeking wise solutions to it. Wildavsky argued that wisdom and safety are born of experience and that we can learn how to be wealthier and healthier as individuals and a society only by first being willing to embrace uncertainty and even occasional failure. I’ve probably quoted this passage from that book in more of my work than anything else I can think of:
“The direct implication of trial without error is obvious: If you can do nothing without knowing first how it will turn out, you cannot do anything at all. An indirect implication of trial without error is that if trying new things is made more costly, there will be fewer departures from past practice; this very lack of change may itself be dangerous in forgoing chances to reduce existing hazards. . . . . Existing hazards will continue to cause harm if we fail to reduce them by taking advantage of the opportunity to benefit from repeated trials.”
In my next book on AI governance, I extend this framework to AI risk.
Ah, thanks, I have read a little bit of Searching for Safety in the past, but had forgotten about this.
I largely agree with this approach. The one problem is when dealing with catastrophic risks, you can’t afford to have an error. In the case of existential risk, there is literally no way to learn or recover from mistakes. In general the worse the risk, the more you need careful analysis and planning up front.
Jason:
If you haven’t already read the work of the late Aaron Wildavsky, I would highly recommend it because he devoted much of his life’s work to the exact issue you tee up here. I’d recommend two of his books to start. The first is Risk and Culture co-authored with Mary Douglas, and the second is his absolutely remarkable Searching for Safety, which served as the inspiration for my book on Permissionless Innovation.
Here are a few choice quotes from Risk and Culture:
“Relative safety is not a static but rather a dynamic product of learning from error over time. . . . The fewer the trails and the fewer the mistakes to learn from, the more error remains uncorrected.” (p. 195)
“The ability to learn from errors and gain experience in coping with a wide variety of difficulties, has proved a greater aid to preservation of the species than efforts to create a narrow band of controlled conditions within which they would flourish for a time… “ (p. 196)
“If some degree of risk is inevitable, suppressing it in one place often merely moves it to another. Shifting risks may be more dangerous than tolerating them, both because those who face new risks may be unaccustomed to them and because those who no longer face old ones may be more vulnerable when conditions change.” (p. 197)
And then in Searching for Safety, Wildavsky went on to build on that logic as he warned of the dangers of “trial without error” reasoning, and contrasted it with the trial-and-error method of evaluating risk and seeking wise solutions to it. Wildavsky argued that wisdom and safety are born of experience and that we can learn how to be wealthier and healthier as individuals and a society only by first being willing to embrace uncertainty and even occasional failure. I’ve probably quoted this passage from that book in more of my work than anything else I can think of:
“The direct implication of trial without error is obvious: If you can do nothing without knowing first how it will turn out, you cannot do anything at all. An indirect implication of trial without error is that if trying new things is made more costly, there will be fewer departures from past practice; this very lack of change may itself be dangerous in forgoing chances to reduce existing hazards. . . . . Existing hazards will continue to cause harm if we fail to reduce them by taking advantage of the opportunity to benefit from repeated trials.”
In my next book on AI governance, I extend this framework to AI risk.
Ah, thanks, I have read a little bit of Searching for Safety in the past, but had forgotten about this.
I largely agree with this approach. The one problem is when dealing with catastrophic risks, you can’t afford to have an error. In the case of existential risk, there is literally no way to learn or recover from mistakes. In general the worse the risk, the more you need careful analysis and planning up front.