I would say both immigration and crime are relevant to progress!

Our primitive monkey brains are good at over-estimating very unlikely risks.^[2]

I think this is presupposing the question isn’t it.

If a risk is indeed very unlikely, then we will tend to overestimate it. (If the probability is 0 it’s impossible to underestimate)

But for risks that are actually quite likely, then we are more likely to underestimate them.

And of course, bias estimates cut both ways. “Our primitive monkey brains are good at ignoring and underestimating abstract and hard to understand risks”.

Thanks Donald, good feedback. I agree about maximizing good over minimizing bad. Curing aging, or extending healthspan, is a great one. Certainly an easier sell than becoming a multiplanetary species.

This is a linkpost for https://​​amistrongeryet.substack.com/​​p/​​alphaproof-and-openai-o1

The latest advances in AI reasoning come from OpenAI’s o1 and Google’s AlphaProof. In this post, I explore how these new models work, and what that tells us about the path to AGI.

Interestingly, unlike GPT-2 → GPT-3 → GPT-4, neither of these models rely on increased scale to drive capabilities. Instead, both systems rely on training data that shows, not just the solution to a problem, but the path to that solution. This opens a new frontier for progress in AI capabilities: how to create that sort of data?

In this post, I review what is known about how AlphaProof and o1 work, discuss the connection between their training data and their capabilities, and identify some problems that remain to be solved in order for capabilities to continue to progress along this path.

Ok. Firstly I do think your “Embodied information” is real. I just think it’s pretty small. You need the molecular structure for 4 base pairs of DNA, and for 30 ish protiens. And this wikipedia page. https://​​en.wikipedia.org/​​wiki/​​DNA_and_RNA_codon_tables

That seems to be in the kilobytes. It’s a rather small amount of information compared to DNA.

Epigenetics is about extra tags that get added. So theoretically the amount of information could be nearly as much as in the DNA. For example, methyization can happen on A and C, so that’s 1 bit per base pair, in theory.

Also, the structure of DNA hasn’t changed much since early micro-organisms existed. Neither has a lot of the other embodied information.

Therefore the information doesn’t contain optimization over intelligence, because all life forms with a brain had the same DNA.

Humans are better than LLM’s at highly abstact tasks like quantum physics or haskel programming.

You can’t argue that this is a result of billions of years of evolution. Sea sponges weren’t running crude haskel programs a billion years ago.

Therefore, whatever data the human brain has, it is highly general information about intelligence.

Suppose we put the full human genome, plus a lot of data about DNA and protein structure, into the LLM training data. In theory, the LLM has all the data that evolution worked so hard to produce. In practice, LLM’s aren’t smart enough to come up with fundamental insights about the nature of intelligence from the raw human genome.

So there is some piece of data, with a length between a few bits and several megabytes, that is implicitly encoded in the human genome, and that describes an algorithm for higher intelligence in general.

If it’s a collection of millions of unintelligible interacting “hacks” tuned to statistical properties of the environment, then maybe not.

Well those “hacks” would have to generalize well. Modern humans operate WAY out of distribution and work on very different problems.

Would interacting hacks that were optimized to hunt mammoths also happen to work in solving abstract maths problems?

So how would this work. There would need to be a set of complicated hacks that work on all sorts of problems, including abstract maths. Abstract maths has limitless training data in theory. And if the hacks apply to all sorts of problems, then data on all sorts of problems is useful in finding the hacks.

If the hacks contain a million bits of information, and help answer a million true/​false questions, then they are in principle findable with sufficient compute.

Also, bear in mind that evolution is INCREADIBLY data inefficient. Yes there are a huge number of ancestors. But evolution only finds out how many children got produced. A human can look at a graph and realize that a 1% increase in parameter X causes a 1% improvement in performance. Evolution randomly makes some individual with 1% more X, and they get killed by a tiger. Bad luck.

And again. Most of the billions of years there were no brains at all. The gap between humans and monkeyish creatures is a few Million years.

AIXI is a theoretical model of an ideal intelligence, it’s a few lines of maths.

I’m not saying it’s totally impossible that there is some weird form of evolution data wall. But mostly this looks like a fairly straightforward insight, possessable, and not possessed by us. I think it’s pretty clear that the human algorithm makes at least a modest amount of sense and isn’t too hard to find with trial and error on the same training dataset. (When the dataset is large, and the amount of outer optimization is fairly modest, the risk of overfitting in the outer stage is small)

https://​​www.lesswrong.com/​​posts/​​ZiRKzx3yv7NyA5rjF/​​the-robots-ai-and-unemployment-anti-faq

Once AI does get that level of intelligence, jobs should be the least of our concerns. Utopia or extinction, our future is up to the AI.

> It also seems vanishingly unlikely that the pressures on middle class jobs, artists, and writers will decrease even if we rolled back the last 5 years of progress in AI—but we wouldn’t have the accompanying productivity gains which could be used to pay for UBI or other programs.

When plenty of people are saying that AGI is likely to cause human extinction, and the worst scenario you can come up with is middle class jobs, your side is the safe one.

I think your notion of “environmental progress” itself is skewing things.

When humans were hunter gatherers, we didn’t have much ability to modify our surroundings.

Currently, we are bemoaning global warming, but if the earth was cooling instead, we would bemoan that too.

Environmentalism seems to only look at part of the effects.

No one boasts about how high the biodiversity is at zoos. No one is talking about cities being a great habitat for pigeons as an environmental success story.

The whole idea around the environmentalist movement is the naturalistic fallacy turned up to 11. Any change made by humans automatically becomes a problem.

It’s goal seems to be “make the earth resemble what it would look like had humans never existed”.

(Name one way humans made an improvement to some aspect of the environment compared to what it was a million years ago)

A goal that kind of gets harder by default as humanities ability to modify the earth increases.

One system I think would be good is issue based voting.

So for example, there would be several candidates for the position of “health minister”, and everyone gets to vote on that.

And independently people get to vote for the next minister for education.

Other interesting add ons include voting for an abstract organization, not a person. One person that decides everything is an option on the balot, but so are various organizations, with various decision procedures. You can vote for the team that listens to prediction markets, or even some sub-democracy system. (Because the organizations can use arbitrary mechanisms, including more votes, teams of people, whatever they like)

Approval voting is good.

An interesting option is to run a 1-of-many election.

So you can cast a ballot in the health-election or in the education election or in the energy election or …, depending on which issue you feel most strongly about. (But you can’t vote on both issues at one time.) This has a nice property that the fewer people care about a topic, the further your vote goes if you decide to vote on that topic.

Solving global warming

Most of the current attempts that interact with everyday people are random greenwashing trying to get people to sort recycling or use paper straws. Yes solar panel tech is advancing, but that’s kind of in the background to most peoples day to day lives.

And all this goal is promising is that things won’t get worse via climate change. It isn’t actually a vision of positive change.

A future with ultra cheap energy, electric short range aviation in common use etc.

Building true artificial intelligence (AGI, or artificial general intelligence)

Half the experts are warning that this is a poisoned chalice. Can we not unite towards this goal until/​unless we come to a conclusion that the risk of human extinction from AGI takeover is low.

Also, if we do succeed in AGI alignment, the line from AGI to good things is very abstract.

What specific nice thing will the AGI do? (The actual answer is also likely to be a bizarre world full of uploaded minds or something. Real utopian futures aren’t obliged to make sense to the average person within a 5 minute explanation.)

Colonizing Mars

Feels like a useless vanity project. (See the moon landings. Lots of PR, not much practical benefit.)

How about something like curing aging? Even the war on cancer was a reasonable vision of a positive improvement.

You are raising good questions, though they are probably beyond the scope for me to answer. My high-level take would be that there are quite a few existing laws that could apply in such a scenario (eg Neuralink-implants to record brain-activity need FDA approval) and that we should expect laws to be adapted to new circumstances caveated with the pacing problem.

within human laws

I have no idea what superintelligence following existing laws even looks like.

Take mind uploading. Is it

1. Murder

2. A (currently unapproved) medical procedure

3. Not something the law makes any mention of, so permitted by default.

The current law is very vague with respect to tech that doesn’t exist yet. And there are a few laws which, depending on interpretation, might be logically impossible to follow.

ASI by default pushes right to the edge of what is technically possible, not a good fit with vague rules.

That is a very fair point! I guess even within human laws there is some point before “God-level” where the “automation overhang” is reduced when AI becomes so good that it can compete with the product/​services of many companies end-to-end rather than relying on integration into business processes. Still, I think it’s fair to say that a) business integration can be/​is a bottleneck to automation and b) “automation overhang” differs between products/​service based on market structure (eg lower in management consulting, higher in public transport)

So, even if Sam Altman would declare tomorrow that he has built a digital God and that he will roll it out for free to everyone, this would not immediately lead to full automation.

Superintelligent AI (whether friendly or not) may not feel obliged to follow human laws and customs that slow down regular automation.

Yeah, I think Oster is great. I think I only differ with her in two respects:

1. As you noted, she sometimes seems to imply “absence of evidence is evidence of absence”.

2. I think she’s too quick to dismiss PROBIT. In Cribsheet, she notes that the IQ measurements at 6.5 might be biased since the evaluators were not blinded. But she doesn’t mention the audit results or the teacher evaluation results. None of these were significant, but every single subtest was positive, and had surprisingly large effects once you account for the fact that the intervention had such a small effect on breastfeeding.

For most of the plausible mechanisms, it seems like partial breastfeeding (and/​or pumping) could capture most of the benefits. The major exceptions would be if the local water was unhealthy, or some of the weirder theories, e.g. that babies feeding in a supine position might cause temporary deafness. (Sounds crazy to me, but seems to be taken seriously).

Regarding benefits for the mother, may I introduce you to the most surprising thing I learned of all: Breastfeeding apparently causes the uterus to contract faster.

The antibodies argument always made the most sense to me. But note that this is an argument for some breast milk, not an all-breast-milk diet—that is, it’s not an argument against formula, just an argument against an all-formula diet. I mention this because when we were in the hospital with our kid, they were pushing against formula very hard.

Also, it’s not an argument for literal feeding at the breast, as opposed to pumping and then bottle-feeding with the breast milk, which is easier for some people.

Emily Oster covers breastfeeding in Chapter 4 of Cribsheet, more extensively than in the 538 article you linked to. IIRC, she notes that there is evidence of benefit for the mother in terms of reduced breast cancer risk (no idea why that would be, though).

(But in general, I agree that Oster is too quick to say “it doesn’t matter” about things that we don’t have rigorous evidence for, rather than trying to make an informed decision about the best course of action based on what data and theories we do have. Other than that minor criticism, though, I am a big fan of her work.)

Here are my initial suggestions, keeping in mind that chaos theory is a subset of dynamical systems theory. For some modern applications it has been subsumed as an element of applied complexity science.

1. The Predictors: How a band of maverick physicists used chaos theory to trade their way to a fortune on Wall Street, Thomas A. Bass, Henry Holt and Company, New York, 1999.

2. Increasing Returns and Path Dependence in the Economy, W. Brian Arthur, The University of Michigan Press, 1994

3. Complexity and the Economy, W. Brian Arthur, Oxford University Press, 2015

4. Complexity Economics, Dialogues of the Applied Complexity Network, W. Brian Arthur, Eric D. Beinhocker, Allison Stanger, Editors, SFI Press, 2020

5. Making Sense of Chaos: A Better Economics for a Better World, J. Doyne Farmer, Penguin Random House UK, 2024

6. Chaos and Nonlinear Dynamics: An Introduction for Scientists and Engineers, 2nd Edition, Robert C. Hilborn, Oxford University Press, 2000. Take a look at Chapter 11 and Chapter 12.

7. Nonlinear Spatio-Temporal Dynamics and Chaos in Semiconductors, Eckehard Scholl, Cambridge University Press, 2001

8. Chaos in Circuits and Systems, Guanrong Chen and Tetsushi Ueta, Editors, World Scientific Press, 2002. Recommended Chapters: 5, 6,10, 12,13, 17, 22, 23, 24, 26, 27.

Many of the engineering applications can be found in application dependent modern circuit designs. Spread spectrum communications and advanced defense electronic applications make use of tools and methods from items 6, 7, and 8.

I hope these are helpful.

That would be great.

The list of fields you gave makes me a little nervous, so I should clarify that I’m looking for specific, practical applications that have become widespread and paid off. ApEn is better than nothing, but was outcompeted in every application. Most of the applications of chaos to ecology were retroactive explanations, with an occasional prediction, but almost never a story like “we implemented chaos-derived management in these fisheries and they did better/​worse than control fisheries”.

The sheer volume of applications that never went anywhere makes it hard to find the successful practical applications, so I’m very grateful for pointers. But I do need pointers to specific successes, not fields where the math was tried.

Elizabeth and Alex—This is an interesting article. More broadly, there is a significant community of scientists/​engineers/​mathematicians that have worked in dynamical systems, nonlinear dynamics, chaos theory, and applications of chaotic dynamics. A reasonable number of technical monographs and textbooks have been written on these areas since the early to mid 1980’s. Applications exist in electrical engineering, mechanical engineering, fluid dynamics, biology, ecology, economics, networks, medical science, applied mathematics,…. I am not sure whether you want to do a follow up article, or devote time to further investigation, but I am happy to provide pointers to relevant literature if you like.

—Evgeny Sedukhin—“Symphony of the sixth blast furnace” (1979)