I expect there's a ton of useful stuff you can learn (that humanity is currently ignorant about) just from looking at existing data on the Internet. 

Thank you for the reply, I agree with this point. Now that I think about it, protein folding is a good example of how the data was already available but before AlphaFold, nobody could predict sequence to structure with high accuracy. Maybe a sufficiently smart AGI can get more knowledge out of existing data on the internet without performing too many new experiments.

How much more can it squeeze out of existing data (which were not generated specifically with the AGI's new hypothesis in mind), and if it that can put a decisive advantage over humanity in a short span of time could be important? I.e. whether existing data out there contains within them enough information to figure out new science that is completely beyond our current understanding and can totally screw us.

Might be a naive question:

For a STEM-capable AGI (or any intelligence for that matter) to do new science, it would have to interact with the physical environment to conduct experiments. Otherwise, how can the intelligent agent discover and validate new theories? For example, an AGI that understands physics and material science may theorize and propose thousands of possible high-temperature superconductors, but actually discovering a working material can happen only after actually synthesizing those materials and performing the experiments, which is time-consuming and difficult to do.

If that's true, then the speed in which the STEM-capable AGI discovers new knowledge, and correspondingly its "knowledge advantage" (not intelligence advantage) over humanity is bottlenecked by the speed in which the AGI can interact and perform experiments in the physical world, which as of now depends almost entirely on human operated equipment and is constrained by various real world physical limitations (wear and tear, speed of chemical reactions, speed of biological systems, energy consumption etc.). Doesn't this significantly throttles the speed of AGI gaining advantage over humanity, giving us more time for alignment?

You probably might have came across it already, but the Good Food Institute (GFI) has compiled a research funding database for alternative proteins (here).

Overall I think having collaborators that have the expertise and facilities to perform biological experiments and bioprocessing may be quite crucial. My experience is that at this moment, publicly available data are scarce and insufficient to support purely computational work  in this field, unlike say cancer research where loads of multi-omics data are available for data mining and modeling.

Thank you for your excellent posts summarizing multiple sources of information across domains of energy and material limits of human development, ecological economics etc. I am still reading through your in depth 3-parts works as I speak, and I am finding many useful sources of information for my further reading.

I happen to chance upon this discussion while browsing around, and decided to create an account to reply to this discussion because it is a topic of great interest to me.

I think the main reaon why you believe that Corentin's argument on EROI affecting percent of GDP required to maintain energy production is a conceptual mistake, is because you have assumed that cost of production (of energy producing equipment) is not linked to energy use. 

 However, the basis of the EROI argument stems from biophysical economics, and is based on the key assumption that the vast majority of economic activity and economic value are in fact embodiement of energy. One may or may not choose to agree with this assumption, but if you do take this assumption to be true, then Corentin's point that for e.g. a 2:1 EROI needs roughly half of society's resources is correct.

So in the simple equation that you described:

let:

• $C$  be the cost of entire energy system
•  $E$  be the total energy produced/demanded
•  $e_{out,eq}$ be the energy produced per unit of equipment
• $c_{eq}$ be the cost per unit of equipment
•  $e_{in,eq}$ be the energy used to produce each unit of equipment

Then,  $C=E/e_{out,eq}\ast c_{eq}$

Because we assume that economic cost of production of anything is directly related to energy, then $c_{eq}=\alpha e_{in,eq}$, where $\alpha$ is some factor describing the economic cost in terms of energy.

Substituting it in the energy cost equation, we get $C=\alpha E/e_{out,eq}\ast e_{in,eq}$

$e_{out,eq}/e_{in,eq}$ is exactly the definition of EROI of the energy producing equipment, and thus $C=\frac{\alpha E}{EROI}$.

Furthermore, with the same key assumption, the total economic output, in other words GDP can be also be expressed in terms of total energy produced or demanded by the economy, i.e. $GDP=\beta E$.

We finally obtain that:

$C=\frac{\alpha \cdot GDP}{\beta \cdot EROI}$

If the scaling factor $\alpha$ and $\beta$ between economic cost and energy is roughly similar for the particular case of energy producing equipment, and for the general case across the whole economy, then EROI approximately determines the proportion of the cost of operating and maintaining the energy system against GDP.

The key assumption put forward by the biophysical economists has been argued both through first principles and empirically (well explored in this textbook of energy and biophysical economics^[1]).  

1. ^{^}

   Hall, C. A., & Klitgaard, K. A. (2011). Energy and the Wealth of Nations. New York: Springer.