What I'm pointing at there is that for strength/weight purposes, using big calcium nuclei to create stronger individual bonds in bone, is like making a steel beam stronger by putting more steel into it; the strength costs weight.

Why is horn weaker than diamond?

I agree that "biology sticks to ionic bonds and static cling" was badly put because lignin, and I'll retire that one.

I'm not sure what's a truer analogy than static cling for hydrophobia as a force holding things together which the general audience has any experience with.  Macroscopic experience of hydrophobia is, like, oil collecting on the surface of water, which isn't experienced as a binding force the way that static cling is.

I'm sort of skeptical that you could write something that works as science communication for a general audience, though lord knows I'm not necessarily succeeding either.  The key valid ideas to be communicated are:

• There exists a level above biology for molecular systems, greatly superior in terms of strength and energy density.  This sets a lower bound on how a very smart and uncaring entity could kill you, which looks like it attacking you with micron-diameter robots, which looks like everyone on Earth falling over dead in the same second.
• The designed micron-diameter thingies can easily kill you, where bacteria can't, because the designed thingies can more easily rip apart human cell membranes or white blood cells made of flimsier materials.  They can do that because human cell membranes are held together by static cling, as are bacterial cells; whereas the ideal limits of what micron-sized engines can be put together are more like "diamond".
• This design space isn't accessible to natural selection despite being physically possible, because evolutionary biology has an incredibly hard time designing systems like freely rotating wheels; for reasons that generalize to evolution not creating airborne cell-engines with solid covalently bonded shells and manipulator ports.  My attempt to compress "Why?" down to something maybe overly pithy is "Because shallow energy gradients are more densely connected in the design space of simple mutations than deep energy gradients."

Now, instead of talking about human cell membranes being held together by static cling, I could talk about extremely thin metallic twisty-tie wires with some magnetized sections that help them fold up together into particular configurations in a barrel of magnetized ball bearings.  Your suggestion above for science communication is that this is a great thing to mention, because it helps convey the following interesting truth: if we churn the ball bearings hard enough to unfold the twisty tie, it'll sometimes fold right back up into the same shape again once we stop churning!

This more complicated metaphor may legit add something to an explanation of organic chemistry.  I don't disagree that it's cool, or important to organic chemistry proper.

From the perspective of explaining how you die when you confront an uncaring mind that thinks smarter and much faster than humanity, it doesn't add anything not already contained in "cell membranes are held together by static cling".

Why is flesh weaker than diamond?  Diamond is made of carbon-carbon bonds.  Proteins also have some carbon-carbon bonds!  So why should a diamond blade be able to cut skin?

I reply:  Because the strength of the material is determined by its weakest link, not its strongest link.  A structure of steel beams held together at the vertices by Scotch tape (and lacking other clever arrangements of mechanical advantage) has the strength of Scotch tape rather than the strength of steel.

Or:  Even when the load-bearing forces holding large molecular systems together are locally covalent bonds, as in lignin (what makes wood strong), if you've got larger molecules only held together by covalent bonds at interspersed points along their edges, that's like having 10cm-diameter steel beams held together by 1cm welds.  Again, barring other clever arrangements of mechanical advantage, that structure has the strength of 1cm of steel rather than 10cm of steel.

Bone is stronger than wood; it runs on a relatively stronger structure of ionic bonds, which are no locally weaker than carbon bonds in terms of attojoules of potential energy per bond.  Bone is weaker than diamond, then, because... why?

Well, partially, IIUC, because calcium atoms are heavier than carbon atoms.  So even if per-bond the ionic forces are strong, some of that is lost in the price you pay for including heavier atoms whose nuclei have more protons that are able to exert the stronger electrical forces making up that stronger bond.

But mainly, bone is so much weaker than diamond (on my understanding) because the carbon bonds in diamond have a regular crystal structure that locks the carbon atoms into relative angles, and in a solid diamond this crystal structure is tesselated globally.  Hydroxyapatite (the crystal part of bone) also tesselates in an energetically favorable configuration; but (I could be wrong about this) it doesn't have the same local resistance to local deformation; and also, the actual hydroxyapatite crystal is assembled by other tissues that layer the ionic components into place, which means that a larger structure of bone is full of fault lines.  Bone cleaves along the weaker fault line, not at its strongest point.

But then, why don't diamond bones exist already?  Not just for the added strength; why make the organism look for calcium and phosphorus instead of just carbon?

The search process of evolutionary biology is not the search of engineering; natural selection can only access designs via pathways of incremental mutations that are locally advantageous, not intelligently designed simultaneous changes that compensate for each other.  There were, last time I checked, only three known cases where evolutionary biology invented the freely rotating wheel.  Two of those known cases are ATP synthase and the bacterial flagellum, which demonstrates that freely rotating wheels are in fact incredibly useful in biology, and are conserved when biology stumbles across them after a few hundred million years of search.  But there's no use for a freely rotating wheel without a bearing and there's no use for a bearing without a freely rotating wheel, and a simultaneous dependency like that is a huge obstacle to biology, even though it's a hardly noticeable obstacle to intelligent engineering.

The entire human body, faced with a strong impact like being gored by a rhinocerous horn, will fail at its weakest point, not its strongest point.  How much evolutionary advantage is there to stronger bone, if what fails first is torn muscle?  How much advantage is there to an impact-resistant kidney, if most fights that destroy a kidney will kill you anyways?  Evolution is not the sort of optimizer that says, "Okay, let's design an entire stronger body."  (Analogously, the collection of faults that add up to "old age" is large enough that a little more age resistance in one place is not much of an advantage if other aging systems or outward accidents will soon kill you anyways.)

I don't even think we have much of a reason to believe that it'd be physically (rather than informationally) difficult to have a set of enzymes that synthesize diamond.  It could just require 3 things to go right simultaneously, and so be much much harder to stumble across than tossing more hydroxyapatite to lock into place in a bone crystal.  And then even if somehow evolution hit on the right set of 3 simultaneous mutations, sometime over the history of Earth, the resulting little isolated chunk of diamond probably would not be somewhere in the phenotype that had previously constituted the weakest point in a mechanical system that frequently failed.  If evolution has huge difficulty inventing wheels, why expect that it could build diamond chainmail, even assuming that diamond chainmail is physically possible and could be useful to an organism that had it?

Talking to the general public is hard.  The first concept I'm trying to convey to them is that there's an underlying physical, mechanical reason that flesh is weaker than diamond; and that this reason isn't that things animated by vitalic spirit, elan vital, can self-heal and self-reproduce at the cost of being weaker than the cold steel making up lifeless machines, as is the price of magic imposed by the universe to maintain game balance.  This is a very natural way for humans to think; and the thing I am trying to come in and do is say, "Actually, no, it's not a mystical balance, it's that diamond is held together by bonds that are hundreds of kJ/mol; and the mechanical strength of proteins is determined by forces a hundred times as weak as that, the part where proteins fold up like spaghetti held together by static cling."

There is then a deeper story that's even harder to explain, about why evolution doesn't build freely rotating wheels or diamond chainmail; why evolutionary design doesn't find the physically possible stronger systems.  But first you need to give people a mechanical intuition for why, in a very rough intuitive sense, it is physically possible to have stuff that moves and lives and self-repairs but is strong like diamond instead of flesh, without this violating a mystical balance where the price of vitalic animation is lower material strength.

And that mechanical intuition is:  Deep down is a bunch of stuff that, if you could see videos of it, would look more like tiny machines than like magic, though they would not look like familiar machines (very few freely rotating wheels).  Then why aren't these machines strong like human machines of steel are strong?  Because iron atoms are stronger than carbon atoms?  Actually no, diamond is made of carbon and that's still quite strong.  The reason is that these tiny systems of machinery are held together (at the weakest joints, not the strongest joints!) by static cling.

And then the deeper question:  Why does evolution build that way?  And the deeper answer:  Because everything evolution builds is arrived at as an error, a mutation, from something else that it builds.  Very tight bonds fold up along very deterministic pathways.  So (in the average case, not every case) the neighborhood of functionally similar designs is densely connected along shallow energy gradients and sparsely connected along deep energy gradients.  Intelligence can leap long distances through that design space using coordinated changes, but evolutionary exploration usually cannot.

And I do try to explain that too.  But it is legitimately more abstract and harder to understand.  So I lead with the idea that proteins are held together by static cling.  This is, I think, validly the first fact you lead with if the audience does not already know it, and just has no clue why anyone could possibly possibly think that there might even be machinery that does what bacterial machinery does but better.  The typical audience is not starting out with the intuition that one would naively think that of course you could put together stronger molecular machinery, given the physics of stronger bonds, and then we debate whether (as I believe) the naive intuition is actually just valid and correct; they don't understand what the naive intuition is about, and that's the first thing to convey.

If somebody then says, "How can you be so ignorant of chemistry?  Some atoms in protein are held together by covalent bonds, not by static cling!  There's even eg sulfur bonds whereby some parts of the folded-spaghetti systems end up glued together with real glue!" then this does not validly address the original point because: the underlying point about why flesh is more easily cleaved than diamond, is about the weakest points of flesh rather than the strongest points in flesh, because that's what determines the mechanical strength of the larger system.

I think there is an important way of looking at questions like these where, at the final end, you ask yourself, "Okay, but does my argument prove that flesh is in fact as strong as diamond?  Why isn't flesh as strong as diamond, then, if I've refuted the original argument for why it isn't?" and this is the question that leads you to realize that some local strong covalent bonds don't matter to the argument if those bonds aren't the parts that break under load.

My main moral qualm about using the Argument From Folded Spaghetti Held Together By Static Cling as an intuition pump is that the local ionic bonds in bone are legitimately as strong per-bond as the C-C bonds in diamond, and the reason that bone is weaker than diamond is (iiuc) actually more about irregularity, fault lines, and resistance to local deformation than about kJ/mol of the underlying bonds.  If somebody says "Okay, fine, you've validly explained why flesh is weaker than diamond, but why is bone weaker than diamond?" I have to reply "Valid, iiuc that's legit more about irregularity and fault lines and interlaced weaker superstructure and local deformation resistance of the bonds, rather than the raw potential energy deltas of the load-bearing welds."

I broadly endorse this reply and have mostly shifted to trying to talk about "covalently bonded" bacteria, since using the term "diamondoid" (tightly covalently bonded CHON) causes people to panic about the lack of currently known mechanosynthesis pathways for tetrahedral carbon lattices.

In this case, the positions from the last bullseyes become reversed. The doomer will argue that that AI might start off incapable, but will quickly evolve into a capable super-AI, following path A. Whereas I will retort that it might get more powerful, but that doesn’t guarantee it will ever actually end up being world domination worthy. 

No, the doomer says, "If that AI doesn't destroy the world, people will build a more capable one."  Current AIs haven't destroyed the world.  So people are trying to build more capable ones.

There is some weird thing here about people trying to predict trajectories, not endpoints; they get as far as describing, in their story, an AI that doesn't end the world as we know it, and then they stop, satisfied that they've refuted the doomer story.  But if the world as we know it continues, somebody builds a more powerful AI.

The first object-level issue the author talks about is whether the brain is close to the Landaeur limit.  No particular issue is cited, only that somebody else claimed a lot of authority and claimed I was wrong about something, what exactly is not shown.

The brain obviously cannot be operating near the Landaeur limit.  Thousands of neurotransmitter molecules and thousands of ions need to be pumped back to their original places after each synaptic flash.  Each of these is a thermodynamically irreversible operation and it staggers the imagination that every ion pumped en masse back out of some long axon or dendrite, after ions flooded en masse into it to propagate electrical depolarization, is part of a well-designed informational algorithm that could not be simplified.  Any calculation saying that biology is operating close to the Landaeur limit has reached a face-value absurdity.

Of course, this may not seem to address anything, since OP failed to state what I was putatively wrong about and admits to not understanding it themselves; I can't refute what isn't shown.

The first substantive criticism OP claims to understand theirself is on Zombies.

I say:

Your "zombie", in the philosophical usage of the term, is putatively a being that is exactly like you in every respect—identical behavior, identical speech, identical brain; every atom and quark in exactly the same position, moving according to the same causal laws of motion—except that your zombie is not conscious.

The author would have you believe this is a ludicrous straw position.

I invite anyone to simply read the opening paragraphs of the SEP encyclopedia entry on P-zombies:

If zombies are to be counterexamples to physicalism, it is not enough for them to be behaviorally and functionally like normal human beings: plenty of physicalists accept that merely behavioral or functional duplicates of ourselves might lack qualia. Zombies must be like normal human beings in all physical respects, and they must have the physical properties that physicalists suppose we have. This requires them to be subject to the causal closure of the physical, which is why their supposed lack of consciousness is a challenge to physicalism. If instead they were to be conceived of as creatures whose behavior could not be explained physically, physicalists would have no reason to bother with the idea: there is plenty of evidence that, as epiphenomenalists hold, our movements actually are explicable in physical terms (see e.g. Papineau 2002).

This is a debate that has gone on for very long in philosophy.  I'd say it's gone on too long.

But whether or not particular thought experiments, by seeming metaphysically possible, license other conclusions about metaphysics, is exactly the entire substance.  The base thought experiment is not or should not be in dispute: it's a being whose physics duplicate the physics of a human being including the causal closure of what is said to be 'physics', i.e., all of the causes of behavior are included into the p-zombie.  Some people go on at fantastic length from this to say that it demonstrates the possibility of an extra consciousness that they call "epiphenomenal", and some say that it demonstrates the possibility of a nonphysical consciousness that they don't call "epiphenomenal", but it's my position that somewhere along the way of a long argument they have dropped the ball on the original thought experiment; whatever they call "consciousness" that isn't in the supposed p-zombie, it can't be among the causes of why we talk about consciousness, or why our verbally reportable stream of thought talks about consciousness, etc, because the zombie behaves outwardly like we do and also includes the minimal closure of the causes of that physical behavior.

The author of the above post has misrepresented what my zombies argument was about.  It's not that I think philosophers openly claim that p-zombies demonstrate epiphenomenalism; it's that I think philosophers are confused about what this thought experiment demonstrates.

The author having been shown to be wrong on the first points addressed, which I chose in order rather than selectively sampling, I hope you accept this as obvious evidence that the rest would be no better if you looked into them in detail or I responded in detail.  For a post claiming to show that I'm often grossly wrong, actual quotes from me, with linked context and dates attached, are remarkably thin on the ground.

You will mark that in this comment I first respond to a substantive point and show it to be mistaken before I make any general criticism of the author; which can then be supported by that previously shown, initial, first-thing, object-level point.  You will find every post of the Less Wrong sequences written the same way.

As the entire post violates basic rules of epistemic conduct by opening with a series of not-yet-supported personal attacks, I will not be responding to the rest in detail.  I'm sad about how anything containing such an egregious violation of basic epistemic conduct got this upvoted, and wonder about sockpuppet accounts or alternatively a downfall of EA.  The relevant principle of epistemic good conduct seems to me straightforward: if you've got to make personal attacks (and sometimes you do), make them after presenting your object-level points that support those personal attacks.  This shouldn't be a difficult rule to follow, or follow much better than this; and violating it this hugely and explicitly is sufficiently bad news that people should've been wary about this post and hesitated to upvote it for that reason alone.