Pan narrans

A few years ago I put up a post on the conjecture that the human ability to throw things far and accurately is one of our distinguishing characteristics, and that developing this ability was likely a key step in our divergence from other lines of the chimp/bonobo family (not my own conjecture, see the post for more details). I think the post holds up fairly well, for whatever that's worth.

Towards the end I mentioned the idea that the advance planning required to throw a projectile at a moving target may have been key to our developing other unique capabilities like speech and "large-scale planning" (I'll call it "complex" planning here since that seems a bit better to me now). My take at the time was that there had to be more to it than that, and that still seems right, but regardless of how they arose, the capacity to make complex plans and describe them in language appears to be another cognitive ability that we have and everyone else seems not to.

Complex planning and the ability to describe it converge, along with a couple of other significant abilities, in one of the most human of all human behaviors: storytelling. I say "converge" because storytelling is not a single ability, much less the basis of other abilities. Rather, it pulls together a number of key abilities that make us who we are.

What makes for a good story? Two key features are

• Multiple things happen, in a given order (for you math majors out there, a partial order, since things can happen at the same time, even if they're generally narrated in some sequential order) ...
• ... but not only that, things happen for a reason. A real story involves cause and effect: "This happened, and because of it, that happened"^*

Cause and effect are the basis of planning: "Do this so that you can do that." To do make complex plans, you need to put together several pairs of causes and effects, with some effects being causes for other effects. This is the basis of plot, one of the key elements of storytelling.

I had originally written "string together a sequence" but then replaced it with "put together several pairs" because cause and effect do not necessarily run in straight lines. An effect can have more than one cause. A cause can have more than one effect. Causes and effect can run in both directions, leading to feedback loops -- to vicious (or virtuous) circles or to longer-term stability. Our brains can conceive of these and describe them. To make up some examples:

• Their success was a combination of skill, preparation and sheer blind luck
• The construction of the new road had far-reaching consequences, with some communities faring better than others
• The new vegetation reduced erosion, retaining more soil, which in turn supported more vegetation
• The shortage led to higher prices, which encouraged more production, eventually eliminating the shortage. Prices fell, leading to more demand, eventually leading to new shortages ...

While I wouldn't call any of those a full-fledged story, it's easy to see how any of them could be made into one.

In the real world, things don't always work out as planned. One way to deal with is to make plans conditional:

• If the weather's good, we'll go through the mountains. If it turns cold, it will be quicker and safer to go around.
• We'll have to wait here until the flood subsides and we can cross the river

Interesting stories involve choices, and many interesting stories revisit the same situation multiple times (the magic number is three, of course).

Both of the sentences above involve cause and effect -- It turned cold, so we went around; The flood subsided, so we could finally cross -- but also something more. They embody the two primitives that, one way or another, underly all models of computing:

• Conditional evaluation: How the result its computed (not just the result itself) depends on the state of the computation. For example, if n is even, the result is n/2, otherwise, it is 3n + 1
• Iteration: The same computation is repeated as the state of the computation evolves, for example: Repeat this computation, at each step using the result as the new n

If n is, say, 8 to begin with, then n becomes 4, then 2, then 1, then 4 (3·1 + 1), then 2, then 1 and so on forever. Other cases are less well-understood.

Given conditional evaluation and iteration, you can compute anything that can be computed (there's a lot more, of course, to what that actually means and implies). In a real sense, modern computers are simply automating something our brains do naturally. Other kinds of brains can handle conditionality and can do the same thing repeatedly, but we appear to be unique in how far we can extend this. To use a fairly shaky metaphor, we have the same kind of CPU but a lot more RAM.

Metaphor, that is, understanding one thing in terms of another, is another building block of storytelling, because it is key to how we understand the world in general. Metaphor is so pervasive that most of the time we don't even notice it until someone uses it deliberately. I've gone into that in several other posts, so I won't go into it here.

Finally, good stories need good characters. That is, they need to show a good understanding of what the people in the story are thinking, what their desires and motivations are, and often what they think about the other characters, including what they think those other characters are thinking. To tell a good story, you need a theory of mind and, for that matter, the ability to convey theories of mind that might be different from yours (the narrator sees one of the characters as a canny operator who's thinking two steps ahead of everyone else, while another of the characters thinks they're a simpleton).

Many interesting and exciting stories hinge on who knows what about whom, and who knows who knows what about whom. Irony, another key element of storytelling, occurs when one person -- in the story or not -- has a different understanding of what's going on from another person. For example, in dramatic irony, the reader/audience knows something that a character doesn't (usually not good news for the character), or the reader only thinks they know what's going on (often good news for the character and, in a different way, for the reader).

This ability to build detailed maps of who wants what and who knows what, including the "meta" level of who knows what about what others know and want, is obviously important for a highly social primate whose survival depends critically on getting along with the rest of the group, which in turn requires navigating its social structures.

Again, other brains are able to do this to at least some extent, but ours seem able to do this much more extensively, particularly at the meta level. More than that, we are able to communicate our understanding of other people: Look out for that person. They'll act like they're your friend, but they're really just trying to take advantage of you. But what they don't know is ...

Stories involve characters, that is, people (usually). Even though they may be fictional, we can identify with them and we care what happens to them. We want the protagonist to attain their goal, so strongly it can lead us to forget whether the goal or the person chasing it are any good. In other words, stories carry an emotional charge, so a good story is fundamentally more memorable than a recitation of facts.

It's striking how many of our characteristic cognitive abilities converge in storytelling. To recap:

• Storytelling is built on an understanding of cause and effect
• This extends to large webs (technically, directed graphs) of cause and effect, which can branch, join and form loops
• Stories involve choices and repetition, the most fundamental elements of computing as we know it
• Stories are built on metaphor, which is key to our ability to learn
• Telling a story with characters requires a detailed understanding of what people want and know, including the meta level of what people know about what other people want and know (or, for that matter, what they want people to know, and so on)
• Irony is built on understanding of that meta level
• Understanding characters requires social understanding
• The ability to communicate complex cognitive structures itself an important cognitive ability
• We care what happens in stories, which makes them memorable

In other words, storytelling is a manifestation of the constellation of cognitive abilities that are a large part of our uniqueness. Moreover, the ability to communicate these things, in detail, to others is itself a key cognitive ability.

I think there's one more thing. Not only are the elements that storytelling is built on key elements of our cognitive abilities, to a large extent we understand the world in terms of stories. That is, not only do we understand the world through cause and effect, choices, repetition, metaphor and theories of mind as capabilities on their own, but to a large extent we understand the world in terms of stories that pull together those elements in a particular way: a protagonist has a goal, encounters obstacles, finds ways to overcome them (or occasionally gets lucky) and (usually) ultimately attains the goal.

Persuasive advertising tells a story (My shirts weren't coming out of the wash looking bright and clean. I started using this detergent (meaning, I bought it). Now my shirts look bright and clean). So does outright propaganda. But even something seemingly factual like an essay on human cognition uses storytelling: I had originally written "string together a sequence" but then replaced it with "put together several pairs" (because the original version didn't fit with my goal of explaining webs of cause and effect).

Given that the ability to communicate complex cognitive structures is important in and of itself, it's not surprising that our minds would favor stories as a means not only of communication, but understanding. 

However, this has its downsides. One common failure mode of our brains is to tell a story where there isn't really one to tell. For example, a financial news item very rarely just says a price went up or down. It almost always gives a reason. Often it will tell of a struggle between bullish factors driving the price up and bearish ones driving it down.

In many cases it's not hard to accept this as a shorthand for something more like "these factors could drive the price up, these could drive it down, both are in operation and we don't have enough information to know in advance what will happen". Often, though, in financial news and elsewhere, the story takes over and continues in episode after episode. Once the narrative is accepted, it's more cognitive work to reconsider whether the narrative actually applies. The consequences of leaving the assumptions unquestioned can range from trivial to tragic.

For better or worse, storytelling isn't just a reflection of individual capabilities like sequencing or theory of mind. It reflects those capabilities being used together in well-established, identifiable patterns. As such, storytelling itself is a fundamental part of our minds, distinct from the many capabilities that enable it.

^* Even the famous six-word story "For sale: baby shoes, never worn" is full of cause and effect.

Apparently, we've just invented AI

The other evening I overheard a conversation about AI and its implications, now that we have it. This morning I read a post saying that it was hard to believe that ChatGPT was released only four years ago and now AI is everywhere, and how that might be the quickest adoption of a new technology in history.

I'm sure you can find plenty of examples the story yourself: Around four years ago, we didn't have AI. Then ChatGPT happened, and we do. What is AI? It's what LLM-based systems like ChatGPT, Claude and Gemini do, especially now that these systems are agentic, that is, they can break down tasks and coordinate a collection of agents and subagents in order to complete them.

I spent some time looking for a word (and asking an AI to find one) to convey the feeling that something is completely predictable and yet still somewhat surprising and came up with droll, ironic, paradoxical and a few others, but none of them quite fit.

A quick look at the AI label on this blog (and the other one) will tell you I've been watching this space for a while. The second Intermittent Conjecture post, back in 2010, mentioned AI in the past tense. The thesis was that there were plenty of applications around that would have been considered to unquestionably be AI in the early days of AI research, but they weren't in 2010 because they just didn't have that "I'll know it when I see it" quality that would cause the general public to say: Yes, that's it, that's Artificial Intelligence.

I gave the examples of "a neural network mining some pile of data, or even a chess program [i.e., that could regularly beat the best human players], or voice-enabled phone," all of which were around by 2010. Facial recognition and machine translation of natural language were also widely available at the time, although current versions work better, as one would hope.

The mechanisms behind the current generation of AI have their roots in research into connectionism in the late 1980s, which is generally considered the second wave of research into computing based on neural networks, the first having run from the 1940s through the 1960s. The general idea of machines mimicking human behavior is ancient, but in the context of modern computers it goes back to Turing's 1950 paper Computing Machinery and Intelligence, which introduced the idea of the "imitation game," which we now call the "Turing test," for better or worse. Systems that people took for human and communicated with accordingly first appeared in the late 1960s and early 1970s and more have appeared in the years since.

So it's natural to have a certain sense of "Wait, now everybody thinks we have AI?" but on the other hand it doesn't seem surprising at all, because something significant did happen when ChatGPT came out. LLM-based systems are simply much better at conversing in natural language than anything that came before.

One reason is fluency. An LLM-based system can answer question after question in fluent natural language. It can spit out page after page of prose that, at least at first glance, looks for all the world like a person wrote it, not just in the syntax and vocabulary but in tone and structure. It's not hard to write a hand-coded Markov chain that will produce at least mostly grammatical sentences, but it won't really sound human and it won't be able to answer questions. LLMs are the first approach that allows an extended two-way conversation without "the man behind the curtain" appearing in short order.

The second reason is the range of topics you can converse about. The first 'L' in LLM is for large. LLMs are trained on trillions, or even hundreds of trillions of tokens (words, more or less). By comparison, the entire US Library of Congress collection comprises a few trillion words. Without digging too deeply into what the numbers actually mean, it's not unreasonable to compare the sheer volume of training data to "all books ever published." Whatever topic you might want to ask about, there's a good chance that something related to it is in the training data, in sufficient quantity that the model can produce a few coherent paragraphs about it.

Before ChatGPT there was nothing widely available -- and hardly anything available at all -- that could converse with an average person in human language and appear knowledgeable on a wide range of topics. Artificial systems were able, and had long been able, to do a variety of tasks that can reasonably said to require some form of intelligence, but until ChatGPT everything was in the category of "Yeah, that's interesting, and even useful, but it's clearly not really intelligence." The particular task of conversing with a human on an open-ended variety of topics was the first AI task that really fit the "I'll know it when I see it" bill.

All this is fine with me. In everyday usage, LLMs are AI and nothing before them was. Common usage makes sense in terms of people's expectations of what artificial intelligence ought to mean. Everyday usage doesn't have to match up with technical definitions. 

Still, I'm uneasy with a couple of aspects of all this. First is the idea that AI is one single thing, whether LLMs or something else. Just as intelligence comprises a number of abilities, artificial intelligence is many different things, each capturing some portion of what we call intelligence in living things.

Tying into this is the idea that artificial intelligence necessarily means artificial general intelligence (AGI) or superhuman intelligence, or even the creation of a new, sentient species. Yes, ChatGPT crossed a threshold into something that people could generally agree could be called AI, but only in one particular sense. It did not cross a threshold into everything that AI could possibly mean.

Fortunately, after spending time actually conversing with the things and observing their behavior, people seem to have generally concluded that, even though LLMs are AI, they're not all-powerful or superhuman, except in the literal sense that they can do some things that humans can't -- as can many things.

How do we anthropomorphize software, and why?

To anthropomorphize something is to attribute human qualities to it, whether it actually has those qualities or not. Compugeeks do this a lot, maybe not everyone but many of us. It's quite common to hear someone talk about what a particular piece of code thinks, or to say that a particular server is unhappy, or that a system is confused, and so on.

This was going on long before there were LLMs and I'm not even sure people do it more with LLMs than with ordinary code, though it has a distinctly different flavor with LLMs ("No, stop trying to patch the kernel so my parser's unit tests will pass. Why do you want to do that? What are you even thinking??")

We do this even though we know perfectly well that fifty lines of Python aren't thinking anything at all, and that an LLM-driven coding agent isn't actually trying to be annoying. Anthropomorphizing isn't treating something non-human as an actual human. We attribute some human qualities but not all.

To take an example, sorting a list of things into a given order, say, alphabetically, is a very useful operation because it enables fast algorithms like binary search. It's much faster to find a particular item in a sorted list than in one in a random order. If you give such an algorithm unsorted input, it won't behave correctly. If you track down a particular case of this happening, you might say "It expected the input to be sorted" and even "That's not its fault. We need to fix whatever gave it the unsorted input".

In a case like this, the code is standing in for the author. The author used an algorithm that required sorted input, with the expectation that anyone calling it would actually provide sorted input. As long as that's clearly noted, the author did their job and it's not their fault if the code fails on unsorted input. In other cases, we interpret behavior in terms of human behavior.

To take another example, if a server that normally runs without incident suddenly starts reporting a lot of errors, you might say "That server is angry," because it is very visibly reporting that something is going wrong. If a server tends to get bogged down due to some internal issue and start rejecting requests, you might say "It's in a mood", as in something like "Our service had to reject requests because the server it calls for FooService was in a mood". Or you could say it's "feeling unwell", or any of a number of other things. The common thread is that the server is not behaving as it usually would, and the reasons aren't clearly understood.

As far as I know there's not really a geek-standard way of saying such things, and what people say may well depend on the details of how a service tends to act. In other words, servers can have personalities.

What sort of human traits do we tend to attribute to software? Some of the more common ones are

• Knowledge: "It doesn't know whether this list could be shared, so it has to make a defensive copy"
• Goals: "It's trying to call FooService, but FooService is down, so it just sits there in a loop"
• Emotional and physical states: "FooServer is angry/unwell" "This code will get confused and panic if you give it unsorted input"
• Communication: "FooServer and BarServer talk to each other" "The executor asks the coordinator for the next task to execute"

Are there any human traits that can't be attributed to software? Probably not. Metaphors are fundamental to human thought and people can be very creative in applying them. Nonetheless, some examples come across as deliberately fanciful

• "FooServer and BarServer are scheming together to make sure this query fails"
• "This system lives in a happy world of unicorns and rainbows"
• "FooServer is lonely because no one wants to talk to it"
• "This code has decided that existence has no meaning and all results are equally valid"

The more I go over this, the more I think there's nothing special here about computing. We anthropomorphize all sorts of things: Other animals ("That cat is happy"), vehicles ("It's not a pretty car, but it's been a loyal friend through the years"), the weather ("The sun is trying to poke its way through the clouds") or really anything that can act on its own, or even appear to. Software fits very comfortably into that category.

As with metaphors in general, anthropomorphizing can be nearly invisible, as with the executor asking for tasks, or deliberately vivid, as with deciding existence has no meaning, or somewhere in between, as with not knowing whether a list could be shared. When a metaphor is conspicuous, we recognize it as such. When it's not, we just use it.

Postscript: In re-reading, it struck me that the examples I gave, like "parser's unit tests" and algorithms that expect sorted input, are more than a little dated. While it's not impossible that a working software engineer would end up writing a parser or a function that requires sorted input, if you're designing a web service you're much probably more concerned about things like UX on the front end and scaling and security on the back end. I stuck with the examples anyway for familiarity. Topics like parsing and binary search have been taught continuously for generations, so they're universally familiar, unlike whatever I was working on last week.

What is the pound measuring?

How much does the Orion capsule (that is, the Crew Module) that splashed down on April 11 weigh? According to NASA's reference guide for Orion, 22,900 pounds.

The guide specifically lists "liftoff weight", and there are a couple of reasons for that. One is that the capsule has reaction control thrusters, which are small rocket engines that allow for fine-tuning the attitude of the craft and small-scale maneuvering, and their propellant is part of that liftoff weight. For this and other reasons, the capsule did not have the exact same contents when it splashed down as when it took off.

The other reason, of course, is that the weight of the capsule depends on where the capsule is in its trajectory. For most of the mission, that weight was essentially zero, since the capsule was coasting in freefall except at a few key points. Units of weight, like pounds, measure force, not mass. At least that's what I was taught in high school physics.

For most practical purposes, though, the pound is a unit of mass. If the door of a bank vault weighs a ton (2,000) pounds, you know it will be a little hard to move, even if it's perfectly mounted on bearings with very low friction so that when you push on it you're not trying to lift its mass. That inertia is due to its mass. If you weigh out a quantity of something, you're interested in how much of it you're getting, that is, the total mass. The force that it exerts on the scale is just a way to deterimine the mass.

You're measuring that mass by way of how much that mass weighs on Earth, but it's still mass that you're measuring. Except in specialized applications like calculating load limits or foot-pounds of torque, the amount of force something exerts under gravity is secondary to how much of it you have.

Yes, it matters that a 22lb bag of something is easier to lift than a 44lb bag, but it matters just the same that a 10kg bag is easier to lift than a 20kg bag. You don't need to know the amount of force involved (about 98 and 196 Newtons, respectively) to make that determination and no one is thinking "Hmm ... that 20kg bag will require 196 Newtons to lift" before trying to pick it up.

There are units, the pound-mass and pound-force, that make the distinction between mass and weight. The pound-mass is now defined as exactly 0.45359237 kg, and the pound-force is the weight of this mass under standard Earth gravity of 9.8m/s².

No one uses this. Well, maybe not absolutely no one, but you won't find anything on a supermarket shelf that says it weighs, say, 1.5 lbm, because no one at the supermarket cares. If you're doing precise engineering or scientific work where the distinction matters, you're not using pounds, but kilograms and Newtons. This is just an example of the distinction I previously discussed between everyday units of measure, which can be pretty much anything, and precisely-defined scientific units of measure.

There are several reasons that SI (metric) units work better than imperial units for scientific work (and why, for example, the telemetry feed that NASA put up during Artemis II showed both SI and imperial units, with SI units first as I recall). One is the consistent use of powers of ten and standard prefixes like mega- and milli-. Another is that SI units have been standard for generations, so anything you're referencing in a scientific context is almost certainly using them. Another is the body of very careful definitions of what each unit means.

A less obvious reason is that SI units carefully make distinctions that we gloss over in everyday use, particularly the mass-weight distinction. During re-entry, when a capsule may be pulling on the order of 5g, it matters quite a bit that the forces on the body of the capsule are much higher than when the capsule is on the launch pad. You want to be talking about Newtons of force and not kilograms of mass when you do those calculations. Using pound interchangeably for pound-mass and pound-force in everyday speech makes good sense when you're buying groceries. Trying to use mass and force interchangeably in mechanical engineering is a recipe for disaster.

To make the distinction completely clear, the Newton is defined as a kilogram-meter per second squared, with no reference to Earth's gravity. A pound-mass weighs a pound-force under standard gravity because we don't really care about the distinction when using pounds. A kilogram weighs about 9.8 Newtons, which helps keep the distinction clear when it matters.

NASA is happy to quote the weight of Orion in pounds and show its speed in miles per hour because the US audience is used to those units. Trying to point out that actually the mass is about 10.4 tonnes and the weight varies is just going to get in the way unless you're specifically talking about the effects of acceleration or microgravity. Using pounds interchangeably for mass and weight is only incorrect if you're doing engineering or science, but then you shouldn't be using pounds at all.

Back to the space age

39,693 km/hYesterday, Artemis II launched four people on a flyby of the Moon, the first such crewed mission in 56 years. I have dim memories of the Apollo program, not so much the missions themselves -- I don't remember whether I heard Neil Armstrong's "One small step" live or later, for example -- but I do remember details like drinking Tang, because astronauts drank it (and still do), and a print on the back of a cereal box (I think?) that you could cut out and fold up into a model of the lunar lander.

The original Apollo program was a truly remarkable engineering feat, particularly considering how much progress there has been since then in fields like materials science and, of course computing. Today, we build massively powerful datacenters (at least, they seem massive now). At the start of the Apollo program in 1961, computers were much, much smaller and the field was so new that the word software had only been coined three years before.

It would be tempting to say that Artemis is just a retread of 50-year-old technology. In the years since the Apollo missions, space flight has become routine. There were 330 orbital launches in 2025, 317 of them successful. The ISS has been in continuous operation for 25 years. A dozen countries have launched satellites into orbit. Spacecraft have gone to all eight planets, Pluto, the Kuiper Belt object Arrokoth and to within about 6 million kilometers of the Sun (harder than it might sound). There were even two lunar landings last year, not to mention ongoing missions on Mars.

Except ...

The vast bulk of space activity has been launches to Low Earth Orbit (LEO for short). An orbital launch is not nothing. It means accelerating whatever you're launching to about 8 kilometers per second (17,500 mph) and handling all the details of tracking exactly where the launch vehicle is at all times, deploying the actual satellite and plenty of stuff I'm leaving out because I don't know any better. Nonetheless, as far as space travel is concerned, LEO is easy mode.

Everything else in the last 50+ years has been uncrewed. No human has been past LEO since Apollo 17 splashed down in 1972.

There are several reasons for this, not all of them technical, but the technical obstacles are considerable. For one thing, crewed missions are much heavier. Besides the mass of the people themselves, you need a life support system, food and water, equipment for a cabin and so on. More mass means a bigger rocket. 

Missions beyond LEO need a significantly higher delta-v budget, which is the total of all speed changes for the maneuvers the mission needs to do. LEO needs about 8 km/s of delta-v. Artemis II will use around 13km/s, about 1.6 times as much. Since it takes more fuel to lift more fuel, that means significantly more than 1.6 times as much fuel. The Space Launch System (SLS) that launched Artemis II was the most powerful rocket that NASA has every launched. The Saturn V rockets used in Apollo are not too far behind.

The stakes are also higher. 13 of the 330 launches in 2025, or about 4%, failed. If you leave out LEO launches (crewed or uncrewed), that number is much higher. There were two successful lunar landings last year, but also at least three failures. Of the two successes, one landed on its side. This sort of thing is OK if it's just expensive equipment getting destroyed, so you can afford to take more risks. For a crewed mission, nothing major can go wrong, and even minor problems like toilet malfunctions require serious attention.

Even with the need to reduce risk, Artemis II is still pushing the envelope a bit, and not only in the power of the SLS. When Artemis II splashes down (assuming everything goes well up to then), it will be traveling at around 11,000 m/s, breaking the previous record held by Apollo 10  [As it turned out, Apollo 10 still holds the record at 11,082 m/s (24,791 mph) vs. 11,026 m/s (24,664 mph) for Artemis II  -- D.H 6 Jul 2026]. It will also go a bit further from Earth than the Apollo missions, so the Artemis crew will be further from Earth than anyone has ever been before [That one panned out. Artemis II reached 406,773 km (252,757 miles) from Earth, vs. 400,171 km (248,655 miles) for Apollo 13].

From a strictly economic perspective, crewed missions make very little sense. The real reason to send people around to the Moon is that we want to send people to the Moon, either for its own sake, or so that we can establish a presence there and eventually send people to Mars and beyond. Whether that's a worthwhile goal is a matter for debate and I'm not going to take a position on it here.

My point here is that a crewed mission to the Moon, or anywhere beyond LEO, wasn't just a major engineering feat 50 years ago. It's still a major engineering feat now. Practically all of the progress in the past 50 years ago has been aimed at solving different problems: Getting equipment and people to LEO, and getting equipment beyond LEO. Crewed missions like the ISS have told us a lot about what happens to people in space, and the Artemis mission reflects that, but not much about how to get them there that we didn't already know.

Suborbital crewed missions like Virgin Galactic's and Blue Origin's are pretty much irrelevant to all this.

As I write this, Artemis has successfully executed all but the last major maneuver in its mission. Before long, if all goes well, it will do the trans-lunar injection burn that will put it on a path to swing by the Moon and return directly to Earth. At that point, the crew needs to survive a bit more than a week in space and splash down safely. They also have a long list of mission goals to accomplish, of course.

I'm a bit surprised by my feelings about this. I've studied enough about the Apollo missions and spaceflight in general to know how much can go wrong, even with the most careful planning. NASA itself has lost crew members on multiple occasions. So while I'm excited for the crew and the many people on the ground, I'm also more nervous about it than I expected to be.

Beyond that, though, is a strange feeling of being in two timelines at once: a young kid in the late 1960s curious about all the moonshot stuff going on, and an adult watching nearly the same things happen 50 years later, almost as though for the first time.

Why don't we still use stone-age tools (or do we)?

While binge-watching a drama set a few hundred years ago in one of the world's empires, I realized that there was something intriguing about the mix of technology in use, which I think was a reasonably accurate rendition of what actually was in use at that time and place. This wasn't a meticulously-researched historical drama, so I'd expect some anachronisms, but the showrunners were constrained by using actual locations and a well-known and well-documented society, so I wouldn't expect to see the equivalent of a Roman senator on an e-bike.

On the one hand, there were ships at sail, intricately woven fabrics, expansive, multistory buildings with carefully crafted architectural details, and a highly-developed administrative state which, among other things, required people across a wide geographical range to carry personal ID. There was an extensive system of roads and detailed maps to show you how to get around them. There were cannon, rifles telescopes, steel farm implements and deadly-sharp swords.

On the other hand, buildings, including heavy-duty structures like jails and military fortifications, were built mainly of wood -- very well built, to be sure, but of wood just the same. Most people lived in small wood-and-stone huts. Some lived in very small dwellings made of sticks. Ladders were made from small logs bound together with rope. The roads were generally traveled on foot or horseback, or in wooden horse-drawn carts. Fighting was mostly hand-to-hand and the main projectile weapon was the bow and arrow.

In other words, while there was quite a bit of technology that would have been state-of-the-art at the time, many if not most of the objects most people dealt with day-to-day could have been made a thousand years before the time of the story, or even five thousand. Consider, for example, a clay cooking pot. While an archeologist could probably tell from the details roughly when and where a particular pot was made, from a purely technical point of view "clay cooking pot" narrows the time and place down to ... "probably not Antarctica" and "probably the last 20,000 years".

Impressive as the more modern-looking items were, few of them seemed essential. An elaborately-embroidered ceremonial robe is not purely decorative. It serves as a signal that the wearer has the resources, including human labor, at their disposal to make such things. Nonetheless, it's far from the only way to keep the wearer warm, and it's actively in the way of the wearer moving around easily (again a feature, not a bug, in context).

Relatively few technological developments are groundbreaking. An aluminum extension ladder is a better general-purpose ladder than a log-and-rope ladder. It's easier to carry. It's probably more weather-resistant and durable though, not knowing any better, I wouldn't want to underestimate how well a well-made wooden ladder would hold up. Likewise, it can probably bear more weight. Because it's easy to change the length of an extension ladder, it can be used in a wider variety of places.

Nonetheless, an aluminum extension ladder is still a ladder, and ladders have probably been around about as long as cooking pots. It's hard to tell for sure since a non-metal ladder is less likely to survive the millennia than a ceramic pot, but there is at least one surviving depiction of a ladder from 10,000 years ago. On a clay pot, of course.

An aluminum extension ladder is also a lot harder to make. While aluminum compounds, particularly alums, have been known for millennia, actually extracting aluminum requires electricity, and quite a bit of it. Aluminum has only been available in commercial quantities for a little more than a century. While we may think of aluminum foil and beverage cans, not so long ago aluminum was rare and expensive. It's probably not true that Napoleon III had a special set of aluminum silverware, but the aluminium statue of Anteros in Piccadilly circus, made in 1893 was the first of its kind, and so kind of a big deal.

Now that we can make aluminum ladders cheaply, they're everywhere. I have a couple in my garage. If someone gave me a carefully crafted wooden ladder, I'd keep it, because cool, and maybe even use it if it happened to come in handy. Like most of us, though, I have absolutely no reason to go out of my way to get one, particularly since it would almost certainly be much more expensive.

As far as technological developments go, I think aluminum ladders are more the rule than the exception. A modern freeway is in most respects a better road than the Via Appia, though not necessarily in durability. A modern car can go much farther and faster than a horse-drawn cart. A coat with a zipper is easier to get in and out of than one with buttons. It's somewhat easier to pay with a card or phone than it is to count coins and notes.

In general, a new development has to offer at least some advantage in order to catch on. That advantage doesn't have to be groundbreaking or even particularly great, however, and it doesn't even have to make sense on a larger scale. Plastic bags have their uses, but whether a new and cheaper way to make plastic bags catches on has little to do with whether we need more plastic bags, as long as we want some plastic bags.

Coming back to the title, in some sense we do still use stone-age tools every day. Ladders are still a thing. You can still buy rope at a hardware store and put it to the same kinds of uses it's been put to for as long as there's been rope. But in most cases, you'll be buying a stone-age tool made with modern technology. The rope you buy at a hardware store was almost certainly made by machine and very likely from a material that didn't exist a hundred years ago, much less several thousand.

I do keep thinking of getting a clay tagine pot, though.

Frankenstein and the Gray Goo

[Ed. note: The original version of this post referred to Mary Wollstonecraft Shelley as Mary Wollstonecraft and said that she "won" the famous Geneva horror story contest. While Mary and Percy Shelley were not married when the contest took place in the summer of 1816, Mary was already going by the name Mary Shelley. The two would marry in December of that year. The novel Frankenstein would be published anonymously in 1818 and later, in 1821, under the name Mary W. Shelley. Before she took the name Shelley, Mary went by Mary Wollstonecraft Godwin, after her mother, Mary Wollstonecraft, and her father, William Godwin. As far as I know, she never went by Mary Wollstonecraft.

There doesn't seem to be a lot available about the contest itself -- sadly, no one seems to have posted about it to their social media at the time, leaving us to rely on whatever diaries and secondhand accounts were preserved -- so it's not really meaningful to talk about winners and losers. However, it is well established that Mary Shelley had the idea for the story in a dream during the party's stay in Geneva, wrote it up as a short story around that time and later expanded it into the novel we know today -- D.H. 6 Sep 2025]

In 1986, so about 40 years ago, K. Eric Drexler's Engines of Creation was published. It was a pretty big deal at the time. I'm not sure whether I read the actual book, but its themes were widely discussed and I do recall reading several articles examining it, and the concepts in it, in depth.

All of which is why it was something of a mild shock to realize I'd forgotten all about it.

Engines of Creation talks about technology that mimics a fundamental process in living things, technology so powerful that it seemed quite possible it would enter a runaway feedback loop, amplifying itself without limit, rapidly evolving beyond human control. Even if that could be prevented, the technology had such profound implications that it was bound to have a major impact on all aspects of human activity. It would make previously impossible things routine and change the lives of every person on the planet in ways we could only hope to anticipate. Best to understand it and get on board, or risk being left behind forever.

If it seems like I'm deliberately using the same kind of language that's now used to talking about AI while avoiding telling you what I'm actually talking about, well, busted.

As you may already know from the book title, I'm talking about nanotechnology. Today, the term refers to any technology that operates on a scale below the somewhat arbitrary limit of 100 nanometers, or 0.1 microns, or about a thousandth of the thickness of a human hair, or about the size of many viruses. This definition came along later and doesn't really capture what Drexler was talking about. The title of Drexler's later book, Nanosystems: molecular machinery, manufacturing, and computation sums it up pretty well, though.

There have been significant developments in nanotechnology since 1986, for example in developing antimicrobial materials and stain-resistant fabrics, but what caught the public attention at the time was the idea of a universal assembler, a hypothetical nanomachine that could put individual atoms together in any desired (physically possible) configuration. Somehow.

Since a universal assembler would itself be an arrangement of atoms, it should be possible for a universal assembler to create copies of itself, and we're off to the races. So as long as the atoms it needed were available, an assembler could rearrange them into more universal assemblers, which in turn could do the same. Exponential growth being what it is, this process would soon produce tons of replicators, and so on up to any quantity you like, assuming, as Drexler says, "the bottle of chemicals [this is happening in] hadn't run dry long before".

A couple of years after the book was published, some people at IBM used a scanning tunneling microscope to spell out the letters "IBM" in xenon atoms on a substrate of nickel. How hard could it be then to build up a universal assembler atom by atom?

Drexeler's book actually covered a number of topics, mostly but not exclusively to do with nanotechnology. As part of that, Drexler spent a couple of paragraphs discussing the idea of universal assemblers assembling more universal assemblers. Once you introduce the idea of a universal assembler, you kind of have to talk about that. He called the scenario gray goo, with the explanation that "Though masses of uncontrolled replicators need not be grey or gooey, the term 'grey goo' emphasizes that replicators able to obliterate life might be less inspiring than a single species of crabgrass."

In other words, we shouldn't assume that something dangerous would be big and spectacular. It might just as well be an amorphous grey goo made up of very tiny, but still dangerous, little machines.

As I understand it, Drexler wasn't claiming that the gray goo scenario was inevitable. Drexler himself later said that there wasn't any good reason to try to build a universal replicator, and later analyses by others suggested that the actual risk of runaway gray goo is quite small. Even so, it's not hard to see why the idea of gray goo might take off anyway.

Drexler's original scenario involved a "dry" replicator that needed a supply of simple chemicals to work with, but surely something that could assemble atoms at will could also disassemble more complex structures into raw material. This gives us the nightmare scenario of a blob of gray goo that could turn whatever was around it into more gray goo, leaving behind only whatever it didn't need to make more assemblers.

Since elements like carbon, hydrogen and oxygen can combine very flexibly into a wide variety of configurations, it's a good bet that a universal assembler would use them as raw material. Since those are also a the main materials that living things like human beings are made of, there's a certain potential for conflict here. Yes, we contain other elements that might not be useful to the goo, but having a small residue of calcium, phosphorous and such make it through unscathed seems like cold comfort in the larger picture.

Unlike, say, time travel or perpetual motion, the gray goo scenario doesn't violate any known laws of physics. In fact, we know that it's possible for collections of atoms to make copies of themselves. That's life (yeah, sorry).

However, at least as far as life is concerned, we also know that the mechanisms to do this are very complex and hard to predict, much less control. It's an interesting situation, really. There's nothing going on in cell metabolism and cell division that doesn't boil down to well-studied physics and chemical reactions. We know quite a bit about many individual reactions, the structure of cells, processes like DNA replication and RNA transcription and so on. In that sense, there's no mystery.

Nonetheless, molecular biology is full of mysteries that molecular biologists have been struggling for generations to get a handle on. For example, given a DNA sequence coding a protein, it's easy to read off exactly what amino acids that protein will consist of. But a protein isn't a simple sequence of amino acids. It's a three-dimensional structure that interacts with other chemicals in the cell, including but not limited to other proteins.

Exactly how a given sequence of amino acids will fold up into a three-dimensional structure (or one of several possible structures) and how it will interact with other chemicals is still a wide-open topic for research. There have been significant advances in recent years, but it's worth noting that the most successful approach to protein folding so far isn't simulating the physics of how the atoms in the protein will interact.

The current state of the art  for protein folding is AlphaFold, a machine-learning model that in some sense is basically going "Meh ... this matches up with this, this and that in the training set, and those folded up this way, so yeah ... it'll probably be something like this." Yes, I'm being very glib here with something that won Nobel prizes (deservedly, I'd say, for whatever my opinion is worth here), but the point is that the best approach so far is to give up on understanding what's going on physically and do very sophisticated pattern matching.

All of this is to say that we only know of one workable way for collections of atoms to produce copies of themselves, and there is an incredible amount we don't know about how that actually works. Even though life is everywhere in our environment, including places that until recently hardly anyone had the imagination to think it might be, almost all of the Earth is non-living matter -- rocks, magma, ocean water, air and such. In other words, after a few billion years of collections of atoms copying themselves, we do not have a gray goo scenario.

The nightmare gray goo scenario depends on a number of assumptions:

• That universal assemblers are even possible. A true universal assembler would be able to arrange atoms into any physically possible arrangement. Living systems can produce more living systems, and they can produce all manner of interesting chemicals and profoundly transform the world around them, but a universal assembler would be able to produce any chemical structure. Living things can assemble collections of particular molecules, such as nucleotides, amino acids, carbohydrates and lipids, but that's pretty much it. There's no microbe that could put a xenon atom in a particular place on a nickel substrate. In other words, the existence of life doesn't demonstrate that universal assemblers are possible, and life is the only thing we know of that can self-reproduce at scale.
• That it would be feasible to build one. The IBM demo, impressive though it was, used a human-scale machine to put a particular kind of atom on a carefully prepared substrate. Xenon is a noble gas, meaning that it's very unreactive. Unlike, say, oxygen, a xenon atom is not going to try to bind to the substrate or whatever else is around while you're maneuvering it into place. The IBM demo arranged 35 atoms on a flat surface. This is a far cry from arranging -- thousands? millions? -- of atoms in a complex three-dimensional structure that can move.
• That we would be able to build an autonomous programmable universal assembler. It would be one thing to have an assembler that could receive instructions from the outside world, on the order of "put this atom here" and then "put that atom there", but a true self-reproducing assembler would have to carry its instructions with it, just as the DNA in a living cell carries the instructions for reproducing the cell.

But even the terms I've been discussing this in are misleading. I've been using phrases like "arrangements of atoms", but what we're really talking about here is chemistry. If you browse the Wikipedia pages on nanotechnology, you'll see illustrations of things that look like tiny machines -- wheels, axles, levers, that sort of thing. But at the scale of individual atoms, our notions of how things move and interact break down entirely.

The six simple machines taught in school operate at a human scale. It's easy to imagine building a molecule shaped like the wheel of a pulley and a polymer to act like rope, but how exactly would you use it? You'd also need an axle for the pulley wheel, and something to hold that axle in place, and something to pull on the rope, and a way to attach something to the other end. All of this is happening at the scale of atoms, which means that everyone's electrons are potentially interacting with everyone else's, trying to find a lower-energy configuration ... my understanding of chemistry is pretty rudimentary, but none of the depictions of nanomachines I've seen look like chemistry as I understand it.

What convinced anyone (though not necessarily Drexler) that we were anywhere near being able to bootstrap a world of universal assemblers that might eventually consume not just all life on Earth, but potentially all matter that could be consumed? I'm not being rhetorical here. I mean, literally what are the thought processes that led to this idea taking off?

For one thing, feedback loops seem to be catnip for a certain kind of brain, my own included. I spent hours and hours as a kid reading and pondering Gödel, Escher, Bach, contemplating self-reference, strange loops, the use/mention distinction and so on.

One fun way to play around with these ideas is to write a Quine, a program which produces itself as its output. Every compugeek should do it from scratch at least once. On a somewhat larger scale, a key milestone in developing a new language is to write a compiler for the language in the language itself and then use the compiler to compile itself (after compiling it with an earlier compiler written in a language that already exists). In a related vein, a post on the other blog mentioned Doug Engelbart's NLS team using NLS to further develop NLS.

In other words, ideas like machines that can build anything, including themselves, or AI systems that can write any code, including code for better AI systems, come naturally to at least some people. I don't think you have to have any training in computing or mathematical logic to hit on ideas like this, but it helps (and on the other hand if you're not already a compugeek it could be a sign that you might enjoy learning more about computing).

The idea of a self-reinforcing feedback loop is compelling and cool, cool enough that it's very easy to get caught up in the implications and brush aside the hidden assumptions that inevitably pile up along the way.

I also think there's also another factor at play.

In 1791, Luigi Galvani published his findings about animals and electricity, including the discovery that applying an electric shock to the nerves of a dead frog would cause the legs to move. Alessandro Volta developed an electric battery a few years later, partly in order to demonstrate that electricity could be created by a chemical reaction, as opposed to it being a "vital force" created specifically by living things, but the idea of Galvanism, as Volta himself called it, continued to be widely discussed.

In 1816, while on holiday in Geneva, Mary Shelley, her soon-to-be-official-husband Percy Shelley, John Polidori and Lord Byron held a contest to see who could write the best horror story (as one does). The details of the contest have faded with time, but we know that it inspired Shelley to write a short story that eventually became the novel Frankenstein, or The Modern Prometheus.

As the subtitle implies, one main theme of the story is humanity dealing with life-changing powers it little understands, in this case electricity. Just as Prometheus stole fire from the gods to bring to humanity and paid dearly for it, so Dr. Frankenstein uses electricity to bring power over life itself, and pays dearly.

Today it may seem silly to think that you could reanimate dead flesh just by shocking the bejeezus out of it, but was this really any more outlandish than thinking that if you just put the right atoms together in the right arrangement you could create something that could reproduce itself without limit? Yes, Volta had demonstrated that you could produce electricity from non-living matter, but if all animals produced electricity as well, surely there was something about electricity that was essential to animal life. If you didn't know any of the details of how electricity is involved in animal life, it would be easy to think that the mere presence of electric current is all there is to it. Animal life means electricity, so electricity must mean life.

When faced with something new that touches on fundamentals like life, matter or thought, it's sensible to consider the implications. When considering something so fundamental, it's natural to see at least the potential for world-shattering changes and even to feel some measure of awe.

Just as Drexel wasn't so much predicting the advent of gray goo as trying to understand the implications of nanotechnology and its potential for escaping our control, Mary Shelley wasn't predicting armies of reanimated corpses, but discussing the implications of our ability to produce electricity and apply it to animal tissue, and the potential for that ability to outrun our ability to control it. This being the Romantic period, she wasn't alone.

These are worthwhile questions to investigate. What if we learn the secrets of bringing the dead back to life? What if we can create tiny devices to arrange matter in any form we like, including the form of those devices themselves? What if we create machines that are more intelligent than us, and those machines figure out how to make more machines that are even more intelligent?

But as we discuss the implications of a new technology, it's important not to lose track of how things would actually happen. It's fine to brush the details aside in a discussion of what's possible in principle. How can you discuss the implications of a universal assembler without assuming that universal assemblers are possible, one way or another? But when the discussion turns back to what do we do now, in the world we actually live in, the assumptions previously brushed aside have to come back into the conversation.

Update: AB and NN chess engines

When I last looked in on computer chess, it hadn't been too long since AlphaZero had made waves by beating Stockfish after spending nine hours training by playing games against itself with no outside interference. As I understand it, the configuration that Stockfish was running wasn't its strongest, but this result was still impressive: A chess engine that looked at relatively few positions but used a neural network to evaluate them (an "NN" engine) beat an engine that looked at billions using a hand-tuned human-written algorithm (an "AB" engine). Soon an open-source engine based on Alpha Zero, Leela Chess Zero (LC0), was doing impressively well in tournaments.

The hallmark of NN engines was that they would play wild-looking moves that neither a human chess master nor an engine like Stockfish would have played at the time, moves which looked risky or even downright reckless, but often turned out to lead to a crushing advantage, all of this because similar-looking moves had led to good results in practice games.

At this writing, LC0 is still doing quite well in tournaments, but not quite as well as Stockfish, which consistently beats it. So AB wins, right?

Well, not quite. At the heart of an AB engine is the evaluation function, which takes a position on the board and returns a number that says how good the position is. The rest of the engine is dedicated to efficiently searching the tree of possible moves, replies to moves, replies to replies and so on typically a few dozen levels, to find the move that leads to the best possible positions against the opponent's best moves, according to the evaluation function.

There is a whole lot of software engineering behind making this as efficient as possible, including a technique called alpha-beta pruning that gave rise to the "AB" designation. The principle behind alpha-beta pruning is simple: Stop looking at the continuations from a move as soon as you know that the opponent can do better than it would with your current best move, but my brain gets completely befuddled when I try to understand the code, probably because the rule is applied recursively for both sides, so the meaning of "better" flips each time you switch sides in the search (alpha represents the score of the player's best move so far in the search; beta represents the same thing for the opponent). 

Until recently, evaluation functions had been carefully crafted to extract features from a position, like how much material each side had, which pieces had good or bad mobility, how each side's pawns were structured and so forth, and combine those using carefully-selected rules to arrive at a final evaluation.

A significant part of this is figuring out how much weight to assign to each feature in what circumstances. Essentially, this means answering questions like "Is it better to have an extra pawn, or better mobility and pawn structure?". The actual answer is "It depends. We need a rule for deciding how much weight to give each of the features we extracted." This in turn might vary depending on the particulars of the situation. Some things are more important in the middlegame, where there is more material on the board, than in the endgame, for example.

One of the reasons for Stockfish's success is its well-designed test framework for evaluating new code, including new evaluation functions. Different versions of the engine, including versions with different evaluation functions, are systematically played against each other and only changes that win make it into the next version.

Extracting features and carefully tuning various parameters that determine how to combine them certainly seems like what I previously called  an "ML-friendly problem", and it didn't take too long for someone to try that out. The result was the NNUE, a neural network that takes the positions of the pieces, with special attention given to the kings, and produces a numerical evaluation. The NNUE was good enough in testing to find its way into the official release, where it remains to this day.

So NN wins, right?

Well, not quite. A pure NN engine like LC0 is applying a large and relatively quite slow neural net to a comparatively tiny number of positions. It doesn't look ahead very far. In principle, an NN engine might look at only the positions after each possible move in the current position, typically a couple dozen. In practice, they look at hundreds of thousands, which is far more than a human player could, but still far fewer than an AB engine does. The power of an NN engine comes from the weightings in its neural net, which in turn come from playing large numbers of training games.

By comparison, the NNUE is tiny. Here's a picture of its weightings for one particular release, and here's a little more technical detail. The NNUE has around a hundred thousand one-bit input parameters and four layers. A parameter file runs to a few dozen megabytes, most of which are for the input weights in the first layer. 

Just as importantly "The efficiency of NNUE is due to incremental update of the input layer outputs in make and unmake move, where only a tiny fraction of its neurons need to be considered in case of [no] king moves." This is the result of hand-optimization, not some emergent property of the neural net.

LC0's network is much larger, though still tiny compared to the ChatGPTs of the world (which don't even really know the rules of chess, as this fairly sharply-worded piece argues). 

If that's all too vague for you (it is for me), the NNUE code runs on a standard CPU and can do hundreds of millions of evaluations per second, while LC0 prefers running its network on a GPU and does tens of thousands of evaluations per second.

By looking at orders of magnitude more positions than LC0, Stockfish is in effect trusting its neural network much, much less than an NN engine does and instead relies on very deep searches to determine which move to play.

Put another way, its actual evaluation is the aggregate of billions of simplistic evaluations, which happen to use a small neural net, rather than a few hundred thousand sophisticated evaluations using a much larger neural net. More simply yet, Stockfish is looking at many, many positions quickly while an NN engine is looking at many fewer positions more carefully.

The NNUE is essentially automating the process of extracting features from a position and deciding how to combine them. There's nothing particularly mysterious going on. As far as I understand it, its evaluations are similar to those produced by the older code, though different enough to lead to better outcomes when fed into the AB algorithm.

Even in the case of NN engines, the neural net isn't doing all the work. It's still running in a framework of "look at the possible moves, look at the replies to each move, and so on, with AB pruning". That framework wasn't created by a neural net. It was coded for computers by humans decades ago, in the 1950, to automate something human chess players already did.

That is, a naturally-evolved neural network, the human brain, developed both the concept of looking at moves and counter-moves and its realization as code. No LLM has developed code for a successful chess engine, or even come anywhere close*. This is, at least so far, a notable difference between LLMs and natural neural nets.

Within the framework that actual chess engines are built on, it turns out that a bit of neural network-based code can be helpful. Past a certain quite small size, though, adding more NN doesn't seem to help.

* To be a really fair test, the LLM would need to have been trained on a corpus that only mentioned, say, tree searches and the rules of chess, without mentioning anything like alpha-beta pruning or the idea of applying tree-searching to the problem of playing chess. I think it's a very good bet that current chatbots don't meet that standard, so if you've seen something like chess-engine code generated by an LLM, the simplest explanation is that there are similar things in the corpus it was trained on. This is to say nothing of actually producing a full chess engine that uses the tree search as its basis.

ML-friendly problems and unexpected consequences

This started out as "one more point before I go" in the previous post, but it grew enough while I was getting that one ready to publish that it seemed like it should have its own post.

Where machine learning systems like LLMs do unexpectedly well, like in mimicking our use of language, it might not be because they've developed unanticipated special abilities. Maybe ML being good at generating convincing text says as much about the problem of generating convincing text as it does about the ML doing it.

The current generation of chatbots makes it pretty clear that producing language that's hard to distinguish from what a person would produce isn't actually that hard a problem, if you have a general pattern-matcher (and a lot of training text and computing power). In that case, the hard part, that people have spent decades trying to perfect and staggering amounts of compute power implementing, is the general pattern-matcher itself.

We tend to look at ML systems as problem solvers, and fair enough, but we can also look at current ML technology as a problem classifier. That is, you can sort problems according to whether ML is good at them. From that point of view, producing convincing text, recognizing faces, spotting tumors in radiological images, producing realistic (though still somewhat funny-looking) images and videos, spotting supernovas in astronomical images, predicting how proteins will fold, along with many other problems, are all examples of pattern-matching that a general ML-driven pattern-matcher can solve as well as, or even better than, our own naturally evolved neural networks can.

Not knowing a better term, I'll call these ML-friendly problems. In the previous post, I argued that understanding the structure of natural languages is a separate problem from understanding what meaning natural language is conveying. Pretty clearly, understanding the structure of natural languages is an ML-friendly problem. If you buy that understanding meaning is a distinct problem, I would argue that we don't know one way or another whether it's ML-friendly, partly, I would further argue, because we don't know nearly as much about what that problem involves.

From about 150 years ago into the early 20th century, logicians made a series of discoveries about what we call reasoning and developed formal systems to describe it. This came out of a school of thought, dating back to Leibniz (and as usual, much farther and wider if you look for it), holding that if we could capture rules describing how reasoning worked, we could use those rules to remove all uncertainty from any kind of thought.

Leibniz envisioned a world where, "when there are disputes among persons, we can simply say: Let us calculate, without further ado, to see who is right". That grand vision failed, of course, both because, as Gödel and others discovered, formal logic has inescapable limitations, but also because formal reasoning captures only a small portion of what our minds actually do and how we reason about the world.

Nonetheless, it succeeded in a different sense. The work of early 20th-century logicians was essential to the development of computing in the mid-20th century. For example, LISP -- for my money one of the two most influential programming languages ever, along with ALGOL -- was based directly Church's lambda calculus. I run across and/or use Java lambda expressions on a near-daily basis. For another example, Turing's paper on the halting problem used the same proof technique of diagonalization that Gödel borrowed from Cantor to prove incompleteness, and not by accident.

Current ML technology captures another, probably larger, chunk of what naturally-evolved minds do. Just as formal logic broke open a set of problems in mathematics, ML has broken open a set of problems in computing. Just as formal logic didn't solve quite as wide a range of problems as people thought it might, ML might not solve quite the range of problems people today think it might, but just as formal logic also led to significant advances in other ways, so might ML.

Embedding and meaning

I a previous post entitled Experiences, mechanisms, behaviors and LLMs, I discussed a couple of strawman objections to the idea that an LLM isn't doing anything particularly intelligent: that it's "just manipulating text" and it's "just doing calculations".

The main argument was that "just" is doing an awful lot of work there. Yes, an LLM is "just" calculating and manipulating text, but it's not "just" doing so in the same way as an early system like ELIZA, which just turned one sentence template into another, or even a 90s-era Markov chain, which just generates text based on how often which words appeared directly after which others in a sample text.

In both of those cases, we can point at particular pieces of code or data and say "those are the templates it's using", or "there's the table of probabilities" and explain directly what's going on. Since we can point at the exact calculations going on, and the data driving them, and we understand how those work, it's easy to say that the earlier systems aren't understanding text the way we do.

We can't do that with an LLM, even if an LLM generating text is doing the same general thing as a simple Markov chain. We can say "here's the code that's smashing tensors to produce output text from input text", and we understand the overall strategy, but the data feeding that strategy is far beyond our understanding. Unlike the earlier systems, there's way, way too much of it. It's structured, but that structure is much too complex to fit in a human brain, at least as a matter of conscious thought. Nonetheless, the actual of behavior shows some sort of understanding of the text without having to stretch the meaning of the word "understanding".

In the earlier post, I also said that even if an LLM encodes a lot about how words are used and in which contexts -- which it clearly does -- the LLM doesn't know the referents of those words -- it doesn't know what it means for water to be wet or what it feels like to be thirsty -- and so it doesn't understand text in the same sense we do.

This feels similar to appeals like "but a machine can't have feelings", which I generally find fairly weak, but that wasn't quite the argument I was trying to make. While cleaning up a different old post (I no longer remember which one), I ran across a reference that sharpens the picture by looking more closely at the calculations/manipulations an LLM is actually doing.

I think the first post I mentioned, on experiences etc. puts a pretty solid floor under what sort of understanding an LLM has of text, namely that it encodes some sort of understanding of how sentences are structured and how words (and somewhat larger units) associate with each other. Here, I hope to put a ceiling over that understanding by showing more precisely in what way LLMs don't understand the meaning of text in the way that we do.

Taking these together, we can roughly say that LLMs understand the structure of text but not the meaning, but the understanding of structure is deep enough that an LLM can extract information from a large body of text that's meaningful to us.

In much of what follows, I'm making use of an article in Quanta Magazine that discusses how LLMs do embeddings, that is, how they turn a text (or other input) into a list of vectors to feed into the tensor-smashing machine. It matches up well with papers I've read and a course I've taken, and I found it well-written, so I'd recommend it even if you don't read any further here.

Despite the name, a Large Language Model doesn't process language directly. The core of an LLM drives the processing of a list of tokens. A token is a vector -- an ordered list of numbers of a given length -- that represents a piece of the actual input.

To totally make up an example, suppose vectors are three numbers long. If the word a maps to (1.2, 3.0, -7.5), list maps to (6.4, -3.2, 1.6), of maps to (27.5, 9.8, 2.0),  and vectors maps to (0.7, 0.3, 6.8), then a list of vectors maps to [(1.2, 3.0, -7.5), (6.4, -3.2, 1.6), (27.5, 9.8, 2.0), (0.7, 0.3, 6.8)].

Here I'm using parentheses for vectors, which in this case always have three numbers, and square brackets for lists, which can have any length (including zero for the empty list, []). In practice, the vectors will have many more than three components. Thousands is typical. The list of vectors encoding a text will be however long the text is.

The particular mapping from input to tokens is called the embedding^*.   The overall idea is to encode similarities along various dimensions. There are (practically) infinitely many ways to do this mapping. Over time this has evolved from a mostly-manual process, to an automated process using hand-written code, to the current state of the art, which uses machine learning techniques on large bodies of text. The first two approaches are pretty easy to understand.

An ML-produced embedding (that is, the procedure for turning an actual list of words into tokens), on the other hand, relies on a mass of numbers created during a training phase. This mass of numbers drives a generic algorithm that turns words into large vectors. While the numbers themselves don't really lend themselves to easy analysis, people have noticed interesting patterns in the results of applying embedding.

Because the model-building phase is looking at streams of text, it's not surprising that the embedding itself captures information about what words appear in what contexts in that text. For example in typical training corpora, dog and cat appear much more often in contexts like my pet ___ than, say, chair does. They are also likely to occur in conjunction with terms like paw and fur, while other words won't, and so forth.

While we don't really understand exactly how the embedding-building stage of training an LLM extracts relations like this, the article in Quanta gives the example that in one particular embedding the vector for king minus the one for man plus the one for woman is approximately equal to the one for queen (you add or subtract vectors component by component, so (1.2, 3.0, -7.5) + (6.4, -3.2, 1.6) = (7.6, -0.2, -5.9) and so on).

It's long been known that use in similar contexts correlates with similarity in meaning. But we're talking about implied similarities in meaning here, not actual meanings.  You can know an analogy like cat : fur :: person : hair without knowing anything about what a cat is, or a person, or fur or hair.

That may seem odd from our perspective. A person would solve a problem like cat : fur :: person : ? by thinking about cats and people, and what about a person is similar to fur for a cat, because we're embodied in the world and we have experience of hair, cats, fur and so forth. Odd as it might seem to know that cat : fur :: person : hair without knowing what any of those things is, that's essentially what's going on with an LLM. It understands relations between words, based on how they appear in a mass of training text, but that's all it understands^**.

But what, exactly, is the difference between understanding how a word relates to other words and understanding what it means? There are schools of thought that claim there is no difference. The meaning of a word is how it relates to other words. If you believe that, then there's a strong argument that an LLM understands words the same way we do, and about as well as we do.

Personally, I don't think that's all there is to it. The words we use to express our reality are not our reality. For one thing, we can also use the same words to express completely different realities. We can use words in new ways, and the meaning of words can and does shift over time. There are experiences in our own reality that defy expression in words.

Words are something we use to convey meaning, but they aren't that meaning. Meaning ultimately comes from actual experiences in the real world. The way words relate to each other clearly captures something about what they actually mean -- quite a bit of it, by the looks of things -- but just as clearly it doesn't capture everything.

I have no trouble saying that the embeddings that current LLMs use encode something significant about how words relate to each other, and that the combination of the embedding and the LLM itself has a human-level understanding of how language works.

That's not nothing. It's something that sets current LLMs apart from anything before them, and it's an interesting result. For one thing, it goes a long way toward clarifying what's understanding of the world and what's just understanding of how language works and what combinations of words people actually use.

If an LLM is good at it, then it's something about how language works. If an LLM isn't good at it, then it's probably something about the world itself. I'll have a bit more to say about that in the next (shorter) post.

Because LLMs know about language, but not what it represents in the real world, we shouldn't be surprised that LLMs hallucinate, and we shouldn't expect them to stop hallucinating just because they're trained on larger and larger corpora of text.

The earlier post distinguished among behavior, mechanism and experience. An LLM is capable of linguistic behavior very similar to a person's.

The mechanism of an LLM may, or may not, be similar as far as language processing. We may well learn rules like the way that we use the in relation to nouns in a way that's similar to training an LLM. Whether that's the case or not, an LLM, by design, lacks a mechanism for tying words to anything in the real world. This probably accounts for much of the difference between what we would say and what an LLM would say.

All of this is separate from subjective experience.  One could imagine a robot that builds up a store of interactions with the world, processes them into some more abstract representation and associates words with them. But even if that is more similar to what we do in terms of mechanism, it says nothing about what the robot might or might not be experiencing subjectively, even if it becomes harder to rule out the possibility that the robot is experiencing the world as we do.

^* Wikipedia seems to think it's only an embedding if it's done using feature learning, but that seems overly strict. Mathematically, an embedding is any map from one domain into another, no matter how it's produced.

^** Technically, it might matter what the actual numbers are, for example, an embedding that doubled every numeric value or added (1.0, 2.0, 3.0) to every token might produce different results. I'm quietly assuming that models are insensitive to this kind of change of coordinates. If you buy that, then it's relations like king - man + woman ~= queen that matter, and not the actual numeric values that king, man, woman and queen map to. Even if that's not the case, I don't think that changes the overall argument that nothing in an embedding or a model is even trying to capture anything about referents in the real world.