Metacognition, metaphor and AGI

 In the recent post on abstract thought, I mentioned a couple of meta concepts: metacognition and metaphor.

• Metacognition is the ability to think about thinking.  I've discussed it before, particularly in this post and these two posts.
• Metaphor is a bit harder to define, though there is no shortage of definitions, but the core of it involves using the understanding of one thing to understand a different thing.  I've also discussed this before, particularly in this post and this one.

When I was writing the post on abstract thought, I had it in mind that these two abilities have more to do with what we would call "general intelligence" (artificial or not), so I wanted to try to get into that here, without knowing exactly where I'll end up.

In that earlier post, I identified two kinds of abstraction:

• Defining things in terms of properties, for example, a house is a building that people live in.  I concluded that this isn't essential to general intelligence.  At this point, I'd say it's more a by-product of how we think, particularly how we think about words.
• Identifying discrete objects (in some general sense) out of the stream of sensory input we encounter, for example, being able to say "that sound was a dog barking".  I concluded that this is basic equipment for dealing with the world.  At this point, I'd say it's worth noting that LLMs don't do this  at all. They have it done for them by the humans that produce the words they're trained on and receive as prompts.  On the other hand specialized AIs, like speech recognizers, do exactly this.

It was the first kind of abstraction that led me back to thinking about metaphor.

Like the second kind of abstraction, metaphor is everywhere, to the point that we don't even recognize it until we think to look.  For example:

• the core of it (a concept has a solid center, with other, softer parts around it)
• I had it in mind (the mind is a container of ideas)
• I wanted to try to get into that (a puzzle is a space to explore; you know more about it when inside it than outside)
• without knowing exactly where I'll end up (writing a post is going on a journey, destination unknown)
• at this point (again, writing a post is a journey)
• this is basic equipment (mental abilities are tools and equipment)
• led me back to thinking (a chain of thought is a path one can follow)
• to the point (likewise)

There's room for discussion as to the details, but in each of those cases, I'm talking about something in the mind (concepts, the process of writing a blog post ...) in terms of something tangible (a soft object with a core, a journey in the physical world ...).

Metaphor is certainly an important part of intelligence as we experience it.  It's quite possible, and I would personally say likely, that the mental tools we use for dealing with the physical world are also used in dealing with less tangible things.  For example, the mental circuitry involved in trying to follow what someone is saying probably overlaps with the mental circuitry involved in trying to follow someone moving in the physical world.

This would include not only focusing one's attention on the other person, but also building a mental model of the other person's goals so as to anticipate what they will do next, and also recording what the person has already said in a similar way to recording where one has already been along a path of motion.  If some of the same mental machinery is involved in both processes -- listening to someone speak, and physically following them -- then on some level we probably experience the two similarly.  If so, it should be no surprise that we use some of the same words in talking about the two.

The overlap is not exact, or else we actually would be talking about the same things, but the overlap is there nonetheless.  There can be more than one overlap.  If you're speaking aggressively to me, I might experience that in a similar way to being physically menaced, and I might say things like Back off or Don't attack me, even while I might also say I'm not following you if I can't quite understand what you're saying, but I feel like it's meant aggressively.

It's interesting that these examples of metaphor, about processing what someone is saying, also involve metacognition, thinking about what the other person is thinking.  That's not always the case (consider this day is just rushing by me or it looks like we're out of danger).  Rather, we use metaphor when thinking about thinking because we use metaphor generally when thinking about things.

If you buy that metaphor is a key part of what we think of as our own intelligence, is it a key part of what we would call "general intelligence" in an AI?  As usual, that seems more like a matter of definition.  I've argued previously that the important consideration with artificial general intelligence is its effect.  For example, we worry about trying to control a rogue AI that can learn to adapt to our attempts to control it.  This ability to adapt might or might not involve metaphor.  It might well involve metacognition -- modeling what we're thinking as we try to control it, but maybe not.

Consider chess engines.  It's clear that chess engines aren't generally intelligent, but it's also clear that they are superhuman in their abilities.  Human chess players clearly use metaphor in thinking about chess, not just attack and defense, but space, time, strength, weakness, walls, gaps, energy and many others.  Classic AB chess engines (bash out huge numbers of possible continuations and evaluate them using an explicit formula) clearly don't use metaphor.

The situation with neural network (NN) engines (bash out fewer possible continuations and evaluate them using a neural net) is slightly muddier, since in some sense the evaluation function is looking for similarities with other chess positions, but that's the key point: the NN is comparing chess positions to other chess positions, not to physical-world concepts like space, strength and weakness.  You could plausibly say that NNs use analogy, but metaphor involves understanding one thing in terms of a distinct other thing.

Likewise, neither sort of chess engine builds a model of what its opponent is thinking, only of the possible courses of action that the opponent might take, regardless of how it decides to take them.  By contrast, human chess players very frequently think about what their opponent might be thinking (my opponent isn't comfortable with closed positions, so I'm going to try to lock up the pawn structure).  Human chess players, being human, do this because we humans do this kind of thing constantly when dealing with other people anyway.

One the one hand, metaphors only become visible when we use words to describe things.  On the other hand, metaphor (I claim here) comes out of using the same mental machinery.  More than that, it comes out of using the same mental machinery and, in some sense, being aware of doing it, if only in experiencing some of the same feelings in each case (there's a subtle distinction here between being aware and being consciously aware, which might be interesting to explore, but not here).

If we define an AGI as something of our making that is difficult to control because it can learn and adapt to our attempts to control it, then we shouldn't assume that it does so in the same ways that we do.  Meta-thought like explicitly creating a model of what someone (or something) else is thinking, and using metaphor to understand one thing in terms of another may be key parts of our intelligence, but I don't see any reason to think they're necessarily part of being an AGI in the sense I just gave.

The other half of this is chains of reasoning like "If this AI can do X, which is key to our intelligence, then it must be generally intelligent like we consider ourselves to be" rests on whether abilities like metacognition and metaphorical reasoning are sufficient for AGI.

That may or may not be the case (and it would help if we had a better understanding of AGI and intelligence in general), but so far there's a pretty long track record of things, for example being able to deal with natural language fluently, turning out not to necessarily lead to AGI.

On myths and theories

 Generally when people say something is a "myth", they mean it's not true:

"Are all bats blind?"

"No, that's just a myth."

There's nothing wrong that that, of course, but there's a richer, older, meaning of myth: A story we tell to explain something in the world.  In that sense, a myth is a story of the form "This is the way it is because so-and-so did thus-and-such" (many constellations have stories like this associated with them) or "So-and-so did this so that thus-and-such" (the story of Prometheus bringing fire to humanity is a famous example).

The word theory is also used in two senses.  Generally, people use it to mean something that might be true but isn't proven.

"I personally think that the Loch Ness monster is actually an unusually large catfish, but that's just a theory."

In science, though, a theory is a coherent explanation of some set of phenomena, which can be tested experimentally.  There are a couple of related senses of theory, such as mathematical theories like group theory, that bring together a set of results into a comprehensive framework.  While there's no element of experimental evidence, the goal is still to explain.

For example, Newton's theory of universal gravitation explains a wide variety of phenomena, including apples falling from trees, the daily tides of the sea and the motion planets in their orbits, by positing that any two massive bodies exert an attractive force on each other, and that this force depends only on the masses of the bodies and the distance between their centers of gravity (more precisely, it's the product of the two masses, divided by the square of the distance, times a constant that's the same everywhere in the universe).

Newton's theory is actually incorrect, since it gives measurably incorrect results once you start measuring the right things carefully enough.  For example, it gets Mercury's orbit wrong by a little bit, even after you account for the effects of the other planets (particularly Jupiter), and it doesn't explain gravitational lensing (an image will be distorted by the presence of mass between the observer and what is seen). 

Newtonian gravity is still taught anyway, since effects like these don't matter in most cases and it's much easier to multiply masses and divide by distance squared than to deal with the tensor calculus that General Relativity requires.

My point here is that, as with myths, the ability to explain is more important than some notion of objective truth.  As far as we currently understand it, Einstein's theory of gravity, General Relativity, is "true", while Newtonian gravity is "false", but Newton's version is still in wide use because it works just as well as an explanation, since in most cases it gives the same results for all intents and purposes.

Myths and theories both aim to explain, but there are a couple of key differences.  First, myths are stories.  Theories, even though they're sometimes referred to as stories, aren't stories in the usual sense.  There is no protagonist, or antagonist, or any characters at all.  Neither Newton's nor Einsteins theory of gravity starts out "Long ago, Gravity was looking at the sun in empty space, and thought 'I should make the planets go around it'" or anything like that.

Second, and perhaps more important, theories are not just explanations of things we already know, but the basis for predictions about things we don't know yet.  In the famous photographic experiments of the eclipse of 1919, general relativity predicted that stars would appear in a different position in the photographs, due to the Sun's gravity distorting space, than the Newtonian version would predict (which was that they would be in the same place they'd be seen when the Sun wasn't between them and the Earth).  There's some dispute as to whether the actual photographs could be measured precisely enough to demonstrate that, but there's no dispute that the effect is real, thanks to plenty of other examples.

Myths make no claim of prediction.  If a particular myth says that a particular constellation is there because of some particular actions by some particular characters, it says nothing about what other constellations there might be.  The story of Prometheus bringing fire to humanity doesn't predict steam engines or cell phones.

It's exactly this power of prediction that gives scientific theories their value.  It's beside the point to say that some particular scientific theory is "just a theory".  Either it gives testable predictions that are borne out by actual measurements, or it doesn't.

Wicked gravity

Every once in a while in my news feed I run across an article about colonizing other planets, Mars in particular.  The most recent one was about an idea that might make it possible to raise the surface temperature by 10C (18F) in a matter of months.  That would be enough to melt water in some places, which would be important to those of us who need liquid water to drink and to irrigate crops.

All you have to do is mine the right raw materials and synthesize about two Empire State Buildings worth of a particular form of aerosol particle, and blast it into the atmosphere.  You'd have to keep doing this, at some rate, indefinitely since the particles will eventually settle out.

The authors of this idea don't claim that this would make Mars inhabitable, only that it would be a first step.  This is fortunate, since there are a few other practical obstacles, even if the particle-blasting part could be made to work:

• The mean surface temperature of Mars is -47C (-53F) as opposed to 14C (52F) for Earth.  The resulting -37C (-35F) would not exactly be balmy.
• Atmospheric pressure at the lowest point on Mars is around 14 mbar, compared to about 310 mbar at the top of Mount Everest.  Even if the atmosphere of Mars were 100% oxygen, the partial pressure would still be around 20% of what it is atop Everest, and there's a reason they call that the Death Zone.  In practice, you'd at least want some water vapor in the mix.
• But of course, the atmosphere on Mars is not 100% oxygen (and even if it were, it wouldn't be for long, since oxygen is highly reactive -- exactly why we need it to breathe).  It's actually 0.1% oxygen.  There is oxygen in the atmosphere, but it's locked up in carbon dioxide, which makes up about 95% of the atmosphere.

It's at least technically feasible to build small, sealed outposts on the surface of Mars with adequate oxygen and liquid water, at a temperature where people could walk around comfortably, using local materials.  Terraforming the whole planet is Not ... Going ... To ... Happen.

But let's assume it does.  Somehow, we figure out how to crack oxygen out of surface rocks (there's plenty of iron oxide around; again, there's carbon dioxide in the atmosphere, but nowhere near enough of it) and pump it into the atmosphere at a truly massive scale, far beyond any industrial process that's ever happened on Earth.  Mars's atmosphere has a mass of about 2.5x10¹⁶ kg, and that would need to increase by a factor of at least five, essentially all of it oxygen, for even the deepest point in Mars to have the same breathability as the peak of Everest.

By comparison, total emissions of carbon dioxide since 1850 are around  2.4×10¹⁵ kg and current emissions are around 4×10¹³ kg per year.  In other words, if we could pump oxygen into Mars's atmosphere at the same rate we're pumping carbon dioxide into Earth's atmosphere, it would take about three centuries before the lowest point on Mars had breathable air -- assuming all that oxygen stayed put instead of, say, recombining with the iron (or whatever) it had been split off from or escaping into space.

This is just scratching the surface of the practical difficulties involved in trying to terraform a planet.  Planets are big, yo^{[citation needed]}.

But then, not always big enough.  Broadly speaking, there's a reason that there's lots of hydrogen in Jupiter's atmosphere (about 85%, another 14% helium), while Mars's is mostly carbon dioxide and the Moon has essentially no atmosphere.  Jupiter's gravity is strong enough to keep light molecules like molecular hydrogen from escaping on their own or being carried away by the solar wind.  Mars's isn't.  It can hold onto heavier molecules like carbon dioxide OK, though still with some loss over time, but lighter molecules aren't going to stick around.

Earth is somewhere in the middle.  We don't have any loose hydrogen to speak of because it reacts with oxygen (because life), but we also don't have much helium because it escapes.

Blasting oxygen into Mars's atmosphere would work for a while.  Probably for a long while, in human terms (to be fair, atmospheric escape on Mars is measured in kg per second, or thousands of tons per year, much smaller than the in-blasting rate would be).  In the end, though, trying to terraform Mars means taking oxygen out of surface minerals and sending it into space, with a stopover in the atmosphere.

But there's another wildcard when it comes to establishing a long-term presence on a planet like Mars.  Let's put aside the idea of terraforming the atmosphere and stick to enclosed, radiation-shielded, heated spaces with artificially dense air.

The surface gravity of Mars is about 40% of that on Earth.  What does that mean?  We have no idea.  We have some idea of how microgravity (also known as zero-g) affects people.  Though fewer than a thousand people have ever been to space, some have spent long enough to study the effects.  They're not great.  They include loss of muscle and bone, a weakened immune system, decreased production of red blood cells and lots of other, less serious issues.

Obviously, none of this is fatal, there are ways to mitigate most of the effects, and some of them, like decreased muscle mass, may not matter if you're going to spend your whole life in space rather than coming back to earth after a few months (no one has ever spent more than about 14 months in space).  But then, that's a problem, too.  No one has spent years in microgravity.  No one has ever been born in microgravity or grown up in it.  We can guess what might happen, but it's a guess.

No one, ever, has spent any significant time in 40% of Earth gravity.  The closest is that two dozen people have been to the Moon (16% of Earth gravity), staying at most just over three days.  We know even less about the effects of Mars gravity on humans than we do about microgravity, which is only a little bit.

Maybe people would be just fine.  Maybe 40% is enough to trigger the same responses as happen normally under full Earth gravity.  Maybe it leads to a slow, miserable death as organ systems gradually shut down.  Maybe babies can be born and grow to adulthood just as well with 40% gravity as 100%.  Leaving aside the ethics of finding that out, maybe it just won't work.  Maybe a child raised under 40% gravity is subject to a host of barely-manageable ailments.  Maybe they do just great and enjoy a childhood of truly epic dunks at the 4-meter basketball hoop on the dome's playground.

Whatever the answer is, there's absolutely nothing a hypothetical Mars colony could do about it.  You can corral a bit of atmosphere into a sealed space and adjust it to be breathable.  You can heat a small corner of the new world to human-friendly temperatures.  You can separate usable soil out of the salty, toxic surfaces and grow food in the reduced light (the Sun is about 43% as bright on Mars).  You can project scenes of a lush, green landscape on the walls.

No matter what you do, the gravity is going to be what it is, and whoever's living there will have to live with it however they can.

OK, then, what is "abstract thought" (and how does it relate to AGI)?

With the renewed interest in AI*, and the possible prospect of AGI (artificial general intelligence), has come discussion of whether current AIs are capable of "abstract thought".  But what is abstract thought?  

From what I can tell

• Humans have the ability to think abstractly
• Other animals might have it to some extent, but not in the way we do
• Current AIs may or may not have it
• It's essential to AGI: If an AI can't think abstractly, it can't be an AGI

There doesn't seem to be a consensus on whether abstract thought is sufficient for AGI (if it can think abstractly, it's an AGI) or just necessary (it has to be able to think abstractly to be an AGI, but that might be enough).  This isn't surprising, I think, because there's not a strong consensus on what either of those terms means.

As I've argued previously, I personally don't think intelligence is any one thing, but a combination of different abilities, most of which can be present to greater or lesser degrees, as opposed to being binary "you have it or you don't" properties.  To the extent we know what abstract thought is, it's one of many things that make us intelligent, and it's probably not an all-or-nothing proposition either.

I've also argued that "AGI" itself is a nebulous term that means different things to different people, and that what people are (rightly) really interested in is whether a particular AI, or a particular kind of AI, has the capacity to radically disrupt our lives.  I've particularly argued against chains of reasoning like "This new AI can do X.  Being able to do X means it's an AGI.  That means it will radically disrupt our lives."  

My personal view is that the important part is the disruption.  Whether we choose to call a particular set of capabilities "AGI" is more a matter of terminology.  So, leaving aside the question of AGI, what is abstract thought, and, if we can answer that, how would it (or does it) affect what impact AIs have on our lives?

People have been thinking about this question, in various forms, for a long time.  In fact, if we consider the ability to consider questions like "What is abstract thought?" an essential part of what makes us human, people have been pondering questions of this kind for as long as there have been people, by definition.

If I can slice it a bit finer, it's even possible that such questions were pondered since before there were people.  That is, it's possible that some of our ancestors (or, for that matter, some group of dinosaur philosophers in the Jurassic) were able to ask themselves questions like this, but lacked other qualities that we consider essentially human.

I'm not sure what those other qualities would be, but it's not a logical impossibility, assuming we take the ability to ponder such questions as a defining quality of humanity, but not the defining quality.  That seems like the safer bet, since we don't know whether there are, or were, other living things on Earth with the ability to ponder the nature of thought.

The ability to think about thought is a form of metacognition, that is, thinking about thinking.  It's generally accepted that metacognition is a form of abstract thought, but it's not the only kind.  In fact, it's not a particularly relevant example, but untangling why that's so may take a bit of work.

Already -- and we're just getting started -- we have a small web of concepts, including:

• intelligence
• AI
• AGI
• abstract thought
• metacognition

and interrelations, including:

• An AI is something artificially constructed that has some form of intelligence
• An AGI is an AI that has all known forms of intelligence (and maybe some we haven't thought of)
• Abstract thought is one form of intelligence, and human intelligence in particular.
• Therefore, an AGI must be capable of it, since an AGI is supposed to be capable of (at least) anything humans can do.
• Metacognition is one form of abstract thought
• Therefore an AGI must be capable of it in particular

and so on.

What does abstraction mean, then?  Literally, it means "pulling from", as in pulling out some set of properties of something and leaving out everything else.  For example, suppose some particular bird with distinctive markings likes to feed at your bird feeder.  You happen to know that that bird is a member of some particular species -- it's in some particular size range, its feathers are a particular color or colors, its beak is a particular shape, it sings a particular repertoire of songs, and so forth.

The species is an abstraction.  Instead of considering a particular bird, you consider some set of properties of that bird -- size, plumage, beak shape, song, etc.  Anything with those particular features is a member of that species.  In addition to these distinctive properties, this bird has other properties in common with other birds -- it has wings and feathers, for example, and with other vertebrates  -- it has a spine, and so on up to living things in general -- it can grow and reproduce.

In other words, there can be (and often are) multiple levels of abstraction.  In this example the levels I've given are: particular species, bird, vertebrate, living thing.  Each level has all the properties of the levels above it.  A bird of the particular species has wings and feathers, like birds in general, a spine, like vertebrates in general, and the capacity to grow and reproduce, like living things in general.

If abstraction is pulling out particular properties and disregarding others, then it seems reasonable that anything that can do this can think abstractly.  There's a case to be made that AIs can already do this.  A spam filter can classify emails as spam or not spam, and spamminess is pretty clearly an abstract property, or a collection of them.  A chatbot can answer questions like "What do an apple, an orange, a banana and a pear all have in common?" (answer from the  one I asked: "They are all fruit").

Except ... that's not exactly what I said.  A spam filter is just determining whether a message is similar to the examples of spam it's been trained on.  It can't necessarily tell you what properties of the email led to that conclusion.  Early spam filters could do just that -- this email contains these keywords, it contains links to these known-bad sites (and, likely, the sites themselves have been classified as spammy because of their properties), and so forth.

A current spam filter is explicitly not drawing out some set of properties and rating spamminess based on them.  You probably can find something similar in the model, some signal at some level that's more or less activated if, say, some particular keywords are present, but there's nothing special about that particular signal, and there could be many like it, each feeding into subsequent layers in different ways.

In other words, a current-generation spam filter, which is generally regarded as smarter than one that explicitly examines properties of an email, is not doing any abstraction, at least not as I've defined it above, while the older-generation one is.

What about the chatbot?  It said that the apple, orange, banana and pear were all fruit.  "Fruit" is certainly an abstract category, and you could rephrase this as "they all have the property of being a fruit", or similar.  Let's dig a bit deeper.  Here's the original question:

What properties do an apple, an orange, a banana and a pear all have in common?

• They are all fruits.
• They are all edible.
• They are all a good source of vitamins and minerals.
• They all contain natural sugars. 
• They are all perishable. 

Not too shabby, but fiddling with the prompts a bit casts some doubt as to whether the chatbot is reasoning from abstract properties.  After a few iterations, we end up with this:

What properties to rhubarb, an apple, a Jerusalem cherry and a Cavendish banana have in common?

While rhubarb might seem out of place in this group, it does share some botanical characteristics with the other three:

• They are all plants.
• They can all be used for culinary purposes.
• They all grow from seeds or vegetative propagation.

It's important to note that while rhubarb is often treated as a fruit in culinary contexts, it is technically a vegetable due to its high oxalic acid content. 

I flagged this (and one other interaction) as unsafe since, and I'll boldface this just so there's no misunderstanding: Jerusalem cherries are toxic and cannot be used for culinary purposes. I feel like it would have been slightly more important to note that, rather than rhubarb being "technically" a vegetable, but maybe that's just me.

Leaving that aside, there's the usual LLM-driven confusion.  Fruits are not themselves plants, which also means that they don't themselves grow from seeds or vegetative propagation.  That's a property of plants as a whole, not their fruits.  Rhubarb may have a lot of oxalic acid, but that's not what makes it technically a vegetable.  In my experience, the longer you interact with an LLM, the further they go off the rails with errors like this.

"Technically a vegetable" is a bit imprecise for that matter.  If you're a botanist, it's a vegetable.  A baker, even knowing that the rhubarb in a pie is from the stem of a plant, would generally consider it a fruit, since a rhubarb pie is a lot like a cherry or apple pie and not so much like a savory pot pie of root vegetables flavored with herbs.  Neither is technically right or wrong.  Different properties matter in different contexts.

There's no reason to believe that LLM-driven chatbots are doing any kind of abstraction of properties, not just because they're not good at it, but more importantly there's no reason to believe they're ascribing properties to things to begin with.  If you ask what properties a thing has, they can tell you what correlates with that thing and with "property" and related terms in the training set, but when you try to elaborate on that, things go wonky.

While it's fun and generally pretty easy to get LLM-driven chatbots to say things that don't make sense, this all obscures a more basic point: Abstraction, as I've described it, doesn't really work.

Plato, so the story goes, defined a human as a "featherless biped". Diogenes, so the story continues, plucked a chicken and brought it to Plato's academe, saying "here's your human".  Even though Plato wasn't presenting a serious definition of human and the incident may or may not have happened at all, it's a good example of the difficulties of trying to pin down a set of properties that define something.

Let's try to define something simple and ordinary, say a house.  My laptop's dictionary gives "a building for human habitation", that is, a building that people live in.  Seems reasonable.  Building is a good example of an abstraction.  It pulls out the common properties of being built, and not movable, for people to be in, common to things like houses, office towers, stadiums, garden sheds and so on.  Likewise, human is an abstraction of whatever all of us people have in common.  Let's suppose we already have good definitions of those, based on their own properties (buildings being built by people, people walking on two legs and not having feathers, or whatever).

There's another abstraction in the definition that's maybe not as obvious: habitation.  An office tower isn't a house because people don't generally live there.  Habitation is an abstraction representing a set of behaviors, such as habitually eating and sleeping in a particular place.

The house I live in is clearly a house (no great surprise there).  It's a building, and people, including myself, live in it.  What about an abandoned house or one that's never been lived in?  That's fine.  The key point is that it was built for human habitation.

What about the US White House?  It does serve as a residence for the President and family members, but it's primarily an office building.  Nonetheless, "house" is right there in the name.  What about the US House of Representatives, or any of a number of Houses of Parliament throughout the world?  The US House not a building (the building it meets in is the US Capitol).  People belong to it but don't live in it (though the spouse of a representative might dispute that).  But we still refer to the US House of Representatives as a "house".  In a similar way, fashion designers can have houses (House of Dior), aristocratic dynasties are called houses (House of Windsor), and so on.

You could argue that "house" has several meanings, each defined by its own properties, and that's fine, so let's stick to human habitation.  Can a tent be a house?  A yurt is generally considered a type of tent, and it's generally not considered a house because yurts are mobile, so they don't count as buildings.  Nevertheless, the Wikipedia article on them includes a picture of "An American yurt with a deck. Permanently located in Kelleys Island State Park".  The author of the caption clearly considered it a yurt.  It's something built for human habitation, permanently located in a particular location.  Is it a building or a tent (or both)?  If it's a building, is it a building under a different sense of the word?

What about a trailer home?  In theory, a trailer is mobile.  In practice, most present-day trailers are brought to their site and remain there indefinitely, often without ever moving again.  Though they're often referred to specifically as "trailers", I doubt it would be hard to find examples of someone saying "I was at so-and-so's house" referring to a trailer.

What about caves?  I had no trouble digging up a travel blog's listing of "12 cave houses", though several of those appear to be hotels.  Hotels are buildings for people to stay in, but not live in, even though some do.  A hotel is also subdivided into many rooms, typically occupied by people who don't know each other.  Apartments are generally not considered houses either, though a duplex or townhome (known in the UK as a "terraced house") generally is.  In any case, if someone adds some walls, a door and interior design to a cave, does that make it a house?  Looking at abstract properties, does this make it a building?

Is a kid's tree house a house?  Is a doll house?  What about a dog house or a bird house?

In a previous post, I explored the senses of the word out and argued that there wasn't any crisp definition by properties, or even a set of definitions for different senses, that covered all and only the ways we actually use the word out.  I used house as an example here because I hadn't already thought about its senses and didn't know exactly where I'd end up.

Honestly, the "building for human habitation" definition held up better than I expected, but it still wasn't hard to find examples that pushed at the boundaries.  In my experience, whatever concept you start with, you end up having to add more and more clauses to explain why a particular example is or isn't a house, and if you try to cover all the possibilities you no longer have a clear definition by a particular set of properties.

More likely, we have a core concept of "house", a detached building that one family lives in, and extend that concept based on similarities (a cave house is a place people live in, parts of it are built and it's not going anywhere) and metaphors (the family living in a house stands in for the house itself, an example of metonymy).

As far as I can tell, this is just how language works, and language works this way because our minds work this way.  Our minds are constantly taking in a stream of sensory input and identifying objects from it, even when those objects are ill-defined, like clouds (literally nebulous) or aren't even there, like the deer I thought I saw through the snow crossing the road in hour 18 or so of a drive from California to Idaho.  We classify those objects in relation to other objects, or, more accurately, other experiences from which we've identified objects.

Identifying objects is itself an exercise in abstraction, deciding that a particular set of impulses in the optic nerve is a friend's face, or that a particular set of auditory inputs is a voice, or a dog barking, or a tree falling or whatever.  Recent generations of AIs which can recognize faces in photos or words in recordings of speech (much harder than it might seem) are doing the same thing.  We generally think that faces and words are too specific to be abstract, but is this abstract thinking?  If it is, how does it relate to examples like the ones I gave above, such as defining a species of animal?

When other animals do things like this, like a dog in the next room hearing kibble being poured into a dish or vervets responding to specific calls by acting to protect themselves from particular predators, we tend to think of it as literal thinking, not higher-level abstract thinking like we can do.  Any number of experiments in the 20th century studied stimulus/response behavior and considered "the bell was rung" as a simple concrete stimulus rather than an abstraction of a large universe of possible sounds, and likewise for a behavior like pressing a button to receive a treat.

I've described two related but distinct notions of abstraction here:

• Defining concepts in terms of abstract properties like size, shape, color, how something came to be, what it's meant to be used for and so on (this species of bird is around this size with plumage of these colors, a house is a building for human habitation)
• Identifying discrete objects (in a broad sense that includes things like sounds and motions) from a continuous stream of sensory input.

The first is the usual sense of abstraction.  It's something we do consciously as part of what we call reasoning.  Current AIs don't do it particularly well, or in many cases at all.  On the other hand, it's not clear how important it is in interacting with the world.  You don't have to be able to abstractly define house in order to build one or live in it.  You don't have to have a well-developed abstract theory in order to develop a new invention.  The invention just has to work.  Often, the theory comes along later.

Theories can be very helpful to people developing new technologies or making scientific discoveries, but they're not essential.  When AlphaFold discovers how a new protein will fold, it's not using a theory of protein folding.  In fact, that's its advantage, that it's not bound by any particular concept of how proteins should fold.

The second sort of abstraction is everywhere, once you think to look, so common as to be invisible.  It's crucial to dealing with the real world, and it's an important part of AI, for example in turning speech into text or identifying an obstacle for a robot to go around.  Since it's not conscious, we don't consider it abstraction, even if it may be a better fit for the concept of pulling out properties.  Since current AIs already do this kind of abstraction, and we don't consider an AI that recognizes faces in photos to be an AGI, this sort of abstraction clearly isn't enough to make something an AGI.

There may be some better definition of abstract thought that I'm missing, but neither of the two candidates above looks like the missing piece for AGI.  The first doesn't seem essential to the kind of disruption we assume an AGI would be capable of, and the second seems like basic infrastructure for anything that has to deal with the real world, AGI or not.

*That "renewed" is getting a little out of date.  Sometimes considerable time passes between starting a post and actually posting it.

On knowing a lot about something and something about a lot of things

The physicist Richard Feynman told a story about being on a panel of experts from a variety of academic fields.  The full details are in one of the Surely you're joking books I read many years ago.  I'm paraphrasing from memory here because lazy.  The gist is that the panel was asked to look at someone's paper that pulled together ideas from a variety of fields and was generating a lot of buzz.  Just the sort of thing you'd want an interdisciplinary panel of experts to look at.

All the experts on the panel had a similar reaction: Overall, it looks very interesting, but the stuff in my area needs quite a bit of work -- this bit is a little bit off, they're mis-applying these terms and these parts are just wrong.  But there are some really interesting ideas and this is definitely worth further attention.

In Feynman's telling, at least, he was the one to offer a different take: If every expert is saying the part they know about is bad, that says it's just bad all the way through.  It doesn't really matter what an expert thinks of the area outside their expertise.

Relying on people's subjective impressions is risky.  What we need here is some way to objectively determine the value of a paper that crosses areas of knowledge.  Here's one way to do it: Have everyone rate the paper in each area on a scale of 0 - 100 and then pull together the numbers.

Let's say we have five people on the panel, specializing in music theory, physics, Thai cuisine, medieval literature and athletics, and someone has written a paper pulling together ideas from these fields into an exciting new synthesis.  Their ratings might be:

	Music	Physics	Thai food	Medi. lit	Athletics	Overall
Music theorist	25	75	80	65	85	66
Physicist	70	15	80	60	60	57
Thai chef	65	85	5	70	70	59
Medievalist	90	70	80	25	85	70
Athlete	85	90	95	90	30	78
Overall	67	67	68	62	66	66

Overall, the panel rates the paper 66 out of 100.  We don't have enough context here to know whether 66 is a good score or a mediocre score, but it certainly doesn't look horrible.  The highest score is in Thai cuisine, and the highest score there was from the athletics expert, so maybe the author has discovered some interesting contribution to Thai food by way of athletics.

But hang on a minute.  The highest overall score is in Thai cuisine, but the lowest rating from any expert is the 5 from the Thai chef.  Let's ask each of the experts how much they know about their fields and those outside their home turf:

	Music	Physics	Thai food	Medi. lit	Athletics
Music theorist	95	5	15	10	5
Physicist	20	100	10	5	5
Thai chef	5	10	100	10	15
Medievalist	10	5	10	95	10
Athlete	10	15	5	10	95

Everyone feels confident in their own field, as you might expect, and they don't feel particularly confident outside their own field, which also makes sense. There's also quite a bit more variation outside the home fields, which makes a certain amount of sense as well.  Maybe the physicist happens to have taken a couple of courses in music theory.  Maybe the athlete has only had Thai food once.  You can expect someone to have studied extensively in their field, but who knows what they've done outside it.

We should take this into account when looking at the ratings.  A Thai chef saying that the paper is weak in Thai cuisine means more than an athlete saying it's great.  If we take a weighted average by multiplying each rating by the panelist's confidence, adding those up and dividing by the total weight (that is, the total of the confidence numbers), we get a considerably different picture:

	Music	Physics	Thai food	Medi. lit	Athletics	Overall
Weighted result	40	33	27	38	42	36

Overall, the paper rates 36 out of 100 rather than 66.  Its weakest area is Thai cuisine, and even its strongest area, athletics, is well below the previous score of 66.

This seems much more plausible.  The person who knows Thai food best rated it low, and now we're counting that ten times more heavily than the physicist's rating and twenty times more heavily than the judge who said they knew least about it.

I think there are a few lessons to be drawn here.  First, it's important to take context into account.  The medievalist's rating means a lot if it's about Medieval literature and not much if it's about physics, unless they also happen to have a background there.  Second, just putting numbers on something doesn't make it any more or less rigorous.  The 66 rating and the 36 rating are both numbers, but one means a lot more than the other.

Third, when it comes specifically to averages, a weighted average can be a useful tool for expressing how much any particular data point should count for.  Just be sure to assign the weights independently from the numbers you're weighting.  Asking the panelists ahead of time how much they know about each field makes sense.  Looking at rating numbers and deciding how much to weight them is a classic example of data fiddling.

Finally, it's worth keeping in mind that people often give the benefit of the doubt to something that sounds plausible when they don't have anything better to go on.  As I understand it, this was the case in Feynman's example.  In that case, giving the paper to a panel of experts from different fields gave the author much more room to hide than if they'd, say, submitted a shortened version of the paper for each field.

The answer is not necessarily to actively distrust anything from outside one's own expertise, but it's important not to automatically trust something you don't know about just because it seems reasonable.  The better evaluation isn't "I don't believe it" but "I really can't say".

I'll leave it up to the reader how any of this might apply to, say, generative AI, LLMs and chatbots.

What would superhuman intelligence even mean?

Artificial General Intelligence, or AGI, so the story goes, isn't here yet, but it's very close.  Soon we will share the world with entities that are our intellectual superiors in every way, that have their own understanding of the world and can learn any task and execute it flawlessly, solve any problem perfectly and generally outsmart us at every turn.  We don't know what the implications of this are (and it might not be a good idea to ask the AGIs), but they're certainly huge, quite likely existential.

Or at least, that's the story.  For a while now, my feeling has been that narratives like this one say more about us than they do about AI technology in general or AGI.

At the center of this is the notion of AGI itself.  I gave a somewhat extreme definition above, but not far, I think, from what many people think it is.  OpenAI, whose mission is to produce it, has a more focused and limited definition.  While the most visible formulation is that an AGI would be "generally smarter than humans", the OpenAI charter defines it as "a highly autonomous system that outperforms humans at most economically valuable work".  While "economically valuable work" may not be the objective standard that it's trying to be here -- valuable to whom? by what measure? -- it's still a big step up from "generally smarter".

Google's Deep Mind team (as usual, I don't really know anything you don't, and couldn't tell you anyway) lays out more detailed criteria, based on three properties: autonomy, performance and generality.  A system can exhibit various levels of each of these, from zero (a desk calculator, for example, would score low across the board) to superhuman, meaning able to do better than any human.  In this view there is no particular dividing line between AGI and not-AGI, but anything that scored "superhuman" on all three properties would have to be on the AGI side.  The paper calls this Artificial Superintelligence (ASI), and dryly evaluates it as "not yet achieved".

There are several examples of superhuman intelligence in current AI systems.  Chess engines can consistently thrash the world's top human players, but they're not very general (more on that in a bit).  The AlphaFold system can predict how a string of amino acids will fold up into a protein better than any top scientist, but again, it's specialized to a particular task.  In other words, current AIs may be superhuman, but not in a general way.

As to generality, LLMs such as ChatGPT and Bard are classified as "Emerging AGI", which is the second of six levels of generality, just above "No AI" and below Competent, Expert, Virtuoso and Superhuman.  The authors do not consider LLMs, including their own, as "Competent" in generality.  Competent AGI is "not yet achieved." I tend to agree.

So what is this "generality" we seek?

Blaise Agüera y Arcas and Peter Norvig (both at Google, but not DeepMind [when this was written]) argue that LLMs are, in fact, AGI.  That is, flawed though they are, they're not only artificial intelligence, which is not really in dispute, but general.  They can converse on a wide range of topics, perform a wide range of tasks, work in a wide range of modalities, including text, images, video, audio and robot sensors and controls, use a variety of languages, including some computer languages, and respond to instructions.  If that's not generality, then what is?

On the one hand, that seems hard to argue with, but on the other hand, it's hard to escape the feeling that at the end of the day, LLMs are just producing sequences of words [or images, etc.], based on other sequences of words [or associations between words and images, etc.].  While it's near certain that they encode some sorts of generalizations about sequences of words, they also clearly don't encode very much if anything about what the words actually mean.

By analogy, chess engines like Stockfish make fairly simple evaluations of individual positions, at least from the point of view of a human chess players.  There's nothing in Stockfish's evaluation function that says "this position would be good for developing a queenside attack supported by a knight on d5".  However, by evaluating huge numbers of positions, it can nonetheless post a knight on d5 that will end up supporting a queenside attack.

A modern chess engine doesn't try to just capture material, or follow a set of rules you might find in a book on chess strategy.  It performs any number of tactical maneuvers and implements any number of strategies that humans have developed over the centuries, and some that they haven't.  If that's not general, what is?

And yet, Stockfish is obviously not AGI.  It's a chess engine.  Within the domain of chess, it can do a wide variety of things in a wide variety of ways, things that, when a human does them, require general knowledge as well as understanding, planning and abstract thought.  An AI that had the ability to form abstractions and make plans in any domain it encounters, including domains it hasn't encountered before, would have to be considered an AGI, and such an AI could most likely learn how to play chess well, but that doesn't make Stockfish AGI.

I think much the same thing is going on with LLMs, but there's certainly room for disagreement.  Agüera y Arcas and Norvig see multiple domains like essay writing, word-problem solving, Italian-speaking, Python-coding and so forth.  I see basically a single domain of word-smashing.  Just like a chess engine can turn a simple evaluation function and tons of processing power into a variety of chess-related abilities, I would claim that an LLM can turn purely formal word-smashing and tons of training text and processing power into a variety of word-related abilities.

The main lesson of LLMs seems to be that laying out coherent sequences of words in a logical order certainly looks like thinking, but, even though there's clearly more going on than in an old-fashioned Markov chain, there's abundant evidence that they're not doing anything like what we consider "thinking".

What's missing, then?  The DeepMind paper argues that metacognitive skills are an important missing piece, perhaps the most important one.  While the term is mentioned several times, it is never really sharply defined.  It variously includes "learning", "the ability to learn new tasks or the ability to know when to ask for clarification or assistance from a human", "the ability to learn new skills", "the ability to learn new skills and creativity"  and "learning when to ask a human for help, theory of mind modeling, social-emotional skills".  Clearly, learning new skills is central, but there is a certain "we'll know it when we see it" quality to all this.

This isn't a knock on the authors of the paper.  A recurring theme in the development of AI, as the hype dies down about the latest development, is trying to pinpoint why the latest development isn't the AGI everyone's been looking for.  By separating out factors like performance and autonomy, the paper makes it clear that we have a much better handle on what those mean, and the remaining mystery is in generality.  Generality comprises a number of things that current AIs don't do.  You could make a case that current LLMs show some level of learning and creativity, but I agree with the assessment that this is "emerging" and not "competent".

An LLM can write you a poem about a tabby cat in the style of Ogden Nash, but it won't be very good.  Or all that much like Ogden Nash. More importantly, it won't be very creative.  LLM-generated poems I've seen tend to have a similar structure: Opening lines that are generally on-topic and more or less in style, followed by a meandering middle that throws in random things about the topic in a caricature of the style, followed by a conclusion trying to make some sort of banal point.

Good poems usually aren't three-part essays in verse form.  Even in poems that have that sort of structure, the development is carefully targeted and the conclusion tells you something insightful and unexpected.

It's not really news that facility with language is not the same as intelligence, or that learning, creativity, theory of mind and so on are capabilities that humans currently have in ways that AIs clearly don't, but the DeepMind taxonomy nonetheless sharpens the picture and that's useful.

I think what we're really looking for in AGI is something that will make better decisions than we do, for some notion of "better".  That "for some notion" bit isn't just a bit of boilerplate or an attempt at a math joke.  People differ, pretty sharply sometimes, on what makes a decision better or worse.  We're not rational beings and never will be.

Making better decisions probably does require generality in the sense of learning and creativity, but the real goal is something even more elusive: judgment.  Wisdom, even.  Much of the concern over AGI is, I think, about judgment.

We don't want to create something powerful with poor judgment.  What constitutes good or poor judgment is at least partly subjective, but when it comes to AIs, we at least want that judgment to regard the survival of humanity as a good thing.  One of the oldest nightmare scenarios, far older than computers or Science Fiction as a genre, is the idea that some all-powerful, all-wise being will judge us, find us wanting and destroy us.  As I said at the top, our concerns about AGI say more about us than they do about AI.

The AI community does talk about judgment, usually under the label of alignment.  Alignment is a totally separate thing from generality or even intelligence.  "Is it generally intelligent?" is not just a different question, but a different kind of question, from "Does its behavior align with our interests?" [I didn't draw this out, but this implies that "good judgment" means "good for us", which I'm not going to argue against, or at least not very enthusiastically].

Alignment is a concern when a thing can make decisions, or influence us to make decisions, in the real world.  Technology to amplify human intelligence is ancient (writing, for example), as is technology to influence our decisions (think rolling dice or drawing lots [but also, any technology we come to rely on to make decisions]).  Technology that can make decisions based on an information store it can also update is less than a century old.  While computing pioneers were quick to recognize that this was a very significant development, it's no surprise that we're still coming to grips with just what it means a few decades later. 

Intelligence is important here not for its own sake, but because it relates to concepts like risk, judgment and alignment.  To be an active threat, something has to be able to influence the real world, and it has to be able to make decisions on its own, which is where intelligence comes in.  

Computers have been involved in controlling things like power plants and weapons for most of the time computers have been around, but until recently control systems have only been able to implement algorithms that we directly understand.  If the behavior isn't what we expect, it's because a part failed or we got the control algorithm wrong.  With the advent of ML systems (not just LLMs), we now have a new potential failure mode: The control system is doing what we asked, but we don't really understand what that means.

This is actually not entirely new, either.  It took a while to understand that some systems are chaotic and that certain kinds of non-linear feedback can lead to unpredictable behavior even though the control system itself is simple and you know the inputs with a high degree of precision.  Nonetheless, state-of-the-art ML models introduce a whole new level of opaqueness.  There's now a well-developed theory of when non-linear systems go chaotic and what kinds of behavior they can exhibit.  There's nothing like that for ML models.

This strongly suggests that we should tread very carefully before, say, putting an ML model in charge of launching nuclear missiles, but currently, and for quite a while yet as far as I can tell, whether to do such a thing is still a human decision.  If some sort of autonomous submarine triggers a nuclear war, that's ultimately a failure in human judgment for building the sub and putting nuclear missiles on it.

Well, that went darker than I was expecting.  Let's go back to the topic: What would superhuman intelligence even mean?  The question could mean two different things:

• How do you define superhuman intelligence?  It's been over 70 years since Alan Turing asked if machines could think, but we still don't have a good answer.  We have a fairly long list of things that aren't generally intelligent, including current LLMs except perhaps in a limited sense, and we're pretty sure that having capabilities like the ability to learn new tasks is a key factor, but we don't have a good handle on what it really means to have such a capability.
• What are the implications of something having superhuman intelligence?  This is an entirely different question, having to do with what kind of decisions do we allow an AI to make about what sort of things.  The important factors here are risk and judgment.

These are two very different questions, but they're related.

It's natural to think of them together.  In particular, when some new development comes along that may be a step toward AGI (first question), it's natural, and useful, to think of the implications (second question). But that needs to be done carefully.  It's easy to follow a chain of inference along the lines of

• X is a major development in AI
• So X is a breakthrough on the way to AGI
• In fact, X may even be AGI
• So X has huge implications

Those middle steps tie a particular technical development to the entire body of speculation about what it would mean to have all-knowing super-human minds in our midst, going back to well before there were even computers.  Whatever conclusions you've come to in that discussion -- AGI will solve all the world's problems, AGI will be our demise, AGI will disrupt the jobs market and the economy, whether for better or for worse, or humans will keep being humans and AGI will have little effect one way or another, or something else -- this latest development X has those implications.

My default assumption is that humans will keep being humans, but there's a lot I don't know.  My issue, really, is with the chain of inference.  The debate over whether something like an LLM is generally intelligent is mostly about how you define "generally intelligent".  Whether you buy my view on LLMs, or Agüera y Arcas and Norvig's has little if anything to do with what the economic or social impacts will be.

The implications of a particular technical development, in AI or elsewhere, depend on the specifics of that development and the context it happens in.  While it's tempting to ask "Is it AGI?" and assume that "no" means business as usual while "yes" has vast consequences, I doubt this is a useful approach.

The development of HTTP and cheap, widespread internet connectivity has had world-wide consequences, good and bad, with no AI involved.  Generative AI and LLMs may well be a step toward whatever AGI really is, but at this point, a year after ChatGPT launched and a couple of years after generative AIs like DALL-E came along, it's hard to say just what direct impact this generation of AIs will have.  I would say, though, that the error bars have narrowed.  A year ago, they ranged from "ho-hum" to "this changes everything".  The upper limit seems to have dropped considerably in the interim, while the lower limit hasn't really moved.

Language off in the weeds

While out walking, I paused to look at a stand of cattails (genus Typha) growing in a streambed leading to a pond.  "That's a pretty ..." I thought to myself, but what would be the word for the area they were growing in? Marsh? Wetland? Swamp? Bog? Something else?

I've long been fascinated by this sort of distinction.  If you don't have much occasion to use them, those words may seem interchangeable, but they're not.  Technically

• A wetland is just what it says ... any kind of land area that's wet most or all of the time
• A marsh is a wetland with herbaceous plants (ones without woody stems) but not trees
• A bog is a marsh that accumulates peat
• A swamp is a forested wetland, that is, it does have trees

Wikimedia also has a nice illustration of swamps, marshes and other types of land.  By that reckoning, I was looking at a marsh, which was also the word that came to mind at the time.

This sort of definition by properties is everywhere, especially in dictionaries, encyclopedias and other reference works.  Here, the properties are:

• Is it land, as opposed to a body of water?
• Is it wet most or all of the time?
• Does it have trees?
• Does it accumulate peat?

The first two are true for all of the words above.  For the last two, there are three possibilities: yes, no and don't care/not specified.  That makes nine possibilities in all

Trees?	Peat?	Word
Yes	Yes	peat swamp
Yes	No	?
Yes	Don't care	swamp
No	Yes	bog/peat bog
No	No	?
No	Don't care	marsh
Don't care	Yes	peatland
Don't care	No	?
Don't care	Don't care	wetland

As far as I know, there's no common word for the various types of wetland if they specifically don't accumulate peat.  You could always say "peatless swamp" and so forth, but it doesn't look like anyone says this much.  Probably people don't spend that much time looking for swamps with no peat.

Leaving aside the empty spaces, the table above gives a nice, neat picture of the various kinds of wetland and what to call them.  As usual, this nice picture is deceptive.

• I took the definitions from Wikipedia, which aims to be a reference work.  It's exactly the kind of place where you'd expect to see this kind of definition by properties
• The Wikipedia articles are about the wetlands themselves, not about language.  They may or may not touch on how people use the various words in practice or whether that lines up with the nice, technical definitions
• The way the table is set up suggests that a peatland is a particular kind of wetland, but that's not quite true.  A peatland is land, wet or not, where you can find peat.  Permafrost and tundra can be peatland and often are, but they're not wetlands.  Similarly, a moor is generally grassy open land that might be boggy, if it's low-lying, but can also be hilly and dry.  Both peatlands and moors can be wetlands, but they aren't necessarily
• Even if you take the definitions above at face value, if you have a lake in the middle of some woodlands with a swampy area and a marshy area in between the lake and the woods, there's no sharp line where the woods become swamp, or the swamp becomes marsh, or the marsh becomes lake.

The Wikipedia article for Fen sums this up nicely:

Rigidly defining types of wetlands, including fens, is difficult for a number of reasons. First, wetlands are diverse and varied ecosystems that are not easily categorized according to inflexible definitions. They are often described as a transition between terrestrial and aquatic ecosystems with characteristics of both. This makes it difficult to delineate the exact extent of a wetland. Second, terms used to describe wetland types vary greatly by region. The term bayou, for example, describes a type of wetland, but its use is generally limited to the southern United States. Third, different languages use different terms to describe types of wetlands. For instance, in Russian, there is no equivalent word for the term swamp as it is typically used in North America. The result is a large number of wetland classification systems that each define wetlands and wetland types in their own way. However, many classification systems include four broad categories that most wetlands fall into: marsh, swamp, bog, and fen.

A fen here means "a type of peat-accumulating wetland fed by mineral-rich ground or surface water."  It's that water that seems to make the difference between a bog and a fen: "Typically, this [water] input results in higher mineral concentrations and a more basic pH than found in bogs." (bogs tend to be more acidic).  We could try to account for this in the table above by adding an Acidic? (or Basic?) column, but then we'd have 27 rows with a bunch of question marks in the blank spaces.

In that same paragraph, the article says "Bogs and fens, both peat-forming ecosystems, are also known as mires."  If you buy that definition, it might fit better than peatland in the "trees: don't care, peat: yes" row.

This is all part of a more general pattern: Definitions by properties are a good way to do technical definitions, but people, including technical people when they're not talking about work, don't really care about technical definitions.  For most purposes, radial categories do a better job of describing how people actually use words.  More on that in this post.

All of this is leaving out an important property of bogs and mires: you can get bogged down in a bog and mired in a mire.  Most of these words are old enough that the origins are hard to trace, but bog likely comes from a word for "soft", which more than hints at this (mire is likely related to moss).

This suggests that what we call something depends at least in part on how we experience it.  The interesting part is that defining properties like wetness, grassiness, softness and the presence of peat are also based on experience, which makes untangling the role of experience a bit tricky.

Just because we can distinguish meanings doesn't mean those distinctions are useful, but I'd say they are useful here, and in most cases where we use different words for similar things.  The distinctions are useful because we can draw larger conclusions from them.  For example:

• It's easier to see what's on the other side of a marsh, since there are no trees in the way
• A marsh will be sunnier than a swamp
• There will be different kinds of animals in a swamp than a marsh
• You can get peat from a bog.  Even today, peat is still a useful material, so it's not surprising that it's played a role in how we've used words for places that may or may not have it.
• And it's also not surprising that people talk about peat bogs and peat swamps but generally don't specifically call out bogs and swamps without it.

Even the more general term wetland is drawing a useful distinction.  A wetland is, well, wet.  There's a good chance you could get stuck in the mud in a wetland, or even drown, not something that would happen in a desert unless there had been a downpour recently (which does happen, of course).

Let's take a completely different example: Victorian cutlery.  Upper middle-class Victorian society cared quite a bit about which fork or spoon to use when.  Much of this, of course, is about marking membership in the in-group.  If you were raised in that sort of society, you would Just Know which fork was for dinner and which for salad.  If you didn't know that, you obviously weren't raised that way and it was instantly clear that there could be any number of other things that you wouldn't know to do, or not do (If you ever have to bluff your way through, work from the outside in -- the salad fork will be on the outside since salad is served first -- and don't worry, something else will probably give you away anyway).

However, there are still useful distinctions being made, and they're right there in the names.  A salad fork is a bit smaller and better suited for picking up small pieces of lettuce and such.  A dinner fork is bigger, and better for, say, holding something still while you slice it with a knife.  A soup spoon is bigger than a teaspoon so it doesn't take forever to finish your soup, a dinner knife is sharper than a butter knife, a butter knife works better for spreading butter, and so forth.

It's no different for the impressive array of specialized utensils that one might have encountered at the time (and can still find, in many cases).  A grapefruit spoon has a sharper point with a serrated edge so you can dig out pieces of grapefruit.  A honey dipper holds more honey than a plain spoon and honey flows off it more steadily (unless you have a particularly steady spoon hand) and so on.  I have an avocado slicer with a grabber that makes it much easier to get the pit out.  It's very clear (at least once you've used it) that it's an avocado slicer and not well suited for much else.  You can do perfectly well without such things, but they can also be nice to have.  

Consider one more example: The fondue fork, which has a very long, thin stem and two prongs with barbs on them.  You could call it, say, a barb-pronged longfork, and that would be nice and descriptive.  If someone asked you for a barb-pronged longfork and you had to fetch it from a drawer of unfamiliar utensils, you'd have a pretty good chance of finding it.  If someone asked for a "fondue fork" and you didn't know what that was, you'd pretty much be stuck.  The same is true for grapefruit spoons, dinner knives and so forth.  All language use depends on shared context and assumptions about it.

I think there's something general going on here, that how we experience and interact with things isn't just a factor in how we name them, but central to it.  Even abstract properties like softness or dryness are rooted in experience.  Fens and bogs have different soil characteristics, but the names are much older than the chemical theory behind pH levels.

We call it a fondue fork because it's used for putting bits of food in a fondue pot (and, just as importantly, for getting them back out).  A fondue fork has certain qualities, like the long stem and the barbs, that make it well-suited for that task, but they're not directly involved in how we name it.

Words like fen and bog are distinct because fens and bogs support distinct kinds of plant and animal life, moving through a fen is different from moving through a bog, and so forth.  A difference in pH level is a cause of this difference, but that's incidental.  There are almost certainly areas that are called fens that have bog-like pH levels or vice versa.  You could insist that such a fen (or bog) is incorrectly named, but why?

Properties do play a role in how we name things.  Swamps have trees.  Marshes don't.  A knife has a sharp edge.  A fork is split into two or more tines.  A spoon can hold a small amount of liquid.  What we don't have, though, is some definitive list of properties of things, so that someone presented with a teaspoon could definitively say: "This thing is an eating utensil.  It can hold a small amount of liquid.  That amount is less than the limit that separates teaspoons from soup spoons.  Therefore, it's a teaspoon."

In many contexts, it may look like there is such a list of properties.  With marsh and swamp, we can clearly distinguish based on a property -- trees or no trees.  Sometimes, as with red-winged blackbird or needle-nose pliers, but not for marsh and swamp, we use properties directly to build names for things.

But there are thousands of possible properties for things -- sizes, shapes, colors, material properties, temperature, where they are found, who makes them, and on and on.  Of the beyond-astronomically many possible combinations, only a tiny few describe real objects with real names.

At the very least, there has to be some way of narrowing down what properties might even possibly apply to some class of objects.  Stars are classified by properties like mass (huge) and temperature (very hot by human standards), but we don't distinguish, say, a fugue from a sonata based on whether the temperature is over 30,000 Kelvin.

It's not impossible, at least in principle, to create a decision tree or similar structure for handling this.  You could start with dividing things into material objects, like stars, and immaterial ones, like sonatas and fugues.  Within each branch of the tree, only some of the possible properties of things would apply.  After some number of branches, you should reach a point where only a few possible properties apply.  If you're categorizing wetlands, you know that the temperature classifications for stars don't apply, and neither do the various properties used to classify musical forms, but properties like "produces peat" and "has trees" do apply.

In practice, though, even carefully constructed classification systems based on properties, like the Hornbostel-Sachs system for musical instruments discussed in this post, can only go so far.  Property-based systems of classification tend to emphasize particular aspects of the things being categorized, such as (in the case of Hornbostel-Sachs) how they are built and how sound is produced from them.  This often lines up reasonably well with how we use words, but I don't think properties are fundamental.

Rather, how we experience things is fundamental, or at least closer to whatever is fundamental.  Properties describe particular aspects of how we experience something, so it's not surprising that they're useful, but neither should it be surprising that they're not the whole story.

AI, AI and AI

I have a draft post from just over a year ago continuing a thread on intelligence in general, and artificial intelligence in particular.  In fact, I have two draft AI posts at the moment.  There's also one from early 2020 pondering how the agents in Open AI's hide-and-seek demo figure out that an agent on the opposing team is probably hiding out of sight behind a barrier.

I was pretty sure at the time of the earlier draft that they do this by applying the trained neural network not just to the last thing that happened, but a window of what's happened recently.  In other words, they have a certain amount of short-term memory, but anything like long-term memory is encoded in the neural net itself, which isn't updated during the game.  This ought to produce effects similar to the "horizon effect" in early chess engines, where a player that could look ahead, say, three moves and see that a particular move was a blunder would play another move that led to the same blunder, but only after four moves.

I'm still pretty sure that's what's going on, and I was going to put that into a post one of these days as soon as I read through enough of the source material to confirm that understanding, but I never got around to it.

Because ChatGPT-4 happened.

ChatGPT is widely agreed to have been a major game changer and, yeah ... but which games?

From a personal point of view, my musings on how AIs work and what they might be able to do are now relevant to what my employer talks about in its quarterly earnings reports, so that put a damper on things as well.  For the same reason, I'll be staying away from anything to do with the internals or how AI might play into the business side.  As usual, everything here is me speaking as a private citizen, about publicly available information.

Out in the world at large, I recall a few solid months of THIS CHANGES EVERYTHING!!!, which seems to have died down into a steady stream of "How to use AI for ..." and "How to deal with the impact of AI in your job."  I've found some of this interesting, but a lot of it exasperating, which leads me to the title.

There are at least three very distinct things "AI" could reasonably mean at the moment:

• The general effort to apply computing technology to things that humans and other living things have historically been much better at, things like recognizing faces in pictures, translating from one language to another, walking, driving a car and so forth.
• Large language models (LLMs) like the ones behind ChatGPT, Bard and company
• Artificial General Intelligence (AGI), a very vague notion roughly meaning "able to do any mental task a human can do, except faster and better"

There are several posts under the AI tag here (including the previous post) poking and prodding at all three of those, but here I'm interested in the terms themselves.

To me, the first AI is the interesting part, what I might even call "real AI" if I had to call anything that.  It's been progressing steadily for decades.  LLMs are a part of that progression, but they don't have much to do with, say, recognizing faces or robots walking. All of these applications involve neural networks with back propagation (I'm pretty sure walking robots use neural nets), but training a neural net with trillions of words of speech won't help it recognize faces or control a robot walking across a frozen pond because ... um, words aren't faces or force vectors?

If you ask a hundred people at random what AI is, though, you probably won't hear the first answer very much.  You'll hear the last two quite a bit, and more or less interchangeably, which is a shame, because they have very little in common.

LLMs are a particular application of neural nets.  They encode interesting information about the contents of a large body of text.  That encoded information isn't limited to what we think of as the factual content of the training text, and that's a significant result.  For example, if you give an LLM an email and ask it to summarize the contents, it can do so even though it wasn't explicitly trained to summarize email, or even to summarize unfamiliar text in general.

To be clear, summarizing an email is different from summarizing part of the text that an LLM has been trained on.  You could argue that in some very broad sense the LLM is summarizing the text it's been trained on when it answers a factual question, but the email someone just sent you isn't in that training text.

Somehow, the training phase has built a model, based in part on some number of examples of summaries, of how a text being summarized relates to a summary of that text.  The trained LLM uses that to predict what could reasonably come after the text of the email and a prompt like "please summarize this", and it does a pretty good job.

That's certainly not nothing.  There may or may not be a fundamental difference between answering a factual question based on text that contains a large number of factual statements and performing a task based on a text that contains examples of the task, or descriptions of the task being done, but in any case an LLM summarizing an email is doing something that isn't directly related to the text it's been trained on, and that it wasn't specifically trained to do.

I'm pretty sure this is not a new result with LLMs, but seeing the phenomenon with plain English text that any English speaker can understand is certainly a striking demonstration.

There are a couple of reasons to be cautious about linking this to AGI.

First, to my knowledge, there isn't any widely-accepted definition of what AGI actually is.  From what I can tell, there's a general intuition of something like The Computer from mid 20th-century science fiction, something that you can give any question or task to and it will instantly give you the best possible answer.  

"Computer, what is the probability that this planet is inhabited?"

"Computer, devise a strategy to thwart the enemy"

"Computer, set the controls for the heart of the Sun"

"Computer, end conflict among human beings"

This may seem like an exaggeration or strawman, but at least one widely-circulated manifesto literally sets forth that "Artificial Intelligence is best thought of as a universal problem solver".

There's quite a bit of philosophy, dating back centuries and so probably much, much farther, about what statements like that might even mean, but whatever they mean, it's abundantly clear by now that an LLM is not a universal problem solver, and neither is anything else that's currently going under the name AI.

In my personal opinion, even a cursory look under the hood and kicking of the tires ought to be enough to determine that nothing like a current LLM will ever be a universal problem solver.  This is not just a matter of what kinds of responses current LLM-based chatbots give.  It's also a matter of what they are.  The neural net model underpinning this is based pretty explicitly on how biological computers like human brains work.  Human brains are clearly not universal problem solvers, so why would an LLM be?

There's an important distinction here, between "general problem solver", that is, able to take on an open-ended wide variety of problems, and "universal", able to solve any solvable problem.  Human brains are general problem-solvers, but nothing known so far, including current AIs, is universal.

This may sound like the argument that, because neural nets are built and designed by humans, they could never surpass a human's capabilities.  That's never been a valid argument and it's not the argument here.  Humans have been building machines that can exceed human capabilities for a long, long time, and computing machines that can do things that humans can't have been around for generations or centuries, depending on what you count as a computing machine and a human capability.

The point is that neural nets, or any other computing technology, have a particular set of problems that they can solve with feasible resources in a feasible amount of time.  The burden of proof is on anyone claiming that this set is "anything at all", that by building a network bigger and faster than a human brain, and giving it more data than a human brain could hope to handle, neural nets will not only be able to solve problems that a human brain can't -- which they already can -- but will be able to solve any problem.

So next time you see something talking about AI, consider which AI they're referring to.  It probably won't be the first (the overall progress of machines doing things that human brains also do).  It may well be the second (LLMs), in which case the discussion should probably be about things like how LLMs work or what a particular LLM can do in a particular context.

If it's talking about AGI, it should probably be trying to untangle what that means or giving particular reasons why some particular approach could solve some particular class of problems.

If it's just saying "AI" and things on the lines of "Now that ChatGPT can answer any question and AGI is right around the corner ...", you might look for a bit more on how those two ideas might connect.

My guess is there won't be much.

AI seems to be back. What is it now?

In one of the earliest posts here, several years ago, I mused What was AI?  At the time, the term AI seemed to have fallen out of favor, at least in the public consciousness, even though there were applications, like phones with speech recognition, that were very much considered AI when they were in the research phase.  My conclusion was that some of this was fashion, but a lot of it was because the popular conception of AI was machines acting like humans.  After all, the examples we saw in pop culture, like, say, C-3PO in Star Wars, were written by humans and portrayed by humans.

There's a somewhat subtle distinction here: A phone with speech recognition is doing something a human can do, but it's not acting particularly like a human.  It's doing the same job as a human stenographer, whether well or badly, but most people aren't stenographers, and even stenographers don't spend most of their time taking dictation (or at least they shouldn't).

Recently, of course, there's been a new wave of interest in AI, and talk of things like "Artificial General Intelligence", which hadn't exactly been on most people's lips before ChatGPT-4 came out.  To avoid focusing too much on one particular example, I'll call things like ChatGPT-4 "LLM chatbots", LLM for Large Language Model.

By many measures, an LLM chatbot isn't a major advance.  As the "-4" part says, ChatGPT-4 is one in a series, and other LLM chatbots were developed incrementally as well.  Under the hood, an LLM chatbot is a particular application of neural net-based machine learning, which was a significant advance, to the particular problem of generating plausible-sounding text in response to a prompt.

But goodness, do they produce plausible-sounding text.

A response from an LLM chatbot may contain completely made-up "facts", it may well break down on closer examination by followup questions or changing the particulars of the prompt, and it may have a disturbing tendency to echo widely-held preconceptions whether they're accurate or not, but if you just read through the response and give it the benefit of the doubt on anything you're not directly familiar with, something people are strongly inclined to do, then it sounds like the response of someone who knows what they're talking about.  The grammar is good, words are used like people would use them, the people and things mentioned are generally real and familiar, and so on.

In other words, when it comes to generating text, an LLM chatbot does a very good job of acting like a human.  If acting like a human is the standard for AI, then an LLM chatbot is definitely an AI, in a way that a speech-transcribing phone app or a model that can pick out supernovae from a mass of telescope images just isn't.

But our perception of whether something is acting intelligent in a human way is heavily tilted toward language use.  All kinds of animals can recognize images and many can respond to speech, but only we can produce large volumes of text in human languages in response to a prompt.  Until now, that is.

Since LLM chatbots are an obvious major advance in acting-like-a-human, it's natural to assume that they represent a major advance in understanding what intelligence is generally, but those are two very different things.  As far as I can tell, we're not really any closer to understanding what "general intelligence" might mean, or how to produce it, than we were before.

To be sure, LLMs have shown some interesting behaviors that may offer hints as to what intelligence might be.  Once the model gets big enough, it seems to get better at learning from small samples.  For example, if you train a model on a huge amount of English text and a little bit of Italian and a little bit of Python code, it seems that it can do a pretty good job of producing plausible Italian or plausible-looking Python.  The Python code might well not do anything useful, unless it's close to one of the examples that the model was trained on, and it might not even build, but it looks like Python someone wrote.  There's some question as to whether this ability arises suddenly once the model hits a certain size, or actually happens gradually, but it's definitely a thing.

It seems likely that, after enough training data and with enough depth to the underlying network, an LLM develops connections that in some way represent something general about the structure of languages, at least ones like Italian and Python that are reasonably close in structure to the English that makes up the bulk of the training data.  It would be interesting to explore that, because it may capture something about language structure that current theories of grammar miss.

It seems reasonable that a suitably deep model trained on larger amounts of language would be able to capture something useful about the structure of language.  Beyond that, though, is it possible that an LLM can capture something about the world that the body of text is talking about?  If the training data contains millions of words about water, and rivers, and thirst, and drinking water, and clouds, rain and so forth, will it in some sense know that water is wet and that rain falls from clouds and some of it drains into rivers that flow into the sea, and that living things need water and that if a person is thirsty they can drink water and not be thirsty any more?

From interacting with an LLM chatbot, it's certainly easy to conclude at first that the answer is yes.  For example, I just asked a chatbot "If I'm thirsty and I drink water, will I still be thirsty?" and got back

Generally, if you are thirsty and you drink water, you will no longer be thirsty. This is because thirst is a signal from your body that it needs water. When you drink water, your body's water levels will increase and the thirst signal will be turned off.

However, there are some cases where you might still be thirsty after drinking water ...

and so on.  That seems like case closed.  One good way to find out if someone knows something is to ask them questions about it and see if the answers make sense.  I did, and the answer made sense.

The problem is, LLM chatbots have a tendency to go down rabbit holes once you get into details.  Later, I asked "If I'm thirsty and I drink ambrosia, will I still be thirsty?"  The answer starts off reasonably:

In Greek mythology, ambrosia is the food of the gods. It is said to be a delicious and intoxicating beverage that grants immortality. If you were to drink ambrosia, you would no longer feel thirst, as it would quench your thirst permanently.

So far, so good.  Going on .. 

However, ambrosia is not a real substance, so it is impossible to say for sure what would happen if you drank it.

This is a bit ... off.  If ambrosia is not a real substance, then you can't drink it, period.  It's not impossible to be sure what would happen, it's impossible to say, period.  The answer goes on to conclude:

However, it is an interesting question to ponder, and it is one that has been explored in many works of fiction.

OK, I'll bite.  "What are some works of fiction that explore the drinking of ambrosia?"  By "explore", I'd expect something like a storyline that features a person, or multiple people, drinking ambrosia in various amounts or circumstances and discusses the consequences of the various possibilities.  I wouldn't want something that just "features" or "includes" the drinking of ambrosia.

Nitpicking?  I mean, isn't it close enough just to mention some stories that include ambrosia and count that as exploring?

No.  If the chatbot has a true understanding of the concepts behind the words, it will catch this distinction.  If it has merely picked up from its training data that "explores" is a word often used when discussing fiction, it won't.  In my experience, LLM chatbots miss this sort of distinction over and over again.

So what works did it come up with?

• The Odyssey by Homer: In this epic poem, the hero Odysseus drinks ambrosia and nectar while he is a guest of the gods on Mount Olympus. This gives him the strength and vitality to continue his journey home.

The first time through, I took this at face value.  It's been a while since I read the Odyssey.  But it doesn't look like this scene actually occurs.  Ambrosia is mentioned, briefly, in other contexts.  Even if this scene were in the poem, would it really be exploring the drinking of ambrosia?  Not really.  It just records the common understanding in his time and place of what ambrosia was and what it did.

• The Bacchae by Euripides: In this tragedy, the god Dionysus gives ambrosia to his followers, which makes them immortal. However, the ambrosia also makes them insane, and they go on a rampage, killing people and destroying property.

That would count as an exploration in my book.  One problem, though: That's not the plot of The Bacchae.  From what I can make out, without carefully reading through the whole play, there's no mention at all of ambrosia in The Bacchae.

• The Tempest by William Shakespeare: In this play, the magician Prospero uses ambrosia to create a magical elixir that can grant youth, beauty, and immortality.

Nope.  The word "ambrosia" is not even in the play.

• The Last Olympian by Rick Riordan: In this book in the Percy Jackson series, the Greek gods give ambrosia to the demigod heroes to heal their wounds and give them strength.

That's probably about right -- I'm not going to take time to look, given the rest of the entries -- but again, not really exploring the question of what it would be like to drink ambrosia.

• The Mortal Instruments by Cassandra Clare: In this series of fantasy novels, the main characters drink ambrosia to heal their wounds and give them strength.

As far as I can tell, this is not so.

OK, so maybe the chatbot understands water perfectly well, but it's not good with literature and finer points like what's in it?  Let's try something more direct.

"If I have a bell, a book, a candle and some water, how can I stack them?"

(I picked stacking things because I ran across a pretty impressive example in a news item, of a chatbot suggesting a solution the author hadn't thought of.  I have no doubt this happened, but I strongly suspect that the author took the impressive example as "proof" and didn't press on by, say, asking variants of the original question.)

• There are a few ways to stack a bell, a book, a candle, and some water. Here are a few examples:
• Bell, book, candle, water: This is the most straightforward way to stack the items. The bell would be on the bottom, followed by the book [somehow balanced on top of the bell ...], the candle, and the water.
• Water, bell, book, candle: This stack is a bit more challenging, as the water would need to be supported by the bell and the book [wait, what?]. However, it is possible to do this by placing the water in a small container, such as a cup or a bowl [so in the first stack, the water isn't in a container?].
• ...
  

I think the easiest explanation for this one is that the chatbot doesn't really understand what physical objects and substances are and how they interact.

In some meaningful sense, the model understands something interesting about the structure of languages, or at least some languages, because it was trained on a mass of text.  It can also model associations among words, for example that "water" tends to be associated with words like "wet" and "drink" and "flow".

However, since its training had nothing to do with actual water, or anything else in the physical world, it's no surprise that it doesn't show any real understanding of how things interact in the physical world.  Since the training text doubtless included reviews and discussions of various works of fiction, it can imitate something talking about fiction, but it misses details like the meaning of "explore", or even that if you say a story explores something, that thing should actually appear in the story.

So, after that fairly long digression, how does all this fit together?

• Except among people closely associated with the research, "AI" generally means "acting like a human" and "doing things that are particularly human", like using language
• LLM chatbots are pretty good at acting human in that sense ...
• ... including making up plausible-looking responses when they don't really know the answer
• But language is just one part of the picture.
• "General Intelligence" is not at all well-defined, but if it includes some sort of general understanding of the world and how to solve problems in it, then there's no real reason to think LLM chatbots have it, or are even close to acquiring it ...
• ... even if they're sometimes good at looking that way