Metacognition, metaphor and AGI

In the recent post on abstract thought, I mentioned a couple of meta concepts: metacognition and metaphor.

• Metacognition is the ability to think about thinking.  I've discussed it before, particularly in this post and these two posts.
• Metaphor is a bit harder to define, though there is no shortage of definitions, but the core of it involves using the understanding of one thing to understand a different thing.  I've also discussed this before, particularly in this post and this one.

When I was writing the post on abstract thought, I had it in mind that these two abilities have more to do with what we would call "general intelligence" (artificial or not), so I wanted to try to get into that here, without knowing exactly where I'll end up.

In that earlier post, I identified two kinds of abstraction:

• Defining things in terms of properties, for example, a house is a building that people live in.  I concluded that this isn't essential to general intelligence.  At this point, I'd say it's more a by-product of how we think, particularly how we think about words.
• Identifying discrete objects (in some general sense) out of the stream of sensory input we encounter, for example, being able to say "that sound was a dog barking".  I concluded that this is basic equipment for dealing with the world.  At this point, I'd say it's worth noting that LLMs don't do this at all. They have it done for them by the humans that produce the words they're trained on and receive as prompts.  On the other hand specialized AIs, like speech recognizers, do exactly this.

It was the first kind of abstraction that led me back to thinking about metaphor.

Like the second kind of abstraction, metaphor is everywhere, to the point that we don't even recognize it until we think to look.  For example:

• the core of it (a concept has a solid center, with other, softer parts around it)
• I had it in mind (the mind is a container of ideas)
• I wanted to try to get into that (a puzzle is a space to explore; you know more about it when inside it than outside)
• without knowing exactly where I'll end up (writing a post is going on a journey, destination unknown)
• at this point (again, writing a post is a journey)
• this is basic equipment (mental abilities are tools and equipment)
• led me back to thinking (a chain of thought is a path one can follow)
• to the point (likewise)

While there's room for discussion as to the details, in each of those cases I'm talking about something in the mind (concepts, the process of writing a blog post ...) in terms of something tangible (a soft object with a core, a journey in the physical world ...).

Metaphor is certainly an important part of intelligence as we experience it.  It's quite possible, and I would personally say likely, that the mental tools we use for dealing with the physical world are also used in dealing with less tangible things.  For example, the mental circuitry involved in trying to follow what someone is saying probably overlaps with the mental circuitry involved in trying to follow someone moving in the physical world.

This would include not only focusing one's attention on the other person, but also building a mental model of the other person's goals so as to anticipate what they will do next, and also recording what the person has already said in a similar way to recording where one has already been along a path of motion.  If some of the same mental machinery is involved in both processes -- listening to someone speak, and physically following them -- then on some level we probably experience the two similarly.  If so, it should be no surprise that we use some of the same words in talking about the two.

The overlap is not exact, or else we actually would be talking about the same things, but the overlap is there nonetheless.  This can happen in more than one way at the same time.  If you're speaking aggressively to me, I might experience that in a similar way to being physically menaced, and I might say things like Back off or Don't attack me, even while I might also say I'm not following you if I can't quite understand what you're saying, but I still feel like it's meant aggressively.

It's interesting that these examples of metaphor, about processing what someone is saying, also involve metacognition, thinking about what the other person is thinking.  That's not always the case (consider this day is just rushing by me or it looks like we're out of danger).  Rather, we use metaphor when thinking about thinking because we use metaphor generally when thinking about things.

If you buy that metaphor is a key part of what we think of as our own intelligence, is it a key part of what we would call "general intelligence" in an AI?  As usual, that seems more like a matter of definition.  I've argued previously that the important consideration with artificial general intelligence is its effect.  For example, we worry about trying to control a rogue AI that can learn to adapt to our attempts to control it.  This ability to adapt might or might not involve metaphor.  It might well involve metacognition -- modeling what we're thinking as we try to control it, but maybe not.

Consider chess engines.  As noted elsewhere, it's clear that chess engines aren't generally intelligent, but it's also clear that they are superhuman in their abilities.  Human chess players clearly use metaphor in thinking about chess, not just attack and defense, but space, time, strength, weakness, walls, gaps, energy and many others.  Classic AB chess engines (bash out huge numbers of possible continuations and evaluate them using an explicit formula) clearly don't use metaphor.

The situation with neural network (NN) engines (bash out fewer possible continuations and evaluate them using a neural net) is slightly muddier, since in some sense the evaluation function is looking for similarities with other chess positions, but that's the key point: the NN is comparing chess positions to other chess positions, not to physical-world concepts like space, strength and weakness.  You could plausibly say that NNs use analogy, but metaphor involves understanding one thing in terms of a distinct other thing.

Likewise, neither sort of chess engine builds a model of what its opponent is thinking, only of the possible courses of action that the opponent might take, regardless of how it decides to take them.  By contrast, human chess players very frequently think about what their opponent might be thinking (my opponent isn't comfortable with closed positions, so I'm going to try to lock up the pawn structure).  Human chess players, being human, do this because we humans do this kind of thing constantly when dealing with other people anyway.

One the one hand, metaphors only become visible when we use words to describe things.  On the other hand, metaphor (I claim here) comes out of using the mental machinery for dealing with one thing to deal with another thing (and in particular, re-using the machinery for dealing with the physical world to deal with something non-physical).  More than that, it comes out of using the same mental machinery and, in some sense, being aware of doing it, if only in experiencing some of the same feelings in each case (there's a subtle distinction here between being aware and being consciously aware, which might be interesting to explore, but not here).

If we define an AGI as something of our making that is difficult to control because it can learn and adapt to our attempts to control it, then we shouldn't assume that it does so in the same ways that we do.  Meta-thought like explicitly creating a model of what someone (or something) else is thinking, and using metaphor to understand one thing in terms of another may be key parts of our intelligence, but I don't see any reason to think they're necessarily part of being an AGI in the sense I just gave.

The other half of this is chains of reasoning like "If this AI can do X, which is key to our intelligence, then it must be generally intelligent like we consider ourselves to be" rests on whether abilities like metacognition and metaphorical reasoning are sufficient for AGI.

That may or may not be the case (and it would help if we had a better understanding of AGI and intelligence in general), but so far there's a pretty long track record of things, for example being able to deal with natural language fluently, turning out not to necessarily lead to AGI.

Words and their senses

What does "out" mean?  Well, it's the opposite of "in", whatever that may mean.  Except, not in all cases.  Sure, "I went out the door" means pretty much the opposite of "I went in the door", but "I've got an in at the State Department" means I know someone there and I might be able to use that connection to advantage, while "I have an out at the State Department" means ... maybe it means I have an escape route from my current predicament via the State Department, maybe because of the in I have there?

Just off the top of my head, "out" might mean any of a dozen things:

1. adj Unconscious ("He's out like a light")
2. adj No longer holding a position ("She's out as CEO of Frobco")
3. adj Retired as a batter according to the rules of baseball ("He's out at third")
4. n An instance of retiring a batter according to the rules of baseball ("That's the third out")
5. adj Openly known to have a particular sexual orientation ("She's an out bisexual")
6. v To reveal someone to have a particular sexual orientation ("He outed her as a bisexual")
7. v To reveal someone to have a clandestine role or identity ("The administration inadvertently outed a CIA agent")
8. adj Not a member of a particular social group ("They always make me feel so out")
9. n A potential means of escape from a situation ("In tense negotiations, it's always good to have an out")
10. adj Not in tune ("I  think your A string is out")
11. adv Deliberately in a different key and/or time signature from the rest of an ensemble ("Vernon likes to play really out when he solos")
12. adj Not at home ("Sorry, Mr. Smith is out")

and so forth.  It's easy to come up with more.  I've by no means hit even all the usual senses.  My laptop's dictionary gives nine adverbial senses, nine adjectival senses, one prepositional, three nouns and two verbs.  I've heard of lists of over a hundred senses of the word.  It depends on how finely you want to slice it.  I'll come back to that.

Clearly some of these senses are more closely related than others.  The baseball senses 3 and 4 are clearly related, with the noun sense derived from the adjective.  Senses 5, 6 and 7 are clearly related, the verb sense has got to be derived from the adjective sense, and that must be related to phrases like "the secret is out".  On the other hand, it's hard to see how senses 3 and 4 have much to do with senses 5 - 7.

Nonetheless, the use of "out" (or any other word) is not completely arbitrary.  There is a well-developed linguistic theory behind this (well, probably several, but I happen to like this particular one).  At the core, a word has a small number of well-defined senses.  In the case of "out", there is a boundary enclosing something.  What's enclosed is in.  On the other side is out.  Thus when you go out a door, you are going from the space you were in, into another space (which may also be enclosed -- you can go out of one room and into another).  You can draw a circle on the ground and say someone or something is in the circle or out of it, and so forth.

From these basic senses, we build new senses metaphorically.  We can imagine a boundary between the members of a social group and anyone out of the group.  We can talk about entering or leaving a state of being ("Moishe led us out of slavery"), and so forth.

Crucially, these metaphors are not just literary fancy.  They are directly meaningful and productive (in the linguistic sense that we can spontaneously create new instances of the metaphor).  If a social group has a boundary, we don't just say someone is in the group or out of it.  We can welcome someone in.  We can kick someone out.  We can expand the group.  We can designate an inner circle, and so forth.  If we're feeling more creative, we can say something that sounds more "metaphorical" in the usual sense ("The boundaries of the group were porous.  People floated in and out with the tides.")

In the case of "out" we can start with the basic boundary-oriented sense and build, well, outwards:

• A container can have something in it, and you can take things out of it
• If the container holds a fluid, you can pour it out of the container
• or it can run out (or flow out, our seep out, or pour out)
• When there's no longer anything in the container, you have run out of whatever was in it (or you're just out of it)
• Any resource can be considered to be a fluid, so you can run out of time (or say time is running out), run out of money, or energy, or whatever.
• The resource need not be physical.  You can run out of patience.
• A burning lamp or candle consumes its oil or wax.  When there's none left -- it has run out -- the lamp or candle has burned out.
• Likewise, any flame can burn out as it runs out of fuel, for example a rocket can burn out
• Someone exhausted is burned out
• An electric light performs the same role as a lamp or candle.  When you shut off the current to it, it is out.
• A person's consciousness is likened to a light or flame, and an unconscious person is out (sense 1 above).  You can also drift in and out of consciousness, but that's considering a state of being as something bounded -- a separate metaphor.

Coming back toward the core senses

• A social group or position is a bounded area, metaphorically.
• As noted above, one can be in or out of a group or position, or leave it (senses 2 and 8; also, being in a job or out of it, leaving a job, etc.)
• Information can be regarded as a substance (you can share it, hide it, have a lot or a little of it, etc.).  You can't see through an opaque container, but once the information is out of the container (out in the open), others can know it (senses 5-7 ... the secret is out)
• Obviously, you can be in or out of your home (sense 12)
• We can speak of some quantity being within or outside of given limits, a special case of a state of being as a bounded area.
• A note too far from its correct pitch is out of tune (sense 10)
• A musician diverging from the key and/or time signature of a piece is clearly crossing those limits (sense 11)
• A difficult situation limits one's potential actions, like being in a confined space.  It's good to have a way out of such a space (sense 9)

That leaves baseball.  The baseball usage derives from the game of cricket, where a batsman stands in his ground, guarding a wicket.  When the wicket falls (for any of a number or reasons I won't even try to enumerate), the batsman is out of his ground, or just out  (well, you can also be out of your ground but not out, for example if you're running between the wickets, but clearly out is related to in/out of one's ground).

So there you have it.  By a series of metaphors and analogies, we can connect seemingly unrelated senses of a word, like senses 1, 4 and 7 above, back to basic, familiar and physically-based concepts.

Of course, if you try hard enough, you can connect anything to anything.  I remember learning the following chain of reasoning as a kid, explaining why fire engines were red, starting with a very basic premise:

• 1+1 = 2
• 2+2 = 4
• 4+4 = 8
• 8+4 = 12
• There are 12 inches in a ruler (at least in the US)
• Queen Mary was a ruler
• The Queen Mary was a ship
• Ships sail in the ocean
• The ocean is full of fish
• Fish have fins
• The Finns fought the Russians
• Russians are red (this was back in the Cold War days)
• Fire engines are always rushin'
• And that's why fire engines are red

If only fire engines were still red.

How do we know that this whole sense-extension exercise isn't just another chain of silly reasoning?  A metaphor doesn't just relate anything to anything else.  There are two basic rules:

• One thing will be more concrete than the other.  E.g., holding a substance in a container is more concrete than holding information in one's mind.
• The metaphor connects the two things coherently, in multiple places.  You can bring things you know about the more concrete thing to the more abstract thing, at least until the metaphor breaks down.  In an example from a previous post "The stop sign was a fire engine" is not a coherent metaphor, even if both are red.  On the other hand, if you have information in your head, you can give it out.  You can have a lot or a little of it.  You can have information crammed into your head until it's about to explode, and so forth.

Even if the explanation above isn't airtight, I hope I've at least presented a plausible case that the senses in which we use words can be explained reasonably well by this sort of metaphoric extension, plus some other rules covering, for example, how we convert adjectival senses like sense 3 to noun senses like sense 4.  Real linguists have, of course, studied this in much more detail.

How many senses does a word have?  Some, like copernicium or Pinophyta, probably only have one, but even highly technical terms like algebraic can have multiple, related senses, following the same core-with-extensions pattern as anything else.  Some very basic words, like the prepositions or head and go have large numbers even by the relatively conservative standards of dictionaries, particularly when you count idioms like head out or go out.  Most words we commonly use have at least a few widely-used senses, and likely several more specialized senses, particularly if you include slang -- which you always should, if you're really trying to understand a language.

But how finely should we really slice?  Senses 6 and 7 above are very nearly the same thing.  Historically, as far as I'm aware, sense 6 precedes sense 7, but is this because outing someone's orientation is a different thing from outing someone's clandestine role, or because the sense of "revealing something that had been secret" was at first only applied to orientation?

I would lean toward that latter, if only for practical reasons.  Otherwise we would have to consider each application of a word its own sense.  Being out of the position of CEO would be a different sense from being out of the position of vice president, or janitor.

All right, then, but then couldn't we argue that all uses of "out" are just different applications of the basic idea of being outside a boundary?  It seems clear that there's something in common at the core of the various senses above, but saying that they're all just uses of the same basic word leaves something out [ahem].

This is especially apparent if you're trying to paraphrase or translate a word.  In senses 6 or 7, we can use basically the same paraphrase: He revealed her to be bisexual; The administration inadvertently revealed the identity of a CIA agent, and sense 5, apart from using a different part of speech, more or less works.  She's a revealed bisexual sounds a bit funny, but it at least makes sense, as opposed to He's revealed like a light for sense 1, or He's revealed at third for sense 3.  On the other hand The secret is revealed works fine for The secret is out.  The paraphrase test isn't foolproof, but it seems like a good starting point.

From the point of view of metaphors, a sense of a word corresponds to a particular metaphor.  Senses 5-7 and The secret is out use the metaphor of information being a substance in an opaque container.  Sense 1 uses a series of metaphors:  consciousness is a light, lights can be out because lamps and candles have fuel and fuel can run out.  If different paraphrases or translations are different ways of expressing the metaphor, then the paraphrase test makes sense.  In short, if you're applying the same metaphor in a new context, you're using the word in the same sense, and if you're not, you're not.

Data, metaphorically

Hmm ... haven't been in here in a while.  Everything still looks OK, just a bit musty.  Let's open up the curtains, blow the dust off the bookends.  Ah ... better.

Now where was I?

In the previous post, I tried to find an ordinary mass noun that behaved like data in its mass noun form, but without great success.  I'm not going to try to fix that here.  In fact, I'm going to try to explain why the effort failed, and to do that I want to explore how data behaves, metaphorically.  But first a bit about metaphor.

Metaphor generally connotes figurative speech used for poetic effect, whether well

That time of year thou mayst in me behold
When yellow leaves, or none, or few, do hang
Upon those boughs which shake against the cold,
Bare ruined choirs, where late the sweet birds sang.

or perhaps not so well

Head down into the storm they went, pressing barehanded to their chests an unshielded sense of peril.

Um ... right.

That's all well and good, but it's not the whole story.  Even the definition your English teacher gave was probably more like "A comparison made by referring to one thing as another."*  That's closer to Aristotle's definition and to the etymology from the Greek for "carry over", and in my view it's an apt one.  There is a strong case to be made that metaphor in this sense is not merely a figure of speech reserved for flowery poetry and purple prose, but rather a fundamental aspect of how we think, whether we put those thoughts into words or not.

Lakoff and Johnson, for example, make this case in Metaphors We Live By, which pulls together dozens of examples of particular metaphors and shows how, taken together, they imply underlying mental metaphors.  Far from grinding away at a desk in English class to produce a figure of speech that will survive the dreaded red pen, we effortlessly produce metaphors -- in Aristotle's sense -- in nearly every sentence.  These metaphors, on the order of "more is up/less is down" and "anger is a hot liquid" (it can boil over, you can get rid of it by blowing off steam, it behaves as a mass noun ...), are so pervasive we don't even see them as metaphors unless we look -- at which point we see them everywhere.

(To get the flavor, go over that last paragraph.  Clearly "grinding away" is metaphoric, but so is "see" in "see them as metaphors", "pulls together" and even "in" in "in this sentence".  Well, a sentence isn't really a container or bounded space, is it?)

It's perfectly normal, indeed probably universal, to have more than one metaphoric view of a concept, and that the different views don't have to be consistent.  For example, we can view ourselves as moving through time ("Let's just get through today.") or ourselves as stationary with time moving past ("What's coming next week?") depending on what works best at the moment.

So, from this point of view, what is data (in the computing sense)?

• It's a fluid.  It can flow or otherwise move from place to place.  It can leak.  It can fill up space.  It can also be compressed, but generally it acts more like a liquid than a gas.  If your data isn't flowing fast enough, you need a bigger pipe.
• It's made up of discrete parts, ultimately bits.  It can be partitioned into chunks of uniform or varying size.  You can change parts independently, but only down to the bit level.
• It's something of value. It can be secured, tampered with, stolen, bought, sold or given away.
• It's a form of text.  It can be written, read, erased and copied.
• You can search through it, organize it and make it universally accessible and useful ... wait, where did that come from?

I'm sure that with a little more thought I could come up with several more metaphors for data, but I think that's enough to make two points: First, that data, like very many other concepts, can be described by internally consistent metaphors, and second, because these metaphors, as with those for other concepts, aren't always consistent with each other, there's no one concrete noun that could serve as a universal metaphor for data.  In other words, trying to fit water or stone or gravel or rice to data as a whole was doomed to failure from the beginning.

That's to be expected, I suppose.  One definition of equality is that two things are the same if they can stand in for each other in all circumstances.  If it looks like rice, tastes like rice and is generally like rice in every observable way, then we may as well say it is rice.  Which leads me to a different definition of metaphor that I don't like nearly as much as the one I used:  A comparison of two unlike things that have something in common.

That's fine as far as it goes, and in particular the things in a metaphor do have to be unlike, but it implies that the things being compared are interchangeable.  They aren't.  One thing is being explained by referring to it as the other.  Moreover, the thing be explained is always more abstract than the thing being referred to.  In the first data example, something very abstract (data), is being referred to as something more concrete (a fluid, for example).

As always, the definition you choose makes a difference.  Seeing metaphor as a comparison between two unlike things with something in common provides a formula for incoherent images (What do you mean "The stop sign was a fire truck." isn't a good metaphor?  They're unlike but they're both red!).  Seeing metaphor as one thing carried over to stand in for another -- the original metaphor for metaphor -- opens up a vast and surprising new world.






* It took me a while to find a definition I liked.  This one is courtesy of Gideon Burton's Silva Rhetoricae.