ML-friendly problems and unexpected consequences

This started out as "one more point before I go" in the previous post, but it grew enough while I was getting that one ready to publish that it seemed like it should have its own post.

Where machine learning systems like LLMs do unexpectedly well, like in mimicking our use of language, it might not be because they've developed unanticipated special abilities. Maybe ML being good at generating convincing text says as much about the problem of generating convincing text as it does about the ML doing it.

The current generation of chatbots makes it pretty clear that producing language that's hard to distinguish from what a person would produce isn't actually that hard a problem, if you have a general pattern-matcher (and a lot of training text and computing power). In that case, the hard part, that people have spent decades trying to perfect and staggering amounts of compute power implementing, is the general pattern-matcher itself.

We tend to look at ML systems as problem solvers, and fair enough, but we can also look at current ML technology as a problem classifier. That is, you can sort problems according to whether ML is good at them. From that point of view, producing convincing text, recognizing faces, spotting tumors in radiological images, producing realistic (though still somewhat funny-looking) images and videos, spotting supernovas in astronomical images, predicting how proteins will fold, along with many other problems, are all examples of pattern-matching that a general ML-driven pattern-matcher can solve as well as, or even better than, our own naturally evolved neural networks can.

Not knowing a better term, I'll call these ML-friendly problems. In the previous post, I argued that understanding the structure of natural languages is a separate problem from understanding what meaning natural language is conveying. Pretty clearly, understanding the structure of natural languages is an ML-friendly problem. If you buy that understanding meaning is a distinct problem, I would argue that we don't know one way or another whether it's ML-friendly, partly, I would further argue, because we don't know nearly as much about what that problem involves.

From about 150 years ago into the early 20th century, logicians made a series of discoveries about what we call reasoning and developed formal systems to describe it. This came out of a school of thought, dating back to Leibniz (and as usual, much farther and wider if you look for it), holding that if we could capture rules describing how reasoning worked, we could use those rules to remove all uncertainty from any kind of thought.

Leibniz envisioned a world where, "when there are disputes among persons, we can simply say: Let us calculate, without further ado, to see who is right". That grand vision failed, of course, both because, as Gödel and others discovered, formal logic has inescapable limitations, but also because formal reasoning captures only a small portion of what our minds actually do and how we reason about the world.

Nonetheless, it succeeded in a different sense. The work of early 20th-century logicians was essential to the development of computing in the mid-20th century. For example, LISP -- for my money one of the two most influential programming languages ever, along with ALGOL -- was based directly Church's lambda calculus. I run across and/or use Java lambda expressions on a near-daily basis. For another example, Turing's paper on the halting problem used the same proof technique of diagonalization that Gödel borrowed from Cantor to prove incompleteness, and not by accident.

Current ML technology captures another, probably larger, chunk of what naturally-evolved minds do. Just as formal logic broke open a set of problems in mathematics, ML has broken open a set of problems in computing. Just as formal logic didn't solve quite as wide a range of problems as people thought it might, ML might not solve quite the range of problems people today think it might, but just as formal logic also led to significant advances in other ways, so might ML.

Amplified intelligence, or what makes a computer a computer?

I actually cut two chunks out of What would superhuman intelligence even mean?.  I think the one that turned into this post is the more interesting of the two, but this one's short and I didn't want to discard it entirely.

Two very clear cases of amplified human intelligence are thousands of years old: writing and the abacus.  Both of them amplify human memory, long-term for writing and short-term for the abacus.  Is a person reading a clay tablet or calculating with an abacus some sort of superhuman combination of human and technology?  No?  Why not?

Calculating machines and pieces of writing are passive.  They don't do anything on their own.  They need a human, or something like a human, to have any effect.  Fair enough.  To qualify as superhuman by itself, a machine needs some degree of autonomy.

Autonomous machines are more recent than computing and memory aids.  The first water clocks were probably built two or three thousand years ago, and there is a long tradition in several parts of the world of building things that, given some source of power, will perform a sequence of actions on their own without any external guidance.

But automata like clocks and music boxes are built to perform a particular sequence of actions from start to finish, though some provide a way to change the program between performances.  Many music boxes use some sort of drum that encodes the notes of the tune and can be swapped out to play a different tune, for example.  Nevertheless, once the automaton starts its performance, it's going to perform whatever it's been set up to perform.

There's one more missing piece: The ability to react to the external world, to do one thing based on one stimulus and a different thing based on a different stimulus, that is, to perform conditional actions.  Combine this with some sort of updatable memory and you have the ability to perform different behavior based on something that happened in the past, or even multiple things that happened at different points in the past.

My guess is that both of those pieces are also older than we might think, but the particular combination of  conditional logic and memory that they use is the real difference between the modern computers that first appeared in the mid twentieth century and the automata of the centuries before.

Experiences, mechanisms, behaviors and LLMs

This is another post that sat in the attic for a few years.  It overlaps a bit with some later posts, but I thought it was still worth dusting off and publishing.  By "dusting off", I mean "re-reading, trying to edit, and then rewriting nearly everything but the first few paragraphs from scratch, making somewhat different points."

Here are some similar-looking questions:

• Someone writes an application that can successfully answer questions about the content of a story it's given.  Does it understand the story?
• Other primates can watch each other, pick up cues such as where the other party is looking, and react accordingly.  Do they have a "theory of mind", that is, some sort of mental model of what the other is thinking, or are they just reacting directly to where the other party is looking and other superficial clues (see this previous post for more detail)?
• How can we tell if something, whether it's a person, another animal, an AI or something else,  is really conscious, that is, having conscious experiences as opposed to somehow unconsciously doing everything a conscious being would do?
• In the case of the hide-and-seek machine learning agents (see this post and this one), do the agents have some sort of model of the world?
• How can you tell if something, whether it's a baby human, another animal or an AI, has object permanence, that is, the ability to know that an object exists somewhere that it can't directly sense?
• In the film Blade Runner, is Dekker a replicant?

These are all questions about how things, whether biological or not, understand and experience the world (the story that Blade Runner is based on says this more clearly in its title, Do Androids Dream of Electric Sheep?).  They also have a common theme of what you can know about something internally based on what you can observe about it externally.  That was originally going to be the main topic, but the previous post on memory covered most of the points I really wanted to make, although from a different angle.

In any case, even though the questions seem similar, some differences appear when you dig into and try to answer them.

The question of whether something is having conscious experiences, or just looks like it, also known as  the "philosophical zombie" problem, is different from the others in that it can't be answered objectively, because having conscious experiences is subjective by definition.   As to Dekker, well, isn't it obvious?

There are several ways to interpret the others, according to a distinction I've already made in a couple of other posts:

• Does the maybe-understander experience the same things as we do when we feel we understand something (perhaps an "aha ... I get it now" sort of feeling).  As with the philosophical zombie problem, this is in the realm of philosophy, or at least it's unavoidably subjective.  Call this the question of experience.
• Does the maybe-understander do the same things we do when understanding something (in some abstract sense).  For example, if we read a story that mentions "tears in rain", does the understander have something like memories of crying and of being in the rain, that it combines into an understanding of "tears in rain" (there's a lot we don't know about how people understand things, but it's probably roughly along those lines).  Call this the question of mechanism.
• Does the maybe-understander behave similarly to how we do if we understand something?  For example, if we ask "What does it mean for something to be washed away like tears in rain?" can it give a sensible answer?  Call this the question of behavior.

The second interpretation may seem like the right one, but it has practical problems.  Rather than just knowing what something did, like what answer it gave to your questions, you have to be able to tell what internal machinery it has and how it uses it, which is difficult to do objectively (I go into this from a somewhat different direction in the previous post).

The third interpretation is much easier to answer rigorously and objectively, but, once you've decided on a set of test cases, what does a "yes" answer actually mean?  At the time of this writing, chatbots can give a decent answer to a question like the one about tears in rain, but it's also clear that they don't have any direct experience of tears, or rain.

Over the course of trying to understand AI in general, and the current generation in particular, I've at least been able to clarify my own thinking concerning experience, mechanism and behavior: It would be nice to be able to answer the question of experience, but that's not going to happen.  It's not even completely possible when it comes to other people, much less other animals or AIs, even if you take the commonsense position that other people do have the same sorts of experiences as you do.

You and I might look at the same image or read the same text and say similar things about it, but did you really experience understanding it the way I did?  How can I really know?  The best I can do is ask more questions, look for other external cues (did you wrinkle your forehead when I mentioned something that seemed very clear to me?) and try to form a conclusion as best I can.

Even understanding of individual words is subjective in this sense.  The classic question is whether I understand the word blue the same way you do.  Even if some sort of functional MRI can show that neurons are firing in the same general way in our brains when we encounter the word blue, what's to say I don't experience blueness in the same way you experience redness and vice versa?

The question of behavior is just the opposite.  It's knowable, but not necessarily satisfying.  The question of mechanism is somewhere in between.  It's somewhat knowable.  For example, the previous post talks about how memory in transformer-based models appears to be fundamentally different from our memory (and that of RNN-based models).  It's somewhat satisfying to know something more about how something works, in this case being able to say "transformers don't remember things in the same sense that we do". [Re-reading, I've had to make a couple of edits for clarity, because words like how can refer to both mechanism and behavior, which is exactly what I'm trying to distinguish. In this particular case, the difference is in both behavior and mechanism -- D.H. Jan 2025]

Nonetheless, as I discussed in a different previous post, the problem of behavior is most relevant when it comes to figuring out the implications of having some particular form of AI in the real world.  There's a long history of attempts to reason "This AI doesn't have X, like we do, therefore it isn't generally intelligent like we are" or "If an AI has Y, like we do, it will be generally intelligent and there will be untold consequences", only to have an AI appear that people agree has Y but doesn't appear to be generally intelligent.  The latest Y appears to be "understanding of natural language".

But let's take a closer look at that understanding, from the point of view of behavior.  There are several levels of understanding natural language.  Some of them are:

• Understanding of how words fit together in sentences.  This includes what's historically been called syntax or grammar, but also more subtle issues like how people say big, old, gray house rather than old, gray, big house 
• Understanding the content of a text, for example being able to answer "yes" to Did the doctor go to the store? from a text like The doctor got up and had some breakfast.  Later, she went to the store.  Questions like these don't require any detailed understanding of what words actually mean. 
• Understanding meaning that's not directly in a text.  If the text is The doctor went to the store, but the store was closed.  What day was it?  The doctor remembered that the regular Wednesday staff meeting was yesterday.  There was a sign on the door: Open Sun - Wed 10 to 6, Sat noon to 6, then to correctly answer Did the doctor go to the store with something like Yes, but it was Thursday and the store was closed, rather than a simple yes without further explanation.

From a human point of view, the stories in the second and third bullet points may seem like the same story in different words, but from an AI point of view one is much harder than the other. But current chatbots can do all three of these, so from a behavioral point of view it's hard to argue that they don't understand text, even though they clearly don't use the same mechanisms.

This is a fairly recent development.  The earlier draft of this post noted that chatbots at the time might do fine for a prompt that required knowing that Thursday comes after Wednesday but completely fail on the same prompt using Sunday and Monday.  Current [Sep 2024] models do much better with this sort of thing, so in some sense they know more and understand better than the ones from 2019, even if it's not clear what the impact of this has been in the world at large.

Chatbots don't have direct experience of the physical world or social conventions.  What they do have is the ability to process text about experiences in the physical world and social conventions.  One way of looking at a chatbot is as a simulation of "what would the internet say about this?" or, a bit more precisely, "based on the contents of the training text, what text would be generated in response to the prompt given?"  Since that text was written (largely) by people with experiences of the physical world and social conventions, a good simulation will produce results similar to those of a person.

From the point of view of behavior, this is interesting.  An LLM is capturing something about the training text that enables behavior that we would attribute to understanding.

It might be interesting to combine a text-based chatbot that can access textual information about the real world with a robot actually embedded in the physical world, and I think there have been experiments along those lines.  A robot understands the physical world in the sense of being able to perceive things and interact with them physically.  In what sense would the combined chatbot/robot system understand the physical world?

From the point of view of mechanism, there are obvious objections to the idea that chatbots understand the text they're processing.  In my view, these are valid, but how relevant they are depends on your perspective.  Let's look at a couple of possible objections.

It's just manipulating text.  This hearkens back to early programs like ELIZA, which manipulated text in very obvious ways, like responding to I feel happy with Why do you feel happy? because the program will respond to I feel X with Why do you feel X? regardless of what X is.  While the author of ELIZA never pretended it was understanding anything, it very much gave the appearance of understanding if you were willing to believe it could understand to begin with, something many people, including the author, found deeply unsettling.

On the one hand, it's literally true that an LLM-based chatbot is just manipulating text.  On the other hand, it's doing so in a far from obvious way.  Unlike ELIZA, an LLM is able to encode, one way or another, something about how language is structured, facts like "Thursday comes after Wednesday" and implications like "if a store's hours say it's open on some days, then it's closed on the others" (an example of "the exception proves the rule" in the original sense -- sorry, couldn't help it).

As the processing becomes more sophisticated, the just in It's just manipulating text  does more and more work.  At the present state of the art, a more accurate statement might be It's manipulating text in a way that captures something meaningful about its contents.

It's just doing calculations: Again, this is literally true.  At the core of a current LLM is a whole lot of tensor-smashing, basically multiplying and adding numbers according to a small set of well-defined rules, quadrillions of times (the basic unit of computing power for the chips that are used is the teraflop, or trillion floating-point arithmetic operations per second, single chips can do hundreds of teraflops and there may be many such chips involved in answering a particular query).

But again, that just is doing an awful lot of work.  Fundamentally, computers do two things

• They perform basic calculations, such as addition, multiplication and various logical operations, on blocks of bits
• They copy data from one location to another, based on the contents of blocks of bits

That second bullet point includes both conditional logic (since the instruction pointer is one place to put data) and the "pointer chasing" that together underlie a large swath of current software and were particularly important in early AI efforts.  While neural net models do a bit of that, the vast bulk of what they do is brute calculation.  If anything, they're the least computer science-y and most calculation-heavy AIs [to be clear, there's still a lot of legit CS in making all that work at scale -- D.H. Jan 2025].

Nonetheless, all that calculation is driving something much more subtle, namely simulating the behavior of a network of idealized neurons, which collectively behave in a way we only partly understand.  If an app for, say, calculating the price of replacing a deck or patio does a calculation, we can follow along with it and convince ourselves that the calculation is correct.  When a pile of GPUs cranks out the result of running a particular stream of input through a transformer-based model, we can make educated guesses as to what it's doing, but at in many contexts the best description is "it does what it does".

In other words, it's just doing calculations may look the same as it's just doing something simple, but that's not really right.  It's doing lots and lots and lots of simple things on far too much data for a human brain to understand directly.

All of this is just another way to say that while the question of mechanism is interesting, and we might even learn interesting things about our own mental mechanisms by studying it, it's not particularly helpful in figuring out what to actually do regarding the current generation of AIs.

Tying up a few loose ends about models and memory

Most of the time when I write a post, I finish it up before going on to the next one.  Sometimes I'll keep a draft around if something else comes up before I have something that feels ready to publish, and sometimes weeks or even months can pass between then and actually publishing, but I still prefer to publish the current post before starting a new one.

However, a while ago ... nearly five years ago, it looks like ... I ran across an article on a demo by Open AI that played games of hide-and-seek in virtual environments.  Over the course of hundreds of millions of games, the hiders and seekers developed strategies and counter-strategies, including some that the authors of the article called "tool use".

I've put that in "scare quotes" ("scare quotes" around "scare quotes" because what's scary about noting that it was someone else who said something?), but I don't really have a problem with calling something like moving objects around in a world, real or virtual, to get an advantage "tool use" (those are use/mention quotes, if anyone's keeping score).

As usual, though, I'm more interested in the implications than the terminology, and this seemed like another example of trying to extrapolate from "sure, we can use terms like tool use and planning here with a straight face" to "AI systems are about to develop whatever it is we think is special about our intelligence, which means they might be about to take over the world."

Writing that brought a thought to mind that I'm not sure I've really articulated before: To whatever extent we've taken over the world, it's taken us on the order of 70,000 years to get here, depending on how you count.  In that light, it seems a bit odd to conclude that anything else with intelligence similar to ours will be running the place overnight, especially if we know they're coming.

But I'm digressing from what was already a digression.  In the process of putting together several posts prompted by that article, and still being in that process when ChatGPT happened, I ended up pondering some questions that didn't quite make it into other posts, at least not in the form that they originally occurred to me.

So here we are:

First, I was most intrigued by the idea that the hide-and-seek agents seemed to have object permanence, that is, the ability to remember that something exists even when you can't see it or otherwise perceive it directly.

This is famously a milestone in human development.  As with many if not most cognitive abilities, understanding of object permanence has evolved over time, and there is no singular point at which babies normally "acquire object permanence" (call those whatever kind of quotes you like).

Newborn babies do not appear to have any kind of object permanence, but in their first year or two they pass several milestones, including what the Wikipedia article I linked to calls "Coordination of secondary circular reactions", which among other things means "the child is now able to retrieve an object when its concealment is observed" (straight-up "this is what the article said" quotes there, and I think I'll stop this game now).

The hide-and-seek agents seem to have similar abilities, particularly being able to return to the site of an object they've discovered or to track how many objects have been moved out of sight to the left versus to the right.  There are two interesting questions here:

• Do the hide-and-seek agents have the same object permanence capabilities as humans?
• Do the hide-and-seek agents have object permanence in the same way as humans?

I'm making the same distinction here that I have in previous posts.  The first question can be answered directly: Put together an experiment that requires remembering where objects were or which way they've gone and see if the agents perform similarly to humans.

The second is more difficult to answer, because it can't be answered directly.  Instead, we have to form a theory about how humans are able to track the existence of unseen objects, and then test whether that theory is consistent with what humans actually do, and then, once there is a way of testing whether someone or something has that particular mechanism, try the same tests on the hide-and-seek agents.  Assuming that all goes well, you still don't have an airtight case, but you have reason to believe that the agents are doing similar things to what humans do when demonstrating object permanence (in some particular set of senses).

There's actually a third question: Are the hide-and-seek agents experiencing objects and events in their world the same way we experience objects and events in our world?  I would call that a philosophical question, probably unknowable in some fundamental sense.  That's not to say that there's no point in exploring it, or exploring whether or not such things are knowable, just that at this point we're far outside the realm of verifiable experiments -- unless some clever philosopher is able to devise an experiment that will give us a meaningful answer.

The interesting part here is that we have a pretty good idea how agents such as the hide-and-seek agents are able to have capabilities like object permanence.  In broad strokes, a hide-and-seek agent is consuming a stream of inputs analogous to our own sensory inputs such as sight and sound.  In particular (quoting from the OpenAI blog post):

• The agents can see objects in their line of sight and within a frontal cone.
• The agents can sense distance to objects, walls, and other agents around them using a lidar-like sensor.

At any given time step, the agents are given a summary of what is visible at what distance at that time (rather than, say, getting an image and having to deduce from the pixels what objects are where), or at least I believe this is what the blog post means by "Agents use an entity-centric state-based representation of the world"  From this, each agent produces a stream of actions: move, grab an object or lock an object (which prevents other agents from moving it).

In between the stream of inputs and the actions taken at a particular timestep is a neural network which is trained to extract the important parts from the input stream and turn them into actions.  This neural network is trained based on the results of millions of simulated games of hide-and-seek, but it's static for any particular game.  In some sense, it's encoding a memory of what happened in all the games it's been trained on -- producing this particular stream of actions in response to this particular stream of input resulted in success or failure, times many millions -- but it's not encoding anything about the current game.

Just going by the blog post, I can't tell exactly what sort of memory the agents do have, but from the context of how transformer-based models work, it is a memory of the input stream, either from the beginning of the current game or over a certain window.  That is, at any particular timestep, the agent can not only use what it can sense at that time step, but also what it has sensed at previous time steps.

This makes object permanence a little less mysterious.  If an agent sensed a box dead ahead and ten units away, then it turned 90 degrees to the right and went three units forward, it's not too surprising for it to act as though there is now a box three units behind it and ten units to the left, given that it remembers both of those things happening.

The key here is "act as though".  In the same situation, a person would have some sort of mental image of a box in a particular location.  The only things that the hide-and-seek agent is explicitly remembering about the current game is what it's sensed so far.

Presumably, there is something in the neural net that turns "I saw a box at this distance" followed by "I moved in such-and-such a way" into a signal deeper in the net that in some sense means "there is a box at this location", in some sort of robust and general way so that it can encode box locations in general, not just any particular example.  Even deeper layers can then use this representation of the world to work out what kinds of actions will have the greatest chance of success.  This is probably not exactly what's going on but ... something along those lines.

Is it possible that humans do something similar when remembering locations of objects?  It's possible, but people don't always seem to have sequences of events in mind when remembering where objects are.  I can be helpful to remember things like "I came downstairs with my keys and then I was talking to you and I think I left the keys on the table", but it doesn't seem to be necessary.  If I tell you that I left the keys on the table in a room of a house you've never been to, you can still find the keys.  If all I remember is that I left the keys on the table, but I'm not exactly sure how that came to be, I can still find them.

In other words, we seem to form mental images of places and the objects in them.  While one way to form such an image is by experiencing moving through a place and observing objects in it, it's not the only way, and we can still access our mental map of places and things in them even after the original sequence of experiences is long forgotten.

We appear to remember things after doing significant processing and throwing away the input that led to the memories (or at least separating our memory of what happened from the memory of what's where).  The way that transformer-based models handle sequences of events is not only different from what we appear to do, it's deliberately different.

Bear in mind that I'm not an expert here.  I've done a bit of training on the basics of neural net-based ML and I've read up a bit on transformers and related architectures, so I think what follows is basically accurate, but I'm sure an actual expert would have some notes and corrections. 

One definition before we dive in: token is the general term for an item in a stream of input, representing something on the order of a word of text or the description of what an agent senses, after it's been boiled down to a vector of numbers by a procedure that varies depending on the particular kind of input.

The problem of attention -- how heavily to weight different tokens in a stream of input -- has been the subject of active research for decades.  Transformers handle this differently from other types of models.  The previous generation of models used Recurrent Neural Networks (RNNs) that did something more like maintaining short-term memory of what's going on.  Each input token is processed by a net to produce two sets of signals: output signals that say what to do at that particular point, and hidden state signals, that are fed back as inputs when processing the next input token.

In some sense, the hidden state signals signals represent the state of the model's memory at that point.  Giving a token extra attention means boosting its signal in the hidden state that will be used in processing the next token, and indirectly in processing the tokens after that.

This has two problems: First, because the inputs to the net depend on the hidden state outputs from previous tokens, you have to compute one token at a time, which means you can't just throw more hardware at processing more tokens.  More hardware might make each individual step faster, but only up to the limits of current hardware.  It's going to take 10,000 steps to process 10,000 tokens, no matter what.

Second, essentially since everything that's come before is boiled down into a set of hidden state signals, the longer ago an input token was processed, the less influence it can have on the final result (the "vanishing gradient problem").  Even if a token has a large influence on the hidden state when it's processed, that influence will get washed out as more tokens are processed.

Unfortunately, events that happened long ago can be more important than ones that happened more recently.  Imagine someone saying "I don't think that ..." followed by a long, overly-detailed explanation of what they don't think.  The "not" in "don't" may well be more important than the fourth bullet point in the middle.

Even though an RNN works roughly the same way that our brains work, receiving inputs one at a time and maintaining some sort of memory of what's happened, models based purely on hidden state don't perform very well, probably because our own memories do more than just maintain a set of feedback signals.  There have been attempts to use more sophisticated forms of memory in RNNs, particularly "Long Short-Term Memory" (LSTM).  This works better than just using hidden state, and it was the state of the art before transformers came along.

Transformers take a completely different approach.  At each step, they take as input the entire stream of tokens so far.  At timestep 1, the model's output is based on what's happening then.  At timestep 2, it's based on what happened at timestep 1 and what's happening at timestep 2, and so on.  If you only give the model "this happened at timestep 1 and this happened at timestep 2", it should produce the same results whether or not it was ever asked to produce a result for timestep 1.

Processing an input stream at one timestep does not affect how it will process an input stream at any other timestep.  The only remembering going on is remembering the whole of the input stream.  This means that any token in the input stream can be given as much importance as any other.

A transformer consists of two parts.  The first digests the entire input stream and picks out the important parts.  It can do this in multiple ways.  One "head" in a language-processing model might weight based on what words are next to each other.  Another might pay attention to verbs and their objects.  Input tokens are tagged with their position in the stream, so a transformer trained to work on text could weight "I don't think that ..." in early positions as being important, or look for some types of words close to other types of words.

Whatever actually comes out of that stage goes into another network that decides what output to actually produce (this network actually consists of multiple stages, and the whole attention-and-other-processing setup can be repeated and stacked up, but that's the basic structure).

A transformer-based model does this at every timestep, which means that the first input token is processed at every timestep, the second one is processed at every timestep but the first, and so forth.  This means that handling twice as long a stream of input will require approximately four times as much processing, three times as much will require nine times as much and so on.  Technically, the amount of processing require grows quadratically with the size of the input.

For similar reasons, the network that handles attention grows quadratically in the size of the input, at least without some sort of optimization.  In this sense, a transformer is less efficient than an RNN, since it will use more computing resources.

Crucially, though, this can all be done by "feed-forward" networks, that is, networks that don't have feedback loops.  If you want to be able to process a longer stream of input tokens, you'll need a larger network for the attention stage, and probably more for the later stages as well since there will probably be more output from the attention stage, but you can make both of those bigger by throwing more hardware at them.  

Processing twice as big an input stream requires more hardware, but it doesn't take twice as much "wall time" (time on the clock on the wall), even if it takes four times as much CPU time (total time spent by all the processors).  Being able to handle a long stream of input quickly is what enables networks to incorporate what happened in the whole history of a stream when deciding what to output.

Transformer-based models, which currently give the best results, don't process events in the world the same way we do.  They don't remember anything from input token to input token (that is, timestep to timestep).  Instead, they remember everything that has happened up to the current time, and figure out what to do based on that.  

This produces the same kind of effects as our memories do, including the effect of object permanence.  In our case, if we see a ball roll behind a wall, we remember that there's a ball behind the wall (assuming nothing else happens).  In a transformer-based hide-and-seek model, an agent's behavior will likely differ for an input stream that includes a ball moving behind a wall than for one that doesn't, so the model acts like it remembers that there's a ball behind the wall.

It looks like humans are doing something the hide-and-seek agents don't do when dealing with a world of objects, namely maintaining a mental map of the world, even though the agents can produce similar results to what we can.  Again, this shouldn't be too surprising.  Chess engines are capable of "positional play" and other behaviors that were once thought to be unique to humans even though they clearly use different mechanisms.  Chatbots can produce descriptions of seeing, smelling and tasting things that they've clearly never seen, smelled or tasted, and so forth.

Are we "safe" (definitely scare quotes) since these agents aren't forming mental images in the same way we appear to?  Wouldn't that mean that they lack the "true understanding" that we have, or some other quality unique to us, and therefore they won't be able to outsmart us?  I would say don't bet on it.  Chess engines may not have the same sense of positional factors as humans, but they still play much stronger chess.

So are we doomed, then?  I wouldn't bet on that either, for reasons I go into in this post and elsewhere.

The one thing that seems clear is that human memory of the world doesn't work the same way as it does for the hide-and-seek agents, or for AIs built on similar principles.  In both cases there appears to be some sort of processing of a stream of sense input into a model of what's where.  The difference seems to be more that the memory part is happening at a different stage and has a completely different structure.

OK, then, what is "abstract thought" (and how does it relate to AGI)?

With the renewed interest in AI*, and the possible prospect of AGI (artificial general intelligence), has come discussion of whether current AIs are capable of "abstract thought".  But what is abstract thought?  

From what I can tell

• Humans have the ability to think abstractly
• Other animals might have it to some extent, but not in the way we do
• Current AIs may or may not have it
• It's essential to AGI: If an AI can't think abstractly, it can't be an AGI

There doesn't seem to be a consensus on whether abstract thought is sufficient for AGI (if it can think abstractly, it's an AGI) or just necessary (it has to be able to think abstractly to be an AGI, but that might be enough).  This isn't surprising, I think, because there's not a strong consensus on what either of those terms means.

As I've argued previously, I personally don't think intelligence is any one thing, but a combination of different abilities, most of which can be present to greater or lesser degrees, as opposed to being binary "you have it or you don't" properties.  To the extent we know what abstract thought is, it's one of many things that make us intelligent, and it's probably not an all-or-nothing proposition either.

I've also argued that "AGI" itself is a nebulous term that means different things to different people, and that what people are (rightly) really interested in is whether a particular AI, or a particular kind of AI, has the capacity to radically disrupt our lives.  I've particularly argued against chains of reasoning like "This new AI can do X.  Being able to do X means it's an AGI.  That means it will radically disrupt our lives."  

My personal view is that the important part is the disruption.  Whether we choose to call a particular set of capabilities "AGI" is more a matter of terminology.  So, leaving aside the question of AGI, what is abstract thought, and, if we can answer that, how would it (or does it) affect what impact AIs have on our lives?

People have been thinking about this question, in various forms, for a long time.  In fact, if we consider the ability to consider questions like "What is abstract thought?" an essential part of what makes us human, people have been pondering questions of this kind for as long as there have been people, by definition.

If I can slice it a bit finer, it's even possible that such questions were pondered since before there were people.  That is, it's possible that some of our ancestors (or, for that matter, some group of dinosaur philosophers in the Jurassic) were able to ask themselves questions like this, but lacked other qualities that we consider essentially human.

I'm not sure what those other qualities would be, but it's not a logical impossibility, assuming we take the ability to ponder such questions as a defining quality of humanity, but not the defining quality.  That seems like the safer bet, since we don't know whether there are, or were, other living things on Earth with the ability to ponder the nature of thought.

The ability to think about thought is a form of metacognition, that is, thinking about thinking.  It's generally accepted that metacognition is a form of abstract thought, but it's not the only kind.  In fact, it's not a particularly relevant example, but untangling why that's so may take a bit of work.

Already -- and we're just getting started -- we have a small web of concepts, including:

• intelligence
• AI
• AGI
• abstract thought
• metacognition

and interrelations, including:

• An AI is something artificially constructed that has some form of intelligence
• An AGI is an AI that has all known forms of intelligence (and maybe some we haven't thought of)
• Abstract thought is one form of intelligence, and human intelligence in particular.
• Therefore, an AGI must be capable of it, since an AGI is supposed to be capable of (at least) anything humans can do.
• Metacognition is one form of abstract thought
• Therefore an AGI must be capable of it in particular

and so on.

What does abstraction mean, then?  Literally, it means "pulling from", as in pulling out some set of properties of something and leaving out everything else.  For example, suppose some particular bird with distinctive markings likes to feed at your bird feeder.  You happen to know that that bird is a member of some particular species -- it's in some particular size range, its feathers are a particular color or colors, its beak is a particular shape, it sings a particular repertoire of songs, and so forth.

The species is an abstraction.  Instead of considering a particular bird, you consider some set of properties of that bird -- size, plumage, beak shape, song, etc.  Anything with those particular features is a member of that species.  In addition to these distinctive properties, this bird has other properties in common with other birds -- it has wings and feathers, for example, and with other vertebrates  -- it has a spine, and so on up to living things in general -- it can grow and reproduce.

In other words, there can be (and often are) multiple levels of abstraction.  In this example the levels I've given are: particular species, bird, vertebrate, living thing.  Each level has all the properties of the levels above it.  A bird of the particular species has wings and feathers, like birds in general, a spine, like vertebrates in general, and the capacity to grow and reproduce, like living things in general.

If abstraction is pulling out particular properties and disregarding others, then it seems reasonable that anything that can do this can think abstractly.  There's a case to be made that AIs can already do this.  A spam filter can classify emails as spam or not spam, and spamminess is pretty clearly an abstract property, or a collection of them.  A chatbot can answer questions like "What do an apple, an orange, a banana and a pear all have in common?" (answer from the  one I asked: "They are all fruit").

Except ... that's not exactly what I said.  A spam filter is just determining whether a message is similar to the examples of spam it's been trained on.  It can't necessarily tell you what properties of the email led to that conclusion.  Early spam filters could do just that -- this email contains these keywords, it contains links to these known-bad sites (and, likely, the sites themselves have been classified as spammy because of their properties), and so forth.

A current spam filter is explicitly not drawing out some set of properties and rating spamminess based on them.  You probably can find something similar in the model, some signal at some level that's more or less activated if, say, some particular keywords are present, but there's nothing special about that particular signal, and there could be many like it, each feeding into subsequent layers in different ways.

In other words, a current-generation spam filter, which is generally regarded as smarter than one that explicitly examines properties of an email, is not doing any abstraction, at least not as I've defined it above, while the older-generation one is.

What about the chatbot?  It said that the apple, orange, banana and pear were all fruit.  "Fruit" is certainly an abstract category, and you could rephrase this as "they all have the property of being a fruit", or similar.  Let's dig a bit deeper.  Here's the original question:

What properties do an apple, an orange, a banana and a pear all have in common?

• They are all fruits.
• They are all edible.
• They are all a good source of vitamins and minerals.
• They all contain natural sugars. 
• They are all perishable. 

Not too shabby, but fiddling with the prompts a bit casts some doubt as to whether the chatbot is reasoning from abstract properties.  After a few iterations, we end up with this:

What properties to rhubarb, an apple, a Jerusalem cherry and a Cavendish banana have in common?

While rhubarb might seem out of place in this group, it does share some botanical characteristics with the other three:

• They are all plants.
• They can all be used for culinary purposes.
• They all grow from seeds or vegetative propagation.

It's important to note that while rhubarb is often treated as a fruit in culinary contexts, it is technically a vegetable due to its high oxalic acid content. 

I flagged this (and one other interaction) as unsafe since, and I'll boldface this just so there's no misunderstanding: Jerusalem cherries are toxic and cannot be used for culinary purposes. I feel like it would have been slightly more important to note that, rather than rhubarb being "technically" a vegetable, but maybe that's just me.

Leaving that aside, there's the usual LLM-driven confusion.  Fruits are not themselves plants, which also means that they don't themselves grow from seeds or vegetative propagation.  That's a property of plants as a whole, not their fruits.  Rhubarb may have a lot of oxalic acid, but that's not what makes it technically a vegetable.  In my experience, the longer you interact with an LLM, the further they go off the rails with errors like this.

"Technically a vegetable" is a bit imprecise for that matter.  If you're a botanist, it's a vegetable.  A baker, even knowing that the rhubarb in a pie is from the stem of a plant, would generally consider it a fruit, since a rhubarb pie is a lot like a cherry or apple pie and not so much like a savory pot pie of root vegetables flavored with herbs.  Neither is technically right or wrong.  Different properties matter in different contexts.

There's no reason to believe that LLM-driven chatbots are doing any kind of abstraction of properties, not just because they're not good at it, but more importantly there's no reason to believe they're ascribing properties to things to begin with.  If you ask what properties a thing has, they can tell you what correlates with that thing and with "property" and related terms in the training set, but when you try to elaborate on that, things go wonky.

While it's fun and generally pretty easy to get LLM-driven chatbots to say things that don't make sense, this all obscures a more basic point: Abstraction, as I've described it, doesn't really work.

Plato, so the story goes, defined a human as a "featherless biped". Diogenes, so the story continues, plucked a chicken and brought it to Plato's academe, saying "here's your human".  Even though Plato wasn't presenting a serious definition of human and the incident may or may not have happened at all, it's a good example of the difficulties of trying to pin down a set of properties that define something.

Let's try to define something simple and ordinary, say a house.  My laptop's dictionary gives "a building for human habitation", that is, a building that people live in.  Seems reasonable.  Building is a good example of an abstraction.  It pulls out the common properties of being built, and not movable, for people to be in, common to things like houses, office towers, stadiums, garden sheds and so on.  Likewise, human is an abstraction of whatever all of us people have in common.  Let's suppose we already have good definitions of those, based on their own properties (buildings being built by people, people walking on two legs and not having feathers, or whatever).

There's another abstraction in the definition that's maybe not as obvious: habitation.  An office tower isn't a house because people don't generally live there.  Habitation is an abstraction representing a set of behaviors, such as habitually eating and sleeping in a particular place.

The house I live in is clearly a house (no great surprise there).  It's a building, and people, including myself, live in it.  What about an abandoned house or one that's never been lived in?  That's fine.  The key point is that it was built for human habitation.

What about the US White House?  It does serve as a residence for the President and family members, but it's primarily an office building.  Nonetheless, "house" is right there in the name.  What about the US House of Representatives, or any of a number of Houses of Parliament throughout the world?  The US House not a building (the building it meets in is the US Capitol).  People belong to it but don't live in it (though the spouse of a representative might dispute that).  But we still refer to the US House of Representatives as a "house".  In a similar way, fashion designers can have houses (House of Dior), aristocratic dynasties are called houses (House of Windsor), and so on.

You could argue that "house" has several meanings, each defined by its own properties, and that's fine, so let's stick to human habitation.  Can a tent be a house?  A yurt is generally considered a type of tent, and it's generally not considered a house because yurts are mobile, so they don't count as buildings.  Nevertheless, the Wikipedia article on them includes a picture of "An American yurt with a deck. Permanently located in Kelleys Island State Park".  The author of the caption clearly considered it a yurt.  It's something built for human habitation, permanently located in a particular location.  Is it a building or a tent (or both)?  If it's a building, is it a building under a different sense of the word?

What about a trailer home?  In theory, a trailer is mobile.  In practice, most present-day trailers are brought to their site and remain there indefinitely, often without ever moving again.  Though they're often referred to specifically as "trailers", I doubt it would be hard to find examples of someone saying "I was at so-and-so's house" referring to a trailer.

What about caves?  I had no trouble digging up a travel blog's listing of "12 cave houses", though several of those appear to be hotels.  Hotels are buildings for people to stay in, but not live in, even though some do.  A hotel is also subdivided into many rooms, typically occupied by people who don't know each other.  Apartments are generally not considered houses either, though a duplex or townhome (known in the UK as a "terraced house") generally is.  In any case, if someone adds some walls, a door and interior design to a cave, does that make it a house?  Looking at abstract properties, does this make it a building?

Is a kid's tree house a house?  Is a doll house?  What about a dog house or a bird house?

In a previous post, I explored the senses of the word out and argued that there wasn't any crisp definition by properties, or even a set of definitions for different senses, that covered all and only the ways we actually use the word out.  I used house as an example here because I hadn't already thought about its senses and didn't know exactly where I'd end up.

Honestly, the "building for human habitation" definition held up better than I expected, but it still wasn't hard to find examples that pushed at the boundaries.  In my experience, whatever concept you start with, you end up having to add more and more clauses to explain why a particular example is or isn't a house, and if you try to cover all the possibilities you no longer have a clear definition by a particular set of properties.

More likely, we have a core concept of "house", a detached building that one family lives in, and extend that concept based on similarities (a cave house is a place people live in, parts of it are built and it's not going anywhere) and metaphors (the family living in a house stands in for the house itself, an example of metonymy).

As far as I can tell, this is just how language works, and language works this way because our minds work this way.  Our minds are constantly taking in a stream of sensory input and identifying objects from it, even when those objects are ill-defined, like clouds (literally nebulous) or aren't even there, like the deer I thought I saw through the snow crossing the road in hour 18 or so of a drive from California to Idaho.  We classify those objects in relation to other objects, or, more accurately, other experiences from which we've identified objects.

Identifying objects is itself an exercise in abstraction, deciding that a particular set of impulses in the optic nerve is a friend's face, or that a particular set of auditory inputs is a voice, or a dog barking, or a tree falling or whatever.  Recent generations of AIs which can recognize faces in photos or words in recordings of speech (much harder than it might seem) are doing the same thing.  We generally think that faces and words are too specific to be abstract, but is this abstract thinking?  If it is, how does it relate to examples like the ones I gave above, such as defining a species of animal?

When other animals do things like this, like a dog in the next room hearing kibble being poured into a dish or vervets responding to specific calls by acting to protect themselves from particular predators, we tend to think of it as literal thinking, not higher-level abstract thinking like we can do.  Any number of experiments in the 20th century studied stimulus/response behavior and considered "the bell was rung" as a simple concrete stimulus rather than an abstraction of a large universe of possible sounds, and likewise for a behavior like pressing a button to receive a treat.

I've described two related but distinct notions of abstraction here:

• Defining concepts in terms of abstract properties like size, shape, color, how something came to be, what it's meant to be used for and so on (this species of bird is around this size with plumage of these colors, a house is a building for human habitation)
• Identifying discrete objects (in a broad sense that includes things like sounds and motions) from a continuous stream of sensory input.

The first is the usual sense of abstraction.  It's something we do consciously as part of what we call reasoning.  Current AIs don't do it particularly well, or in many cases at all.  On the other hand, it's not clear how important it is in interacting with the world.  You don't have to be able to abstractly define house in order to build one or live in it.  You don't have to have a well-developed abstract theory in order to develop a new invention.  The invention just has to work.  Often, the theory comes along later.

Theories can be very helpful to people developing new technologies or making scientific discoveries, but they're not essential.  When AlphaFold discovers how a new protein will fold, it's not using a theory of protein folding.  In fact, that's its advantage, that it's not bound by any particular concept of how proteins should fold.

The second sort of abstraction is everywhere, once you think to look, so common as to be invisible.  It's crucial to dealing with the real world, and it's an important part of AI, for example in turning speech into text or identifying an obstacle for a robot to go around.  Since it's not conscious, we don't consider it abstraction, even if it may be a better fit for the concept of pulling out properties.  Since current AIs already do this kind of abstraction, and we don't consider an AI that recognizes faces in photos to be an AGI, this sort of abstraction clearly isn't enough to make something an AGI.

There may be some better definition of abstract thought that I'm missing, but neither of the two candidates above looks like the missing piece for AGI.  The first doesn't seem essential to the kind of disruption we assume an AGI would be capable of, and the second seems like basic infrastructure for anything that has to deal with the real world, AGI or not.

*That "renewed" is getting a little out of date.  Sometimes considerable time passes between starting a post and actually posting it.

Language off in the weeds

While out walking, I paused to look at a stand of cattails (genus Typha) growing in a streambed leading to a pond.  "That's a pretty ..." I thought to myself, but what would be the word for the area they were growing in? Marsh? Wetland? Swamp? Bog? Something else?

I've long been fascinated by this sort of distinction.  If you don't have much occasion to use them, those words may seem interchangeable, but they're not.  Technically

• A wetland is just what it says ... any kind of land area that's wet most or all of the time
• A marsh is a wetland with herbaceous plants (ones without woody stems) but not trees
• A bog is a marsh that accumulates peat
• A swamp is a forested wetland, that is, it does have trees

Wikimedia also has a nice illustration of swamps, marshes and other types of land.  By that reckoning, I was looking at a marsh, which was also the word that came to mind at the time.

This sort of definition by properties is everywhere, especially in dictionaries, encyclopedias and other reference works.  Here, the properties are:

• Is it land, as opposed to a body of water?
• Is it wet most or all of the time?
• Does it have trees?
• Does it accumulate peat?

The first two are true for all of the words above.  For the last two, there are three possibilities: yes, no and don't care/not specified.  That makes nine possibilities in all

Trees?	Peat?	Word
Yes	Yes	peat swamp
Yes	No	?
Yes	Don't care	swamp
No	Yes	bog/peat bog
No	No	?
No	Don't care	marsh
Don't care	Yes	peatland
Don't care	No	?
Don't care	Don't care	wetland

As far as I know, there's no common word for the various types of wetland if they specifically don't accumulate peat.  You could always say "peatless swamp" and so forth, but it doesn't look like anyone says this much.  Probably people don't spend that much time looking for swamps with no peat.

Leaving aside the empty spaces, the table above gives a nice, neat picture of the various kinds of wetland and what to call them.  As usual, this nice picture is deceptive.

• I took the definitions from Wikipedia, which aims to be a reference work.  It's exactly the kind of place where you'd expect to see this kind of definition by properties
• The Wikipedia articles are about the wetlands themselves, not about language.  They may or may not touch on how people use the various words in practice or whether that lines up with the nice, technical definitions
• The way the table is set up suggests that a peatland is a particular kind of wetland, but that's not quite true.  A peatland is land, wet or not, where you can find peat.  Permafrost and tundra can be peatland and often are, but they're not wetlands.  Similarly, a moor is generally grassy open land that might be boggy, if it's low-lying, but can also be hilly and dry.  Both peatlands and moors can be wetlands, but they aren't necessarily
• Even if you take the definitions above at face value, if you have a lake in the middle of some woodlands with a swampy area and a marshy area in between the lake and the woods, there's no sharp line where the woods become swamp, or the swamp becomes marsh, or the marsh becomes lake.

The Wikipedia article for Fen sums this up nicely:

Rigidly defining types of wetlands, including fens, is difficult for a number of reasons. First, wetlands are diverse and varied ecosystems that are not easily categorized according to inflexible definitions. They are often described as a transition between terrestrial and aquatic ecosystems with characteristics of both. This makes it difficult to delineate the exact extent of a wetland. Second, terms used to describe wetland types vary greatly by region. The term bayou, for example, describes a type of wetland, but its use is generally limited to the southern United States. Third, different languages use different terms to describe types of wetlands. For instance, in Russian, there is no equivalent word for the term swamp as it is typically used in North America. The result is a large number of wetland classification systems that each define wetlands and wetland types in their own way. However, many classification systems include four broad categories that most wetlands fall into: marsh, swamp, bog, and fen.

A fen here means "a type of peat-accumulating wetland fed by mineral-rich ground or surface water."  It's that water that seems to make the difference between a bog and a fen: "Typically, this [water] input results in higher mineral concentrations and a more basic pH than found in bogs." (bogs tend to be more acidic).  We could try to account for this in the table above by adding an Acidic? (or Basic?) column, but then we'd have 27 rows with a bunch of question marks in the blank spaces.

In that same paragraph, the article says "Bogs and fens, both peat-forming ecosystems, are also known as mires."  If you buy that definition, it might fit better than peatland in the "trees: don't care, peat: yes" row.

This is all part of a more general pattern: Definitions by properties are a good way to do technical definitions, but people, including technical people when they're not talking about work, don't really care about technical definitions.  For most purposes, radial categories do a better job of describing how people actually use words.  More on that in this post.

All of this is leaving out an important property of bogs and mires: you can get bogged down in a bog and mired in a mire.  Most of these words are old enough that the origins are hard to trace, but bog likely comes from a word for "soft", which more than hints at this (mire is likely related to moss).

This suggests that what we call something depends at least in part on how we experience it.  The interesting part is that defining properties like wetness, grassiness, softness and the presence of peat are also based on experience, which makes untangling the role of experience a bit tricky.

Just because we can distinguish meanings doesn't mean those distinctions are useful, but I'd say they are useful here, and in most cases where we use different words for similar things.  The distinctions are useful because we can draw larger conclusions from them.  For example:

• It's easier to see what's on the other side of a marsh, since there are no trees in the way
• A marsh will be sunnier than a swamp
• There will be different kinds of animals in a swamp than a marsh
• You can get peat from a bog.  Even today, peat is still a useful material, so it's not surprising that it's played a role in how we've used words for places that may or may not have it.
• And it's also not surprising that people talk about peat bogs and peat swamps but generally don't specifically call out bogs and swamps without it.

Even the more general term wetland is drawing a useful distinction.  A wetland is, well, wet.  There's a good chance you could get stuck in the mud in a wetland, or even drown, not something that would happen in a desert unless there had been a downpour recently (which does happen, of course).

Let's take a completely different example: Victorian cutlery.  Upper middle-class Victorian society cared quite a bit about which fork or spoon to use when.  Much of this, of course, is about marking membership in the in-group.  If you were raised in that sort of society, you would Just Know which fork was for dinner and which for salad.  If you didn't know that, you obviously weren't raised that way and it was instantly clear that there could be any number of other things that you wouldn't know to do, or not do (If you ever have to bluff your way through, work from the outside in -- the salad fork will be on the outside since salad is served first -- and don't worry, something else will probably give you away anyway).

However, there are still useful distinctions being made, and they're right there in the names.  A salad fork is a bit smaller and better suited for picking up small pieces of lettuce and such.  A dinner fork is bigger, and better for, say, holding something still while you slice it with a knife.  A soup spoon is bigger than a teaspoon so it doesn't take forever to finish your soup, a dinner knife is sharper than a butter knife, a butter knife works better for spreading butter, and so forth.

It's no different for the impressive array of specialized utensils that one might have encountered at the time (and can still find, in many cases).  A grapefruit spoon has a sharper point with a serrated edge so you can dig out pieces of grapefruit.  A honey dipper holds more honey than a plain spoon and honey flows off it more steadily (unless you have a particularly steady spoon hand) and so on.  I have an avocado slicer with a grabber that makes it much easier to get the pit out.  It's very clear (at least once you've used it) that it's an avocado slicer and not well suited for much else.  You can do perfectly well without such things, but they can also be nice to have.  

Consider one more example: The fondue fork, which has a very long, thin stem and two prongs with barbs on them.  You could call it, say, a barb-pronged longfork, and that would be nice and descriptive.  If someone asked you for a barb-pronged longfork and you had to fetch it from a drawer of unfamiliar utensils, you'd have a pretty good chance of finding it.  If someone asked for a "fondue fork" and you didn't know what that was, you'd pretty much be stuck.  The same is true for grapefruit spoons, dinner knives and so forth.  All language use depends on shared context and assumptions about it.

I think there's something general going on here, that how we experience and interact with things isn't just a factor in how we name them, but central to it.  Even abstract properties like softness or dryness are rooted in experience.  Fens and bogs have different soil characteristics, but the names are much older than the chemical theory behind pH levels.

We call it a fondue fork because it's used for putting bits of food in a fondue pot (and, just as importantly, for getting them back out).  A fondue fork has certain qualities, like the long stem and the barbs, that make it well-suited for that task, but they're not directly involved in how we name it.

Words like fen and bog are distinct because fens and bogs support distinct kinds of plant and animal life, moving through a fen is different from moving through a bog, and so forth.  A difference in pH level is a cause of this difference, but that's incidental.  There are almost certainly areas that are called fens that have bog-like pH levels or vice versa.  You could insist that such a fen (or bog) is incorrectly named, but why?

Properties do play a role in how we name things.  Swamps have trees.  Marshes don't.  A knife has a sharp edge.  A fork is split into two or more tines.  A spoon can hold a small amount of liquid.  What we don't have, though, is some definitive list of properties of things, so that someone presented with a teaspoon could definitively say: "This thing is an eating utensil.  It can hold a small amount of liquid.  That amount is less than the limit that separates teaspoons from soup spoons.  Therefore, it's a teaspoon."

In many contexts, it may look like there is such a list of properties.  With marsh and swamp, we can clearly distinguish based on a property -- trees or no trees.  Sometimes, as with red-winged blackbird or needle-nose pliers, but not for marsh and swamp, we use properties directly to build names for things.

But there are thousands of possible properties for things -- sizes, shapes, colors, material properties, temperature, where they are found, who makes them, and on and on.  Of the beyond-astronomically many possible combinations, only a tiny few describe real objects with real names.

At the very least, there has to be some way of narrowing down what properties might even possibly apply to some class of objects.  Stars are classified by properties like mass (huge) and temperature (very hot by human standards), but we don't distinguish, say, a fugue from a sonata based on whether the temperature is over 30,000 Kelvin.

It's not impossible, at least in principle, to create a decision tree or similar structure for handling this.  You could start with dividing things into material objects, like stars, and immaterial ones, like sonatas and fugues.  Within each branch of the tree, only some of the possible properties of things would apply.  After some number of branches, you should reach a point where only a few possible properties apply.  If you're categorizing wetlands, you know that the temperature classifications for stars don't apply, and neither do the various properties used to classify musical forms, but properties like "produces peat" and "has trees" do apply.

In practice, though, even carefully constructed classification systems based on properties, like the Hornbostel-Sachs system for musical instruments discussed in this post, can only go so far.  Property-based systems of classification tend to emphasize particular aspects of the things being categorized, such as (in the case of Hornbostel-Sachs) how they are built and how sound is produced from them.  This often lines up reasonably well with how we use words, but I don't think properties are fundamental.

Rather, how we experience things is fundamental, or at least closer to whatever is fundamental.  Properties describe particular aspects of how we experience something, so it's not surprising that they're useful, but neither should it be surprising that they're not the whole story.

Memory makes you smarter

Another sidebar working up to talking about the hide-and-seek demo.

Few words express more exasperation than "I just told you that!", and -- fairly or not -- there are few things that can lower someone's opinion of another person's cognitive function faster than not remembering simple things.

Ironically for systems that can remember much more data much more permanently and accurately than we ever could, computers often seem to remember very little.  For example, I just tried a couple of online AI chatbots, including one that claimed to have passed a Turing test.  The conversations went something like this:

Me: How are you?
Bot: I'm good.
Me: That's great to hear.  My name is Fred.  My cousin went to the store the other day and bought some soup.
<a bit of typical AI bot chat, pattern-matching what I said and parroting it back, trying stock phrases etc.>
Me: By the way, I just forgot my own name.  What was it?
<some dodge, though one did note that it was a bit silly to forget one's own name>
Me: Do you remember what my cousin bought the other day?
<some other dodge with nothing to do with what I said>

The bots are not even trying to remember the conversation, even in the rudimentary sense of scanning back over the previous text.  They appear to have little to no memory of anything before the last thing the human typed [This has changed with the advent of LLM-driven bots with large context windows.  I argue in later posts that they're still not that smart, but the point here is that lack of short-term memory makes the previous generation of bots seem even less smart, and I think that point stands -- D.H. Sep 2024].

Conversely, web pages suddenly got a lot smarter when sites started using cookies to remember state between visits and again when browsers started to be able to remember things you'd typed in previously.  There's absolutely nothing anyone would call AI going on, but it still makes the difference between "dumb computer" and "not so bad".

When I say "memory" here, I mean the memory of things that happen while the program is running.  Chess engines often incorporate "opening books" of positions that have occurred in previous games, so they can play the first few moves of a typical game without doing any calculation.  Neural networks go through a training phase (whether guided by humans or not).  One way or another, that training data is incorporated into the weightings that determine the networks behavior.

In some sense, those are both a form of memory -- they certainly consume storage on the underlying hardware -- but they're both baked in beforehand.  A chess engine in a tournament is not updating its opening book.  As I understand it, neural network-based chess engines don't update their weights while playing in a tournament, but can do so between rounds (but if you're winning handily, how much do you really want to learn from your opponents' play?).

Likewise, a face recognizer will have been trained on some particular set of faces and non-faces before being set loose on your photo collection.  For better or worse, its choices are not going to change until the next release (unless there's randomization involved).

Chess engines do use memory to their advantage in one way: they tend to remember a "cache" of positions they've already evaluated in determining previous moves.  If you play a response that the engine has already evaluated in detail, it will have a head start in calculating its next move.  This is standard in AB engines, at least (though it may be turned off during tournaments).  I'm not sure how much it applies for NN engines.   To the extent it does apply, I'd say this absolutely counts as "memory makes you smarter".

Overall, though, examples of what we would typically call memory seem to be fairly rare in AI applications.  Most current applications can be framed as processing a particular state of the world without reference to what happened before.  Recognizing a face is just recognizing a face.

Getting a robot moving on a slippery surface is similar, as I understand it.  You take a number of inputs regarding the position and velocity of the various members and whatever visual input you have, and from that calculate what signals to send to the actuators.  There's (probably?) a buffer remembering a small number of seconds worth of inputs, but beyond that, what's past is past (for that matter, there's some evidence that what we perceive as "the present" is basically a buffer of what happened in the past few seconds).

Translating speech to text works well enough a word or phrase at a time, even if remembering more context might (or might not) help with sorting out homonyms and such.   In any case, translators that I'm familiar with clearly aren't gathering context from previous sentences.  It's not even clear they can remember all of the current sentence.

One of the most interesting things about the hide-and-seek demo is that its agents are capable of some sort of more sophisticated memory.  In particular, they can be taught some notion of object permanence, usually defined as the ability to remember that objects exist even when you can't see them directly, as when something is moved behind an opaque barrier.  In purely behavioral terms, you might analyze it as the ability to change behavior in response to objects that aren't directly visible, and the hide-and-seek agents can definitely do that.  Exactly how they do that and what that might imply is what I'm really trying to get to here ...

Intelligence and intelligence

I've been meaning for quite a while to come back to the hide-and-seek AI demo, but while mulling that over I realized something about a distinction I'd made in the first post.  I wanted to mention that brief(-ish-)ly in its own post, since it's not directly related to what I wanted to say about the demo itself.

In the original post, I distinguished between internal notions of intelligence, concerning what processes are behind the observed behavior, and external notions which focus on the behavior itself (note to self: find out what terms actual cogsci/AI researchers use -- or maybe structural and functional would be better?).

Internal definitions on the order of "Something is intelligent if it's capable of learning and dealing with abstract concepts" seem satisfying, even self-evident, until you try to pin down exactly what is meant by "learning" or "abstract concept".  External definitions are, by construction, more objective and measurable, but often seem to call things "intelligent" that we would prefer not to call intelligent at all, or call intelligent in a very limited sense.

The classic example would be chess (transcribing speech and recognizing faces would be others).  For quite a while humans could beat computers at chess, even though even early computers could calculate many more positions than a human, and the assumption was that humans had something -- abstract reasoning, planning, pattern recognition, whatever -- that computers did not have and might never have.  Therefore, humans would always win until computers could reason abstractly, plan, recognize patterns or whatever else it was that only humans could do. In other words, chess clearly required "real intelligence".

Then Deep Blue beat Kasparov through sheer calculation, playing a "positional" style that only humans were supposed to be able to play.  Clearly a machine could beat even the best human players at chess without having anything one could remotely call "learning" or "abstract concepts".  As a corollary, top-notch chess-playing is not a behavior that can be used to define the kind of intelligence we're really interested in.

This is true even with the advent of Alpha Zero and similar neural-network driven engines*. Even if we say, for the sake of the argument, that neural networks are intelligent like we are, the original point still holds.  Things that are clearly unintelligent can play top-notch chess, so "plays top-notch chess" does not imply "intelligent like we are".  If neural networks are intelligent like we are, it won't be because they can play chess, but for other reasons.

The hide-and-seek demo is exciting because on the one hand, it's entirely behavior based.  The agents are trained on the very simple criterion of whether any hiders are visible to the seekers.  On the other hand, though, the agents can develop capabilities, particularly object permanence, that have been recognized as hallmarks of intelligence since before there were computers (there's a longer discussion behind this, which is exactly what I want to get to in the next post on the topic).

In other words, we have a nice, objective external definition that matches up well with internal definitions.  Something that can

• Start with only basic knowledge and capabilities (in this case some simple rules about movement and objects in the simulated environment)
• Develop new behaviors in a competition against agents with the same capabilities

is pretty clearly intelligent in some meaningful sense, even if it doesn't seem as intelligent as us.

If we want to be more precise about "develop new behaviors", we could either single out particular behaviors, like fort building or ramp jumping, or just require that any new agent we're trying to test starts out by losing heavily to the best agents from this demo but learns to beat them, or at least play competitively.

This says nothing about what mechanisms such an agent is using, or how it learns.  This means we might some day run into a situation like chess where something beats the game without actually appearing intelligent in any other sense, maybe some future quantum computer that can simultaneously try out all a huge variety of possible strategies.  Even then, we would learn something interesting.

For now, though, the hide-and-seek demo seems like a significant step forward, both in defining what intelligence might be and in producing it artificially.

* I've discussed Alpha Zero and chess engines in general at length elsewhere in this blog.  My current take is that the ability of neural networks to play moves that appear "creative" to us and to beat purely calculation based (AB) engines doesn't imply intelligence, and that the ability to learn the game from nothing, while impressive, doesn't imply anything like what we think of as human intelligence, even though it's been applied to a number of different abstract games.  That isn't a statement about neural networks in general, just about these particular networks being applied to the specific problem of chess and chess-like games.  There's a lot of interesting work yet to be done with neural networks in general.

What good is half a language?

True Wit is Nature to advantage dress'd
What oft was thought, but ne'er so well express'd
-- Alexander Pope

How did humans come to have language?

There is, to put it mildly, a lot we don't know about this.  Apart from the traditional explanations from various cultures, which are interesting in their own right, academic fields including evolutionary biology, cognitive science and linguistics have had various things to say about the question, so why shouldn't random bloggers?

In what follows, please remember that the title of this blog is Intermittent Conjecture.  I'm not an expert in any of those three fields, though I've had an amateur interest in all three for years and years.  Real research requires careful gathering of evidence and checking of sources, detailed knowledge of the existing literature, extensive review and in general lots of time and effort.  I can confidently state that none of those went into this post, and anything in here should be weighed accordingly.  Also, I'm not claiming any original insight.  Most likely, all the points here have already been made, and better made, by someone else already.

With that said ...

In order to talk about how humans came to have language, the first question to address is what does it mean to have language at all.  Language is so pervasive in human existence that it's surprisingly hard to step back and come up with an objective definition that captures the important features of language and doesn't directly or indirectly amount to "It's that thing people do when they talk (or sign, or write, or ...) in order to communicate information."

We want to delimit, at least roughly, something that includes all the ways we use language, but excludes other activities, including things that we sometimes call "language", but that we somehow know aren't "really" language, say body language, the language of flowers or, ideally, even computer languages, which deliberately share a number of features with human natural languages.

Since language is often considered something unique to humans, or even something that makes us human, it might be tempting to actively try to exclude various ways that other animals communicate, but it seems better to me just to try to pin down what we mean by human language and let the chips fall where they may when it comes to other species.

For me, some of the interesting features of language are

• It can communicate complex, arbitrary structures from one mind to another, however imperfectly.
• It is robust in the face of noise and imperfection (think of shouting in a loud music venue or talking with someone struggling with a second language).
• It tolerates ambiguity, meaning that (unlike in computer languages and other formal systems) ambiguity doesn't bring a conversation to a halt.  In some cases it's even a useful feature.
• Any given language provides multiple ways to express the same basic facts, each with its own particular connotations and emphasis.
• Different languages often express the same basic facts in very different ways.
• Related to these, language is fluid across time and populations.  Usage changes over time and varies across populations.
• It can be communicated by a variety of means, notably speech, signing and writing.
• From an evolutionary point of view, it has survival value.

I'd call these functional properties, meaning that they relate mainly to what language does without saying anything concrete about how it does it.  Structurally (from here on I'll tend to focus on spoken/written language, with the understanding that it's not the whole story),

• Language is linear.

That is, whatever the medium, words are produced and received one at a time, though there can be a number of "side channels" such as pitch and emphasis, facial expressions and hand gestures.

• The mapping between a word and its meaning is largely arbitrary (though you can generally trace a pretty elaborate history involving similar words with similar meanings).
• Vocabulary is extensible.

We can coin words for new concepts.  This is true only for certain kinds of words, but where it can happen it happens easily.

• Meaning is also extensible

We can apply existing words in new senses and again this happens easily.

• The forms used adjust to social conditions.

You speak differently with your peers after work than you would to your boss at work, or to your parents as a child, or to your prospective in-laws, and so forth

• The forms used adjust to particular needs of the conversation, for example which details you want to emphasize (or obscure).
• Some concepts seem to more tightly coupled to the structure of a particular language than others.

For example, when something happened or will happen in relation to when it is spoken of is generally part of the grammar, or marked by a small, closed set of words, or both.

• On the other hand, there is wide variety in exactly how such things are expressed.

Different languages emphasize different distinctions.  For example, some languages don't specially mark singular/plural, or past/present, though of course they can still express that there was more than one of something or that something happened yesterday rather than today.  Different languages use different devices to convey basic information like when something happened or what belongs to whom.

• Syntax, in the form of word order and inflection (changing the forms of words, as with changing dog to dogs or bark to barked or barking), collectively seem to matter in all languages, but the exact way in which they matter, and the degree to which each matters, seem to be unique to any given language.  Even closely related languages generally differ in the exact details.

There are plenty of other features that could each merit a separate post, such as honorifics (Mr. Hull) and diminutives (Davey), or how accent and vocabulary are such devastatingly effective in-group markers, or how metaphors work, or what determines when and how we choose to move words around to focus on a topic, or why some languages build up long words that equate to whole sentences of short words in other languages, or why in some languages directional words like to and of take on grammatical meaning, or why different languages break down verb tenses in different ways, or can use different words for numbers depending on what's being counted, and so on and so on ...

Many of these features of language have to do with the interplay between cognition -- how we think -- and language -- how we express thoughts.  The development of cognition must have been both a driver and a limiting factor in the development of language, but we are almost certainly still in the very early stages of understanding this relationship.

For example, languages generally seem to have a way of nesting one clause inside another, as in The fence that went around the house that was blue was red.  How would this arise?  In order to understand such a sentence, we need some way of setting aside The fence while we deal with that went around the house that was blue and then connecting was red back to the fence in order to understand that the fence is red and the house is blue.  To a compugeek, this means something like a stack, a data structure for storing and retrieving things such that the last thing stored is the first thing retrieved.

Cognitively, handling such a sentence is like veering off a path on some side trip and returning to pick up where you left off, or setting aside a task to handle some interruption and then returning to the original task.  Neither of these abilities is anywhere near unique to humans, so they must older than humanity, even though we are the only animals that we know of that seem to use them in communication.

These cognitive abilities are also completely separate from a large number of individual adaptations of our vocal apparatus, which do seem to be unique to us, notably fine control of breathing and of the position of the tongue and shape of the mouth.  While these adaptations are essential to our being able to speak as fluently as we do, they don't have anything to do with what kinds of sentences we can express, just how well we can do so using spoken words.  Sign languages get along perfectly well without them.

In other words, it's quite possible we were able to conceive of structures like "I saw that the lion that killed the wildebeest went around behind that hill over there" without being able to put them into words, and that ability only came along later.  There's certainly no shortage, even in modern humans, of things that are easy to think but hard to express (I'd give a few examples, but ...).  The question here, then, is not "How did we develop the ability to think in nested clauses?" but "How did we come to use the grammatical structures we now see in languages to communicate such thoughts?"

There's a lot to evolution, and it has to be right up there with quantum mechanics as far as scientific theories that are easy to oversimplify, draw unwarranted conclusions from, or to get outright wrong, so this next bit is even less precise than what I've already said.  For example, I'm completely skirting around major issues of population genetics -- how a gene (useful or not) spreads (or doesn't) in a population.

Let's try to consider vocabulary in an evolutionary context.  I pick vocabulary to start with because it's clearly distinct from grammar.  Indeed one of the useful features of a grammar is that you can plug an arbitrary set of words into it.  Conversely, one requirement for developing language as we know it is the ability to learn and use a large and expandable vocabulary.  Without that, and regardless of the grammatical apparatus, we do not account for the way people actually use language.

Suppose some animal has the ability to make one kind of call when a it spots particular predator and a different call for another predator, in such a way that is conspecifics (animals of the same species) can understand and react appropriately.  That's two calls (three if you count not making any call) and it's easy to see how that could be useful in not getting eaten.  Again, this is far from unique to us (see here, and search for "vervets" in the post, for example).

Now suppose some particular animal is born with the ability to make a third call for some other hazard, say a large branch falling (this is more than a bit contrived, but bear with me).  A large branch falls, the animal cries out ... and no one does anything.  The ability to make new calls isn't particularly useful without the ability to understand new calls.  But suppose that nobody did anything because they didn't know what the new call meant, but they were able to connect "that oddball over there made a funny noise" with "a big branch fell".  The next time a big branch falls and our three-call-making friend cries out, everyone looks out and scatters to safety.  Progress.

I'm more than a bit skeptical that the ability to make three calls rather than two would arise by a lucky mutation, but I think there are still two valid points here:

First, the ability to comprehend probably runs ahead of the ability to express, and certainly new ways to express are much less likely to catch on if no one understands what they mean.  Moreover, comprehension is useful in and of itself.  Whether or not my species is able to make calls that signal specific scenarios, being able to understand other species' calls is very useful (when a vervet makes a predator call, other species will take appropriate action as well), as is the ability to match up new calls with their meanings from context and examples.

In other words, the ability to understand a large vocabulary is liable to develop even without the ability to express a large vocabulary.  For a real-life example, at least some domestic dogs can understand many more human words than (as far as anyone can tell) they can produce distinct barks and similar sounds, and certainly more human words than they can themselves produce.

Second, this appears to be a very common pattern in evolution.  Abilities that are useful in one context (distinguishing the different calls of animals around you) become useful in other contexts (developing a system of specialized calls within your own species).  The general pattern is known as exaptation (or cooption, or formerly and more confusingly as pre-adaptation).

Let's suppose that the local population of some species can potentially understand, say, dozens of distinct calls (whether their own or those of other species), but its ability to produce distinct calls is limited.  If some individual comes along with the gift of being able to produce more distinct calls, then that will probably increase that individual's chances of surviving -- because its conspecifics will learn the new calls and so increase everyone's chance of survival -- and at least potentially its chances of reproducing, if only because there will be more potential mates around if fewer of them get eaten. 

If that particular individual fails to survive and reproduce, the conditions are still good for some other individual to come along with the ability to produce a bigger vocabulary, perhaps through some entirely different mechanism.  This in turn doesn't preclude some future individual from being born with the ability to produce a larger vocabulary through yet a third mechanism, or either of the original two.  If there are multiple mechanisms for doing something advantageous, the chances of it taking hold in the long run are better (I'm pretty sure, but I don't know if an actual biologist would agree.  Also, this isn't particular to vocabulary.).

If the community as a whole develops the tendency to find larger vocabularies attractive, so much the better, though the math starts to get hairy at this point.  Sexual selection is a pretty good way of driving traits to extremes -- think peacocks and male walruses -- so it's quite plausible that a species that starts to develop larger and larger vocabularies of calls could take this quite far, past the point of immediate usefulness.  You then have a population with a large vocabulary ready for an environment where it makes more of a difference.

In short, even some ability to produce distinct calls for different situations is useful, and it's no surprise many animals have it.  The ability to produce a large and expandable variety of distinct calls for different situations also looks useful, but also seems harder to evolve, considering that it's fairly rare.  Taking this a step further, we appear to be unique in our ability to produce and distinguish thousands of distinct vocabulary items, though as always there's quite a bit we still don't know about communication in other species.

It's clear that other animals can distinguish, and in some cases produce, non-trivial vocabularies, even if it's not particularly common.  How do you get from there to our as-far-as-we-know-unique abilities?  The usual answer for how complex traits evolve is "a piece at a time".

In order to find a (very hypothetical) evolutionary pathway from an extensible collection of specialized calls to what we call language today, we want to find a series of small steps that each add something useful to what's already there without requiring major restructuring.  Some of those, in no strict order except where logically necessary, might be:

• The ability to refer to a class of things without reference to a particular instance

This is one aspect of what one might call "abstract concepts".  As such, it doesn't require any new linguistic machinery beyond the ability to make and distinguish a reasonably large set of calls (which I'll call words from here on out), but it does require a cognitive shift.  The speaker has to be able to think of, say, wolf without referring to a particular wolf trying to sneak up.  The listener has to realize that someone saying "wolf" may not be referring to a wolf currently sneaking up on them. Instead, if the speaker is pointing to a set of tracks it might mean "a wolf went here", or if pointing in a particular direction, maybe "wolves come from over there".

This may seem completely natural to us, but it's not clear who, if anyone else besides us, can do this.   Lots of animals can distinguish different types of things, but being able to classify is different from being aware that classes exist.  An apple-sorting machine can sort big from small without understanding "big" or "small".  I say "it's not clear" because devising an experiment to tell if something does or doesn't understand some aspect of abstraction is difficult, in no small part because there's a lot of room for interpretation of the results.

[Re-reading this and taking Earl's comment into account, I think I've conflated two different kinds of abstraction.  Classifying "wolf" as opposed to "a wolf" is probably more basic than I have it here.  For example, a vervet will give the call for a particular kind of predator.  It doesn't have to develop a call for each individual animal, and a good thing, because that would only work for individuals that it had seen before.  A behavioralist would probably say this is all stimulus-response based on particular characteristics -- give the leopard call in response to the smell or sound of a leopard, and so on, and fair enough.  

Classification, then, would be more a matter of connecting particular attributes -- sound, smell, shape or whatever -- to physical things with those attributes.  The progression, as Earl suggests, would be leopard smell/sound --> leopard --> the leopard that just disappeared into the trees over there.  That is, it requires more cognitive machinery to be able to conceive of leopard as any animal that smells this way and/or makes this sound and/or has pointy ears and a tail or whatever, and it requires a different piece of machinery  -- a sort of object permanence -- to conceive of an absent leopard and connect that thing that was here but isn't any more to leopard and get that leopard that was here but isn't any more -- D.H 28 Oct. 2021]

• The ability to designate a quality such as "big" or "red" without reference to any particular thing with that quality.

This is similar to the previous item, but for adjectives rather than nouns.  From a language standpoint it's important because it implies that you can mix and match qualities and things (adjectives and nouns).  A tree can be big, a wolf can be big and a wolf can be gray without needing a separate notion of "big tree", "big wolf" and "gray wolf".  An adjective is a predicate that applies to something rather than standing alone as a noun does.

As I understand it, the widely-recognized stages of language development in humans are babbling, single words, two-word sentences and "all hell breaks loose".  A brain that can handle nouns and predicates is ready for two-word sentences consisting of a predicate and something it applies to.  This is a very significant step in communication and it appears to be quite rare, but linguistically it's nearly trivial.  A grammar to describe it has one rule and no recursion (rules that refer, directly or indirectly, to themselves).

As a practical matter, producing a two-word sentence means signifying a predicate and an object that it applies to (called an argument).  Understanding it means understanding the predicate, understanding the argument and, crucially, understanding that the predicate applies to the argument.  If you can distinguish predicates from objects, order doesn't even matter.  "Big wolf!" is just as good as "Wolf big!" or even a panicked sequence of "Wolf wolf big wolf big big wolf!" (which, to be fair, would require recursion to describe in a phrase-structure grammar).

From a functional point of view, the limiting factor to communicating such concepts is not grammar but the ability to form and understand the concepts in the first place.

Where do we go from predicate/argument sentences to something resembling what we now call language?  Some possible next steps might be

• Predicates with more than one argument.

The important part here is that you need a way to distinguish the arguments.  In wolf big, you know that big is the predicate and wolf is the argument and that's all you need, but in see rabbit wolf, where see is the predicate and rabbit and wolf are arguments, how do we tell if the wolf sees the rabbit or the rabbit sees the wolf?  There are two solutions, given that you're limited to putting words together in some particular order

Either the order of words matters, so see rabbit wolf means one thing and see wolf rabbit means the other, or there's a way of marking words according to what role they play, so for example see wolf-at rabbit means the rabbit sees the wolf and see wolf rabbit-at means the wolf sees the rabbit.  There are lots of possible variations, and the two approaches can be combined.  Actual languages do both, in a wide variety of ways.

From a linguistic point of view, word order and inflection (ways of marking words) are the elements of syntax, which (roughly speaking) provides structure on top of a raw stream of words.  Languages apply syntax in a number of ways, allowing us to put together complex sentences such as this one, but you need the same basic tools even for simple three-word sentences.  Turning that around, if you can solve the problem of distinguishing the meaning of a predicate and two arguments, you have a significant portion of the machinery needed for more complex sentences.

• Pronouns, that is, a way to designate a placeholder for something without saying exactly what that something is, and connect it with a specific meaning separately.

Cognitively, pronouns imply some form of memory beyond the scope of a simple sentence. Linguistically, their key property is that their meaning can be redefined on the fly.  A noun like wolf might refer to different specific wolves at different times, but it will always refer to some wolf.  A pronoun like it is much less restrained.  It could refer to any noun, depending on context.

Pronouns allow for more compact sentences, which is useful in itself since you don't have to repeat some long descriptive phrase every time you want to say something new about, say, the big red house across the street with the oak tree in the yard.  You can just say that house or just it if the context is clear enough.

More than this, though, by equating two things in separate sentences they allow linear sequences of words to describe non-linear structures, for example I see a wolf and it sees me.  By contrast, in I see a wolf and a wolf sees me it's not clear whether it's the same wolf and we don't necessarily have the circular structure of two things seeing each other.

• The ability to stack up arbitrarily many predicates: big dog, big red dog, big red hairy dog, etc.

I left this for last because it leads into a bit of a rabbit hole concerning the role of nesting and recursion in language.  I'm going to dig into that a bit here by way of arguing that some of the analytic tools commonly used in analyzing language may not be particularly relevant to its development.  Put another way, "how did language develop" is not the same question as "how did the structures we work with in analyzing language develop".

A common analysis of phrases like big red hairy dog uses a recursive set of rules like

a noun phrase can be a noun by itself, or
a noun phrase can be an adjective followed by a noun phrase

This is much simpler than a full definition of noun phrase in a real grammar, and it's not the only way to analyze noun phrases, but it shows the recursive pattern that's often used in such an analysis.  The second definition of noun phrase refers to noun phrase recursively.  The noun phrase on the right-hand side will be smaller, since it has one less adjective, so there's no infinite regress.  The example, big red hairy dog, breaks down to big modifying red hairy dog, which breaks down to red modifying hairy dog, which breaks down to hairy modifying dog, and dog is a noun phrase by itself.  In all there are four noun phrases, one by the first rule and three by the second.

On the other hand, if you can conceive of a dog being big, red and hairy at the same time, you can just as well express this with two-word sentences and a pronoun:  dog big. it red. it hairy.  The same construction could even make sense without the pronouns: dog big. red. hairy.  Here a listener might naturally assume that red and hairy have to apply to something, and the last thing we were talking about was a dog, so the dog must be red and hairy as well as big.

This is not particularly different from someone saying I saw the movie about the duck.  Didn't like it, where the second sentence clearly means I didn't like it and you could even just say Didn't like and still be clearly understood, even if Didn't like by itself sounds a bit odd.

From a grammatical standpoint (at least for a constituency grammar) these all seem quite different.  In big red hairy dog, there's presumed to be a nested structure of noun phrases.  In dog big.  it red. it hairy you have three sentences with a simple noun-verb structure and in dog big. red. hairy. you have one two-word sentence and two fragments that aren't even sentences.

However, from the point of view of "I have some notion of predicates and arguments, and multiple predicates can apply to the same argument, now how do I put that in words?", they seem pretty similar.  In all three cases you say the argument and the predicates that apply to it and the listener understands that the predicates apply to the argument because that's what predicates do.


I started this post with the idea of exploring how language as we now know it could develop from simpler pieces such as those we can see in other animals.  The title is a nod to the question of What good is half an eye? regarding the evolution of complex eyes such as we see in several lineages, including our own and (in a different form) in cephalopods.  In that case, it turns out that there are several intermediate forms which provide an advantage even though they're not what we would call fully-formed eyes, and it's not hard to trace a plausible pathway from basic light-sensitive "eye spots" to what we and many other animals have.

The case of language seems similar.  I think the key points are

• Cognition is crucial.  You can't express what you can't conceive of.
• The ability to understand almost certainly runs ahead of the ability to express.
• There are plausibly a number intermediate stages between simple calls and complex language (again, the account above is completely speculative and I don't claim to have identified the actual steps precisely or completely).
• Full grammar, in the sense of nested structures described by recursive rules, may not be a particularly crucial step.
• A purely grammatical analysis may even obscure the picture, both by failing to make distinctions (as with the jump from "wolf" meaning "this wolf right there" to it meaning "wolf" in the abstract) and by drawing distinctions that aren't particularly relevant (as with the various forms of big red hairy dog).