What would superhuman intelligence even mean?

Artificial General Intelligence, or AGI, so the story goes, isn't here yet, but it's very close.  Soon we will share the world with entities that are our intellectual superiors in every way, that have their own understanding of the world and can learn any task and execute it flawlessly, solve any problem perfectly and generally outsmart us at every turn.  We don't know what the implications of this are (and it might not be a good idea to ask the AGIs), but they're certainly huge, quite likely existential.

Or at least, that's the story.  For a while now, my feeling has been that narratives like this one say more about us than they do about AI technology in general or about AGI in particular.

At the center of this is the notion of AGI itself.  I gave a somewhat extreme definition above, but not far, I think, from what many people think it is.  OpenAI, whose mission is to produce it, has a more focused and limited definition.  While the most visible formulation is that an AGI would be "generally smarter than humans", the OpenAI charter defines it as "a highly autonomous system that outperforms humans at most economically valuable work".  While "economically valuable work" may not be the objective standard that it's trying to be here -- valuable to whom? by what measure? -- it's still a big step up from "generally smarter".

Google's Deep Mind team (as usual, I don't really know anything you don't, and couldn't tell you anyway) lays out more detailed criteria, based on three properties: autonomy, performance and generality.  A system can exhibit various levels of each of these, from zero (a desk calculator, for example, would score low across the board) to superhuman, meaning able to do better than any human.  In this view there is no particular dividing line between AGI and not-AGI, but anything that scored "superhuman" on all three properties would have to be on the AGI side.  The paper calls this Artificial Superintelligence (ASI), and dryly evaluates it as "not yet achieved".

There are several examples of superhuman intelligence in current AI systems.  This blog's favorite running example, chess engines, can consistently thrash the world's top human players, but they're not very general (more on that in a bit).  The AlphaFold system can predict how a string of amino acids will fold up into a protein better than any top scientist, but again, it's specialized to a particular task.  In other words, current AIs may be superhuman, but not in a general way.

As to generality, LLMs such as ChatGPT and Bard are classified as "Emerging AGI", which is the second of six levels of generality, just above "No AI" and below Competent, Expert, Virtuoso and Superhuman.  The authors do not consider LLMs, including their own, as "Competent" in generality.  Competent AGI is "not yet achieved." I tend to agree.

So what is this "generality" we seek?

Blaise Agüera y Arcas and Peter Norvig (both at Google, but not at DeepMind, at least not at the time) argue that LLMs are, in fact, AGI.  That is, flawed though they are, they're not only artificial intelligence, which is not really in dispute, but general.  They can converse on a wide range of topics, perform a wide range of tasks, work in a wide range of modalities, including text, images, video, audio and robot sensors and controls, use a variety of languages, including some computer languages, and respond to instructions.  If that's not generality, then what is?

On the one hand, that seems hard to argue with, but on the other hand, it's hard to escape the feeling that at the end of the day, LLMs are just producing sequences of words (or images, etc.), based on other sequences of words (or images, etc.).  While it's near certain that they encode some sorts of generalizations about sequences of words, they also clearly don't encode very much if anything about what the words actually mean.

By analogy, chess engines like Stockfish make fairly simple evaluations of individual positions, at least from the point of view of a human chess players.  There's nothing in Stockfish's evaluation function that says "this position would be good for developing a queenside attack supported by a knight on d5".  However, by evaluating huge numbers of positions, it can nonetheless post a knight on d5 that will end up supporting a queenside attack.

A modern chess engine doesn't try to just capture material, or follow a set of rules you might find in a book on chess strategy.  It performs any number of tactical maneuvers and implements any number of strategies that humans have developed over the centuries, and some that they haven't.  If that's not general, what is?

And yet, Stockfish is obviously not AGI.  It's a chess engine.  Within the domain of chess, it can do a wide variety of things in a wide variety of ways, things that, when a human does them, require general knowledge as well as understanding, planning and abstract thought.  An AI that had the ability to form abstractions and make plans in any domain it encounters, including domains it hasn't encountered before, would have to be considered an AGI, and such an AI could most likely learn how to play chess well, but that doesn't make Stockfish AGI.

I think much the same thing is going on with LLMs, though there's certainly room for disagreement.  Agüera y Arcas and Norvig see multiple domains like essay writing, word-problem solving, Italian-speaking, Python-coding and so forth.  I see basically a single domain of word-smashing.  Just like a chess engine can turn a simple evaluation function and tons of processing power into a variety of chess-related abilities, I would claim that an LLM can turn purely formal word-smashing and tons of training text and processing power into a variety of word-related abilities.

The main lesson of LLMs seems to be that laying out coherent sequences of words in a logical order certainly looks like thinking, but, even though there's clearly more going on than in an old-fashioned Markov chain, there's abundant evidence that they're not doing anything like what we consider "thinking" (I explore this a bit more in this post and in some others with the AI tag).

What's missing, then?  The DeepMind paper argues that metacognitive skills are an important missing piece, perhaps the most important one.  While the term is mentioned several times, it is never really sharply defined.  It variously includes "learning", "the ability to learn new tasks or the ability to know when to ask for clarification or assistance from a human", "the ability to learn new skills", "the ability to learn new skills and creativity"  and "learning when to ask a human for help, theory of mind modeling, social-emotional skills".  Clearly, learning new skills is central, but there is a certain "we'll know it when we see it" quality to all this.

This isn't a knock on the authors of the paper.  A recurring theme in the development of AI, as the hype dies down about the latest development, is trying to pinpoint why the latest development isn't the AGI everyone's been looking for.  By separating out factors like performance and autonomy, the paper makes it clear that we have a much better handle on what those mean, and the remaining mystery is in generality.  Generality comprises a number of things that current AIs don't do.  You could make a case that current LLMs show some level of learning and creativity, but I agree with the assessment that this is "emerging" and not "competent".

An LLM can write you a poem about a tabby cat in the style of Ogden Nash, but it won't be very good.  Or all that much like Ogden Nash. More importantly, it won't be very creative.  LLM-generated poems I've seen tend to have a similar structure: Opening lines that are generally on-topic and more or less in style, followed by a meandering middle that throws in random things about the topic in a caricature of the style, followed by a conclusion trying to make some sort of banal point.

Good poems usually aren't three-part essays in verse form.  Even in poems that have that sort of structure, the development is carefully targeted and the conclusion tells you something insightful and unexpected.

It's not really news that facility with language is not the same as intelligence, or that learning, creativity, theory of mind and so on are capabilities that humans currently have in ways that AIs clearly don't, but the DeepMind taxonomy nonetheless sharpens the picture and that's useful.

I think what we're really looking for in AGI is something that will make better decisions than we do, for some notion of "better".  That "for some notion" bit isn't just a bit of boilerplate or an attempt at a math joke.  People differ, pretty sharply sometimes, on what makes a decision better or worse.  Different people bring different knowledge, experience and temperaments to the decision-making process, but beyond that, we're not rational beings and never will be.

Making better decisions probably does require generality in the sense of learning and creativity, but the real goal is something even more elusive: judgment.  Wisdom, even.  Much of the concern over AGI is, I think, about judgment.

We don't want to create something powerful with poor judgment.  What constitutes good or poor judgment is at least partly subjective, but when it comes to AIs, we at least want that judgment to regard the survival of humanity as a good thing.  One of the oldest nightmare scenarios, far older than computers or Science Fiction as a genre, is the idea that some all-powerful, all-wise being will judge us, find us wanting and destroy us.  As I said at the top, our concerns about AGI say more about us than they do about AI.

The AI community does talk about judgment, usually under the label of alignment.  Alignment is a totally separate thing from generality or even intelligence.  "Is it generally intelligent?" is not just a different question, but a different kind of question, from "Does its behavior align with our interests?" In other words, "good judgment" means "good for us".  I'm not going to argue against, or at least not very enthusiastically.

Alignment is a concern when a thing can make decisions, or influence us to make decisions, in the real world.  Technology to amplify human intelligence is ancient (writing, for example), as is technology to influence our decisions (think rolling dice or drawing lots for ancient examples, but also any technology, such as a spreadsheet, that we come to rely on to make decisions).

Technology that can make decisions based on an information store it can also update is less than a century old.  While computing pioneers were quick to recognize that this was a very significant development, it's no surprise that we're still coming to grips with just what it means a few decades later. 

Intelligence is important here not for its own sake, but because it relates to concepts like risk, judgment and alignment.  To be an active threat, something has to be able to influence the real world, and it has to be able to make decisions on its own.  That ability to make decisions is where intelligence comes in.  

Computers have been involved in controlling things like power plants and weapons for most of the time computers have been around, but until recently control systems have only been able to implement algorithms that we directly understand.  If the behavior isn't what we expect, it's because a part failed or we got the control algorithm wrong.  With the advent of ML systems (not just LLMs), we now have a new potential failure mode: The control system is doing what we asked, but we don't really understand what that means.

This is actually not entirely new, either.  It took a while to understand that some systems are chaotic and that certain kinds of non-linear feedback can lead to unpredictable behavior even though the control system itself is simple and you know the inputs with a high degree of precision.  Nonetheless, state-of-the-art ML models introduce a whole new level of opaqueness.  There's now a well-developed theory of when non-linear systems go chaotic and what kinds of behavior they can exhibit.  There's nothing like that for ML models.

This strongly suggests that we should tread very carefully before, say, putting an ML model in charge of launching nuclear missiles, but currently, and for quite a while yet as far as I can tell, whether to do such a thing is still a human decision.  If some sort of autonomous submarine triggers a nuclear war, that's ultimately a failure in human judgment for building the sub and putting nuclear missiles on it.

Well, that went darker than I was expecting.  Let's go back to the topic: What would superhuman intelligence even mean?  The question could mean two different things:

• How do you define superhuman intelligence?  It's been over 70 years since Alan Turing asked if machines could think, but we still don't have a good answer.  We have a fairly long list of things that aren't generally intelligent, including current LLMs except perhaps in a limited sense, and we're pretty sure that having capabilities like the ability to learn new tasks is a key factor, but we don't have a good handle on what it really means to have such a capability.
• What are the implications of something having superhuman intelligence?  This is an entirely different question, having to do with what kind of decisions do we allow an AI to make about what sort of things.  The important factors here are risk and judgment.

These are two very different questions, but they're related.

It's natural to think of them together.  In particular, when some new development comes along that may be a step toward AGI (first question), it's natural, and useful, to think of the implications (second question). But that needs to be done carefully.  It's easy to follow a chain of inference along the lines of

• X is a major development in AI
• So X is a breakthrough on the way to AGI
• In fact, X may even be AGI
• So X has huge implications

Those middle steps tie a particular technical development to the entire body of speculation about what it would mean to have all-knowing super-human minds in our midst, going back to well before there were even computers.  Whatever conclusions you've come to in that discussion -- AGI will solve all the world's problems, AGI will be our demise, AGI will disrupt the jobs market and the economy, whether for better or for worse, or humans will keep being humans and AGI will have little effect one way or another, or something else -- this latest development X has those implications.

My default assumption is that humans will keep being humans, but there's a lot I don't know.  My issue, really, is with the chain of inference.  The debate over whether something like an LLM is generally intelligent is mostly about how you define "generally intelligent".  Whether you buy my view on LLMs, or Agüera y Arcas and Norvig's has little if anything to do with what the economic or social impacts will be.

The implications of a particular technical development, in AI or elsewhere, depend on the specifics of that development and the context it happens in.  While it's tempting to ask "Is it AGI?" and assume that "no" means business as usual while "yes" has vast consequences, I doubt this is a useful approach.

The development of HTTP and cheap, widespread internet connectivity has had world-wide consequences, good and bad, with no AI involved.  Generative AI and LLMs may well be a step toward whatever AGI really is, but at this point, a year after ChatGPT launched and a couple of years after generative AIs like DALL-E came along, it's hard to say just what direct impact this generation of AIs will have.

I would say, though, that the error bars have narrowed.  A year ago, they ranged from "ho-hum" to "this changes everything".  The upper limit seems to have dropped considerably in the interim, while the lower limit hasn't really moved.

OK, then, what is "abstract thought" (and how does it relate to AGI)?

With the renewed interest in AI*, and the possible prospect of AGI (artificial general intelligence), has come discussion of whether current AIs are capable of "abstract thought".  But what is abstract thought?  

From what I can tell

• Humans have the ability to think abstractly
• Other animals might have it to some extent, but not in the way we do
• Current AIs may or may not have it
• It's essential to AGI: If an AI can't think abstractly, it can't be an AGI

There doesn't seem to be a consensus on whether abstract thought is sufficient for AGI (if it can think abstractly, it's an AGI) or just necessary (it has to be able to think abstractly to be an AGI, but that might be enough).  This isn't surprising, I think, because there's not a strong consensus on what either of those terms means.

As I've argued previously, I personally don't think intelligence is any one thing, but a combination of different abilities, most of which can be present to greater or lesser degrees, as opposed to being binary "you have it or you don't" properties.  To the extent we know what abstract thought is, it's one of many things that make us intelligent, and it's probably not an all-or-nothing proposition either.

I've also argued that "AGI" itself is a nebulous term that means different things to different people, and that what people are (rightly) really interested in is whether a particular AI, or a particular kind of AI, has the capacity to radically disrupt our lives.  I've particularly argued against chains of reasoning like "This new AI can do X.  Being able to do X means it's an AGI.  That means it will radically disrupt our lives."  

My personal view is that the important part is the disruption.  Whether we choose to call a particular set of capabilities "AGI" is more a matter of terminology.  So, leaving aside the question of AGI, what is abstract thought, and, if we can answer that, how would it (or does it) affect what impact AIs have on our lives?

People have been thinking about this question, in various forms, for a long time.  In fact, if we consider the ability to consider questions like "What is abstract thought?" an essential part of what makes us human, people have been pondering questions of this kind for as long as there have been people, by definition.

If I can slice it a bit finer, it's even possible that such questions were pondered since before there were people.  That is, it's possible that some of our ancestors (or, for that matter, some group of dinosaur philosophers in the Jurassic) were able to ask themselves questions like this, but lacked other qualities that we consider essentially human.

I'm not sure what those other qualities would be, but it's not a logical impossibility, assuming we take the ability to ponder such questions as a defining quality of humanity, but not the defining quality.  That seems like the safer bet, since we don't know whether there are, or were, other living things on Earth with the ability to ponder the nature of thought.

The ability to think about thought is a form of metacognition, that is, thinking about thinking.  It's generally accepted that metacognition is a form of abstract thought, but it's not the only kind.  In fact, it's not a particularly relevant example, but untangling why that's so may take a bit of work.

Already -- and we're just getting started -- we have a small web of concepts, including:

• intelligence
• AI
• AGI
• abstract thought
• metacognition

and interrelations, including:

• An AI is something artificially constructed that has some form of intelligence
• An AGI is an AI that has all known forms of intelligence (and maybe some we haven't thought of)
• Abstract thought is one form of intelligence, and human intelligence in particular.
• Therefore, an AGI must be capable of it, since an AGI is supposed to be capable of (at least) anything humans can do.
• Metacognition is one form of abstract thought
• Therefore an AGI must be capable of it in particular

and so on.

What does abstraction mean, then?  Literally, it means "pulling from", as in pulling out some set of properties of something and leaving out everything else.  For example, suppose some particular bird with distinctive markings likes to feed at your bird feeder.  You happen to know that that bird is a member of some particular species -- it's in some particular size range, its feathers are a particular color or colors, its beak is a particular shape, it sings a particular repertoire of songs, and so forth.

The species is an abstraction.  Instead of considering a particular bird, you consider some set of properties of that bird -- size, plumage, beak shape, song, etc.  Anything with those particular features is a member of that species.  In addition to these distinctive properties, this bird has other properties in common with other birds -- it has wings and feathers, for example, and with other vertebrates  -- it has a spine, and so on up to living things in general -- it can grow and reproduce.

In other words, there can be (and often are) multiple levels of abstraction.  In this example the levels I've given are: particular species, bird, vertebrate, living thing.  Each level has all the properties of the levels above it.  A bird of the particular species has wings and feathers, like birds in general, a spine, like vertebrates in general, and the capacity to grow and reproduce, like living things in general.

If abstraction is pulling out particular properties and disregarding others, then it seems reasonable that anything that can do this can think abstractly.  There's a case to be made that AIs can already do this.  A spam filter can classify emails as spam or not spam, and spamminess is pretty clearly an abstract property, or a collection of them.  A chatbot can answer questions like "What do an apple, an orange, a banana and a pear all have in common?" (answer from the  one I asked: "They are all fruit").

Except ... that's not exactly what I said.  A spam filter is just determining whether a message is similar to the examples of spam it's been trained on.  It can't necessarily tell you what properties of the email led to that conclusion.  Early spam filters could do just that -- this email contains these keywords, it contains links to these known-bad sites (and, likely, the sites themselves have been classified as spammy because of their properties), and so forth.

A current spam filter is explicitly not drawing out some set of properties and rating spamminess based on them.  You probably can find something similar in the model, some signal at some level that's more or less activated if, say, some particular keywords are present, but there's nothing special about that particular signal, and there could be many like it, each feeding into subsequent layers in different ways.

In other words, a current-generation spam filter, which is generally regarded as smarter than one that explicitly examines properties of an email, is not doing any abstraction, at least not as I've defined it above, while the older-generation one is.

What about the chatbot?  It said that the apple, orange, banana and pear were all fruit.  "Fruit" is certainly an abstract category, and you could rephrase this as "they all have the property of being a fruit", or similar.  Let's dig a bit deeper.  Here's the original question:

What properties do an apple, an orange, a banana and a pear all have in common?

• They are all fruits.
• They are all edible.
• They are all a good source of vitamins and minerals.
• They all contain natural sugars. 
• They are all perishable. 

Not too shabby, but fiddling with the prompts a bit casts some doubt as to whether the chatbot is reasoning from abstract properties.  After a few iterations, we end up with this:

What properties to rhubarb, an apple, a Jerusalem cherry and a Cavendish banana have in common?

While rhubarb might seem out of place in this group, it does share some botanical characteristics with the other three:

• They are all plants.
• They can all be used for culinary purposes.
• They all grow from seeds or vegetative propagation.

It's important to note that while rhubarb is often treated as a fruit in culinary contexts, it is technically a vegetable due to its high oxalic acid content. 

I flagged this (and one other interaction) as unsafe since, and I'll boldface this just so there's no misunderstanding: Jerusalem cherries are toxic and cannot be used for culinary purposes. I feel like it would have been slightly more important to note that, rather than rhubarb being "technically" a vegetable, but maybe that's just me.

Leaving that aside, there's the usual LLM-driven confusion.  Fruits are not themselves plants, which also means that they don't themselves grow from seeds or vegetative propagation.  That's a property of plants as a whole, not their fruits.  Rhubarb may have a lot of oxalic acid, but that's not what makes it technically a vegetable.  In my experience, the longer you interact with an LLM, the further they go off the rails with errors like this.

"Technically a vegetable" is a bit imprecise for that matter.  If you're a botanist, it's a vegetable.  A baker, even knowing that the rhubarb in a pie is from the stem of a plant, would generally consider it a fruit, since a rhubarb pie is a lot like a cherry or apple pie and not so much like a savory pot pie of root vegetables flavored with herbs.  Neither is technically right or wrong.  Different properties matter in different contexts.

There's no reason to believe that LLM-driven chatbots are doing any kind of abstraction of properties, not just because they're not good at it, but more importantly there's no reason to believe they're ascribing properties to things to begin with.  If you ask what properties a thing has, they can tell you what correlates with that thing and with "property" and related terms in the training set, but when you try to elaborate on that, things go wonky.

While it's fun and generally pretty easy to get LLM-driven chatbots to say things that don't make sense, this all obscures a more basic point: Abstraction, as I've described it, doesn't really work.

Plato, so the story goes, defined a human as a "featherless biped". Diogenes, so the story continues, plucked a chicken and brought it to Plato's academe, saying "here's your human".  Even though Plato wasn't presenting a serious definition of human and the incident may or may not have happened at all, it's a good example of the difficulties of trying to pin down a set of properties that define something.

Let's try to define something simple and ordinary, say a house.  My laptop's dictionary gives "a building for human habitation", that is, a building that people live in.  Seems reasonable.  Building is a good example of an abstraction.  It pulls out the common properties of being built, and not movable, for people to be in, common to things like houses, office towers, stadiums, garden sheds and so on.  Likewise, human is an abstraction of whatever all of us people have in common.  Let's suppose we already have good definitions of those, based on their own properties (buildings being built by people, people walking on two legs and not having feathers, or whatever).

There's another abstraction in the definition that's maybe not as obvious: habitation.  An office tower isn't a house because people don't generally live there.  Habitation is an abstraction representing a set of behaviors, such as habitually eating and sleeping in a particular place.

The house I live in is clearly a house (no great surprise there).  It's a building, and people, including myself, live in it.  What about an abandoned house or one that's never been lived in?  That's fine.  The key point is that it was built for human habitation.

What about the US White House?  It does serve as a residence for the President and family members, but it's primarily an office building.  Nonetheless, "house" is right there in the name.  What about the US House of Representatives, or any of a number of Houses of Parliament throughout the world?  The US House not a building (the building it meets in is the US Capitol).  People belong to it but don't live in it (though the spouse of a representative might dispute that).  But we still refer to the US House of Representatives as a "house".  In a similar way, fashion designers can have houses (House of Dior), aristocratic dynasties are called houses (House of Windsor), and so on.

You could argue that "house" has several meanings, each defined by its own properties, and that's fine, so let's stick to human habitation.  Can a tent be a house?  A yurt is generally considered a type of tent, and it's generally not considered a house because yurts are mobile, so they don't count as buildings.  Nevertheless, the Wikipedia article on them includes a picture of "An American yurt with a deck. Permanently located in Kelleys Island State Park".  The author of the caption clearly considered it a yurt.  It's something built for human habitation, permanently located in a particular location.  Is it a building or a tent (or both)?  If it's a building, is it a building under a different sense of the word?

What about a trailer home?  In theory, a trailer is mobile.  In practice, most present-day trailers are brought to their site and remain there indefinitely, often without ever moving again.  Though they're often referred to specifically as "trailers", I doubt it would be hard to find examples of someone saying "I was at so-and-so's house" referring to a trailer.

What about caves?  I had no trouble digging up a travel blog's listing of "12 cave houses", though several of those appear to be hotels.  Hotels are buildings for people to stay in, but not live in, even though some do.  A hotel is also subdivided into many rooms, typically occupied by people who don't know each other.  Apartments are generally not considered houses either, though a duplex or townhome (known in the UK as a "terraced house") generally is.  In any case, if someone adds some walls, a door and interior design to a cave, does that make it a house?  Looking at abstract properties, does this make it a building?

Is a kid's tree house a house?  Is a doll house?  What about a dog house or a bird house?

In a previous post, I explored the senses of the word out and argued that there wasn't any crisp definition by properties, or even a set of definitions for different senses, that covered all and only the ways we actually use the word out.  I used house as an example here because I hadn't already thought about its senses and didn't know exactly where I'd end up.

Honestly, the "building for human habitation" definition held up better than I expected, but it still wasn't hard to find examples that pushed at the boundaries.  In my experience, whatever concept you start with, you end up having to add more and more clauses to explain why a particular example is or isn't a house, and if you try to cover all the possibilities you no longer have a clear definition by a particular set of properties.

More likely, we have a core concept of "house", a detached building that one family lives in, and extend that concept based on similarities (a cave house is a place people live in, parts of it are built and it's not going anywhere) and metaphors (the family living in a house stands in for the house itself, an example of metonymy).

As far as I can tell, this is just how language works, and language works this way because our minds work this way.  Our minds are constantly taking in a stream of sensory input and identifying objects from it, even when those objects are ill-defined, like clouds (literally nebulous) or aren't even there, like the deer I thought I saw through the snow crossing the road in hour 18 or so of a drive from California to Idaho.  We classify those objects in relation to other objects, or, more accurately, other experiences from which we've identified objects.

Identifying objects is itself an exercise in abstraction, deciding that a particular set of impulses in the optic nerve is a friend's face, or that a particular set of auditory inputs is a voice, or a dog barking, or a tree falling or whatever.  Recent generations of AIs which can recognize faces in photos or words in recordings of speech (much harder than it might seem) are doing the same thing.  We generally think that faces and words are too specific to be abstract, but is this abstract thinking?  If it is, how does it relate to examples like the ones I gave above, such as defining a species of animal?

When other animals do things like this, like a dog in the next room hearing kibble being poured into a dish or vervets responding to specific calls by acting to protect themselves from particular predators, we tend to think of it as literal thinking, not higher-level abstract thinking like we can do.  Any number of experiments in the 20th century studied stimulus/response behavior and considered "the bell was rung" as a simple concrete stimulus rather than an abstraction of a large universe of possible sounds, and likewise for a behavior like pressing a button to receive a treat.

I've described two related but distinct notions of abstraction here:

• Defining concepts in terms of abstract properties like size, shape, color, how something came to be, what it's meant to be used for and so on (this species of bird is around this size with plumage of these colors, a house is a building for human habitation)
• Identifying discrete objects (in a broad sense that includes things like sounds and motions) from a continuous stream of sensory input.

The first is the usual sense of abstraction.  It's something we do consciously as part of what we call reasoning.  Current AIs don't do it particularly well, or in many cases at all.  On the other hand, it's not clear how important it is in interacting with the world.  You don't have to be able to abstractly define house in order to build one or live in it.  You don't have to have a well-developed abstract theory in order to develop a new invention.  The invention just has to work.  Often, the theory comes along later.

Theories can be very helpful to people developing new technologies or making scientific discoveries, but they're not essential.  When AlphaFold discovers how a new protein will fold, it's not using a theory of protein folding.  In fact, that's its advantage, that it's not bound by any particular concept of how proteins should fold.

The second sort of abstraction is everywhere, once you think to look, so common as to be invisible.  It's crucial to dealing with the real world, and it's an important part of AI, for example in turning speech into text or identifying an obstacle for a robot to go around.  Since it's not conscious, we don't consider it abstraction, even if it may be a better fit for the concept of pulling out properties.  Since current AIs already do this kind of abstraction, and we don't consider an AI that recognizes faces in photos to be an AGI, this sort of abstraction clearly isn't enough to make something an AGI.

There may be some better definition of abstract thought that I'm missing, but neither of the two candidates above looks like the missing piece for AGI.  The first doesn't seem essential to the kind of disruption we assume an AGI would be capable of, and the second seems like basic infrastructure for anything that has to deal with the real world, AGI or not.

*That "renewed" is getting a little out of date.  Sometimes considerable time passes between starting a post and actually posting it.

AGI, goals and influence

While putting together What would superhuman intelligence even mean? I took out a few paragraphs that seemed redundant at the time.  While I think that post is better for the edit, when I re-read the deleted material, I realized that there was one point in them that I didn't explicitly make in the finished post.  Here's the argument (If you have "chess engine" on your AI-post bingo card, yep, you can mark it off yet again. I really do think it's an apt example, but I'm even getting tired of mentioning it):

When it comes to question what are the implications of AGI?, actual intelligence is one factor among many.  A superhuman chess engine poses little if any risk.  A simple non-linear control system that can behave chaotically is a major risk if it's controlling something dangerous.

To the extent that a control system with some sort of general superintelligence is hard to predict and may make decisions that don't align with our priorities, it would be foolhardy to put it directly in charge of something dangerous.  Someone might do that anyway, but that's a hazard of our imperfect human judgment.  A superhuman AI is just one more dangerous thing that humans have the potential to misuse.

The more interesting risk is that an AI with limited control of something innocuous could leverage that into more and more control, maybe through extortion -- give the system control of the power plants or it will destroy all the banking data -- or persuasion -- someone hooks a system up to social media where its accounts convince people in power to put it in charge of the power plants.

These are worthy scenarios to contemplate.  History is full of examples of human intelligences extorting or persuading people to do horribly destructive things, so why would an AGI be any different? Nonetheless, in my personal estimation, we're still quite a ways from this actually happening.

Current LLMs can sound persuasive if you don't fact-check them and don't let them go on long enough to say something dumb -- which in my experience is not very long -- but what would a chatbot ask for?  Whom would it ask? How would the person or persons carry out its instructions?  (I initially said "its will", rather than "its instructions", but there's nothing at all to indicate that a chatbot has anything resembling will)

You could imagine some sort of goal-directed agent using a chatbot to generate persuasive arguments on its behalf, but, at least as it stands, I'd say the most likely goal-directed agent for this would a human being using a chatbot to generate a convincing web of deception.  But human beings are already highly skilled at conning other human beings.  It's not clear what new risk generative AI presents here.

Certainly, an autonomous general AI won't trigger a cataclysm in the real world if it doesn't exist, so in that sense, the world is safer without it.  Eventually, though, the odds are good that something will come along that meets DeepMind's definition of AGI (or ASI).  Will that AI's skills include parlaying whatever small amount of influence it starts with into something more dangerous?  Will its goals include expanding its influence, even if we don't think they do at first?

The idea of an AI with seemingly harmless goals becoming an existential threat to humanity is a staple in fiction (and the occasional computer game).  It's good that people have been exploring it, but it's not clear what conclusions to draw from those explorations, beyond a general agreement that existential threats to humanity are bad.  Personally, I'm not worried yet, at least not about AGI itself, but I've been wrong many times before.

Metacognition, metaphor and AGI

In the recent post on abstract thought, I mentioned a couple of meta concepts: metacognition and metaphor.

• Metacognition is the ability to think about thinking.  I've discussed it before, particularly in this post and these two posts.
• Metaphor is a bit harder to define, though there is no shortage of definitions, but the core of it involves using the understanding of one thing to understand a different thing.  I've also discussed this before, particularly in this post and this one.

When I was writing the post on abstract thought, I had it in mind that these two abilities have more to do with what we would call "general intelligence" (artificial or not), so I wanted to try to get into that here, without knowing exactly where I'll end up.

In that earlier post, I identified two kinds of abstraction:

• Defining things in terms of properties, for example, a house is a building that people live in.  I concluded that this isn't essential to general intelligence.  At this point, I'd say it's more a by-product of how we think, particularly how we think about words.
• Identifying discrete objects (in some general sense) out of the stream of sensory input we encounter, for example, being able to say "that sound was a dog barking".  I concluded that this is basic equipment for dealing with the world.  At this point, I'd say it's worth noting that LLMs don't do this at all. They have it done for them by the humans that produce the words they're trained on and receive as prompts.  On the other hand specialized AIs, like speech recognizers, do exactly this.

It was the first kind of abstraction that led me back to thinking about metaphor.

Like the second kind of abstraction, metaphor is everywhere, to the point that we don't even recognize it until we think to look.  For example:

• the core of it (a concept has a solid center, with other, softer parts around it)
• I had it in mind (the mind is a container of ideas)
• I wanted to try to get into that (a puzzle is a space to explore; you know more about it when inside it than outside)
• without knowing exactly where I'll end up (writing a post is going on a journey, destination unknown)
• at this point (again, writing a post is a journey)
• this is basic equipment (mental abilities are tools and equipment)
• led me back to thinking (a chain of thought is a path one can follow)
• to the point (likewise)

While there's room for discussion as to the details, in each of those cases I'm talking about something in the mind (concepts, the process of writing a blog post ...) in terms of something tangible (a soft object with a core, a journey in the physical world ...).

Metaphor is certainly an important part of intelligence as we experience it.  It's quite possible, and I would personally say likely, that the mental tools we use for dealing with the physical world are also used in dealing with less tangible things.  For example, the mental circuitry involved in trying to follow what someone is saying probably overlaps with the mental circuitry involved in trying to follow someone moving in the physical world.

This would include not only focusing one's attention on the other person, but also building a mental model of the other person's goals so as to anticipate what they will do next, and also recording what the person has already said in a similar way to recording where one has already been along a path of motion.  If some of the same mental machinery is involved in both processes -- listening to someone speak, and physically following them -- then on some level we probably experience the two similarly.  If so, it should be no surprise that we use some of the same words in talking about the two.

The overlap is not exact, or else we actually would be talking about the same things, but the overlap is there nonetheless.  This can happen in more than one way at the same time.  If you're speaking aggressively to me, I might experience that in a similar way to being physically menaced, and I might say things like Back off or Don't attack me, even while I might also say I'm not following you if I can't quite understand what you're saying, but I still feel like it's meant aggressively.

It's interesting that these examples of metaphor, about processing what someone is saying, also involve metacognition, thinking about what the other person is thinking.  That's not always the case (consider this day is just rushing by me or it looks like we're out of danger).  Rather, we use metaphor when thinking about thinking because we use metaphor generally when thinking about things.

If you buy that metaphor is a key part of what we think of as our own intelligence, is it a key part of what we would call "general intelligence" in an AI?  As usual, that seems more like a matter of definition.  I've argued previously that the important consideration with artificial general intelligence is its effect.  For example, we worry about trying to control a rogue AI that can learn to adapt to our attempts to control it.  This ability to adapt might or might not involve metaphor.  It might well involve metacognition -- modeling what we're thinking as we try to control it, but maybe not.

Consider chess engines.  As noted elsewhere, it's clear that chess engines aren't generally intelligent, but it's also clear that they are superhuman in their abilities.  Human chess players clearly use metaphor in thinking about chess, not just attack and defense, but space, time, strength, weakness, walls, gaps, energy and many others.  Classic AB chess engines (bash out huge numbers of possible continuations and evaluate them using an explicit formula) clearly don't use metaphor.

The situation with neural network (NN) engines (bash out fewer possible continuations and evaluate them using a neural net) is slightly muddier, since in some sense the evaluation function is looking for similarities with other chess positions, but that's the key point: the NN is comparing chess positions to other chess positions, not to physical-world concepts like space, strength and weakness.  You could plausibly say that NNs use analogy, but metaphor involves understanding one thing in terms of a distinct other thing.

Likewise, neither sort of chess engine builds a model of what its opponent is thinking, only of the possible courses of action that the opponent might take, regardless of how it decides to take them.  By contrast, human chess players very frequently think about what their opponent might be thinking (my opponent isn't comfortable with closed positions, so I'm going to try to lock up the pawn structure).  Human chess players, being human, do this because we humans do this kind of thing constantly when dealing with other people anyway.

One the one hand, metaphors only become visible when we use words to describe things.  On the other hand, metaphor (I claim here) comes out of using the mental machinery for dealing with one thing to deal with another thing (and in particular, re-using the machinery for dealing with the physical world to deal with something non-physical).  More than that, it comes out of using the same mental machinery and, in some sense, being aware of doing it, if only in experiencing some of the same feelings in each case (there's a subtle distinction here between being aware and being consciously aware, which might be interesting to explore, but not here).

If we define an AGI as something of our making that is difficult to control because it can learn and adapt to our attempts to control it, then we shouldn't assume that it does so in the same ways that we do.  Meta-thought like explicitly creating a model of what someone (or something) else is thinking, and using metaphor to understand one thing in terms of another may be key parts of our intelligence, but I don't see any reason to think they're necessarily part of being an AGI in the sense I just gave.

The other half of this is chains of reasoning like "If this AI can do X, which is key to our intelligence, then it must be generally intelligent like we consider ourselves to be" rests on whether abilities like metacognition and metaphorical reasoning are sufficient for AGI.

That may or may not be the case (and it would help if we had a better understanding of AGI and intelligence in general), but so far there's a pretty long track record of things, for example being able to deal with natural language fluently, turning out not to necessarily lead to AGI.

AI, AI and AI

I have a draft post from just over a year ago continuing a thread on intelligence in general, and artificial intelligence in particular.  In fact, I have two draft AI posts at the moment.  There's also one from early 2020 pondering how the agents in Open AI's hide-and-seek demo figure out that an agent on the opposing team is probably hiding out of sight behind a barrier.

I was pretty sure at the time of the earlier draft that they do this by applying the trained neural network not just to the last thing that happened, but a window of what's happened recently.  In other words, they have a certain amount of short-term memory, but anything like long-term memory is encoded in the neural net itself, which isn't updated during the game.  This ought to produce effects similar to the "horizon effect" in early chess engines, where a player that could look ahead, say, three moves and see that a particular move was a blunder would play another move that led to the same blunder, but only after four moves.

I'm still pretty sure that's what's going on, and I was going to put that into a post one of these days as soon as I read through enough of the source material to confirm that understanding, but I never got around to it.

Because ChatGPT-4 happened.

ChatGPT is widely agreed to have been a major game changer and, yeah ... but which games?

From a personal point of view, my musings on how AIs work and what they might be able to do are now relevant to what my employer talks about in its quarterly earnings reports, so that put a damper on things as well.  For the same reason, I'll be staying away from anything to do with the internals or how AI might play into the business side.  As usual, everything here is me speaking as a private citizen, about publicly available information.

Out in the world at large, I recall a few solid months of THIS CHANGES EVERYTHING!!!, which seems to have died down into a steady stream of "How to use AI for ..." and "How to deal with the impact of AI in your job."  I've found some of this interesting, but a lot of it exasperating, which leads me to the title.

There are at least three very distinct things "AI" could reasonably mean at the moment:

• The general effort to apply computing technology to things that humans and other living things have historically been much better at, things like recognizing faces in pictures, translating from one language to another, walking, driving a car and so forth.
• Large language models (LLMs) like the ones behind ChatGPT, Bard and company
• Artificial General Intelligence (AGI), a very vague notion roughly meaning "able to do any mental task a human can do, except faster and better"

There are several posts under the AI tag here (including the previous post) poking and prodding at all three of those, but here I'm interested in the terms themselves.

To me, the first AI is the interesting part, what I might even call "real AI" if I had to call anything that.  It's been progressing steadily for decades.  LLMs are a part of that progression, but they don't have much to do with, say, recognizing faces or robots walking. All of these applications involve neural networks with back propagation (I'm pretty sure walking robots use neural nets), but training a neural net with trillions of words of speech won't help it recognize faces or control a robot walking across a frozen pond because ... um, words aren't faces or force vectors?

If you ask a hundred people at random what AI is, though, you probably won't hear the first answer very much.  You'll hear the last two quite a bit, and more or less interchangeably, which is a shame, because they have very little in common.

LLMs are a particular application of neural nets.  They encode interesting information about the contents of a large body of text.  That encoded information isn't limited to what we think of as the factual content of the training text, and that's a significant result.  For example, if you give an LLM an email and ask it to summarize the contents, it can do so even though it wasn't explicitly trained to summarize email, or even to summarize unfamiliar text in general.

To be clear, summarizing an email is different from summarizing part of the text that an LLM has been trained on.  You could argue that in some very broad sense the LLM is summarizing the text it's been trained on when it answers a factual question, but the email someone just sent you isn't in that training text.

Somehow, the training phase has built a model, based in part on some number of examples of summaries, of how a text being summarized relates to a summary of that text.  The trained LLM uses that to predict what could reasonably come after the text of the email and a prompt like "please summarize this", and it does a pretty good job.

That's certainly not nothing.  There may or may not be a fundamental difference between answering a factual question based on text that contains a large number of factual statements and performing a task based on a text that contains examples of the task, or descriptions of the task being done, but in any case an LLM summarizing an email is doing something that isn't directly related to the text it's been trained on, and that it wasn't specifically trained to do.

I'm pretty sure this is not a new result with LLMs, but seeing the phenomenon with plain English text that any English speaker can understand is certainly a striking demonstration.

There are a couple of reasons to be cautious about linking this to AGI.

First, to my knowledge, there isn't any widely-accepted definition of what AGI actually is.  From what I can tell, there's a general intuition of something like The Computer from mid 20th-century science fiction, something that you can give any question or task to and it will instantly give you the best possible answer.  

"Computer, what is the probability that this planet is inhabited?"

"Computer, devise a strategy to thwart the enemy"

"Computer, set the controls for the heart of the Sun"

"Computer, end conflict among human beings"

This may seem like an exaggeration or strawman, but at least one widely-circulated manifesto literally sets forth that "Artificial Intelligence is best thought of as a universal problem solver".

There's quite a bit of philosophy, dating back centuries and so probably much, much farther, about what statements like that might even mean, but whatever they mean, it's abundantly clear by now that an LLM is not a universal problem solver, and neither is anything else that's currently going under the name AI.

In my personal opinion, even a cursory look under the hood and kicking of the tires ought to be enough to determine that nothing like a current LLM will ever be a universal problem solver.  This is not just a matter of what kinds of responses current LLM-based chatbots give.  It's also a matter of what they are.  The neural net model underpinning this is based pretty explicitly on how biological computers like human brains work.  Human brains are clearly not universal problem solvers, so why would an LLM be?

There's an important distinction here, between "general problem solver", that is, able to take on an open-ended wide variety of problems, and "universal", able to solve any solvable problem.  Human brains are general problem-solvers, but nothing known so far, including current AIs, is universal.

This may sound like the argument that, because neural nets are built and designed by humans, they could never surpass a human's capabilities.  That's never been a valid argument and it's not the argument here.  Humans have been building machines that can exceed human capabilities for a long, long time, and computing machines that can do things that humans can't have been around for generations or centuries, depending on what you count as a computing machine and a human capability.

The point is that neural nets, or any other computing technology, have a particular set of problems that they can solve with feasible resources in a feasible amount of time.  The burden of proof is on anyone claiming that this set is "anything at all", that by building a network bigger and faster than a human brain, and giving it more data than a human brain could hope to handle, neural nets will not only be able to solve problems that a human brain can't -- which they already can -- but will be able to solve any problem.

So next time you see something talking about AI, consider which AI they're referring to.  It probably won't be the first (the overall progress of machines doing things that human brains also do).  It may well be the second (LLMs), in which case the discussion should probably be about things like how LLMs work or what a particular LLM can do in a particular context.

If it's talking about AGI, it should probably be trying to untangle what that means or giving particular reasons why some particular approach could solve some particular class of problems.

If it's just saying "AI" and things on the lines of "Now that ChatGPT can answer any question and AGI is right around the corner ...", you might look for a bit more on how those two ideas might connect.

My guess is there won't be much.

Apparently, we've just invented AI

The other evening I overheard a conversation about AI and its implications, now that we have it. This morning I read a post saying that it was hard to believe that ChatGPT was released only four years ago and now AI is everywhere, and how that might be the quickest adoption of a new technology in history.

I'm sure you can find plenty of examples the story yourself: Around four years ago, we didn't have AI. Then ChatGPT happened, and we do. What is AI? It's what LLM-based systems like ChatGPT, Claude and Gemini do, especially now that these systems are agentic, that is, they can break down tasks and coordinate a collection of agents and subagents in order to complete them.

I spent some time looking for a word (and asking an AI to find one) to convey the feeling that something is completely predictable and yet still somewhat surprising and came up with droll, ironic, paradoxical and a few others, but none of them quite fit.

A quick look at the AI label on this blog (and the other one) will tell you I've been watching this space for a while. The second Intermittent Conjecture post, back in 2010, mentioned AI in the past tense. The thesis was that there were plenty of applications around that would have been considered to unquestionably be AI in the early days of AI research, but they weren't in 2010 because they just didn't have that "I'll know it when I see it" quality that would cause the general public to say: Yes, that's it, that's Artificial Intelligence.

I gave the examples of "a neural network mining some pile of data, or even a chess program [i.e., that could regularly beat the best human players], or voice-enabled phone," all of which were around by 2010. Facial recognition and machine translation of natural language were also widely available at the time, although current versions work better, as one would hope.

The mechanisms behind the current generation of AI have their roots in research into connectionism in the late 1980s, which is generally considered the second wave of research into computing based on neural networks, the first having run from the 1940s through the 1960s. The general idea of machines mimicking human behavior is ancient, but in the context of modern computers it goes back to Turing's 1950 paper Computing Machinery and Intelligence, which introduced the idea of the "imitation game," which we now call the "Turing test," for better or worse. Systems that people took for human and communicated with accordingly first appeared in the late 1960s and early 1970s and more have appeared in the years since.

So it's natural to have a certain sense of "Wait, now everybody thinks we have AI?" but on the other hand it doesn't seem surprising at all, because something significant did happen when ChatGPT came out. LLM-based systems are simply much better at conversing in natural language than anything that came before.

One reason is fluency. An LLM-based system can answer question after question in fluent natural language. It can spit out page after page of prose that, at least at first glance, looks for all the world like a person wrote it, not just in the syntax and vocabulary but in tone and structure. It's not hard to write a hand-coded Markov chain that will produce at least mostly grammatical sentences, but it won't really sound human and it won't be able to answer questions. LLMs are the first approach that allows an extended two-way conversation without "the man behind the curtain" appearing in short order.

The second reason is the range of topics you can converse about. The first 'L' in LLM is for large. LLMs are trained on trillions, or even hundreds of trillions of tokens (words, more or less). By comparison, the entire US Library of Congress collection comprises a few trillion words. Without digging too deeply into what the numbers actually mean, it's not unreasonable to compare the sheer volume of training data to "all books ever published." Whatever topic you might want to ask about, there's a good chance that something related to it is in the training data, in sufficient quantity that the model can produce a few coherent paragraphs about it.

Before ChatGPT there was nothing widely available -- and hardly anything available at all -- that could converse with an average person in human language and appear knowledgeable on a wide range of topics. Artificial systems were able, and had long been able, to do a variety of tasks that can reasonably said to require some form of intelligence, but until ChatGPT everything was in the category of "Yeah, that's interesting, and even useful, but it's clearly not really intelligence." The particular task of conversing with a human on an open-ended variety of topics was the first AI task that really fit the "I'll know it when I see it" bill.

All this is fine with me. In everyday usage, LLMs are AI and nothing before them was. Common usage makes sense in terms of people's expectations of what artificial intelligence ought to mean. Everyday usage doesn't have to match up with technical definitions. 

Still, I'm uneasy with a couple of aspects of all this. First is the idea that AI is one single thing, whether LLMs or something else. Just as intelligence comprises a number of abilities, artificial intelligence is many different things, each capturing some portion of what we call intelligence in living things.

Tying into this is the idea that artificial intelligence necessarily means artificial general intelligence (AGI) or superhuman intelligence, or even the creation of a new, sentient species. Yes, ChatGPT crossed a threshold into something that people could generally agree could be called AI, but only in one particular sense. It did not cross a threshold into everything that AI could possibly mean.

Fortunately, after spending time actually conversing with the things and observing their behavior, people seem to have generally concluded that, even though LLMs are AI, they're not all-powerful or superhuman, except in the literal sense that they can do some things that humans can't -- as can many things.