What's a Fermi paradox?

So far, we haven't detected strong, unambiguous signs of extraterrestrial intelligence.  Does that mean there isn't any?

The usual line of attack for answering this question is the Drake equation [but see the next post for a bit on its origins --D.H Oct 2018], which breaks the question of "How many intelligent civilizations are there in our galaxy?" down into a series of factors that can then be estimated and combined into an overall estimate.

Let's take a simpler approach here.

The probability of detecting extraterrestrial intelligence given our efforts so far is the product of:

• The probability it exists
• The probability that what we've done so far would detect it, given that it exists

(For any math geeks out there, this is just the definition of conditional probability)

Various takes on the Fermi paradox (why haven't we seen anyone, if we're pretty sure they're out  there?) address these two factors

• Maybe intelligent life is just a very rare accident.  As far as we can tell, Earth itself has lacked intelligent life for almost all of its history (one could argue it still does, so feel free to substitute "detectable" for "intelligent").
• Maybe intelligent life is hard to detect for most of the time it's around (See this post for an argument to that effect and this one for a bit on the distinction between "intelligent" and "detectable").  A particularly interesting take on this is the "dark forest" hypothesis, that intelligent civilizations soon figure out that being detectable is dangerous and deliberately go dark, hoping never to be seen again.  I mean to take this one on in a bit, but not here.
• One significant factor when it comes to detecting signs of anything, intelligent or otherwise: as far as we know detectability drops with the square of distance, that is, twice as far away means four times harder to detect.  Stars are far away.  Other galaxies are really far away.
• Maybe intelligent life is apt to destroy itself soon after it develops, so it's not going to be detectable for very long and chances are we won't have been looking when they were there .  This is a popular theme in the literature.  I've talked about it here and here.
• Maybe the timing is just wrong.  Planetary time scales are very long.  Maybe we're one of the earlier ones and life won't develop on nearby planets for another million or billion years (basically low probability of detection again, but also an invitation to be more rigorous about the role of timing)

At first blush, the logic of the Fermi paradox seems airtight: Aliens are out there.  We'd see them if they were out there.  We haven't seen them.  QED.  But we're not doing a mathematical proof here.  We're dealing in probabilities (also math, but a different kind).  We're not trying to explain a mathematically impossible result.  We're trying to determine how likely it is that our observations are compatible with life being out there.

I was going to go into a longish excursion into Bayesian inference here, but ended up realizing I'm not very adept at it (note to self: get better at Bayesian inference).  So in the spirit of keeping it at least somewhat simple, let's look at a little badly-formatted table with, granted, a bunch of symbols that might not be familiar:

	We see life (S)	We don't see life (¬S)
Life exists (L)	P(L ∧ S)	P(L ∧ ¬S)	P(L)
No life (¬L)	P(¬L ∧ S)	P(¬L ∧ ¬S)	P(¬L)
	P(S)	P(¬S)	100%

P is for probability.  P(L) is the probability that there's intelligent life out there we could hope to detect as such, at all.  P(S) is the probability that we see evidence strong enough that the scientific community (whatever we mean by that, exactly) agrees that intelligent life is out there.  The ¬ symbol means "not" and the ∧ symbol means "and".  The rows sum to the right, so

• P(L ∧ S) + P(L ∧ ¬S) = P(L) (the probability life exists is the probability that life exists and we see it plus the probability it exists and we don't see it)
• P(S + ¬S) = 100% (either we see life or we don't see it)

Likewise the columns sum downward.  Also "and" means multiply (as long as the two probabilities are independent; they are here, since we allow for false positives), so P(L ∧ S) = P(L)×P(S).  This all puts restrictions on what numbers you can fill in.  Basically you can pick any three and those determine the rest.

Suppose you think it's likely that life exists, and you think that it's likely that we'll see it if it's there.  That means you think P(L) is close to 100% and P(L ∧ S) is a little smaller but also close to 100% (see conditional probability for more details) .  You get to pick one more.  It actually turns out not to matter that much, since we've already decided that life is both likely and likely to be detected.  One choice would be P(¬L ∧ S), the chance of a "false positive", that is, the chance that there's no life out there but we think we see it anyway.  Again, in this scenario we're assuming false positives should be unlikely overall, but choosing exactly how unlikely locks in the rest of the numbers.

It's probably worth calling out one point that kept coming up while I was putting this post together: The chances of finding signs of life depend on how much we've looked and how we've done it.  A lot of SETI has centered around radio waves, and in particular radio waves in a fairly narrow range of frequencies.  There are perfectly defensible reasons for this approach, but that doesn't mean that any actual ETs out there are broadcasting on those frequencies.  In any case we're only looking at a small portion of the sky at any given moment, our current radio dishes can only see a dozen or two light years out and there's a lot of radio noise from our own technological society to filter out.

I could model this as a further conditional probability, but it's probably best just to keep in mind that P(S) is the probability of having detected life after everything we've done so far, and so includes the possibility that we haven't really done much so far. 

To make all this concrete, let's take an optimistic scenario: Suppose you think there's a 90% chance that life is out there and a 95% chance we'll see it if it's out there.  If there's no chance of a false positive, then there's an 85.5% chance that we'll see signs of life and so a 14.5% chance we won't (as is presently the case, at least as far as the scientific community is concerned).  If you think there's a 50% chance of a false positive, then there's a 90.5% chance we'll see signs of life, including the 5% chance it's not out there but we see it anyway.  That means a 9.5% chance of not seeing it, whether or not it's actually there.

This doesn't seem particularly paradoxical to me.  We think life is likely.  We think we're likely to spot it.  So far we haven't.  By the assumptions above, there's about a 10% chance of that outcome.  You generally need 99.99994% certainty to publish a physics paper, that is, a 0.00006% chance of being wrong.  A 9.5% chance isn't even close to that

Only if you're extremely optimistic and you think that it's overwhelmingly likely that detectable intelligent life is out there, and that we've done everything possible to detect it do we see a paradox in the sense that our present situation seems very unlikely.  But when I say "overwhelmingly likely" I mean really overwhelmingly likely.  For example, even if you think both are 99% likely, then there's still about a 1-2% chance of not seeing evidence of life, depending on how likely you think false positives are.  If, on the other hand, you think it's unlikely that we could detect intelligent life even if it is out there, there's nothing like a paradox at all.

My personal guess is that we tend to overestimate the second of the two bullet points at the beginning.  There are good reasons to think that life on other planets is hard to detect, and our efforts so far have been limited.  In this view,  the probability that detectably intelligent life is out there right now is fairly low, even if the chance of intelligent life being out there somewhere in the galaxy is very high and the chance of it being out there somewhere in the observable universe is near certain.

As I've argued before, there aren't a huge number of habitable planets close enough that we could hope to detect intelligent life on them, and there's a good chance that we're looking at the wrong time in the history of those planets -- either intelligent life hasn't developed yet or it has but for one reason or another it's gone dark.

Finding out that there are potentially habitable worlds in our own solar system is exciting, but probably doesn't change the picture that much.  There could well be a technological civilizations in the oceans of Enceladus, but proving that based on what molecules we see puffing out of vents on the surface many kilometers above said ocean seems like a longshot.

With that in mind, let's put some concrete numbers behind a less optimistic scenario.  If there's a 10% chance of detectable intelligent life (as opposed to intelligent life we don't currently know how to detect), and there's a 5% chance we'd have detected it based on what we've done so far and a 1% chance of a false positive (that is, of the scientific community agreeing that life is out there when in fact it's not), then it's 98.6% likely we wouldn't have seen clear signs of life by now.   That seems fine.

While I'm conjecturing intermittently here, my own wild guess is that it's quite likely that some kind of detectable life is out there, something that, while we couldn't unequivocally say it was intelligent, would make enough of an impact on its home world that we could hope to say "that particular set of signatures is almost certainly due to something we would call life".   I'd also guess that it's pretty likely that in the next, say, 20 or 50 or 100 years we would have searched enough places with enough instrumentation to be pretty confident of finding something if it's there.  And it's reasonably likely that we'd get a false positive in the form of something that people would be convinced it was a sign of life when there in fact wasn't -- maybe we'd figure out our mistake in another 20 or 50 or 100 years.

Let's say life of some sort is 90% likely, there's a 95% chance of finding it in the next 100 years if it's there and a 50% chance of mistakenly finding life when it's not there, that is, a 50% chance that at some point over those 100 years we mistakenly convince ourselves we've found life and later turn out to be wrong.  Who knows?  False positives are based on the idea that there's no detectable life out there, which is another question mark.  But let's go with it.

I actually just ran those numbers a few paragraphs ago and came up with a 9.5% chance of not finding anything, even with those fairly favorable odds.

All in all, I'd say we're quite a ways from any sort of paradoxical result.

One final thought occurs to me:  The phrase "Fermi paradox" has been in the lexicon for quite a while, long enough to have taken on a meaning of its own.  Fermi himself, being one of the great physicists, was quite comfortable with uncertainty and approximation, so much so that the kind of "How many piano tuners are there in Chicago?" questions given to interview candidates are meant to be solved by "Fermi estimation".

I should go back and get Fermi's own take on the "Fermi paradox".  My guess was he wasn't too bothered by it and probably put it down to some combination of "we haven't really looked" and "maybe they're not out there".

If I find out I'll let you know.

[As noted above, I did in fact come across something --D.H Oct 2018]

Are we alone in the face of uncertainty?

I keep seeing articles on the Drake equation and the Fermi Paradox on my news feed, and since I tend to click through and read them, I keep getting more of them.  And since I find at least some of the ideas interesting, I keep blogging about them.  So there will probably be a few more posts on this topic.  Here's one.

One of the key features of the Drake equation is how little we know, even now, about most of the factors.  Along these lines, a recent (preprint) paper by Anders Sandberg, Eric Drexler and Toby Ord claims to "dissolve" the Fermi Paradox (with so many other stars out there why haven't we heard from them?), claiming to find "a substantial ex ante probability of there being no other intelligent life in our observable universe".

As far as I can make out, "ex ante" (from before) means something like "before we gather any further evidence by trying to look for life".  In other words, there's no particular reason to believe there should be other intelligent life in the universe, so we shouldn't be surprised that we haven't found any.

I'm not completely confident that I understand the analysis correctly, but to the extent I do, I believe it goes like this (you can probably skip the bullet points if math makes your head hurt -- honestly, some of this makes my head hurt):

• We have very little knowledge of the some of the factors in the Drake equation, particularly fl (probability of life on a planet that might support life) fi (probability of a planet with life developing intelligent life) and L (the length of time a civilization produces a detectable signal)
• Estimates of those range over orders of magnitude.
• Estimates for L range from 50 years to a billion or even 10 billion years.
• The authors do some modeling and come up with a range of uncertainty of 50 orders of magnitude for fl.  That is, it might be close to 1 (that is, close to 100% certain), or it might be more like 1 in 100,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000.  Likewise they take fi to range over three orders of magnitude, from near 1 to 1 in 1,000.
• Rather than assigning a single number to every term, as most authors do, it makes more sense to assign a probability distribution.  That is, instead of saying "the probability of life arising on a suitable planet is 90%", or 0.01% or whatever, assign probability for each possible value (the actual math is a bit more subtle, but that should do for our purposes).  Maybe the most likely probability of life developing intelligence is 1 in 20, but there's a possibility, though not as likely, that it's actually 1 in 10 or 1 in 100, so take that into account with a probability distribution..
• (bear in mind that the numbers were looking at are themselves probabilities, so we're assigning a probability that the probability is a given number -- this is the part that makes my head hurt a bit)
• Since we're looking very wide ranges of values, a reasonable distribution is the "log normal" distribution -- basically "the number of digits fits a bell curve".
• These distributions have very long tails, meaning that if, say, 1 in a thousand is a likely value for the chance of life evolving into intelligent life, then (depending on the exact parameters) 1 in a million may be reasonably likely, 1 in a billion not too unlikely and 1 in trillion is not out of the question.
• The factors in the Drake equation multiply, following the rules of probability, so it's quite possible that the aggregate result is very small.
• For example if it's reasonably likely that fl is 1 in a trillion and fi is 1 in a million, then we can't ignore the chance that the product of the two is 1 in a quintillion.
• Numbers like that would make it unlikely that there is any life in our galaxy's few hundred billion stars and that ours just happened to get lucky.
• Putting it all together, they estimate that there's a significant chance that we're alone in the observable universe.

I'm not sure how much of this I buy.

There are two levels of probability here.  The terms in the Drake equation represent what has actually happened in the universe.  An omniscient observer that knew the entire history of every planet in the universe (and exactly what was meant by "life" and "intelligent") could count the number of planets, the number that had developed life and so forth and calculate the exact values of each factor in the equation.

The probability distributions in the paper, as I understand it, represent our ignorance of these numbers.  For all we know, the portion of "habitable" planets with intelligent life is near 100%, or near 1 in a quintillion or even lower.  If that's the case, then the paper is exploring to what extent our current knowledge is compatible with there being no other life in the universe.  The conclusion is that the two are fairly compatible -- if you start with what (very little) we know about the likelihood of life and so forth, there's a decent chance that the low estimates are right, or even too optimistic, and there's no one but us.

Why?  Because low probabilities are more plausible than we think, and multiplying probabilities increases that effect.  Again, the math is a bit subtle, but if you have a long chain of contingencies, any one of them failing breaks the whole chain.  If you have several unlikely links in the chain, the chances of the chain breaking are even better.

The conclusion -- that for all we know life might be extremely rare -- seems fine.  It's the methodology that makes me a bit queasy.

I've always found the Drake equation a bit long-winded.  Yes, the probability of intelligent life evolving on a planet is the probability of life evolving at all multiplied by the probability of life evolving into intelligent life, but does that really help?

On the one hand, it seems reasonable to separate the two.  As far as we know it took billions of years to go from one to the other, so clearly they're two different things.

But we don't really know the extent of our uncertainty about these things.  If you ask for an estimate of any quantity like this, or do your own estimate based on various factors, you'll likely* end up with something in the wide range of values people consider plausible enough to publish (I'm hoping to say more on this theme in a future post).  No one is going to say "zero ... absolutely no chance" in a published paper, so it's a matter of deriving a plausible really small number consistent given our near-complete ignorance of the real number -- no matter what that particular number represents or how many other numbers it's going to be combined with.

You could almost certainly fit the results of surveying several good-faith attempts into a log-normal distribution.  Log-normal distributions are everywhere, particularly where the normal normal distribution doesn't fit because the quantity being measured has something exponential about it -- say, you're multiplying probabilities or talking about orders of magnitude.

If the question is "what is the probability of intelligent life evolving on a habitable planet?" without any hints as to how to calculate it, that is, one not-very-well-determined number rather than two, then the published estimates, using various methodologies, should range from a small fraction to fairly close to certainty depending on the assumptions used by the particular authors.  You could then plug these into a log normal distribution and get some representation of our uncertainty about the overall question, regardless of how it's broken down.

You could just as well ask "What is the probability of any self-replicating system arising on a habitable planet?", "What is the probability of a self-replicating system evolving into cellular life?"  "What is the probability of cellular life evolving into multicellular life?" and so forth, that is, breaking the problem down into several not-very-well-determined numbers.  My strong suspicion is that the distribution for any one of those sub-parts will look a lot like the distribution for the one-question version, or the parts of the two-question version, because they're basically the same kind of guess as any answer to the overall question.  The difference is just in how many guesses your methodology requires you to make.

In particular, I seriously doubt that anyone is going to cross-check that pulling together several estimates is going to yield the same distribution, even approximately, as what's implied by a single overall estimate.  Rather, the more pieces you break the problem into, the more likely really small numbers become, as seen in the paper.

I think this is consistent with the view that the paper is quantifying our uncertainty.  If the methodology for estimating the number of civilizations requires you to break your estimate into pieces, each itself with high uncertainty, you'll get an overall estimate with very high uncertainty.  The conclusion "we're likely to be alone" will lie within that extremely broad range, and may even take up a sizable chunk of it.  But again, I think this says much more about our uncertainty than about the actual answer.

I suspect that if you surveyed estimates of how likely intelligent life is using any and all methodologies*, the distribution would imply that we're not likely to be alone, even if intelligent life is very rare.  If you could find estimates of fine-grained questions like "what is the probability of multicellular life given cellular life?" you might well get a distribution that implied we're an incredibly unlikely fluke and really shouldn't be here at all.  In other words, I don't think the approach taken in the paper is likely to be robust in the face of differing methodologies.  If it's not, it's hard to draw any conclusions from it about the actual likelihood of life.

I'm not even sure, though, how feasible it would be to survey a broad sample of methodologies.  The Drake formulation dominates discussion, and that itself says something.  What estimates are available to survey depends on what methods people tend to use, and that in turn depends on what's likely to get published.  It's not like anyone somehow compiled a set of possible ways to estimate the likelihood of intelligent life and prospective authors each picked one at random.

The more I ponder this, the more I'm convinced that the paper is a statement about the Drake equation and our uncertainty in calculating the left hand side from the right.  It doesn't "dissolve" the Fermi paradox so much as demonstrate that we don't really know if there's a paradox or not.  The gist of the paradox is "If intelligent life is so likely, why haven't we heard from anyone?", but we really have no clear idea how likely intelligent life is.

* So I'm talking about probabilities of probabilities about probabilities?

Did clickbait kill the aliens?

Disclaimer: This post is on a darker topic than most.  I've tried to adjust the tone accordingly, but if anything leads you to ask "How can he possibly say that so casually?", rest assured that I don't think any of this is a casual matter.  It's just that if we're talking at the scale of civilizations and stars we have to zoom out considerably from the everyday human scale, to the point where a truly horrible cataclysm becomes just another data point.


As I've noted elsewhere, the Fermi paradox is basically "It looks likely that there's life lots of other places in the universe, so why haven't we been able to detect it -- or why haven't they made it easy by contacting us?"  Or, as Fermi put it, "Where is everybody?"

One easy answer, though something of a downer, is "They're all dead."^*

This is the idea that once a species gets to a certain level of technological ability, it's likely to destroy itself.  This notion has been floated before, in the context of the Cold War: Once it became technically possible, it took shockingly little time for humanity to develop enough nuclear weapons to pose a serious threat to itself.  One disturbingly ready conclusion from that was that other civilizations hadn't contacted us because they'd already blown themselves up.

While this might conjure up images of a galaxy full of  the charred, smoking cinders of once vibrant, now completely sterile planets, that's not exactly what the hypothesis requires.  Before going into that in detail, it's probably worth reiterating here that most planets in the galaxy are much too far away to detect directly against the background noise, or to be able to carry on a conversation with (assuming that the speed of light is the cosmic speed limit we think it is).  In order to explain why we haven't heard from anyone, we're really trying to explain why we haven't heard from anyone within, say, a hundred light years.  I've argued elsewhere that that narrows the problem considerably (though maybe not).


A full-scale nuclear exchange by everyone with nuclear weapons would not literally kill all life on Earth.  There are a lot of fungi and bacteria, and a lot of faraway corners like hydrothermal vents for all kinds of life to hide.  It probably wouldn't even kill all of humanity directly, but -- on top of the indescribable death and suffering from the bombing itself -- it would seriously damage the world economy and make life extremely difficult even in areas that weren't directly affected by the initial exchange.  Behind the abstraction of the "world economy" is the flow of food, medicine, energy and other essentials.

There is an extensive literature concerning just how bad things would get under various assumptions, but at some point we're just quibbling over levels of abject misery.  In no realistic case is bombing each other better for anyone involved than not bombing each other.

For our purposes here, the larger point is clear: a species that engages in a full-scale nuclear war is very unlikely to be sending out interstellar probes or operating radio beacons aimed at other stars.  It may not even be sending out much in the way of stray radio signals at all.  It might well be possible for a species in another star system to detect life in such a case without detecting signs of a technological civilization, much less communicating with it.

So how likely is a full-scale nuclear war?  We simply don't know.  So far we've managed to survive several decades of the nuclear age without one, but, as I've previously discussed, that's no time at all when it comes to estimating the likelihood of finding other civilizations.  To totally make up some numbers, suppose that, once nuclear weapons are developed, a world will go an average of a thousand years without seriously using them and then, after the catastrophe, take a couple of centuries to get back to the level of being able to communicate with the rest of the universe.

Again, who knows?  We (fortunately) have very little data to go on here.  In the big picture, though, this would mean that a planet with nuclear weaponry or something similarly dangerous would be 10-20% less likely to be detected than one without.  We also have to guess what portion of alien civilizations would be subject to this, but how likely is it, really, that someone would develop the ability to communicate with the stars without also figuring out how to do anything destructive with its technology?

My guess is  that "able to communicate across interstellar distances" is basically the same as "apt to destroy that ability sooner or later".  This applies particularly strongly to anyone who could actually send an effective interstellar probe.  The  kinetic energy of any macroscopic object traveling close to light speed is huge.  It's hard to imagine being able to harness that level of energy for propulsion without also learning how to direct it toward destruction.

For purposes of calculation, it's probably best to assume a range of scenarios.  In the worst case, a species figures out how to genuinely destroy itself, and perhaps even life on its planet, and is never heard from.  In a nearly-as-bad case, a species spends most of its time recovering from the last major disaster and never really gets to the point of being able to communicate effectively across interstellar distances, and is never heard from.  The upshot is a reduction in the amount of time a civilization might produce a detectable signal (or, in a somewhat different formulation, the average expected signal strength over time).

Our own case is, so far, not so bad, and let's hope it continues that way.  However, along with any other reasons we might not detect life like us on other planets, we can add the possibility that they're too busy killing each other to say hello.


With all that as context, let's consider a recent paper modeling the possibility that a technological civilization ends up disrupting its environment with (from our point of view here, at least) pretty much the same result as a nuclear war.   The authors build a few models, crunch through the math and present some fairly sobering conclusions: Depending on the exact assumptions and parameters, it's possible for a (simulated) civilization to reach a stable equilibrium with its (simulated) environment, but several other outcomes are also entirely plausible: There could be a boom-and-bust that reduces the population to, say, 10% of its peak.  The population could go through a repeating boom/bust cycle.  It could even completely collapse and leave the environment essentially unlivable.

So what does this add to the picture?  Not much, I think.

The paper reads like as a proof-of-concept of the idea of modeling an alien civilization and its environment using the same mathematical tools (dynamical system theory) used to model anything from weather to blood chemistry to crowd behavior and cognitive development.  Fair enough.  There is plenty of well-developed math to apply here, but the math is only as good as the assumptions behind it.

The authors realize this and take care only to make the most general, uncontroversial assumptions possible.  They don't assume anything about what kind of life is on the planet in question, or what kind of resources it uses, or what exact effect using those resources has on the planet.  Their assumptions are on the order of "there is a planet", "there is life on it", "life consumes resources" and so forth.

Relying on few assumptions means that any conclusions you do reach are very general.  On the other hand, if the assumptions support a range of conclusions, how do you pick from amongst them?  Maybe once you run through all the details, any realistic set of assumption leads to a particular outcome -- whether stability or calamity.  Maybe most of the plausible scenarios are in a chaotic region where the slightest change in inputs can make an arbitrarily large difference in outputs.  And so forth.

As far as I can make out, the main result of the paper is that planets, civilizations and their resources can be modeled as dynamical systems.  It doesn't say what particular model is appropriate, much less make any claims about what scenarios are most likely for real civilizations on real exoplanets.  How could it?   Only recently has there been convincing evidence that exoplanets even exist.  The case that there is life on at least some of them is (in my opinion) reasonably persuasive, but circumstantial.  It's way, way too early to make any specific claims about what might or might not happen to civilizations, or even life in general, on other planets.

To be clear, the authors don't seem to be making any such claims, just to be laying some groundwork for eventually making such claims.  That doesn't make a great headline, of course.  The article I used to find the paper gives a more typical take: Climate change killed the aliens, and it will probably kill us too, new simulation suggests.

Well, no.  We're still in the process of figuring out exactly what effect global warming and the resulting climate change will have on our own planet, where we can take direct measurements and build much more accurate models than the authors of the paper put forth.  All we can do for an alien planet is lay out the general range of possibilities, as the authors have done.  Trying to draw conclusions about our own fate from our failure (so far) to detect others like us seems quite premature, whether the hypothetical cause of extinction is war or a ruined environment.



There's a familiar ring to all this.   When nuclear destruction was on everyone's mind, people saw an obvious, if depressing, answer to Fermi's question.  As I recall, papers were published and headlines written.  Now that climate-related destruction is on everyone's mind, people see an obvious, if depressing, answer to Fermi's question, with headlines to match.  It's entirely possible that fifty years from now, if civilization as we know it is still around (as I expect it will be) and we haven't heard directly from an alien civilization (as I suspect we won't), people will see a different obvious, if depressing, answer to Fermi's question.  Papers will be written about it, headlines will do what headlines do, and it will all speak more to our concerns at the time than to the objective state of any alien worlds out there.


I want to be clear here, though.  Just because headlines are overblown doesn't mean there's nothing to worry about.  Overall, nuclear weapons take up a lot less cultural real estate than they did during the height of the cold war, but they're very much still around and just as capable of wreaking widespread devastation.  Climate change was well underway during that period as well, and already recognized as a hazard, but not nearly as prominent in the public consciousness as it is today.

It's tempting to believe in an inverse relationship between the volume of headlines and the actual threat: If they're making a big deal out of it, it's probably nothing to worry about.  But that's an empirical question to be answered by measurement.  It's not a given.  Without actually taking measurements, the safest assumption is the two are unrelated, not inversely related.  That is, how breathless the headlines are is no indication one way or another as to how seriously to take the threat.

My own guess, again without actually measuring, is that there's some correlation between alarming headlines and actual danger.  People study threats and publish their findings.  By and large, and over time, there is significant signal in the noise.  If a range of people working in various disciplines say that something is cause for concern, then it most likely is -- nuclear war and climate change are real risks.  Some part of this discussion finds its way into the popular consciousness, with various shorthands and outright distortions, but if you take the time to read past the headlines and go back to original sources you can get a reasonable picture, and one that will bear at least some resemblance to the headlines.

Going back to original sources and getting the unruly details may not be as satisfying as a nice, punchy one-sentence summary, but I'd say it's worth the effort nonetheless.



(*) A similar but distinct notion is the "Dark forest" hypothesis: They're out there, but they're staying quiet so no one else kills them -- and we had best follow suit.  That's fodder for another post, though I think at least some of this post applies.

Cookies, HTTPs and OpenId

I finally got around to looking at the various notices that have accumulated on the admin pages for this blog.  As a result:

• This blog is supposed to display a notice regarding cookies if you access it from the EU.  I'm not sure that this notice is actually appearing when it should (I've sent feedback to try to clarify), but as far as I can tell blogspot is handling cookies for this blog just like any other.  I have not tried to explicitly change that behavior.
• I've turned on "redirect to https".  This means that if you try to access this blog via http://, it will be automatically changed to https://.  This shouldn't make any difference.  On the one hand, https has been around for many years and all browsers I know of handle it just fine.  On the other hand, this is a public blog, so there's no sensitive private information here.  It might maybe make a difference if you have to do some sort of login to leave comments, but I doubt it.
• Blogger no longer supports OpenID.  I think this would only matter if I'd set up "trust these web sites" under the OpenId settings, but I didn't.

In other words, this should all be a whole lot of nothing, but I thought I'd let people know.

The stuff of dreams

... and then our hero woke up and it was all a dream ...

... has to rank among the most notorious pulled-out-of-thin-air deus ex machina twist endings in the book, along with "it was actually twins" and "it was actually the same person with multiple personalities".  As with all such tropes, there's nothing wrong with these plot twists per se.  The problem is generally the setup.

In a well-written "it was actually twins" twist, you have clues all along that there were actually two people -- maybe subtle shifts in behavior, or a detail of clothing that comes and goes, or the character showing up in an unexpected place that it seemed unlikely they'd be able to get to.  With a good setup, you're reaction is "Oh, so that's why ..." and not "Wait ... what?  Seriously?"

The same goes for "it was all a dream".  In a good setup, there are clues that it was all a dream.  Maybe things start out ok, then something happens that doesn't quite make sense, then towards the end things get seriously weird, but more in a "wait, what's going on here, why did they do that?" kind of way, as opposed to a "wait, was that a flying elephant I just saw with a unicyclist on its back?" kind of way, though that can be made to work as well.

There's a skill to making things dreamlike, particularly if you're trying not to give the game away completely.  Dream logic doesn't just mean randomly bizarre things happening.  Dreams are bizarre in particular ways which are not particularly well understood, even though people have been talking about and interpreting dreams probably for as long as there have been talking and dreams.

A while ago I ran across a survey by Jennifer Windt and Thomas Metzinger that has quite a bit to say about dreams and the dream state, both ordinary dreams and "lucid" dreams where the rules are somewhat different.  They compare the three states of ordinary dreaming, lucid dreaming and waking consciousness to try to tease out what makes each one what it is with, I found, fair success.  I'm not going to go into a detailed analysis of that paper here, but I did want to acknowledge it, if only as a starting point.


First, though, some more mundane observations about dreams.  We tend to dream several times a night, in cycles lasting around 90 minutes.  We don't typically remember this, but a subject who is awakened while exhibiting signs of a dream state can generally recall dreaming while a subject awakened under other conditions doesn't.  The dream state is marked by particular patterns of electrical activity in the brain, near-complete relaxation of the skeletal muscles and, probably best-known, Rapid Eye Movement, or REM.  REM is not a foolproof marker, but the correlation is high.

Dreams in early sleep cycles tend to be closely related to things that happened during the waking day.  Subjects who studied a particular skill prior to going to sleep, for example, tended to have dreams about that skill.  I've personally had dreams after coding intensely that were a sort of rehash of how I'd been thinking about the code in question, not so much in the concrete sense of writing or reading particular pieces as more abstractly navigating data and control structures.

Later dreams -- those closer to when you wake up -- tend to be more emotional and less closely associated with recent memories.  Since these are more likely to be the ones you remember unless someone is waking you up as part of a sleep experiment, these are the kind of dreams we tend to think of as "dreamlike".  These are the "I was in this restaurant having dinner with such-and-such celebrity, except it didn't look like them, and I could hear my third-grade teacher yelling something, but everyone just ignored it and then a huge ocean wave came crashing in and we all had to swim for it, even though the restaurant was in the Swiss Alps" kind of dreams.

In my experience this kind of dream can often be linked back to relevant events, but in a sort of mashed-up, piecemeal, indirect way.  Maybe you heard a news story about a tidal wave yesterday and a couple of days ago some relative or old friend had mentioned something that happened to you in grade school.  Celebrities, by definition, are frequently in the news, and it was the Swiss Alps just because.  That doesn't really explain what the dream might mean, if indeed it meant anything, but it does shed some light on why those particular elements might have been present.

But why that particular assemblage of elements?  Why wasn't the third grade teacher your dinner companion?  Why did all the other diners ignore the teacher?  Why wasn't the restaurant on the beach? And so on.

My personal theory on such things is pretty unsatisfying: it just is.  Whatever part of the mind is throwing dream elements together is distinct from the parts of the mind concerned with cause and effect and pulling together coherent narratives.

To draw a very crude analogy, imagine memory as a warehouse.  From time to time things have to be shuffled around in a warehouse in for various logistical reasons.  For example, if something that's been stored in the back for months now needs to be brought out, you may have to move other items around to get at it.  Those items were put there for their own reasons that may not have anything to do with the item that's being brought out.

Now suppose someone from management in a different part of the company -- say media relations -- comes in and starts observing what's going on.  A pallet of widgets gets moved from section 12D, next to the gadgets, to section 4B, next to the thingamajigs.  This goes on for a while and our curious manager may even start to notice patterns and make tentative notes on them.

Suppose upper-level management demands, for its own inscrutable reasons, a press release on the warehouse activity.  The media relations person writing the release is not able to contact the warehouse people to find out what's really going on and just has to go by the media relations manager's notes about widgets moving from next to the gadgets to next to the thingamajigs.  The resulting press release is going to try to tell a coherent story, but it's not going to make much sense.  It's almost certainly not going to say "We had to get the frobulator out of long-term storage for an upcoming project so we moved a bunch of stuff to get at it."

My guess is that something similar is going on in the brain with dreams.  In normal waking consciousness, the brain is receiving a stream of inputs from the outside world and putting them together into a coherent picture of what's going on.  There are glitches all the time for various reasons. The input we get is generally incomplete and ambiguous.  We can only pay attention to so much at a time.

In order to cope with this we constantly make unconscious assumptions based on expectations, and these vary from person to person since we all have different experiences.  The whole concept of consciousness is slippery and by no means completely understood, but for the purpose of this post consciousness (as opposed to any particular state of consciousness) means whatever weaves perception into a coherent picture of what's going on.

Despite all the difficulties in turning perception into a coherent reality, we still do pretty well.  Different people perceiving the same events can generally agree on at least the gist of what happened, so in turn we agree that there is such a thing as "objective reality" independent of the particular person observing it.  Things fall down.  The sun rises in the morning.  It rains sometimes.  People talk to each other, and so on.  Certainly there's a lot we disagree on, sometimes passionately with each person firmly believing the other just doesn't know the simple facts, but this doesn't mean there's no such thing as objective reality at all.



In the dream state, at least some of the apparatus that builds conscious experience is active, but it's almost completely isolated from the outside world (occasionally people will incorporate outside sounds or other sensory input into a dream, but this is the exception).  Instead it is being fed images from memories which, as in the warehouse analogy, are being processed according to however memory works, without regard to the outside world.  Presented with this, consciousness tries to build a narrative anyway, because that's what it does, but it's not going to make the same kind of sense as waking consciousness because it's not anchored to the objective, physical world.

If the early memory-processing is more concerned with organizing memories of recent events, early-cycle dreams will reflect this.  If later memory processing deals in larger-scale rearrangement and less recent, less clearly correlated memories, later-cycle dreams will reflect this.


As I understand it, Windt and Metzinger's analysis is broadly compatible with this description, but they bring in two other key concepts that are important to understanding the various states of consciousness: agency and phenomenal transparency.

Agency is just the power to act.  In waking consciousness we have a significant degree of agency.  In normal circumstances we can control what we do directly -- I choose to type words on a keyboard.  We can influence the actions of others to some extent, whether by force or persuasion.  We can move physical objects around, directly or indirectly.  If I push over the first domino in a chain, the others will fall.

In a normal dream the dreamer has no agency.  Things just happen.  Even things that the dreamer experiences as doing just happen.  You can recall "I was running through a field", but generally that's just a fact.  Even if your dream self decides to do something, as in "The water was rushing in so I started swimming", it's not the same as "I wanted to buy new curtains so I looked at a few online and then I picked these out".  Your dream self is either just doing things, or sometimes just doing things in a natural reaction to something that happened.

Even that much is a bit suspect.  It wouldn't be a surprise to hear "... a huge ocean wave came crashing in and then I was walking through this city, even though it was underwater".  In some fundamental way, in a dream you're not making things happen.  They just happen.

Likewise, one of the most basic forms of agency is directing one's attention, but in a dream you don't have any choice in that, either.  Instead, attention is purely salience based, meaning, more or less, that in a dream your attention is directed where it needs to be -- if that ocean wave bursts in you're paying attention to the water -- rather than where you want it to be.

Phenomenal transparency concerns knowing what state of consciousness you're in.  Saying that dreaming is phenomenally transparent is just a technical way of saying "when you're in a dream you don't know you're dreaming" (So why coin such a technical term for such a simple thing?  For the usual reasons.  On the one hand, repeating that whole phrase every time you want to refer to the concept -- which will be a lot if you're writing a paper on dreaming -- is cumbersome at best.  It's really convenient to have a short two-word phrase for "the-quality-of-not-knowing-you're-dreaming-when-you're dreaming".  On the other hand, defining a phrase and using it consistently makes it easier for different people to agree they're talking about the same thing.  But I digress.)

If someone is recalling a dream, they don't recall it as something that they dreamed.  The recall it as something that happened, and happened in a dream.  It "happened" just the same as something in waking consciousness "happened".  During the dream itself, it's completely real.  Only later, as we try to process the memory of a dream, do we understand it as a dream.  I've personally had a few fairly unsettling experiences of waking up still in a dreamlike state and feeling some holdover from the dream as absolutely real, before waking up completely and realizing ... it was all a dream (more on this below).  I expect this is something most people have had happen and this is why the "it was all a dream trope" can work at all.

In some sense this seems related to agency.  When you say "I dreamed that ..." it doesn't mean that you consciously decided to have thus-and-such happen in your dream.  It means that you had a dream, and thus-and-such happened in it.

Except when  it doesn't ...

Windt and Metzinger devote quite a bit of attention to lucid dreams. While the term lucid might suggest vividness and clarity, and this can happen, lucidity generally refers to being aware that one is dreaming (phenomenal transparency breaks down).  Often, but not always, the dreamer has a degree of control (agency) over the action of the dream.  In a famous experiment, lucid dreamers were asked to make a particular motion with their eyes, something like "when you realize you're in a dream, look slowly left and then right, then up, then left and right again", something that would be clearly different from normal REM.  Since the eyes can still move during a dream, even if the rest of the body is completely relaxed, experimenters were able to observe this and confirm that the dreamers were indeed aware and able to act.

Not everybody has lucid dreams, or at least not everyone is aware of having had them.  I'm not sure I've had any lucid dreams in the "extraordinarily clear and vivid" sense, but I've definitely had experiences drifting off to sleep and working through some problem or puzzle to solve, quite consciously, but blissfully unaware that I'm actually asleep and snoring.  I've also had experiences waking up where I was able to consciously replay what had just been happening in a dream and at least to some extent explore what might happen next.  I'm generally at least somewhat aware of my surroundings in such cases, at least intermittently, so it's not clear what to call dreaming and what to call remembering a dream.

In any case, I think this all fits in reasonably well with the idea of multiple parts of the brain doing different things, or not, none of them in complete control of the others.  Memory is doing whatever memory sorting it needs to do during sleep (it's clear that there's at least something essential going on during sleep, because going without sleep for extended periods is generally very bad for one's mental health).  Some level of consciousness's narrative building is active as well, doing its best to make sense of the memories being fed to it.  Some level of self awareness that "I'm here and I can do things" may or may not be active as well, depending on the dreamer and the particular circumstances.

This is nowhere near a formal theory of dreams.  Working those out is a full-time job.  I do think it's interesting, though, to try to categorize what does and doesn't happen in dream states and compare that to normal waking consciousness.  In particular, if you can have A happen without B happening and vice versa, then in some meaningful sense A and B are produced by different mechanisms.

If we draw up a little table of what can happen with or without what else...

Can there be ... without ...	Consciousness	Agency	Phenomenal transparency
Consciousness	—	yes1	yes2
(Conscious) Agency	no	—	?
Phenomenal transparency	no	yes3	—

¹ In ordinary dreams, but also, e.g., if paralyzed by fear

² In ordinary dreams

³ In a lucid dream, if you're aware that you're dreaming but can't influence the dream

... it looks like things are pretty wide open.  I didn't mention it in the table, but agency doesn't require consciousness.  We do things all the time without knowing why, or even that, we're doing them.  However, conscious agency requires consciousness by definition.  So does phenomenal transparency -- it's consciousness of one's own state.

Other than that, everything's wide open except for one question mark: Can you have conscious agency without phenomenal transparency?  That is, can you consciously take an action without knowing whether you're awake or dreaming (or in some other mental state).  This isn't clear from lucid dreaming, since lucid dreaming means you know you're dreaming.  It isn't clear from ordinary dreaming.  Ordinary dreams seem passive in nature.

In a related phenomenon, though, namely false awakening, the dreamer can, while actually remaining asleep, awaken and start to do ordinary things.  In some cases, the dreamer becomes aware of the dream state, but in other cases the illusion of being awake lasts until the dreamer awakens for real.

All of this is just a long way of saying that our various faculties like consciousness, agency and awareness of one's state of consciousness seem to be mix and match.  The normal states are waking consciousness and ordinary dreaming, but anything between seems possible.  In other words, while these faculties generally seem to operate either together (waking consciousness), or with only consciousness (ordinary dreaming) they're actually independent.  It's also worth noting that nothing in the table above distinguishes waking from dreaming.  The difference there would seem to be in whether we're processing the real world or memories of it.

This is an interesting piece of information, one which would have been considerably harder to come by if we didn't have the alternate window into consciousness provided by dreams.

Getting off the ground

Not long after I published the previous post about the Drake Equation, a couple of headlines surfaced about a paper by Michael Hippke with the admirably straightforward title Spaceflight from Super-Earths is difficult.  The paper is actually a light rewrite of what was originally an April Fool's joke, but the analysis is real, even if the author originally considered the topic frivolous.

The term Super-Earth itself is fairly loosely defined.  For concreteness, Hippke chooses Kepler-20b, with a radius of about 1.87R_⊕ (Earth radii) and a mass of about 9.7 M_⊕ (Earth masses).  Since gravity is proportional to mass and inversely proportional to the square of distance, the surface gravity of this planet would be about 2.8g (Earth gravity).  This is assuming that the measured radius is actually the radius of the surface.  There's a good chance that Kepler-20b is actually a "Mini-Neptune" with an extensive atmosphere rather than a Super-Earth with a rocky surface, but let's assume the Earth-like scenario here.

Hippke argues that it would be impractical for a civilization on such a planet to build rockets because the amount of fuel you need to reach escape velocity* increases exponentially in relation to that velocity.  This is exponential in the literal sense that doubling the velocity of a rocket means squaring the ratio of fuel to mass, not in the colloquial sense of "a lot".  Escape velocity in turn increases as the square root of the surface gravity.  For example, four times the surface gravity means twice the escape velocity, so square the ratio of fuel to dry mass.  Taking the square root doesn't make a lot of difference in the big picture.  The exponential part still dominates everything else.

In short, a somewhat bigger planet doesn't mean somewhat more fuel to get to escape velocity.  It can mean a lot more.

On Earth, a chemical rocket which magically had a weightless engine, fuel tank etc. would need to have 26 times as much fuel as payload in order to reach Earth's escape velocity of about 11 km/s**.  In real life that ratio is more like 50 or even 83 since the engine and so forth actually do weigh something.

Escape velocity for Kepler-20b would be about 2.3 times Earth's escape velocity, or around 25 km/s.  Hippke calculates that for a typical chemical rocket, that 26:1 ideal mass ratio is more like 2700:1 and the more realistic ratio of 83:1 would correspond to something like 9000:1.  To send a 1-ton payload out of the planet's gravity well would take 9000 tons of fuel.  By contrast, the Saturn V -- the largest rocket actually put into service so far -- had a mass of around 3000 tons, not all of which was fuel.

All this is fine, and surely more than enough for something that started out thoroughly tongue-in-cheek.  So let's take it at face value and try to poke holes in it anyway.

First, the calculations are for a single-stage rocket, though the real-life rockets used for comparison purposes are multi-stage.  In a multi-stage rocket you use a rocket with plenty of thrust (the first stage) to boost another rocket (the second stage) through the atmosphere quickly and then jettison that first stage.  At that point you no longer have to worry about the mass of the first stage and you consequently get more acceleration out of your remaining fuel.  You don't have to stop there.  The Saturn V, for example, was a three-stage rocket.  Five-stage rockets have been successfully launched.

This doesn't just make a difference in that a multi-stage rocket allows you get more acceleration out of the same mass ratio.  It also means that you don't have to use chemical rockets for all stages.  You could, for example, use an ion drive, which has a much higher effective velocity and therefore a much lower mass ratio, for the final stage and use chemical rockets to get it into orbit.  Ion drives produce very low thrust, far too little to launch from the ground, but they can do it for a very long time using very little fuel, eventually reaching much higher speeds than chemical rockets.  Once in orbit, a modestly-sized ion-driven vehicle could easily escape even Kepler 20b's gravity well.

In other words, getting to escape velocity in a single stage is a red herring.  You really just have to get a reasonable mass to orbital velocity, and you can use multiple stages if that helps.  At a given distance from the planet's center of mass, the orbital velocity is smaller than the escape velocity at the same distance by a factor of the square root of two.  In real life the orbit is -- of course -- further from the center of mass than the surface is.  If escape velocity at the surface is 25 km/s, a more reasonable orbital velocity would be 17 km/s, depending on how high up you have to go to get out of the atmosphere.  That would mean a mass ratio of more like 150 for an ideal rocket and 500 for a more realistic one.

That's still considerably more expensive than here on earth, but not nearly as discouraging as the 9000 figure in the paper.  A 500 ton rocket could put a ton in orbit, and you wouldn't even need to do that to get out of the gravity well.  Japan's ion-driven Hayabusa craft had a mass of about half a ton.  It was able to get to an asteroid, grab a sample and bring it back to Earth -- a pretty impressive piece of engineering if you ask me.  Our counterparts on Kepler 20b could do that with something like a 250 ton rocket.

The rocket that launched Sputnik was 267 tons (the rocket that actually launched Haybusa was around 140 tons, for a mass ratio of around 280).  Sputnik itself was only 84 kg, for a mass ratio of somewhat over 3000.  Small payloads generally mean higher mass ratios because it's not practical to shrink the launch system proportionately.

Leaving all that aside, you could also do multiple launches and assemble the final craft in orbit, if your robotics were good enough.  If you can launch half a ton with a reasonable-sized rocket, you can launch five tons with ten such rockets, and so forth.

Which brings up another point.  In the early stages of space exploration, before Kepler 20b puts its ion drive into orbit, they'll want to start small, using relatively big rockets to put relatively small things in orbit, and before that, to blast relatively small objects -- on Earth, that mainly meant weapons -- across large portions of the planet.

There doesn't seem to be any reason intelligent beings on Kepler 20b couldn't do that, assuming they're there.  Start with toy rockets, then weather rockets to explore the upper atmosphere, work up to ICBM-style systems, then orbit, then out of the gravity well, just as we did.  As far as I can tell, the difference on Kepler 20b would mainly be a matter of time, not a night-and-day difference between plausible and clearly impractical.  The benchmark of putting a ton or more directly on an escape trajectory doesn't seem particularly relevant to the question of whether or not this could happen, though, being concrete and understandable, it's still useful to think about.

There's another way to bring down the mass ratio: faster rocket fuel.  Hippke's calculations use an effective velocity of 3430 m/s, but hydrogen/oxygen delivers more like 4400.  That brings our ideal mass ratio down closer to 50 as opposed to 150.  As I understand it we only use hydrogen/oxygen in specific situations, due to various engineering considerations, but the tradeoffs will be different on Kepler 20b.  It might make sense to find ways to make the faster fuel work in more situations.

Even if chemical rockets weren't a practical way of getting into orbit, there are plenty of other options, some more speculative than others, for doing so.  Space elevators ... mass drivers ... blast wave accelerators ... space fountains.  Some of these require materials we don't know how to make yet or other not-so-proven technologies, but to some extent this is all a matter of economics.  Rockets are easy and cheap enough for us, so we use rockets.

Finally, it's probably worth pointing out that escaping a planet's gravity well is necessary for sending an interstellar mission, but hardly sufficient.  Kepler 20 is 950 light-years away.  To get here from there in, say, less than 10,000 years, you'll need to be going about a tenth the speed of light, or 30,000 km/s.  If you can do that, getting into orbit or even to escape velocity doesn't seem like a major problem.  Conversely, the most likely reason not to receive a visit from Kepler 20b is that it's just too far, not that it's too hard to get off the ground.





* I suppose I should acknowledge that "velocity" here actually means "speed" since it's a magnitude with no particular direction.  But everyone says "velocity" anyway.

** In real life you also have to deal with gravity losses until you reach orbital velocity.  For example, for every second you spend going straight up against Earth's surface gravity, you lose 9.8 m/s.  For Kepler 20b, that's more like 28 m/s.  If your initial stages take 180 seconds (three minutes), that's an extra 4000 m/s or so, except it's not really that simple since you don't spend all your time going straight up, particularly if the goal is to reach orbit.  I'm handwaving that, though it's quite a bit to handwave, just to keep the comparison with the ideal mass ratio of 26.  Part of the reason real rockets, even with multiple stages, needed a higher mass ratio than just the change in speed would suggest was to deal with gravity loss.

Detectability and the Drake Equation

I've argued in posts on the Drake Equation (really more a framework for trying to work out the odds of finding extraterrestrial life), that the L factor, representing the amount of time for which an intelligent civilization is detectable on a planet, is both underappreciated and overestimated.  That is, it's not just important whether a planet can develop life -- which is were a lot of attention is -- but just how long a planet with intelligent life is detectable as such.  If that time span is not very long, then there might well be intelligent civilizations out there that we don't know about, or have much hope of ever knowing about.

Radio transmissions are often used as a proxy for intelligent life.  Clearly, if we detect a radio signal coming from planet X with a structure we can't explain by natural means, we have to seriously consider the possibility that some intelligent life form sent the signal.  Artificial-looking radio signals strongly imply intelligent life, but lack of them doesn't imply lack of intelligent life.

Radio isn't the only way to go looking for intelligent life.  We're already able to get some idea of the atmospheres of exoplanets based on their effect on light from the parent star as they transit between us and that star, provided everything is in a favorable alignment.  That ability is liable to improve over time, to the point where we'll be able to detect whether a planet has a chemical composition that's likely to be produced by something like life as we know it.  That's pretty impressive, if you think about what it entails, and we're just getting started.  Astronomy has gotten really good at gleaning ridiculously faint signals from vast fields of noise, and while there are some fundamental limits to what we can gather, there's clearly a lot more we can do within those limits.

Likewise, if we can detect some signal related to a planet's surface, and we can observe the same planet from different angles (which is often possible since planets rotate) we can get some idea of any changes in the surface as seen from different angles.  Similar techniques were used to get a very rough map of Pluto's surface prior to the New Horizons mission.  It may also be possible to detect the polarization of light coming from a planet, and there are probably other sources of data.  Put together enough such hints and we may be able to measure whether a planet has anomalously dull or shiny or hot or cold regions or similar that might indicate ... something, maybe enough to say that there's probably a civilization something like ours on a given planet, and not just an odd configuration of protoplanetary dust.


So suppose that some twin Earth in the general vicinity -- say a few dozen or a few hundred light-years -- develops along similar lines to ours.  Suppose that some time after a species like ours arises it discovers radio, but not long after that it finds more efficient ways of communicating than blasting radio waves in all directions (including a tiny fraction headed towards us).  In my previous posts I argued that that's probably about it.  We won't be able to detect any signs of intelligent life (life, yes, civilization no) except for a tiny portion of the planet's existence, and so the odds are very low that we happen to be listening at the same time they're broadcasting.

But surely twin Earth will have cities and other large artifacts for longer than it has detectable radio emissions, and have them both before and after their radio era.  Our ability to detect such artifacts will only improve over time.  Suppose our techniques get to the point where we could detect the analog of a city of, say, 100,000 people by way of its structures and overall impact on the surrounding environment.  There are thousands of those on Earth now and, more to the point, there have been for quite a while.  Thousands of years, versus decades for radio transmission.  It's at least possible that there will continue to be cities for thousands of years more.  This is a couple of orders of magnitude longer than our detectable radio era might end up being, not something easy to write off.

Nonetheless, I don't know that it changes the picture much.  On the one hand, even this larger time window is still pretty small on a planetary time scale.  On the other hand, it's not at all clear to me that detecting a signature consistent with cities means that there are cities there.  I'd want to see a lot of work to rule out natural formations that we haven't thought of, and even then city-like collections of life don't necessarily mean intelligent civilization.  We are not the only life forms on Earth that can gather in numbers or have a significant environmental effect.

It's also entirely possible that we'd see the same effect with cities on Earth as with radio, just on a slower scale.  That is, we or our counterparts might have cities for a long time, but not have detectable cities for very long.  I'm not going to predict that humanity will necessarily lessen its overall impact on the environment over time, but it's possible.  If we become cleaner and (much the same thing) more efficient, we become harder to spot, and likewise for a hypothetical alien civilization.

Nonetheless, it seems dangerous to assume that whatever impact we do have, or an alien civilization has, will be undetectable from interstellar distances.  It will probably be detectable as an overall signature.  The question is what could we make of such a signature.  We'd probably be able to associate it with life, but what kind of life?

[ Re-reading an earlier post I see I already took this point into account, although in a more abstract way -- D.H.]

Where should I file this, and do I care?

I used to love to browse the card catalog at the local library (so yep ... geek).  This wasn't just for the books, but for the way they were organized.  The local library, along with my middle and high school libraries, used the Dewey Decimal Classification (or "Dewey Decimal System" as I remember it actually being called).

This was, to my eyes, a beautiful way of sorting books.  The world was divided into ten categories, each given a range of a hundred numbers, from 000-099 for "information and general works" (now also including computer science) to 900-999 for history and geography.  Within those ranges, subjects were further divided by number.  Wikipedia gives a good example:

500 Natural sciences and mathematics
510 Mathematics
516 Geometry
516.3 Analytic geometries
516.37 Metric differential geometries
516.375 Finsler geometry

Finsler geometry is pretty specialized (a Finsler manifold is a differentiable manifold whose metric has particular properties -- I had to look that up).  Clearly you could keep adding digits as long as you like, slicing ever finer, though in practice there are never more than a few (maybe just three?) after the decimal point.

With the Dewey classification in place, you could walk into libraries around the country, indeed around the world, and quickly figure out where, say, the books on gardening, medieval musical instruments or truck repair were located.  If you or the librarian found a book lying around, you could quickly put it back in its proper place on the shelves.  If you found a book you liked, you could find other books on related topics near it, whether on the shelves or in the card catalog (what's that, Grandpa?).

On top of that, the field of library science, in which the Dewey classification and others like it* play a central role is one of the precursors of computer science as we know it.  This is true at several levels, from the very idea of organizing large amounts of information (and making it universally accessible and useful), to the idea of using an index that can easily be modified as new items are added.

There's one other very significant aspect of library classification systems like Dewey: hierarchy.

It's almost too obvious to mention, but in the Dewey Classification, and others like it, the world is organized into high-level categories (natural sciences and mathematics), which contain smaller, more specific categories (mathematics), and so on down to the bottom level (Finsler geometry).  There are lots and lots of systems like this -- federal/state/local government in the US and similar systems elsewhere; domain/kingdom/phylum/class/order/family/genus/species in taxonomy; supercluster/galaxy cluster/galaxy/star system/star in astronomy; top-level-domain/domain/.../host and so forth.

Strictly speaking, this sort of structure is a containment hierarchy, where higher levels contain lower levels.  There are other sorts of hierarchies, for example primary/secondary/tertiary colors.  However, containment hierarchies are the most prominent kind.  Even hierarchies such as rank generally have containment associated with them -- if a colonel reports to a general, then that general is ultimately in command of the colonel's units (and presumably others).  The term hierarchy itself comes from the Greek for "rule of a high priest".  One of the most notable examples, of course, is the hierarchy of the Catholic church.

Containment hierarchies organize the world into units that our minds seem pretty good at comprehending, which probably why we're willing to overlook a major drawback: containment hierarchies almost always leak.

There are some possible exceptions.  One that comes to mind is the hierarchy of molecule/atom/subatomic particle/quark implied by the Standard Model.  Molecules are always composed of atoms and atoms of subatomic particles.  Of the subatomic particles in an atom, electrons (as far as we know) are elementary, having no simpler parts, while protons and neutrons are composed of quarks which (as far as we know) are also elementary.

Even here there are some wrinkles.  There are other elementary particles besides electrons and quarks that are not parts of atoms.  Electrons, protons and neutrons can all exist independently of atoms.  Some elements can exist without forming molecules.  Electrons in some types of molecule may not belong to particular atoms.  Even defining which atoms belong to which molecules can get tricky.

Perhaps a better example would be the classification of the types of elementary particles.  All (known) particles are unambiguously quarks, leptons, gauge bosons or scalar bosons.  Leptons and quarks are subdivided into generations, again with no room for ambiguity.  There are similar hierarchies in mathematics and other fields.

For most hierarchies, though, you have more than a bit of a mess.  Cities cross state lines, and while the different parts are administratively part of separate states, there will typically be citywide organizations, some with meaningful authority, that cross state lines.  Defining species and other taxonomic groups is notoriously contentious**.  One of the key points of Darwin's Origin is that you can't always find a satisfactory boundary -- the whole point of Origin is to explain why we so often can.

In astronomy, the designations of supercluster, galaxy cluster, galaxy and star system can all become murky or even arbitrary when several are interacting -- is that one merged galaxy, or two galaxies in the process of merging?  The distinction between star and planet can be troublesome as well, so it may not always be clear whether you have a planet orbiting a star or two companion stars.

On the internet, the distinction in notation between nested domains and hosts is clear, but the same (physical or virtual) computer can have multiple identities, even in different domains, and multiple computers can share the same host identity.  On the internet, what matters is which packets you respond to (and no one knows you're a dog).

And, of course, organization charts, arguably the prototypical example of a containment hierarchy, are in real life more what you'd call guidelines.  Beyond "dotted-line reports" and such, most real work crosses team boundaries and if everyone waited for every decision to percolate up and down the chain of command appropriately, nothing would get done (I've seen this attempted.  It did not go well).


So why group things into hierarchies anyway?

Again, there's clearly something about our minds that finds them natural.  In the early days of PCs, some of the prominent players originally started out storing files in one "flat" space.  If a floppy disk typically only held a handful of files, or even a few dozen, there was no harm in just listing them all out.  It didn't take long, however, until that got unwieldy.  People wanted to group related files together and, just as importantly, keep unrelated files separate.  Before long, all the major players had ported the concept of a "directory" or "folder" from the earlier "mainframe" operating systems -- which had themselves gone through roughly the same evolution.

Since computer scientists love nothing more than recursion, folders themselves could contain folders, and so on as far as you liked.  Somehow it didn't seem to bother anyone that this couldn't possibly work in a physical folder in a physical file drawer.

This all brought a new problem -- how to put things into folders.  There are at least two varieties of this problem (hmm ... problem subdivided into varieties ...).

For various reasons, some files needed to appear in multiple folders in identical form.  This is a problem not only for space reasons, but because you'd really like a change in a common file to show up everywhere instead of having to make the same change in an unknown number of copies.   This led to the rediscovery of "shortcuts" and "symbolic links", again already part of older operating systems, which allowed you to show the same physical file under multiple folders at the same time.

When it comes to organizing human-readable information, there's a different problem -- it's not always clear what folder to put things in.  Does a personal financial document go in the "personal" folder or the "financial" folder?  This problem leads us right back to ontology (the study of, among other things, how to categorize things) and library science.  Library science has always had to deal with this problem as well.  Does a book on the history of mathematics go under history (900s) or mathematics (510s).

There are always cases where you just have to decide one way or another, and then try to make the same arbitrary decision consistently in the future, hoping that some previously-unseen common thread will emerge that can then be codified into a rule.


The upshot, I think, is that hierarchies are a useful tool for organizing things for the convenience of human minds, not a property of the universe itself (except, arguably, in cases such as subatomic particles as discussed above).  As with any tool, there are costs and benefits to its use and it's best to weigh them before charging ahead.  Imposing a hierarchy that doesn't fit well isn't just wasted effort.   It can actively obscure what's really going on.

Interestingly enough, I now work for a company that takes an entirely different approach to organizing knowledge.  Don't worry about where something should be, or what grouping it should be in.  Just search for what's in it.

This has been remarkably successful.  It may be hard to remember, but for a while there was a brisk business in manually curating and categorizing information.  It's still done, of course, because it's still a useful exercise in some contexts, but it's no longer the primary way we find information on the web.  Now we just search.

OK, time to hit Publish.  Oh wait ... what labels should I put on this post?



* Dewey isn't the only game in town, just the one most widely used in US primary and secondary schools.  The local university library uses the Library of Congress Classification, which uses letters and numbers in a way that made my brain melt, not so much for looking more complex, I think, as for not being Dewey.

** My understanding is that the idea of a clade -- all (living) organisms descended from a given ancestor -- has come to be at least as important as the traditional taxonomic groupings, at least in some contexts, but I'm not a biologist.

syl·lab·i·fi·ca·tion

[Author's note: When I started this, I thought it was going to touch on deep questions of language and cognition.  It ended up kinda meandering around some random bits of computer word-processing.  This happens sometimes.  I'm posting it anyway since, well, it's already written.  --D.H.]

Newspaper and magazine articles are traditionally typeset in narrow, justified columns. "Justified" here means that every line is the same width (unlike, say, with most blog posts).  If the words aren't big enough to fill out a line, the typesetter will widen the spaces to fill it out.  If the words are a bit too long, the typesetter might move the last word to the next line and then add space to what's left.

Originally, a typesetter was a person who physically inserted pieces of lead type into a form.  Later, it was a person operating a Linotype™ or similar machine to do the same thing.  These days it's mostly done by software.

Technically, laying out a paragraph to minimize the amount of extra space is not trivial, but certainly feasible, the kind of thing that would make a good undergraduate programming exercise.  Several algorithms are available.  They may not always produce results as nice as an experienced human typesetter, but they do well enough for most purposes.

One option for getting better line breaks and better-looking paragraphs is to hyphenate.  If your layout looks funny because you've got floccinaucinihilipilification in the middle of a line, you might try breaking it up as, say floccinaucinihili-
pilification.  It will probably be easier to lay out those two pieces rather than trying to make room for one large one.

You can't just throw a hyphen in anywhere.  There's a strong tendency to read whatever comes before and after the hyphen as independent units, so you don't want to break at wee-
knights or pre-
aches.

In many languages, probably most, this isn't a big problem.  For example, Spanish has an official set of rules that gives a clear hyphenation for any word (actually there are several of these, depending on what country you're in).  It's hard for English, though, for the same reason that spelling is hard for English -- English spelling is historical, not phonetic, and has so far resisted attempts at standardisation standardization and fonetissizing.

So instead we have the usual suspects, particularly style guides produced by various academic and media organizations.  This leads to statements like this one from the Chicago Manual of Style:

Chicago favors a system of word division based on pronunciation and more or less demonstrated by the recommendations in Webster’s tenth.

The FAQ list that that comes from has a few interesting cases, though I'm not sure that "How should I hyphenate Josephine Bellver's last name?" actually qualifies as a frequently asked question.  The one that interests me here concerns whether it should be "bio-logy" or "biol-ogy".  CMOS opts for "biol-ogy", going by pronunciation rather than etymology.

Which makes sense, in that consistently going by pronunciation probably makes reading easiest.  But it's also a bit ironic, in that English spelling is all about etymology over pronunciation.

Either approach is hard for computers to cope with, since they both require specific knowledge that's not directly evident from the text.  It's common to teach lists of rules, which computers do deal with reasonably well, but the problem with lists of rules for English is that they never, ever work.  For example, it's going to be hard to come up with a purely rule-based approach that divides "bark-ing" but also "bar-keeper".

This is why style guides tend to fall back on looser guidance like "divide the syllables as they're pronounced".  Except -- whose pronunciation?  When I was a kid I didn't pronounce an l in also or an n in government (I've since absorbed both of those from my surroundings).  I'm pretty sure most American speakers don't pronounce a t in often.  So how do you hyphenate those according to pronunciation?


Fortunately, computers don't have to figure this out.  A hyphenation dictionary for 100,000 words will cost somewhere around a megabyte, depending on how hard you try to compress it.  That's nothing in modern environments where a minimal "Hello world" program can run into megabytes all by itself (it doesn't have to, but it's very easy to eat a few megabytes on a trivial program without anyone noticing).

But what if the hyphenator runs across some new coinage or personal name that doesn't appear in the dictionary -- for example, whoever put the dictionary together didn't know about Josephine Bellver?  One option is just not to try to hyphenate those.  A refinement of that would be to allow the author to explicitly add a hyphen.  This should be the special "optional hyphen" character, so that you don't get hyphens showing up in the middle of lines if you later edit the text.  That way if you invent a really long neologism, it doesn't have to mess up your formatting.

If there's a point to any of this, it's that computers don't have to follow specific rules, except in the sense that anything a computer does follows specific rules.  While it might be natural for a compugeek to try to come up with the perfect hyphenation algorithm, the better engineering solution is probably to treat every known word as a special case and offer a fallback (or just punt) when that fails.

This wasn't always the right tradeoff.  Memory used to be expensive, and a tightly-coded algorithm will be much smaller than a dictionary.  But even then, there are tricks to be employed.  One of my all-time favorite hacks compressed a spelling dictionary down to a small bitmap that didn't even try to represent the actual words.  I'd include a link, but the only reference I know for it, Programming Pearls by Jon Bentley, isn't online.

Surrender, puny humans!

A while ago, Deep Mind's AlphaGo beat human champion Lee Sedol at the game of go.  This wasn't just another case of machines beating humans at games of skill.

Granted, from a mathematical point of view it was nothing special.  Games like go, chess, checkers/draughts and tic-tac-toe, can in theory be "solved" by simply bashing out all the possible combinations of moves and seeing which ones lead to wins for which players.

Naturally the technical definition of "games like go, etc." is a bit, well, technical, but the most important stipulations are

• perfect information -- each player has the same knowledge of the game as the others
• no random elements

That leaves out card games like poker and bridge (imperfect information, random element) and Parcheesi (random element) and which-hand-did-I-hide-the-stone-in (imperfect information), but it includes most board games (Reversi, Connect 4, Pente, that game where you draw lines to make squares on a field of dots, etc. -- please note that most of these are trademarked).

From a practical point of view, there is sort of a pecking order:

• Tic-tac-toe is so simple that you can write down the best strategy on a piece of paper.   Most people grow bored of it quickly since the cat always wins if everyone plays correctly, and pretty much everyone can.
• Games like ghost or Connect 4 have been "strongly solved", meaning that there's a known algorithm for determining whether a given position is a win, loss or draw for the player whose turn it is.  Typically the winning strategy is fairly complex, in some cases too complex for a human to reasonably memorize.  A human will have no chance of doing better than a computer for such games (unless the computer is programmed to make mistakes), but might be able to do as well.
• Checkers is too complex for humans to play perfectly, but it has been "weakly solved".  This means that it's been proved that with perfect play, the result is always a draw, but, not all legal positions have been analyzed, and there is currently nothing that will always be able to tell you if a particular position is a win for either side, or a draw.  In other words, for a weakly solved game, we can answer win/loss/draw for the initial position, and typically many others, but not for an arbitrary position.
• Chess has not been solved, even in theory, but computer chess players that bash out large numbers of sequences of moves can consistently beat even the best human players.

In most cases the important factor in determining where a game fits in this order is the "branching factor", which is the average number of moves available at any given point.  In tic-tac-toe, there are nine first moves, eight second moves, and so on, and since the board is symmetrical there are effectively even fewer.  In many positions there's really only one (win with three-in-a-row or block your opponent from doing that).

In Connect 4, there are up to six legal moves in any position.  In checkers there can be a dozen or so.  In chess, a couple dozen is typical.  As with tic-tac-toe there are positions where there is only one legal move, or only one that makes sense, but those are relatively rare in most games.

In go, there are typically more than a hundred different possible moves, and go positions tend not to be symmetrical.  Most of the time a reasonably strong human player will only be looking at a small portion of the possible moves.  In order to have any hope of analyzing a situation, a computer has to be able to narrow down the possibilities by a similar amount.  But to beat a human, it has to be able to find plays that a human will miss.

I've seen go described as a more "strategic" game, one that humans can develop a "feel" for that computers can't emulate, but that's not entirely true.  Tactics can be quite important.  Much of the strategy revolves around deciding which tactical battles to pursue and which to leave for later or abandon entirely.  At least, that's my understanding.  I'm not really a go player.

AlphaGo, and programs like it, solved the narrowing-down problem by doing what humans do: collecting advice from strong human players and studying games played by them.  Historically this has meant a programmer working with an expert player to formulate rules that computers can interpret, along with people combing through games to glean more rules.

As I understand it (and I don't know anything more about Deep Mind or AlphaGo than the public), AlphaGo used machine learning techniques to automate this process, but the source material was still games played by human players.  [Re-reading this in light of a more recent post, I see I left out a significant point: AlphaGo (and AlphaZero) encode their evaluation of positions -- their understanding of the game -- as neural networks rather than explicit rules.  While a competent coder could look at the code for explicit rules and figure out what they were doing, no on really knows how to decode what a neural network is doing, at least not to the same level of detail -- D.H. Jan 2019]

The latest iteration (AlphaGo Zero, of course) dispenses with human input.  Rather than studying human games, it plays against itself, notes what works and what doesn't, and tries again after incorporating that new knowledge.  Since it's running on a pretty hefty pile of hardware, it can do this over and over again very quickly.

This approach worked out rather well.  AlphaGo Zero can beat the AlphaGo that beat Lee Sedol, making it presumably the strongest go player in the world.  [and it has since done the same thing with chess and shogi, though its superiority in chess is not clear-cut.  See the link above for more details.  -- D.H. Jan 2019]

On the one hand, this is not particularly surprising.  It's a classic example of what I call "dumb is smarter" on the other blog, where a relatively straightforward approach without a lot of built in assumptions can outperform a carefully crafted system with lots of specialized knowledge baked in.  This doesn't mean that dumb is necessarily smartest, only that it often performs better than one might expect, because the downside to specialized knowledge is specialized blind spots.

On the other hand, this is all undeniably spooky.  An AI system with no baked-in knowledge of human thought is able, with remarkably little effort, to outperform even the very best of us at a problem that had long been held up as something unreachable by AI, something that only human judgement could deal with effectively.  If computers can beat us at being human, starting essentially from scratch (bear in mind that the hardware that all this is running on is largely designed and built by machine these days), then what, exactly are we meat bags doing here?

So let's step back and look at the actual problem being solved: given a position on a go board, find the move that is most likely to lead to capturing the most stones and territory at the end of the game.

Put that way, this is a perfectly well-posed optimization problem of the sort that we've been using computers to solve for decades.  Generations, really, at this point.  Granted, one particular solution -- bashing out all possible continuations from a given position -- is clearly not best suited, but so what?  Finding the optimum shape -- or at least a better one -- for an airplane wing isn't well suited to that either, but we've made good progress on it anyway using different kinds of algorithms.

So "chess-style algorithms suck at go, therefore go is inherently hard" was a bad argument from the get-go.

From what I've seen in the press, even taking potential hype with a grain of salt, AlphaGo Zero is literally rewriting the books on go, having found opening moves that have escaped human notice for centuries.  But that doesn't mean this is an inherently hard problem.  Humans failing to find something they're looking for for centuries means it's a hard problem for humans.

We humans are just really bad at predicting what kinds of problems are inherently hard, which I'd argue is the same as being hard to solve by machine*.  Not so long ago the stereotype of a genius was someone who "knew every word in the dictionary" or "could multiply ten-digit numbers immediately", both of which actually turned out to be pretty easy to solve by machine.

Once it was clear that some "genius" problems were easy for machines, attention turned to things that were easy for people but hard for machines.  There have been plenty of those -- walking, recognizing faces, translating between speech and text, finding the best move on the go board.  Those held out for quite a long time as "things machines will never be able to do", but the tide has been turning on them as well thanks, I think, to two main developments:

• We can now build piles of hardware that have, in a meaningful sense, more processing power than human brains.
• With these new piles of hardware, techniques that looked promising in the past but never really performed are now able to perform well, the main example being neural network-style algorithms

At this point, I'm convinced that trying to come up with ever fuzzier and more human things that only human brains will ever be able to do is a losing bet.  Maybe not now, but in the long run.  I will not be surprised at all if I live to see, say

• Real time speech translation that does as well as a human interpreter.
• Something that can write a Petrarchan sonnet on a topic of choice, say the futility of chasing perfection, that an experienced and impartial reviewer would describe as "moving", "profound" and "original".
• Something that could read a novel and write a convincing essay on it comparing the story to specific experiences in the real world, and answer questions about it in a way that left no choice but to say that in some meaningful sense the thing "understood" what it read.
• Something that it would be hard to argue didn't have emotions -- though the argument would certainly be made.

[On the other hand, I also won't be shocked if these don't totally pan out in the next few decades --D.H. Feb 2019]

These all shade into Turing test territory.  I've argued that, despite Alan Turing's genius and influence, the Turing test is not necessarily a great test of whatever we mean by intelligence, and in particular it's easy to game because people are predisposed to assume intelligence.  I've also argued that "the Singularity" is an ill-defined concept, but that's really a different thread.  Nevertheless, I expect that, sooner or later, we will be able to build things that pass a Turing test with no trickery, in a sense that most people can agree on.

And that's OK.

Or at least, we're going to have to figure out how to be OK with it.  Stopping it from happening doesn't seem like a realistic option.

This puts us firmly in the territory of I, Robot and other science fiction of its era and more recently (the modern Westworld reboot comes to mind), which is one reason I chose the cheesy title I did.  Machines can already do a lot of things better than we can, and the list will only grow over time.  At the moment we still have a lot of influence over how that happens, but that influence will almost certainly decrease over time (the idea behind the Singularity is that this will happen suddenly, in fact nearly instantaneously, once the conditions are right).

The question now is how to make best use of what influence we still have while we still have it.  I don't really have any good, sharp answers to that, but I'm pretty sure it's the right question.

* There's a very well-developed field, complexity theory, dealing in what kinds of problems are hard or easy for various models of computing in an absolute, quantifiable sense.  This is largely distinct from the question of what kinds of games or other tasks computers should be good at, or at least better than humans at, though some games give good examples of various complexity classes.  One interesting result is that it's often easy (in a certain technical sense) to produce good-enough approximate solutions to problems that are provably very hard to solve exactly.  Another interesting result is that it can be relatively tricky to find hard examples of problems that are known to be hard in general.